An intelligent multi-modal dialog system and method

Abstract

The application relates to an intelligent multi-modal dialogue system and method, wherein the system comprises: a multi-modal input processing module responsible for real-time collection and processing of voice and video inputs of a user, special data marking mechanism is used to ensure accurate information transmission; a voice recognition processing module: innovative segmentation accumulation mechanism and double filtering strategy are used to ensure the accuracy and continuity of voice recognition; an LLM controller module: intelligent analysis is used to realize dynamic selection of a model, balance system performance and resource consumption; and a response generation module: responsible for integrating model outputs and generating a final response, ensuring the natural and smooth dialogue. The application can intelligently judge the use of a suitable model type according to dialogue content, and realize efficient integration of multi-modal information through special data marking and processing mechanisms.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-modal interaction, and in particular to an intelligent multi-modal dialogue system and method. BACKGROUND

[0002] Currently, intelligent dialogue systems are mainly divided into three categories: the first category is a pure text dialogue system, which interacts through text; the second category is a voice dialogue system, which supports voice input and output; and the third category is a multi-modal dialogue system, which can handle multiple input forms such as voice and video. With the development of large language model technology, the understanding ability and response quality of dialogue systems have been significantly improved.

[0003] However, the existing intelligent dialogue system lacks intelligent judgment mechanism in model selection, either using heavy visual models throughout, causing resource waste, or failing to provide visual understanding ability when needed. SUMMARY

[0004] In view of the problems existing in the prior art, the purpose of the present application is to provide an intelligent multi-modal dialogue system and method, which realizes on-demand switching of text LLM and visual LLM through an intelligent model selection mechanism, avoids the heavy model training process in traditional schemes, and realizes efficient reuse of existing models.

[0005] To achieve the above purpose, the technical solution adopted by the present application is:

[0006] An intelligent multi-modal dialogue system comprises:

[0007] A multi-modal input processing module, comprising an audio input unit, a video input unit, an audio processor and a video processor, the audio input unit being in communication connection with the audio processor, and the video input unit being in communication connection with the video processor; the audio input unit is used to collect voice input of a user, and the audio processor packs the collected voice input into an audio data packet; the video unit is used to collect video input of a user, and the video processor performs frame processing on the collected video input and packs it into a video data packet containing visual features;

[0008] A speech recognition module comprising an ASR processor, the ASR processor performs validity detection on the audio data packet, judges whether it contains valid speech by calculating the RMS value and short-time energy features of the audio stream, then processes the valid speech segment using a segment accumulation strategy, and finally recognizes the processed speech segment as text information;

[0009] The LLM controller module includes a controller LLM, a text LLM and a visual LLM, the LLM controller performs deep semantic analysis on the text information output by the ASR processor, and identifies whether visual understanding capability is required; when the visual understanding capability is not required, the text LLM is called to process the text information, and a response and a response mark are output; when the visual understanding capability is required, the visual LLM is called, and visual features of a video data packet are obtained, the visual features and the text information are taken as inputs of the visual LLM, and a response and a response mark are output after processing of the visual LLM; the response mark includes “[S.LISTEN]”, “[C.LISTEN]” and “[S.SPEAK]”, [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, and [S.LISTEN] indicates that the system is waiting for user response;

[0010] The response generation module includes a system response unit and a speech synthesis unit, the system response unit detects the response mark output by the text LLM and the visual LLM, if “[S.LISTEN]” is output, no speech synthesis processing is performed, and if “[S.SPEAK]” is output, the speech synthesis unit is triggered: “[S.SPEAK]” is replaced with “ ”, and then a speech synthesis service is input to perform speech response to the user.

[0011] The ASR processor performs effectiveness checking, specifically: an adaptive double threshold mechanism is set, when the RMS value of the audio exceeds a times of the background noise level and the short-time energy exceeds a preset threshold, it is determined that it is an effective speech segment; the segment accumulation strategy is: audio processing is performed in a basic unit of b seconds, and an overlapping interval of c seconds is maintained between adjacent segments, where c < b.

[0012] The ASR processor adopts a double filtering mechanism to filter the recognized text information: the time interval of adjacent recognition results is required to be greater than d seconds; and the similarity of new and old recognized texts is calculated, and only when the similarity is lower than m% can the new recognized text information be retained.

[0013] The speech recognition module further includes a interrupt control unit, the interrupt control unit matches the text information recognized by the ARS processor through a preset high-priority keyword library, and when the preset keyword is matched, an interrupt operation is performed; when the text LLM or the visual LLM is processed, deep semantic analysis is performed in combination with user input and dialogue context information to identify the implicit interrupt intention of the user, and when the interrupt intention is detected, a response mark “[S.STOP]” is output, indicating that the user wants to stop or pause the dialogue; the system response unit detects the response mark “[S.STOP]”, and immediately empties the TTS output queue, and the system remains silent.

[0014] The multi-modal dialogue system adopts a WebSocket communication mechanism for data transmission between the multi-modal input processing module, the speech recognition processing module, the LLM controller module and the response generation module. The audio processor adds a 0x00 mark to the header of the data packet when packaging the audio data. The video processor adds a 0xFF mark to the header of the data packet when packaging the video data.

[0015] An intelligent multi-modal dialogue method, comprising the following steps:

[0016] Step S1, collecting user input information and packaging;

[0017] Audio information input by the user is collected, sampled and packaged to form an audio data packet.

[0018] Step S2, information preprocessing;

[0019] First, the audio data packet is subjected to validity detection, and whether it contains valid speech is determined by calculating the RMS value and short-time energy feature of the audio stream.

[0020] Then, the valid speech segment is processed using a segmented accumulation strategy.

[0021] Finally, the processed audio is recognized and converted into text information.

[0022] Step S3, using the controller LLM to perform deep semantic analysis on the text information to determine whether visual understanding capability is needed, and if so, calling the text LLM to process the text information, and then outputting a response and a response mark; if not, calling the visual LLM, taking the text information as the input of the visual LLM, and collecting video data to obtain visual features as the input of the visual LLM, and the visual LLM model processes the text information and the visual features and outputs a response and a response mark.

[0023] The response mark includes “[S.LISTEN]”, “[C.LISTEN]”, “[S.SPEAK]”, and “[S.STOP]”, where “[S.SPEAK]” indicates that the system is outputting, “[C.LISTEN]” indicates that the system is receiving user input, and “[S.LISTEN]” indicates that the system is waiting for user response.

[0024] Step S4, detecting the response character output by the visual LLM or the text LLM, and if the response character is “[S.LISTEN]”, no speech synthesis processing is performed, and if it is “[S.SPEAK]”, speech synthesis processing is triggered: replacing “[S.SPEAK]” with “null”, and then inputting the speech synthesis service to respond to the user in speech.

[0025] The effectiveness identification in the step S2 is specifically: setting an adaptive double threshold mechanism, when the RMS value of the audio exceeds a times of the background noise level and the short-time energy exceeds a preset threshold, the audio is determined as an effective speech segment; the segment accumulation strategy is: taking b seconds as a basic unit for audio processing, and maintaining an overlapping interval of c seconds between adjacent segments, wherein c < b.

[0026] In the step S3, after obtaining the text information, a double filtering mechanism is used to filter the recognized text information: the time interval of adjacent recognition results is required to be greater than d seconds; and the similarity of new and old recognized texts is calculated, and only when the similarity is lower than m% the new recognized text information is retained.

[0027] The method further includes interrupting the control flow, and the specific process is as follows:

[0028] The first level of interrupting control is started during the execution of the step S2, the text information recognized by the ARS processor is matched by using a preset high-priority keyword library, when the preset keyword is matched, the interrupting operation is performed; when the preset keyword is not matched, the second level of interrupting control is started.

[0029] The second level of interrupting control is: when the text LLM or the visual LLM in the step S3 is processing, the user input and the dialogue context information are combined for deep semantic analysis, the implicit interrupting intention of the user is recognized, when the interrupting intention is detected, a response mark [S.STOP] is output, indicating that the user wants to stop or pause the dialogue;

[0030] In the step 4, if the response mark [S.STOP] is detected, the TTS output queue is immediately emptied, and the system remains silent.

[0031] The method uses a WebSocket communication mechanism for data transmission, and in the step S1, when the audio data is packaged, a 0x00 mark is added to the header of the data packet; in the step S3, when the video data is packaged, a 0xFF mark is added to the header of the data packet.

[0032] After the above scheme is used, the application has the following beneficial effects:

[0033] First, in terms of deployment efficiency, since the no-training design idea is used, the system can directly reuse the existing open source model, and the deployment cost is significantly reduced. Compared with the traditional method which needs to collect data and train the model for a specific scene, the system can realize plug and play, and the deployment time is shortened from several weeks to several hours.

[0034] Second, in terms of resource utilization efficiency, the system achieves the goal of calling visual LLM on demand through an intelligent model selection mechanism. Test data shows that in typical dialogue scenarios, about 70% of interactions can be completed using only text LLM, compared with the scheme of always enabling visual LLM, the system's computing resource consumption is reduced by about 50%, and the average response delay is reduced by 40%.

[0035] Third, in terms of system adaptability, the no-training architecture adopted by the invention enables it to flexibly adapt to different types of application scenarios. Experimental verification shows that the same system architecture can be seamlessly applied to customer service consultation, education guidance, medical inquiry and other fields without targeted training. In actual application tests, the system's dialogue completion rate in different scenarios exceeds 95%, and user satisfaction is improved by 15% compared with traditional solutions.

[0036] Fourth, in terms of maintenance cost, due to the adoption of modular design and standardized interface, the system can easily update or replace individual functional modules. In particular, in terms of model iteration, when a new open-source model is released, it can be integrated directly without the need for retraining, which greatly reduces the system's maintenance cost and upgrade difficulty. Through the analysis of the above technical effects, it can be seen that the no-training multi-modal dialogue framework proposed by the invention not only has innovation in technical implementation, but also exhibits significant cost advantage and use value in actual application. Through innovative architectural design and control mechanisms, the system effectively solves the training dependency problem in traditional solutions, providing a new solution for large-scale application of intelligent dialogue systems. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a schematic diagram of the principles of the invention. DETAILED DESCRIPTION

[0038] The invention discloses an intelligent multi-modal dialogue system, which realizes efficient and accurate human-computer dialogue interaction through an innovative real-time data processing architecture and dynamic model control mechanism. The system adopts modular design and includes complete data acquisition, processing, analysis and response generation processes. The core feature of the system is the ability to intelligently determine the appropriate model type (text LLM or visual LLM) based on dialogue content, and to efficiently integrate multi-modal information through special data labeling and processing mechanisms.

[0039] As Figure 1As shown, the intelligent multi-modal dialogue system of the present application includes: (1) a multi-modal input processing module responsible for real-time collection and processing of user voice and video input, ensuring accurate information transmission through a special data marking mechanism. (2) A speech recognition processing module: innovative segmented accumulation mechanism and double filtering strategy are adopted to ensure the accuracy and continuity of speech recognition. (3) LLM (Large Language Model) controller module: through intelligent analysis, the dynamic selection of the model is realized, balancing system performance and resource consumption. (4) Response generation module: responsible for integrating model output and generating the final response, ensuring the natural flow of the dialogue.

[0040] The multi-modal input processing module adopts an innovative parallel processing architecture to realize real-time collection and processing of front-end data. The multi-modal input processing module includes an audio input unit and a video input unit in the input layer, and an audio processor and a video processor in the processing layer, wherein the audio input unit is connected to the audio processor, and the video input unit is connected to the video processor.

[0041] In this embodiment, the audio input unit calls the device microphone using WebAPI, acquires data using a 16kHz sampling rate and a single-channel format, and ensures audio quality through real-time volume monitoring, automatic gain control, and noise suppression preprocessing steps. In order to realize streaming processing, the audio processor performs packaging once every 4096 sampling points (the number is a set value N, which can be set accordingly according to actual needs), forming an audio data packet, which is transmitted to the speech recognition processing module for processing.

[0042] The video input unit calls the device camera using WebAPI, sets the collection resolution to 320x240 and the sampling frequency to 1fps. The video processor optimizes image quality through automatic exposure adjustment and contrast optimization, and then compresses it into a data packet using JPEG format, forming a video data packet containing visual features.

[0043] At the data transmission level, this embodiment establishes a WebSocket channel to realize bidirectional real-time communication, designs a unified data packet format containing type markers, timestamps, and data lengths, and implements a complete transmission control mechanism, including automatic reconnection, heartbeat maintenance, flow control, and packet loss detection functions, to ensure the reliability of data transmission. Therefore, the header of the audio data packet needs to add a 0x00 marker, and the header of the video data packet needs to add a 0xFF marker. Through this design, the system can efficiently and stably collect and transmit multi-modal data, providing high-quality input for subsequent processing modules.

[0044] The voice recognition processing module includes an ASR processor and an interruption control unit, wherein the ASR processor realizes high-quality voice recognition through an innovative multi-level processing mechanism. First, the ASR processor performs validity detection on the received audio data packet, and judges whether it contains valid speech by calculating the RMS value and the short-time energy feature of the audio stream. Specifically, an adaptive double threshold mechanism is set, and when the RMS value of the audio exceeds a times (1.5 times in this embodiment) of the background noise level and the short-time energy exceeds the preset threshold, it is determined as a valid speech segment. Second, the ASR processor adopts an innovative segmented accumulation strategy, processes the audio in a basic unit of b seconds, and maintains an overlapping interval of c seconds (wherein c<b, in this embodiment, c is 0.5 and b is 2) between adjacent segments. This design not only ensures the real-time performance of voice recognition, but also effectively avoids the problem of speech segment breakage. In terms of voice recognition quality control, the ASR processor realizes a double filtering mechanism based on time interval and text similarity: by requiring the time interval of adjacent recognition results to be greater than d seconds (1 second in this embodiment), it avoids too frequent recognition; at the same time, the similarity of new and old recognition texts is calculated, and when the similarity is less than m% (80% in this embodiment), the new recognition result is retained, effectively solving the problem of repeated recognition. Through this multiple processing mechanism, the user's voice input can be accurately extracted and converted into high-quality text information, providing reliable input data for the subsequent LLM controller module.

[0045] The interrupt control unit adopts an innovative double-level interrupt detection architecture, and realizes an efficient and intelligent interrupt mechanism through a hierarchical judgment strategy. In the first level, the interrupt control unit performs rapid matching through a pre-set high-priority keyword library, including keywords such as "stop", "stop down", "don't say any more", "wait", "wait a moment", etc. When these keywords are detected, the system immediately performs an interrupt operation, including stopping the current speech synthesis, clearing the audio buffer, resetting the dialogue state, etc., to ensure millisecond-level response speed. When the keyword matching in the first level does not hit, the system starts the semantic analysis mechanism of the second level, sends the user input together with the dialogue context to the LLM for deep semantic analysis, and guides the LLM to recognize the user's implicit interrupt intention through specific system prompt words. When the LLM detects the interrupt intention, it will include special control markers in the response, and the system continuously analyzes the LLM output through the state monitoring mechanism and triggers the interrupt process. To ensure the reliability of the interrupt mechanism, the system implements a complete state synchronization control mechanism, establishing a bidirectional state communication channel between the ASR processor and the speech synthesis processor, and synchronously processing the state in real time. At the same time, the system introduces an interrupt cooling period mechanism, which maintains a high detection threshold for a short time after the speech synthesis stops, effectively avoiding false triggering. This double-level interrupt control mechanism with hierarchical judgment ensures immediate response to explicit interrupt instructions through keyword matching, and can intelligently understand various implicit interrupt intentions through LLM even when the keywords do not hit, significantly improving the naturalness and intelligence of the system's interaction.

[0046] The LLM controller module realizes intelligent model selection and dialogue management mechanism. The LLM controller module includes a controller LLM, a text LLM, and a visual LLM, and the controller LLM is connected to the text LLM and the visual LLM. The text information output by the ASR processor is input into the controller module, and the controller module performs deep semantic analysis on the text information (i.e. user input), and identifies whether visual understanding capability is needed through keyword matching and context understanding. For example, it detects whether visual indicating words such as "see", "in the image", "in the video" appear in the input, or analyzes the dialogue history to determine whether the current dialogue is in a visual-related context. When it is determined that visual understanding capability is not needed, a lightweight text LLM is called to process the text information, significantly reducing the consumption of computing resources; when visual understanding is indeed needed, the system seamlessly switches to the visual LLM and processes the text information and the visual features of the video data packet as input to ensure accurate understanding of the visual content. In order to maintain the coherence of the dialogue, the system maintains the last 10 dialogue records as context information, and records whether each dialogue involves visual content through special markers, which enables the system to accurately track the dialogue state and avoid understanding breaks caused by model switching. Through this intelligent control mechanism, the system realizes optimization of resource use while ensuring functional integrity.

[0047] The text LLM and the visual LLM output a response and a response mark after processing. The response mark includes [S.LISTEN], [C.LISTEN], [S.SPEAK], [S.STOP], etc. [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, [S.LISTEN] indicates that the system is waiting for user response, and [S.STOP] indicates that the user wants to stop or pause the dialogue. Precise dialogue flow control is achieved through these state marks.

[0048] The response generation module ensures the coherence and naturalness of the system output through an innovative state management and multi-level processing mechanism. The response generation module includes a system response unit of the output layer and a text-to-speech (TTS) unit. The system response unit normalizes the output of the text LLM and the visual LLM, including removing special characters, unifying punctuation, adjusting mood words, etc., to ensure the normalization of the output text. Specifically, the response mark of the text LLM and the visual LLM output is detected. If the output is [S.LISTEN], no speech synthesis processing is performed. If it is [S.SPEAK], speech synthesis processing is triggered: [S.SPEAK] is replaced with "empty", and then the speech synthesis service is input to respond to the user in voice. If it is [S.STOP], the TTS output queue is immediately emptied, and the system remains silent. The speech synthesis unit selects the appropriate output form according to the current dialogue state, for example, in a video-related answer, the system adjusts the use of demonstrative words in combination with the picture information, making the answer more natural. In order to realize real-time voice output, the system uses a streaming synthesis strategy to perform speech synthesis while generating text, and transmits it to the front end in real time through the WebSocket channel. This design not only improves the timeliness of the system response, but also ensures the synchronization of multi-modal output through precise state control. Through this multi-level response generation mechanism, the system can provide a smooth and natural interactive experience, effectively improving user satisfaction.

[0049] Based on the above multi-modal dialogue system, the present application also discloses an intelligent multi-modal dialogue method, which comprises the following steps:

[0050] Step S1, collecting user input information and packaging.

[0051] Specifically, the audio information input by the user is collected, sampled and packaged, and a 0x00 mark is added to the header of the data packet to form an audio data packet.

[0052] Step S2, information preprocessing.

[0053] The effectiveness detection is performed on the audio data packet, and whether valid speech is contained is determined by calculating the RMS value and the short-time energy feature of the audio stream. Specifically, an adaptive double threshold mechanism is set, and when the RMS value of the audio exceeds 1.5 times of the background noise level and the short-time energy exceeds a preset threshold, it is determined that the valid speech segment.

[0054] Then, the valid speech segment is processed by using an innovative segmented accumulation strategy: audio processing is performed in units of m (for example, 2) seconds, and an overlapping interval of n (for example, 0.5) seconds is maintained between adjacent segments. This design ensures the real-time performance of speech recognition and effectively avoids the fragmentation problem of speech segments.

[0055] Finally, the processed audio is recognized and converted into text information.

[0056] In order to improve the recognition effect, the audio is recognized based on the dual filtering of time interval and text similarity. Specifically, the time interval of adjacent recognition results is required to be greater than 1 second to avoid too frequent recognition; at the same time, the similarity between the new and old recognition texts is calculated, and when the similarity is less than 80%, the new recognition result is retained, effectively reducing the problem of repeated recognition.

[0057] In step S3, the controller LLM is used to perform deep semantic analysis on the text information to determine whether visual understanding capability is needed. If visual understanding capability is needed, the text LLM is called to process the text information, and then the response and response mark are output. If visual understanding capability is not needed, the visual LLM is called, the text information is taken as the input of the visual LLM, and the video data is collected to obtain the visual features as the input of the visual LLM. After the text information and the visual features are processed by the visual LLM model, the response and the response mark are output.

[0058] Whether visual understanding capability is needed can be identified by keyword matching and context understanding. For example, whether visual indicating words such as "see", "in the image", "in the video" appear in the input is detected, or whether the current conversation is in a visual-related context is analyzed. When it is determined that visual understanding capability is not needed, the lightweight text LLM is called to process the conversation, which significantly reduces the consumption of computing resources; when visual understanding is indeed needed, the system seamlessly switches to the visual LLM, ensuring that the visual content can be accurately understood. In order to maintain the coherence of the conversation, the system maintains the last 10 conversation records as context information, and records whether each conversation involves visual content through special marks. This design enables the system to accurately track the conversation state and avoid understanding breaks caused by model switching. Through this intelligent control mechanism, the function integrity is guaranteed while the resource usage is optimized.

[0059] In this embodiment, the device camera is called by WebAPI to collect video, and the collection resolution of 320x240 and the sampling frequency of 1fps are set. The collected video is input into the visual LLM after frame processing by the video processor.

[0060] The response marks include "[S.LISTEN]", "[C.LISTEN]", "[S.SPEAK]", "[S.STOP]", etc., [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, [S.LISTEN] indicates that the system is waiting for user response, and [S.STOP] indicates that the user wants to stop or pause the dialogue. Through these state marks, accurate dialogue flow control is realized.

[0061] Step S4, detect the response character output by the visual LLM or the text LLM. If the response character is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", speech synthesis processing is triggered: replace "[S.SPEAK]" with "empty", then input the speech synthesis service, and respond to the user with voice. If it is "[S.STOP]", immediately empty the TTS output queue, and the system remains silent.

[0062] The application adopts a double-level interrupt detection architecture, and realizes an efficient and intelligent interrupt mechanism through a hierarchical judgment strategy. In the first level, during the execution of step S2, a preset high-priority keyword library is used for rapid matching. The keywords include "stop", "stop down", "don't say", "wait", "wait a moment", etc. When these keywords are detected, the system immediately performs an interrupt operation, including stopping the current speech synthesis, emptying the audio buffer, resetting the dialogue state, etc., to ensure millisecond-level response speed. When the keyword matching of the first level does not hit, the system starts the semantic analysis mechanism of the second level, sends the user input together with the dialogue context to the text LLM or visual LLM for deep semantic analysis, and guides the text LLM or visual LLM to recognize the user's implicit interrupt intention through specific system prompt words. When the interrupt intention is detected, a special control mark [S.STOP] will be included in the response, and the system will continuously analyze the LLM output and trigger the interrupt process through the state monitoring mechanism. To ensure the reliability of the interrupt mechanism, the system realizes a complete state synchronization control mechanism, establishes a two-way state communication channel between the ASR processor and the speech synthesis processor, and synchronously processes the state in real time. At the same time, the system introduces an interrupt cooling period mechanism, which maintains a high detection threshold for a short time after the speech synthesis is stopped, effectively avoiding false triggering. This double-level interrupt control mechanism of hierarchical judgment can not only ensure the immediate response to explicit interrupt instructions through keyword matching, but also understand various implicit interrupt intentions through LLM intelligent understanding when the keywords do not hit, significantly improving the naturalness and intelligence level of the system interaction.

[0063] The application solves the model dependence problem in the prior art through an intelligent model selection mechanism and a fine control strategy. Unlike traditional technologies that require model training or fine-tuning for specific scenarios, the present solution achieves efficient reuse of existing models through an innovative dynamic control framework. At the model selection level, the system automatically determines the dialogue requirements based on deep semantic analysis, enabling on-demand switching of text LLM and visual LLM, and avoiding the heavy model training process in traditional solutions. In terms of data processing, the system uses a special marking mechanism (0x00 / 0xFF) to accurately classify audio and video data, and ensures data quality through differentiated preprocessing strategies. The speech processing module innovatively designs a dual threshold detection mechanism based on RMS value and short-time energy, combined with a 2-second segment accumulation strategy and a 0.5-second overlap interval, effectively solving the quality problem of speech recognition. The core innovation of the system is the design of a complete training-free adaptation mechanism, which intelligently schedules the capabilities of existing models through an LLM controller, ensuring the adaptability of the system to different scenarios, and avoiding the heavy training overhead in traditional solutions. In the response generation link, the system manages the state through special markers and uses stream processing to ensure real-time performance. This design enables the system to be quickly deployed and adapted to different application scenarios. Through the organic combination of these key technologies, the present application not only solves the training dependence problem in existing multi-modal dialogue systems, but also exhibits excellent versatility and scalability in practical applications.

[0064] The above untrained multi-modal dialogue system can significantly improve the overall performance of the system. In terms of technical effects, the following aspects are mainly embodied: first, in terms of deployment efficiency, since the untrained design idea is adopted, the system can directly reuse the existing open source model, which significantly reduces the deployment cost. Compared with the traditional method of collecting data and training models for specific scenarios, the system can achieve plug and play, and the deployment time is shortened from several weeks to several hours. Secondly, in terms of resource utilization efficiency, through the intelligent model selection mechanism, the system realizes the goal of calling the visual LLM on demand. Test data shows that in a typical dialogue scenario, about 70% of the interaction can be completed using only the text LLM, compared with the scheme of always enabling the visual LLM, the system's computing resource consumption is reduced by about 50%, and the average response delay is reduced by 40%. Third, in terms of system adaptability, the untrained architecture adopted by the present application can be flexibly adapted to different types of application scenarios. Experimental verification shows that the same system architecture can be seamlessly applied to customer service consultation, education guidance, medical inquiry and other fields without targeted training. In actual application tests, the dialogue completion rate of the system in different scenarios is more than 95%, and the user satisfaction is improved by 15% compared with the traditional scheme. Finally, in terms of maintenance cost, since the modular design and standardized interface are adopted, the system can easily update or replace each functional module. Especially in model iteration, when a new open source model is released, it can be integrated directly without retraining, which greatly reduces the maintenance cost and upgrade difficulty of the system. Through the above technical effect analysis, it can be seen that the untrained multi-modal dialogue framework proposed by the present application not only has innovation in technical implementation, but also shows significant cost advantage and use value in practical application. Through the innovative architecture design and control mechanism, the system effectively solves the training dependence problem in the traditional scheme, and provides a new solution for the large-scale application of intelligent dialogue systems.

[0065] The above is only an embodiment of the present application, and does not limit the technical scope of the present application in any way. Any minor modification, equivalent change and modification made according to the technical essence of the present application to the above embodiment are still within the scope of the technical solution of the present application.

Claims

1. An intelligent multimodal dialogue system, characterized in that, include: A multimodal input processing module includes an audio input unit, a video input unit, an audio processor, and a video processor. The audio input unit is communicatively connected to the audio processor, and the video input unit is communicatively connected to the video processor. The audio input unit is used to collect the user's voice input, and the audio processor packages the collected voice input into audio data packets. The video input unit is used to collect the user's video input, and the video processor performs frame processing on the collected video input and packages it into video data packets containing visual features. The speech recognition module includes an ASR processor, which performs validity detection on audio data packets by calculating the RMS value and short-time energy characteristics of the audio stream to determine whether it contains valid speech. Then, it uses a segmented accumulation strategy to process the valid speech segments and finally recognizes the processed speech segments as text information. The LLM controller module includes a controller LLM, a text LLM, and a visual LLM. The LLM controller performs deep semantic analysis on the text information output by the ASR processor and identifies whether visual understanding capability is required. When visual understanding capability is not required, the text LLM is invoked to process the text information and output a response and response marker. When visual understanding capability is required, the visual LLM is invoked, and the visual features of the video data packets are acquired. These visual features and the text information are used as input to the visual LLM. After processing, the visual LLM outputs a response and response marker. The response markers include "[S.LISTEN]", "[C.LISTEN]", and "[S.SPEAK]", where "[S.SPEAK]" indicates that the system is outputting, "[C.LISTEN]" indicates that the system is receiving user input, and "[S.LISTEN]" indicates that the system is waiting for a user response. The response generation module includes a system response unit and a speech synthesis unit. The system response unit is used to detect the response markers of the text LLM and visual LLM outputs. If the output is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", the speech synthesis unit is triggered: "[S.SPEAK]" is replaced with "empty", and then the speech synthesis service is input to provide a speech response to the user.

2. The intelligent multimodal dialogue system according to claim 1, characterized in that, The ASR processor performs validity checks by setting an adaptive dual threshold mechanism. When the RMS value of the audio exceeds a times the background noise level and the short-time energy exceeds a preset threshold, it is determined to be a valid speech segment. The segment accumulation strategy is to process audio in b-second units and maintain an overlap of c seconds between adjacent segments, where c < b.

3. The intelligent multimodal dialogue system according to claim 1, characterized in that, The ASR processor employs a dual filtering mechanism to filter the identified text information: it requires that the time interval between adjacent identification results be greater than d seconds; it also calculates the similarity between the new and old identified text, and only retains the new identified text information when the similarity is less than m%.

4. The intelligent multimodal dialogue system according to claim 1, characterized in that, The speech recognition module also includes an interruption control unit. The interruption control unit matches the text information recognized by the ARS processor using a preset high-priority keyword library. When a preset keyword is matched, an interruption operation is performed. When the text LLM or visual LLM is processed, it performs deep semantic analysis by combining user input and dialogue context information to identify the user's implicit interruption intention. When an interruption intention is detected, a response flag [S.STOP] is output, indicating that the user wants to stop or pause the dialogue. When the system response unit detects the response flag "[S.STOP]", it immediately clears the TTS output queue, and the system remains silent.

5. The intelligent multimodal dialogue system according to claim 1, characterized in that, The multimodal dialogue system uses the WebSocket communication mechanism to transmit data between the multimodal input processing module, the speech recognition processing module, the LLM controller module, and the response generation module. When packaging audio data, the audio processor adds a 0x00 flag to the header of the data packet; when packaging video data, the video processor adds a 0xFF flag to the header of the data packet.

6. An intelligent multimodal dialogue method, characterized in that, Includes the following steps: Step S1: Collect user input information and package it; Collect user-input audio information, sample and package the audio information to form audio data packets; Step S2: Information preprocessing; First, the validity of the audio data packets is checked by calculating the RMS value and short-time energy characteristics of the audio stream to determine whether it contains valid speech. Then, a segmented accumulation strategy is used to process the effective speech segments; Finally, the processed audio is recognized and converted into text information; Step S3: Use the controller LLM to perform deep semantic analysis on the text information to determine whether visual understanding capability is required. If visual understanding capability is required, call the text LLM to process the text information and then output the response and response tag. If visual understanding is not required, then Visual LLM is invoked, taking text information as input to Visual LLM and simultaneously acquiring visual features from video data as input to Visual LLM. After processing the text information and visual features, the Visual LLM model outputs the response and response label. Response flags include "[S.LISTEN]", "[C.LISTEN]", "[S.SPEAK]", and "[S.STOP]", where [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, and [S.LISTEN] indicates that the system is waiting for a user response. Step S4: Detect the response character output by the visual LLM or text LLM. If the response character is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", speech synthesis processing is triggered: replace "[S.SPEAK]" with "empty", and then input it into the speech synthesis service to provide a speech response to the user.

7. The intelligent multimodal dialogue method according to claim 6, characterized in that, In step S2, the validity identification specifically involves setting an adaptive dual threshold mechanism. When the RMS value of the audio exceeds a times the background noise level and the short-time energy exceeds a preset threshold, it is determined to be a valid speech segment. The segment accumulation strategy is to process audio in b-second units and maintain an overlap interval of c seconds between adjacent segments, where c < b.

8. The intelligent multimodal dialogue method according to claim 6, characterized in that, In step S3, after obtaining the text information, a dual filtering mechanism is used to filter the identified text information: the time interval between adjacent identification results is required to be greater than d seconds; at the same time, the similarity between the new and old identified texts is calculated, and the new identified text information is retained only when the similarity is less than m%.

9. The intelligent multimodal dialogue method according to claim 6, characterized in that, The method also includes interrupting the control flow, as detailed below: During the execution of step S2, the first level of interruption control is initiated. The text information recognized by the ARS processor is matched using a preset high-priority keyword library. When a preset keyword is matched, an interruption operation is performed. When no preset keyword is matched, the second level of interruption control is initiated. The second level of interruption control is as follows: When the text LLM or visual LLM in step S3 is processed, it combines user input and dialogue context information to perform deep semantic analysis, identify the user's implicit interruption intention, and when the interruption intention is detected, output the response flag [S.STOP], indicating that the user wants to stop or pause the dialogue. In step 4, if the response flag "[S.STOP]" is detected, the TTS output queue is immediately cleared and the system remains silent.

10. The intelligent multimodal dialogue method according to claim 6, characterized in that, The method uses the WebSocket communication mechanism for data transmission. In step S1, when packaging audio data, a 0x00 flag is added to the header of the data packet. In step S3, when acquiring video data, the video data is packaged and a 0xFF flag is added to the header of the data packet.

Images