Multi-modal inquiry data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a multi-mode inquiry data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: firstly, obtaining video data of a target task, carrying out the data decoding of the video data, and obtaining a plurality of decoded data frames, the target task comprising a task frame rate and a task minimum frame rate; then, obtaining processing parameters of algorithm consumption decoding data frames in the target equipment, determining a frame extraction rate based on the processing parameters, and when the frame extraction rate is not less than the task lowest frame rate, selecting at least one decoding data frame from the plurality of decoding data frames according to the frame extraction rate and sending the at least one decoding data frame to a cache region, selecting the decoded data frames from the cache region one by one for data processing to obtain target output data; and when the frame extraction rate is smaller than the lowest frame rate of the task, the cooperative processing device matched with the target task is selected to carry out data processing, so that the reliability, the real-time performance and the smoothness of processing the target task are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model data processing, and in particular to a multi-modal inquiry data processing method and device, an electronic device and a storage medium. BACKGROUND

[0002] Existing intelligent customer service systems, especially those based on large language models, mostly adopt the "LLM+RAG" (i.e., Retrieval Augmented Generation) architecture. These systems retrieve relevant documents through semantic matching and generate answers based on them to respond to user inquiries. With the development of technology, such customer service systems need to handle increasingly complex interactions, not limited to single text input, but extended to receiving and processing multi-modal data including text, speech, images or videos, and often need to maintain the continuity of context in multiple rounds of dialogue.

[0003] Although the existing "LLM+RAG" architecture can handle multiple types of information, they have significant deficiencies in handling complex inputs, especially "insufficient cross-modal and cross-round consistency". When a user inputs information of different modalities (such as voice instructions and image credentials) in one or more interactions, the system lacks effective mechanisms to identify and handle potential conflicts between these modalities. At the same time, the system also has difficulty in determining whether there is a logical conflict or dependency relationship between the current intent and the historical confirmed intent. The lack of comprehensive assessment capabilities for input quality, inter-modal consistency and inter-round conflicts results in the system being unable to form a reliable intent confidence, which can easily lead to deviations, conflicts or errors in understanding, thereby reducing the accuracy and practicality of intelligent customer service systems. SUMMARY

[0004] The embodiments of the present application provide a multi-modal inquiry data processing method, device, electronic device and storage medium, which can improve the accuracy and practicality of data processing by intelligent customer service systems.

[0005] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application provides a multi-modal inquiry data processing method, which comprises: acquiring multi-modal data input into the multi-modal large language model, and calculating a multi-modal quality score of the multi-modal data; determining at least one candidate intent corresponding to the multi-modal data and a corresponding intent probability, and calculating a multi-modal consistency indicator of the multi-modal data based on the intent probability; comparing the candidate intent with historical confirmed intents to obtain a round conflict indicator; calculating an intent confidence corresponding to the multi-modal data based on the multi-modal consistency indicator, the multi-modal quality score and the round conflict indicator; In the multi-modal large language model, based on the comparison relationship between the intent confidence and a preset confidence threshold and the candidate intent, corresponding output data is generated.

[0006] In some embodiments, the multi-modal data includes at least one of text data, speech data, and visual data, and the multi-modal quality score of the multi-modal data is calculated, including: Based on at least one of the invalid character ratio corresponding to the text data and the spelling correction edit distance normalization, the corresponding text quality is obtained; Based on at least one of the average confidence, the word error rate, and the signal-to-noise ratio corresponding to the speech data, the corresponding speech quality is calculated; Based on at least one of the average confidence and the clarity corresponding to the visual data, the corresponding visual quality is calculated; Based on at least one of the text quality, the speech quality, and the visual quality, weighted sum processing is performed to obtain the multi-modal quality score.

[0007] In some embodiments, the multi-modal data includes at least one modal data, and the determination of at least one candidate intent and the corresponding intent probability corresponding to the multi-modal data includes: Text encoding processing is performed on each of the modal data to obtain a text encoding vector; Each of the text encoding vectors is input into a pre-trained intent recognition model for data processing to obtain an initial candidate intent corresponding to each of the modal data and an intent probability corresponding to each of the initial candidate intents; From a plurality of the initial candidate intents, a plurality of the initial candidate intents with high intent probability are selected as the candidate intents.

[0008] In some embodiments, when the multi-modal data includes a plurality of modal data, the multi-modal consistency indicator of the multi-modal data is calculated based on the intent probability, including: The probability difference degree between the intent probabilities corresponding to the candidate intents of each two of the modal data is calculated; All of the probability difference degrees are averaged to obtain the multi-modal consistency indicator.

[0009] In some embodiments, the candidate intent is a plurality, and the comparison processing between the candidate intent and the historical confirmed intent is performed to obtain a round conflict indicator, including: Each of the candidate intents is compared with the historical confirmed intent to obtain an exclusive comparison result corresponding to each of the candidate intents; Confirm a precedent dependent intent corresponding to each of the candidate intents based on the precedent dependency information, and obtain a precedent state result of each of the candidate intents based on a completion state of the corresponding precedent dependent intent; Obtain the round conflict indicator corresponding to each of the candidate intents based on the mutual exclusion comparison result and / or the precedent state result.

[0010] In some embodiments, the obtaining, based on the multi-modal consistency indicator, the multi-modal quality score, and the round conflict indicator, of an intent confidence corresponding to the multi-modal data comprises: Perform weighted summation processing based on the round conflict indicator and the multi-modal consistency indicator corresponding to the candidate intent to obtain a comprehensive conflict metric corresponding to the candidate intent; Perform weighted summation processing based on the comprehensive conflict metric, the multi-modal quality score, and the intent probability to obtain the intent confidence corresponding to the candidate intent.

[0011] In some embodiments, the generating, based on the comparison relationship between the intent confidence and a preset confidence threshold and the candidate intent, of corresponding output data comprises: When the intent confidence is less than the preset confidence threshold, generating, based on the candidate intent, inquiry information corresponding to a clarification inquiry to the user, and obtaining reply information input by the user according to the inquiry information, and determining a target intent based on the reply information; When the intent confidence is not less than the preset confidence threshold, taking the candidate intent as the target intent; Performing data execution based on the target intent, and generating the output data based on an execution result.

[0012] In some embodiments, the generating, based on the candidate intent, of inquiry information corresponding to a clarification inquiry to the user comprises: Performing clarification reason analysis on the candidate intent to obtain an analysis result; When the analysis result indicates that the candidate intent has a conflict, generating, based on a plurality of the candidate intents and a selection template prompt word, inquiry information corresponding to a selection clarification inquiry to the user; When the analysis result indicates that the candidate intent has a missing priority, determining an uncompleted precedent dependent intent corresponding to the candidate intent based on precedent dependency information; Marking the candidate intent as a suspended state, and recording it as a to-be-resumed intent in a state machine; Generating, based on the precedent dependent intent, inquiry information for guiding the user to complete the precedent dependent intent.

[0013] In some embodiments, the method further comprises: obtaining a state update result of the user after completing the preceding dependent intent according to the query information; extracting the to-be-resumed intent when a completion state of the preceding dependent intent is monitored to change to completed; resuming an execution context of the to-be-resumed intent, and taking the to-be-resumed intent as the target intent for data execution.

[0014] In some embodiments, the data execution based on the target intent and the generation of the output data based on the execution result comprise: compliance verification based on the target intent and user information of the user, to obtain a compliance verification result; when the compliance verification result represents that the target intent and the user information are compliant information, data execution based on the target intent, and generation of the output data based on the execution result; when the compliance verification result represents that the target intent and the user information are non-compliant information, generating the output data for rejection prompt.

[0015] To achieve the above-mentioned purpose, a second aspect of the embodiment of the present application proposes a multi-modal query data processing device, the device comprises: an input data acquisition module configured to acquire multi-modal data input into the multi-modal large language model, and calculate a multi-modal quality score of the multi-modal data; a consistency index calculation module configured to determine at least one candidate intent corresponding to the multi-modal data and a corresponding intent probability, and calculate a multi-modal consistency index of the multi-modal data based on the intent probability; a conflict index calculation module configured to compare the candidate intent with a historical confirmed intent to obtain a round conflict index; a confidence calculation module configured to calculate an intent confidence corresponding to the multi-modal data based on the multi-modal consistency index, the multi-modal quality score, and the round conflict index; an output module configured to generate corresponding output data based on the intent confidence and a preset confidence threshold comparison relationship and the candidate intent in the multi-modal large language model.

[0016] To achieve the above-mentioned purpose, a third aspect of the embodiment of the present application proposes an electronic device, the electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the multi-modal query data processing method of the first aspect when executing the computer program.

[0017] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal query data processing method described in the first aspect.

[0018] The multimodal query data processing method, apparatus, electronic device, and storage medium proposed in this application include: First, acquiring multimodal data input to a multimodal large language model and calculating the multimodal quality score of the multimodal data; then, determining at least one candidate intent and its corresponding intent probability corresponding to the multimodal data, and calculating a multimodal consistency index based on the intent probability; next, comparing the candidate intent with historical confirmed intents to obtain a round conflict index; then, calculating the intent confidence level corresponding to the multimodal data based on the multimodal consistency index, the multimodal quality score, and the round conflict index; finally, generating corresponding output data in the multimodal large language model based on the comparison relationship between the intent confidence level and a preset confidence threshold and the candidate intent. This application first calculates a multimodal quality score to assess the reliability of the input data itself. Simultaneously, by calculating a multimodal consistency index, it proactively identifies and quantifies potential intent conflicts between different modalities such as voice and images. Combined with a round-of-turn conflict index, it judges the logical consistency between the current intent and historical dialogues. Then, it comprehensively quantifies these three key dimensions (quality, modal consistency, and round-of-turn conflict) into a reliable intent confidence level. Based on this confidence level, the system can proactively initiate clarification inquiries (e.g., when the confidence level is low) instead of executing incorrectly, thereby effectively avoiding subsequent operation failures caused by misunderstandings or conflicts. This significantly improves the accuracy, reliability, and practicality of the intelligent customer service system in complex, multimodal, and multi-turn dialogue scenarios.

[0019] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0020] Figure 1 This is a flowchart of a multimodal query data processing method provided in an embodiment of this application.

[0021] Figure 2 yes Figure 1 The flowchart for step 101.

[0022] Figure 3 yes Figure 1 The flowchart for step 102.

[0023] Figure 4 yes Figure 1 Another flowchart for step 102.

[0024] Figure 5 yes Figure 1 The flowchart for step 103.

[0025] Figure 6 yes Figure 1 The flowchart for step 104.

[0026] Figure 7 yes Figure 1 The flowchart for step 105.

[0027] Figure 8 yes Figure 7 The flowchart for step 701.

[0028] Figure 9 This is a flowchart of a process for resuming execution of an intent to be restored, provided in another embodiment of this application.

[0029] Figure 10 yes Figure 7 The flowchart for step 703.

[0030] Figure 11 This is a schematic diagram of an intent dictionary provided in another embodiment of this application.

[0031] Figure 12 This is a schematic diagram of intent confidence provided in another embodiment of this application.

[0032] Figure 13 This is a schematic diagram of the structure of a multimodal query data processing device provided in another embodiment of this application.

[0033] Figure 14 This is a schematic diagram of the hardware structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0037] Existing intelligent customer service systems, especially those based on large language models, mostly adopt an "LLM+RAG" (Retrieval Enhanced Generation) architecture. These systems retrieve relevant documents through semantic matching and generate answers based on these documents to respond to user queries. With technological advancements, the interactions that these customer service systems need to handle are becoming increasingly complex, no longer limited to single text input, but extending to receiving and processing multimodal data including text, voice, images, or video, and often requiring the maintenance of contextual continuity throughout multi-turn dialogues.

[0038] While existing "LLM+RAG" architectures can handle a variety of information, they have significant shortcomings when dealing with complex inputs, particularly in terms of insufficient cross-modal and cross-round consistency. When users input information from different modalities (such as voice commands and image credentials) in one or more interactions, the system lacks an effective mechanism to identify and handle potential intent conflicts between these modalities. Simultaneously, the system struggles to determine whether there are logical conflicts or dependencies between the current intent and previously confirmed intents. This lack of comprehensive assessment capabilities regarding input quality, intermodal consistency, and inter-round conflicts prevents the system from forming a reliable intent confidence level, making it prone to misunderstandings, conflicts, or erroneous execution, thus reducing the accuracy and usability of the intelligent customer service system.

[0039] To improve the accuracy and practicality of data processing in intelligent customer service systems, this application first calculates a multimodal quality score to assess the reliability of the input data itself. Simultaneously, by calculating a multimodal consistency index, it proactively identifies and quantifies potential intent conflicts between different modalities such as voice and images. Combined with a round-of-turn conflict index, it determines the logical consistency between the current intent and historical dialogues. These three key dimensions (quality, modal consistency, and round-of-turn conflict) are then comprehensively quantified into a reliable intent confidence level. Based on this confidence level, the system can proactively initiate clarification inquiries (e.g., when the confidence level is low) instead of executing incorrectly, thereby effectively avoiding subsequent operational failures due to misunderstandings or conflicts. This significantly improves the accuracy, reliability, and practicality of the intelligent customer service system in complex, multimodal, and multi-turn dialogue scenarios.

[0040] The following will further describe the multimodal query data processing method, apparatus, electronic device, and storage medium provided in the embodiments of this application. The multimodal query data processing method provided in the embodiments of this application can be applied to processing systems (such as smart terminals, servers, computers, etc.) that are loaded with or associated with intelligent customer service systems, or processing systems (such as smart terminals, servers, computers, etc.) that are loaded with or associated with large language models.

[0041] Reference Figure 1 This is an optional flowchart of the multimodal query data processing method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 105. It is also understood that this embodiment... Figure 1 The order of steps 101 to 105 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0042] Step 101: Obtain the multimodal data of the input multimodal large language model and calculate the multimodal quality score of the multimodal data.

[0043] Step 101 will be described in detail below.

[0044] In some embodiments, the system first performs the operation of "acquiring multimodal data of the input multimodal large language model." Here, "multimodal data" refers to the processing object of this scheme, specifically input containing multiple information formats, such as text, speech, image, or video data provided by the user simultaneously or sequentially. After acquiring the data, the system immediately "calculates the multimodal quality score of the multimodal data." This "multimodal quality score" is used to quantify the clarity, completeness, or recognizability of the input data itself (e.g., signal-to-noise ratio of speech, image clarity, or effective character ratio of text). This score will serve as one of the basic weights for subsequently determining the reliability of the intent.

[0045] The following section will further describe how to calculate the multimodal quality score corresponding to the multimodal data.

[0046] Reference Figure 2 The multimodal data includes at least one of text data, voice data, and visual data. The multimodal quality score of the multimodal data is calculated, including the following steps 201 to 204.

[0047] Step 201: Obtain the corresponding text quality based on at least one of the proportion of invalid characters in the text data and the spell correction edit distance normalization.

[0048] Step 202: Calculate the corresponding speech quality based on at least one of the average confidence level, word error rate, and signal-to-noise ratio corresponding to the speech data.

[0049] Step 203: Calculate the corresponding visual quality based on at least one of the average confidence level and sharpness corresponding to the visual data.

[0050] Step 204: Perform a weighted summation based on at least one of the text quality, speech quality, and visual quality to obtain a multimodal quality score.

[0051] Steps 201 to 204 are described in detail below.

[0052] In some embodiments, the system performs a calculation based on at least one of the proportion of invalid characters corresponding to the text data and the spell correction edit distance normalization to quantify text quality. The "invalid character ratio" here is a technical metric used to measure the proportion of meaningless, garbled, or non-standard characters (i.e., noise) in the text. The "spell correction edit distance normalization" is another technical metric. It calculates the minimum number of editing operations (such as additions, deletions, and modifications) required to correct the input text to standard spelling, and then normalizes this number (e.g., by dividing by the total text length) to reflect the spelling accuracy and standardization of the text. Through any one of these metrics or a combination of both, the system ultimately "obtains the corresponding text quality," and this "text quality" score characterizes the reliability and usability of the "text data."

[0053] Simultaneously, the system performs this calculation "based on at least one of the following: average confidence score, word error rate, and signal-to-noise ratio (SNR) corresponding to the speech data." Specifically, the "average confidence score" typically comes from an Automatic Speech Recognition (ASR) engine and is the engine's average confidence score for the accuracy of its transcription results; the "word error rate" (WER) is an industry-standard measure of the difference between ASR-transcribed text and the human-annotated ground truth, with a lower error rate indicating higher quality; and the "signal-to-noise ratio" (SNR) is a physical quantity used to measure the contrast between the strength of the speech signal and the strength of background noise, with a high SNR indicating clear speech with less interference. The system uses one or more of these metrics to "calculate the corresponding speech quality." This allows for the assessment of the intelligibility and credibility of the "voice data".

[0054] In addition, the system "calculates the corresponding visual quality based on at least one of the average confidence level and sharpness corresponding to the visual data". In this technological context, "average confidence" can be derived from the recognition model applied to "visual data" (such as optical character recognition OCR or object detection models), representing the model's confidence in the accuracy of the recognition results (such as extracted text or detected objects). "Sharpness," on the other hand, is an image quality metric used to assess the sharpness of an image or the presence of blur (e.g., it can be obtained by calculating the Laplacian variance of the image). "Visual data" with low "sharpness" will directly affect the accuracy of subsequent recognition and understanding.

[0055] Finally, the system performs a weighted summation process on one or more quality scores, based on "at least one of text quality, speech quality, and visual quality." This weighted summation is a fusion technique that allows the system to assign different importance weights to the quality of different modalities (i.e., "text quality," "speech quality," and "visual quality"). This design can flexibly adapt to scenarios where "multimodal data" contains only some modalities (e.g., only text). Through this fusion calculation, the system ultimately obtains a "multimodal quality score," i.e. This score is the final quantitative assessment of the overall reliability of the user's multimodal input.

[0056] Through steps 201 to 204 above, a comprehensive and refined multimodal input quality assessment mechanism is achieved. It abandons the assumption of equal input data quality and instead conducts independent and in-depth quantitative analysis of "text quality" (based on factors such as "invalid character ratio"), "speech quality" (based on factors such as "word error rate" and "signal-to-noise ratio"), and "visual quality" (based on factors such as "average confidence" and "clarity"). Finally, through "weighted summation," these factors are integrated into a unified "multimodal quality score." This "multimodal quality score" provides a crucial and quantifiable reliable basis for subsequent intent understanding and decision-making (e.g., when calculating intent confidence), enabling the system to identify and properly handle low-quality or noisy inputs, avoiding the risk of making erroneous responses based on unreliable data, thereby significantly improving the robustness and interactive accuracy of the entire multimodal large language model system.

[0057] Furthermore, in this embodiment, intent dictionary construction is also required. This step is the foundation for building the entire intelligent customer service system. Its core is to define the boundaries of "what can be done" (intent dictionary) and provide the knowledge of "what to know" (business knowledge base). In the technical field, this stage is called the system initialization and knowledge engineering stage, which deeply binds unstructured artificial intelligence (such as large models) with structured, highly controllable business logic.

[0058] The "intent dictionary" provided in this application is constructed by domain experts, clearly defining the business scope, required slots, optional slots, business prerequisites, and backend APIs. The "business knowledge base" is used to store document text resources and multimodal resources. Multimodal embedding models (such as CLIP, Video CLIP, etc.) are used to perform unified semantic representation of text, images, video, and audio, which is then stored in a multimodal vector database.

[0059] First, the construction of the "Intent Dictionary" is explained in detail. This is not merely a simple dictionary, but rather the "business rule engine" and "state machine blueprint" of the entire system. It is constructed by "domain experts" (rather than AI) to ensure the absolute accuracy and authority of the business logic. "Defining the business scope" acts as the first layer of guardrail for the large model, ensuring that the system only responds to business that it is authorized to handle. "Defining required and optional slots" establishes a standardized data structure (Schema) for each "business intent," directly supporting the subsequent "clarification query" (used to fill missing "required slots") and the "parameter integrity check" in step 7. "Business prerequisite dependencies" are the core solution to the "missing business dependencies" defect in existing technologies, providing underlying rule support for the subsequent "suspend and resume" mechanism. Finally, the definition of the "backend API" is the bridge connecting intents and actions, enabling the system to trigger real-world operations (such as in step 7) when the "intent confidence" is high enough.

[0060] Secondly, the establishment of a "business knowledge base" is a prerequisite for realizing the RAG (Retrieval Enhanced Generation) function. The key to this solution lies in its being a "multimodal" knowledge base, storing not only "document text resources" but also "images, videos, and audio." To enable the system to understand and retrieve this heterogeneous data, the solution employs "multimodal embedding models (such as CLIP, VideoCLIP, etc.)." In the technical field, the revolutionary aspect of such models (such as CLIP) is their ability to map different modalities (such as an image of a "prescription" and the text of "querying a prescription") to the same high-dimensional "semantic representation" space. This means that the system can "understand" images and videos and comprehend their semantic relationships with text queries. After storing these representations in a "multimodal vector database," the system can efficiently recall the knowledge fragments most relevant to the user's multimodal input through vector similarity search during the RAG retrieval stage.

[0061] Furthermore, the intent recognition model was fine-tuned beforehand: the intent classification model was fine-tuned using real user question-and-answer data for subsequent intent recognition.

[0062] "Fine-tuning the intent recognition model" is crucial for adapting general AI capabilities to specific business scenarios. The system fine-tunes a general intent classification model by collecting real user question-and-answer data. The goal of this process is to make the model highly familiar with the specific terminology and expression habits of the business, enabling it to accurately generate "candidate intents" and their "probability distributions" when faced with real and complex user input in subsequent steps. This fine-tuned model is the starting point for all subsequent intent recognition, consistency verification, and confidence calculations; its accuracy directly determines the quality of the system's upstream processing.

[0063] Step 102: Determine at least one candidate intent and its corresponding intent probability corresponding to the multimodal data, and calculate the multimodal consistency index of the multimodal data based on the intent probability.

[0064] Step 102 is described in detail below.

[0065] In some embodiments, the system begins preliminary intent parsing after acquiring the data. This step first performs the process of "determining at least one candidate intent and its corresponding intent probability corresponding to the multimodal data," which involves analyzing each modality of data using a pre-trained intent recognition model to generate one or more possible business requests (i.e., "candidate intents") and assigning an initial probability value (i.e., "intent probability") to the accuracy of each "candidate intent." Subsequently, the key step in this process is "calculating a multimodal consistency index based on the intent probability." This "multimodal consistency index" is a quantitative measure used to assess whether there are conflicts between the intents indicated by different modalities (such as voice and image); if the probability distributions of the "candidate intents" pointed to by different modalities differ significantly, this index will reflect this inconsistency.

[0066] Reference Figure 3 The multimodal data includes at least one modality of data. Determining at least one candidate intent and the corresponding intent probability corresponding to the multimodal data includes the following steps 301 to 303.

[0067] Step 301: Perform text encoding processing on each modality of data to obtain a text encoding vector.

[0068] Step 302: Input each text encoding vector into the pre-trained intent recognition model for data processing to obtain the initial candidate intent corresponding to each modality and the intent probability corresponding to each initial candidate intent.

[0069] Step 303: Select multiple initial candidate intents with high intent probabilities from multiple initial candidate intents as candidate intents.

[0070] Steps 301 to 303 are described in detail below.

[0071] In some embodiments, the system first receives "modal data," which can be one or more types, such as text data, speech data, image data, or video data. A key operation involves "text encoding" of this data. This processing is a normalization process designed to convert all non-textual "modal data" into a unified text representation; for example, performing Automatic Speech Recognition (ASR) transcription on speech data, or performing Optical Character Recognition (OCR) on image / video data to extract text. The output of this process is a "text-encoded vector," a high-dimensional digital representation that uniformly encodes raw information from different sources into a standardized input format that subsequent models (such as "intent recognition models") can understand and process.

[0072] The multimodal data to be acquired, including text data, speech data, and visual data (i.e., image data or video data), will be uniformly encoded into text encoding vectors. .

[0073] Text data: Encoded into vector representations using this table or a pre-trained language model. .

[0074] Speech data: Speech is transcribed into text using ASR, and the transcribed text is then processed using the same text encoding model to obtain a vector representation. .

[0075] Image or video data: Use OCR to extract text information from images or videos; for videos, subtitles or keyframe text can be extracted, and vectors can be obtained using a text encoding model. or .

[0076] The modal vectors are fused into a text encoding vector. It includes the text encoding vector corresponding to the text data. The text encoding vector corresponding to the speech data Text encoding vectors corresponding to visual data (including image data and / or video data). or .

[0077] Next, the generated "each text encoding vector" (i.e., the standardized input from each modality) is "inputted into the pre-trained intent recognition model for data processing." This "pre-trained intent recognition model" is a classifier fine-tuned with domain-specific data (e.g., real user question-and-answer sessions), whose technical function is to parse the user intent implied in the input vector. The result of the processing is "obtaining the initial candidate intents corresponding to each modality of data and the intent probability corresponding to each initial candidate intent." This means that the system generates a list of intents (i.e., "initial candidate intents") for each input modality (e.g., speech and image), and assigns a quantified confidence score (i.e., "intent probability") to each intent in the list (e.g., "appoint an appointment" or "query history").

[0078] Next, given the "multiple initial candidate intents" and their corresponding "intent probabilities" from one or more modalities, the system will perform the operation of "selecting multiple initial candidate intents with high intent probabilities" to improve the efficiency and accuracy of subsequent processing. Technically, this operation typically corresponds to a Top-k (select the top k) algorithm, which retains only those intents with the highest "intent probability" values ​​that are most likely to represent the user's true needs. The intents remaining after this filtering step are identified as "candidate intents," and they will be passed to subsequent processes (e.g., for calculating consistency metrics or confidence levels) as the most relevant set of intents.

[0079] Through steps 301 to 303 above, a complete processing chain is formed from the original multimodal input to the generation of highly relevant candidate intents. First, the standardization problem of heterogeneous modal data is solved by "text encoding processing". Then, the intent inference of each modality is performed by "intent recognition model" to obtain preliminary "initial candidate intents" with probabilities. Finally, through the filtering mechanism of "selecting intents with high probability", the set of "candidate intents" that are most worthy of the system's attention is refined from the complex preliminary results. This process ensures that the system can efficiently and accurately extract the user's core intent from complex multimodal input, providing a clear and reliable data foundation for subsequent consistency verification, conflict detection and generation of the final response.

[0080] The following section will further describe how to obtain the multimodal consistency index for multimodal data.

[0081] Reference Figure 4 When multimodal data includes multiple modalities, the multimodal consistency index of the multimodal data is calculated based on the intent probability, including the following steps 401 to 402.

[0082] Step 401: Calculate the probability difference between the candidate intent probabilities corresponding to each of the two modalities.

[0083] Step 402: Average all probability differences to obtain the multimodal consistency index.

[0084] Steps 401 to 402 are described in detail below.

[0085] In some embodiments, when the system receives "multi-modal data" (e.g., including both voice and image data), the system performs the operation of "calculating the probability difference between the probability of candidate intents corresponding to each of the two modalities of data". Here, "intention probability" refers to the probability of the candidate intent inferred independently for each individual modality (e.g., voice modality, image modality) in previous steps. The probability distribution of " Therefore, the "Probability Difference Degree" is a key quantitative metric (e.g., achievable through methods such as Jensen-Shannon Divergence and JSD) specifically designed to measure the difference or distance between the "intention probability" distributions of any two modalities (e.g., speech and image, or image and text). A lower probability difference value indicates a highly similar distribution of intention probabilities between two modalities (i.e., both pointing to the same intention); conversely, a higher probability difference value indicates significantly different distributions (i.e., pointing to different or conflicting intentions).

[0086] Then, the generated multiple pairwise discrepancy measures are aggregated and merged. The system "averages all probability discrepancies." Here, "all probability discrepancies" refers to the calculated set of probability discrepancy values ​​that cover all pairwise combinations of modal data (e.g., speech-image, speech-text, image-text). By performing "averaging" (e.g., taking the arithmetic mean), the system merges these scattered, pairwise conflict measures into a single, global metric score. The final output of this process is the "multimodal consistency index." The single value of this indicator represents the level of inherent consistency in the overall intent expression of the "multi-modal data" input. The lower the value, the higher the consistency, and the higher the value, the greater the conflict between modalities.

[0087] Through steps 401 and 402 above, an effective mechanism for quantifying multimodal input conflicts is implemented. By calculating the probability difference on a pairwise basis, potential inconsistencies between all modalities are systematically identified, rather than relying solely on the input of a single modality. Subsequently, by averaging all probability differences, this complex and multidimensional conflict information is integrated into a single, clear "multimodal consistency index." This "multimodal consistency index" provides a key, quantifiable input for subsequent intent confidence calculation, enabling the system to accurately determine the inherent consistency of user input. This triggers a clarification mechanism when high conflict is detected (i.e., high index value), effectively avoiding business errors caused by executing contradictory multimodal instructions and significantly improving the robustness and accuracy of the system.

[0088] Step 103: Compare the candidate intents with the historical confirmed intents to obtain the round conflict index.

[0089] Step 103 will be described in detail below.

[0090] In some embodiments, the consideration of dialogue history context is further introduced to ensure the logical coherence of multi-turn interactions. The system performs a "comparison processing based on candidate intents and historically confirmed intents," where "historically confirmed intents" refer to business intents that have been jointly confirmed and recorded by the system and the user in previous dialogue rounds. By comparing the current "candidate intents" with these "historically confirmed intents" (e.g., checking for mutual exclusion or pre-sequence dependencies), the system ultimately "obtains a round conflict index." This "round conflict index" is used to characterize the degree of conflict between the currently identified "candidate intents" and the dialogue history context in terms of time sequence or business logic, as described below.

[0091] Reference Figure 5 There are multiple candidate intentions. The candidate intentions are compared with historical confirmed intentions to obtain the round conflict index, including the following steps 501 to 503.

[0092] Step 501: Perform a mutual exclusion comparison between each candidate intent and the historical confirmed intent to obtain the mutual exclusion comparison result corresponding to each candidate intent.

[0093] Step 502: Based on the preceding dependency information, identify the preceding dependent intent corresponding to each candidate intent, and based on the completion status of the corresponding preceding dependent intent, obtain the preceding status result of each candidate intent.

[0094] Step 503: Based on the mutual exclusion comparison results and / or the previous state results, obtain the round conflict index corresponding to each candidate intent.

[0095] Steps 501 to 503 are described in detail below.

[0096] In some embodiments, conflict determination is based on the "mutually exclusive / preceding" rules in the intent dictionary, first performing a process of direct logical conflict detection between the "candidate intent" and the dialogue history. The system performs a "mutually exclusive comparison of each candidate intent with historical confirmed intents." This "mutually exclusive comparison" is a key logical check used to determine whether a newly identified "candidate intent" (e.g., "refund") and a "historical confirmed intent" (e.g., "confirmed receipt") have an irreconcilable contradiction with predefined business rules (e.g., stored in the intent dictionary). The output of this comparison is "the mutually exclusive comparison result corresponding to each candidate intent," which is typically a Boolean value or status flag indicating whether the "candidate intent" contradicts the historical record.

[0097] Then, the "candidate intents" are evaluated from the perspective of the sequence of business processes. The system first "confirms the preceding dependent intents corresponding to each candidate intent based on the preceding dependency information." Here, "preceding dependency information" refers to a set of business rules predefined in the system (e.g., an intent dictionary), which clarifies the other intents (i.e., "preceding dependent intents," such as "credit card activation") that must be completed before executing a certain intent (e.g., "credit card limit adjustment"). After confirming this dependency relationship, the system then "obtains the preceding state result for each candidate intent based on the completion status of the corresponding preceding dependent intents." This "preceding state result" is a judgment result used to characterize whether the preconditions have been met (e.g., determining whether "credit card activation" has been completed by querying the state machine).

[0098] Subsequently, the system makes a comprehensive judgment "based on the mutual exclusion comparison results and / or the results of preceding states." This "and / or" expression provides technical flexibility, meaning the system can rely solely on "mutual exclusion comparison results," or solely on "preceding state results," or combine both for comprehensive consideration (e.g., through weighted summation). Ultimately, the system "obtains the round-based conflict index corresponding to each candidate intent." The “round conflict index” is a quantified score or status value that integrates two types of information: direct logical conflict and business process sequence conflict, comprehensively reflecting the compatibility between “candidate intent” and dialogue history.

[0099] Through steps 501 to 503 above, a dialogue history consistency verification mechanism is constructed to evaluate the logical rationality of "candidate intents" in the context of multi-turn dialogues. It no longer understands the current intent in isolation, but prevents direct logical contradictions through "mutually exclusive comparison" and ensures the compliance and timing of business processes by checking the "completion status of preceding dependent intents". Finally, by integrating the results of these two checks to generate "turn conflict indicators", a key quantitative input on the contextual logic is provided for subsequent intent confidence calculation. This enables the system to accurately identify and process requests that are inappropriate or violate business rules in specific dialogue turns, significantly improving the accuracy and logical coherence of multi-turn dialogues.

[0100] Step 104: Calculate the intent confidence level corresponding to the multimodal data based on the multimodal consistency index, multimodal quality score, and round conflict index.

[0101] Step 104 will be described in detail below.

[0102] In some embodiments, the system obtains the evaluation results of all the aforementioned dimensions and performs the following: "Calculate the intent confidence level corresponding to the multimodal data based on the multimodal consistency index, multimodal quality score, and round conflict index." The “intent confidence” here is a comprehensive weighted score that no longer relies solely on the intent probability output of a single model. Instead, it integrates the reliability of the input data itself (i.e., the “multimodal quality score”), the inherent consistency between different modalities (i.e., the “multimodal consistency index”), and the logicality of the dialogue context (i.e., the “turn conflict index”), thus deriving a more comprehensive and robust final confidence score, as described below.

[0103] Reference Figure 6 Based on the multimodal consistency index, multimodal quality score, and round conflict index, the intent confidence corresponding to the multimodal data is calculated, including the following steps 601 to 602.

[0104] Step 601: Based on the round conflict index and multimodal consistency index corresponding to the candidate intent, perform weighted summation to obtain the comprehensive conflict metric corresponding to the candidate intent.

[0105] Step 602: Based on the comprehensive conflict metric, multimodal quality score and intent probability, perform weighted summation to obtain the intent confidence corresponding to the candidate intent.

[0106] Steps 601 to 602 are described in detail below.

[0107] In some embodiments, the system performs "round conflict index based on candidate intent". and multimodal consistency index The system performs a weighted summation process. Here, the "round conflict index" characterizes the degree of conflict between the "candidate intent" and the dialogue history context (e.g., pre-order dependencies or mutual exclusion relationships); while the "multimodal consistency index" characterizes the degree of intent conflict between different modalities (e.g., speech and image) in the current input. By performing a weighted summation process on these two indices reflecting conflict in different dimensions, the system obtains a comprehensive conflict metric corresponding to the candidate intent. ,in As a weighted factor, the "Comprehensive Conflict Measure" is an intermediate result that uniformly quantifies all the conflicts and uncertainties faced by the "candidate intentions".

[0108] Then, the system executes "based on comprehensive conflict measurement". Multimodal mass fraction and the probability of intent The system then performs a weighted summation process. During this fusion, three key inputs are considered: the "Comprehensive Conflict Metric" (representing the overall conflict risk), the "Multimodal Quality Score" (representing the physical reliability of the input data itself), and the "Intent Probability" (representing the initial inference probability of the intent recognition model for the "candidate intent"). This is followed by another weighted summation process (which technically allows the conflict metric and quality score to correct or penalize the intent probability). ,in , as well as All are weighted factors, and the system ultimately “obtains the intent confidence level corresponding to the candidate intent”. This “intent confidence level” is the final and most reliable quantitative score used for subsequent decisions (such as clarification or execution).

[0109] Through steps 601 and 602 above, a layered, multi-dimensional information fusion mechanism is implemented to calculate a highly reliable intent confidence score. The final intent confidence score is generated through a two-step fusion process (first fusing all conflicts, then fusing conflicts, quality, and model probabilities). This intent confidence score is no longer a single score that relies solely on model inference (intent probability), but a comprehensive score that has been fully corrected and adjusted by "multimodal quality score" (input reliability), "multimodal consistency index" (intermodal conflict), and "round conflict index" (contextual conflict). This allows the intent confidence score to more accurately reflect the system's true understanding of the user's intent, providing a more robust and intelligent decision-making basis for whether to initiate clarification or execute directly, and significantly improving the accuracy and security of the system in complex interaction scenarios.

[0110] Step 105: In the multimodal large language model, based on the comparison between intent confidence and pre-set confidence threshold and candidate intent, generate corresponding output data.

[0111] Step 105 is described in detail below.

[0112] In some embodiments, the system will base its decisions on a comprehensive "intent confidence level". The final response strategy is determined based on this information. Specifically, "in a multimodal large language model, corresponding output data is generated based on the comparison between intent confidence and a pre-set confidence threshold, and candidate intents." Here, the "pre-set confidence threshold" is a criterion for decision-making. If the intent confidence is not less than this threshold, it indicates that the system has a high degree of certainty in understanding the candidate intent, and the "output data" may directly execute the business logic or provide an answer. Conversely, if the intent confidence is less than this threshold, it indicates ambiguity, conflict, or low-quality input. In this case, the generated "output data" may be a specific script used to initiate a clarification inquiry or confirmation request to the user, as described below.

[0113] Reference Figure 7 Based on the comparison between intent confidence and a preset confidence threshold and candidate intents, corresponding output data is generated, including the following steps 701 to 703.

[0114] Step 701: When the confidence level of the intent is less than the preset confidence threshold, generate the corresponding inquiry information for clarifying the inquiry to the user based on the candidate intent, obtain the response information input by the user based on the inquiry information, and then determine the target intent based on the response information.

[0115] Step 701 will be described in detail below.

[0116] In some embodiments, the system first describes a "clarification" processing branch when it detects a low-confidence intent. This branch is triggered when the intent confidence level is less than a preset confidence threshold. This condition indicates that, after comprehensive evaluation, the system believes the current "candidate intent" has significant conflict, ambiguity, or input quality issues and is not suitable for direct execution. In this case, the system will "generate query information based on the candidate intent to clarify the question to the user," for example, providing a confirmation question with two options. Subsequently, the system "obtains the response information input by the user based on the query information," and after receiving the user's explicit "response information," "determines the target intent based on the response information." This "target intent" is an intent that has been actively confirmed by the user and whose ambiguity has been eliminated, replacing the original ambiguous "candidate intent."

[0117] The following section will describe in detail how to generate this query information.

[0118] Reference Figure 8The process involves generating inquiry information for clarifying questions to users based on candidate intents, including steps 801 to 805.

[0119] Step 801: Clarify the reasons for the candidate intent and obtain the analysis results.

[0120] Step 802: When the analysis results represent conflicting candidate intentions, based on multiple candidate intentions and selection template prompts, generate the corresponding query information for clarifying the selection to the user.

[0121] Step 803: When the analysis results indicate that the candidate intent has a priority deficiency, determine the incomplete preceding dependent intent corresponding to the candidate intent based on the preceding dependency information.

[0122] Step 804: Mark the candidate intent as suspended and record it as an intent to be resumed in the state machine.

[0123] Step 805: Generate the query information based on the preceding dependency intent to guide the user to complete the preceding dependency intent.

[0124] Steps 801 to 805 are described in detail below.

[0125] In some embodiments, the necessary diagnostic phase prior to initiating a clarification inquiry is described first. The system first performs a "clarification cause analysis of the candidate intent." This "clarification cause analysis" is an internal diagnostic process designed to accurately pinpoint the specific root causes that led to the previous steps determining the "intent confidence level" to be too low. These causes may include: intermodal conflict (e.g., an excessively high "multimodal consistency index" calculated in step 402), inter-round conflict (e.g., an excessively high "round conflict index" calculated in step 503), or a situation where, although the "candidate intent" itself is clear, the key information (i.e., "priority") upon which its execution depends is missing. Finally, the system "obtains an analysis result," which is a classification label that clarifies the clarification objective (e.g., whether it is "resolving conflict" or "filling in the gaps"), providing a decision-making basis for selecting the correct clarification strategy in subsequent steps.

[0126] The system then further describes specific response strategies for "conflicting" diagnostic results. The trigger condition is "when the analysis results represent conflicting candidate intentions." This "conflict" is a key reason for low confidence levels. For example, in one embodiment, the voice intention "appointment booking" and the image intention "querying historical registration" coexist, constituting a direct conflict between "candidate intentions." In this case, the system will construct a response "based on multiple candidate intentions and selection template prompts." Here, "multiple candidate intentions" refers to these conflicting intention options (such as "appointment" and "query"), while "selection template prompts" are pre-set phrases used to guide the user to make a choice between two or more options (e.g., "Do you prefer... or...?"). Finally, the system "generates inquiry information corresponding to the user's choice clarification question." This "inquiry information" is a multiple-choice clarification script designed to force the user to explicitly eliminate ambiguity.

[0127] In traditional intelligent customer service, if a user's intent (such as "credit card limit adjustment") depends on a precondition that has not yet been met (such as "credit card activation"), the system often fails to execute or gives an invalid response. This application's embodiment introduces an intelligent, dynamic process control logic to ensure that complex business flows are executed in the correct order and in compliance with regulations, as described below.

[0128] When the analysis results indicate a lack of priority in a candidate intent, the system determines the incomplete preceding dependent intent based on the preceding dependency information. Specifically, "lack of priority" here refers to a situation where, although the current candidate intent is clear, its execution, according to business logic rules, depends on another business node that has not yet been completed. The system consults pre-set "preceding dependency information" (such as the business logic chain stored in the intent dictionary) to trace back and locate the specific business intent that the current candidate intent must depend on, and whose current state is incomplete; this is the "preceding dependent intent." For example, in a credit card limit adjustment scenario, if an activation status is detected as not being met, the system will use this step to accurately identify "credit card activation" as a preceding dependent intent that must be processed with priority.

[0129] Then, the system marks the candidate intent as suspended and records it as an intent to be resumed in the state machine. Specifically, this step involves dynamic changes in dialogue state management. The system does not simply discard candidate intents that cannot be executed at present, but rather performs a "suspend" operation on them. At this time, the system allocates dedicated storage space or sets specific state flags in the background "state machine" to completely save the candidate intent and its filled slot information, and marks its attribute as "intent to be resumed". This mechanism ensures that the user's original needs are not lost during the processing of previous tasks, provides a data foundation for automatic context recovery after the completion of subsequent tasks, and realizes context protection when multiple tasks are processed in parallel.

[0130] Subsequently, the system generates prompts based on the preceding dependent intent to guide the user in completing that intent. Specifically, the system's focus automatically shifts from the original intent to the preceding dependent intent. The system dynamically generates targeted prompts based on the key slot information required by the preceding dependent intent (such as the required ID number, verification code, etc.) and the corresponding prompt template. The purpose of these prompts is to drive the user to prioritize completing the preceding business, thereby eliminating the obstacle of "priority deficiency." For example, the system might generate a prompt like "Please enter your card number to activate your credit card," guiding the user into the preceding business processing flow, thus ensuring that the business flow conforms to the preset logical order.

[0131] Through steps 801 to 805 above, by introducing in-depth clarification of cause analysis and a dynamic state machine management mechanism, the automated processing capability and logical robustness of the intelligent customer service system in complex business scenarios are significantly improved. For the "priority missing" scenario, an automated closed loop of "diagnosing dependencies - suspending business - guiding priors" is realized, effectively solving the technical defects of traditional systems that directly report errors or cause infinite loops due to unmet preconditions. By suspending the blocked business and guiding users to resolve prior dependencies first, the system not only ensures the compliance and rationality of the business execution order, but also achieves seamless switching and continuous connection of multi-round dialogue intents, greatly improving user experience and the success rate of business processing.

[0132] Reference Figure 9 The multimodal query data processing method also includes the following steps 901 to 903.

[0133] Step 901: Obtain the state update result after the user completes the preceding dependency intent based on the query information.

[0134] Step 902: When the completion status of the preceding dependent intent is detected to have changed to "completed", extract the intent to be restored.

[0135] Step 903: Restore the execution context of the intent to be restored, and perform data execution on the intent to be restored as the target intent.

[0136] Steps 901 to 903 are described in detail below.

[0137] In some embodiments, after outputting query information to the user to guide them in completing the preceding dependent intent, the system performs an operation to obtain the status update result after the user completes the preceding dependent intent based on the query information. The system receives and parses the user's interaction data (such as the activation code or confirmation command entered by the user) in real time to determine whether the user has completed the previously identified missing "preceding dependent intent". The "status update result" here refers to the feedback signal after the preceding business logic is completed. It not only includes whether the business was successfully completed, but also triggers the system's internal change record of the completion status of the business node, providing the necessary triggering conditions for subsequent process regression.

[0138] When the system detects that the completion status of a preceding dependent intent has changed to "completed," it automatically retrieves the intent to be restored. Specifically, this step relies on a background state machine or similar process monitoring mechanism to continuously monitor the status of nodes in the business dependency chain. Once the system confirms that the status flag of the "preceding dependent intent" that is hindering the process has been flipped to "completed," the monitoring mechanism will immediately respond, retrieving and locking the "intent to be restored" that was previously temporarily shelved due to priority issues from the pending queue or temporary storage space. This process is automatically triggered without requiring the user to initiate a new request, thus achieving a seamless logical callback from preceding business to the original business.

[0139] Afterward, the system restores the execution context of the intent to be restored and executes it as the target intent. Specifically, this step is the core of the business interruption resumption process. "Restoring the execution context" means that the system reloads the intent parameters and filled slot information (such as amount, date, etc.) saved when the process was suspended into the current active memory, ensuring that the user does not need to re-enter known information. Subsequently, the system formally confirms the "intent to be restored" as the current compliant and executable "target intent," and accordingly calls the corresponding backend interface or generates a final response to complete the "data execution" operation, thereby closing the entire business processing chain.

[0140] In one example, when the system recognizes the user's "target intent" When it encounters a request such as "(e.g., credit card limit adjustment)," it immediately "checks the intent dictionary." When the system detects this "preceding transaction,"... When "(e.g., credit card activation)" is "not yet complete", the "preceding state result" will cause the system to generate a high "round conflict index". This high conflict index will lower the final "intent confidence", thereby triggering the clarification logic.

[0141] Next, the system will not simply reject the request; the system will first consider the target intent. The system is "suspended" and recorded as "intention to be restored" in the state machine. Subsequently, the system switches to the preceding service. This "switching" and "driving" action involves the system generating a query message (such as prompting the user to "please enter the activation credit card number") when the analysis results indicate a lack of priority in the candidate intent.

[0142] Finally, after the user completes the "preceding business" under the system's guidance (i.e. (If the "complete state" has been satisfied), the state machine detects this state change and will automatically "restore to" the previous state. The system "continues to execute" the user's initial "credit card limit adjustment" request. This dynamic management mechanism of "suspend-switch-resume" ensures that the "business operation sequence is reasonable" and "avoids operation failures or risk events due to unmet business dependencies," thereby achieving automated, continuous, and compliant management of complex business flows.

[0143] Through steps 901 to 903 above, a complete business interruption recovery and automated connection mechanism is constructed. By real-time monitoring of the completion status of preceding tasks and context recovery of the original intent, the system successfully achieves closed-loop management of "suspend-execute-resume" in complex business scenarios. This design not only ensures the strict sequential execution of multi-step business logic (i.e., compliance), but also effectively avoids user session loss or repeated operations caused by interruptions in intermediate steps. It significantly improves the continuity, intelligence level, and user interaction experience of the intelligent customer service system when handling long-process and highly dependent businesses.

[0144] Step 702: When the confidence level of the intent is not less than the preset confidence threshold, the candidate intent is taken as the target intent.

[0145] Step 703: Perform data execution based on the target intent, and generate output data based on the execution results.

[0146] Steps 702 to 703 are described in detail below.

[0147] In some embodiments, a "pass-through" processing branch is further described when the system is highly confident in the intent. This branch is triggered when the intent confidence is not less than a preset confidence threshold. This condition indicates that the overall intent confidence calculated by the system (e.g., based on step 104) has reached a preset reliability standard, proving that the input quality is high, the modalities are consistent, and the contextual logic is coherent. In this high-confidence scenario, the system does not need to perform clarification operations but directly "treats the candidate intent as the target intent." Here, the "target intent" is the user request that the system determines to be accurate and directly executable, and is then passed to the subsequent execution stage.

[0148] Following this, the system executes the final operation and generates feedback based on the "target intent" determined in the previous steps. The system first "executes data based on the target intent." Here, "data execution" is a general term that technically corresponds to various backend operations, such as calling backend APIs bound to the intent dictionary (e.g., registration, money transfer), or performing RAG retrieval and security compliance verification. After "data execution" is completed, the system "generates output data based on the execution results." This "output data" is the final natural language feedback or operation receipt generated by the system based on the "execution results" (e.g., successful API call, compliance verification rejection, or retrieved knowledge), presented to the user, as described below.

[0149] Reference Figure 9 The data is executed based on the target intent, and output data is generated based on the execution results, including the following steps 1001 to 1003.

[0150] Step 1001: Perform compliance verification based on the target intent and user information to obtain the compliance verification results.

[0151] Step 1002: When the compliance inspection result indicates that the target intent and user information are compliant information, perform data execution based on the target intent and generate output data based on the execution result.

[0152] Step 1003: When the compliance inspection result indicates that the target intent and user information are non-compliant, generate output data for rejection prompts.

[0153] Steps 1001 to 1003 are described in detail below.

[0154] In some embodiments, a key security and compliance "gate" is first described before the final execution of the business operation. The system performs a "compliance check based on the target intent and the user's user information." Here, the "target intent" is the business request confirmed in the preceding steps (e.g., "query account balance" or "transfer"), while the "user's user information" contains the context necessary to execute the intent, such as the user's identity, permission level, or parameters involved in the operation (e.g., amount, target account). The "compliance check" is a crucial verification process (i.e., "security compliance check") that determines, based on preset business rules (e.g., sensitive markers or risk control rules in the intent dictionary), whether the combination of the "target intent" and "user information" constitutes unauthorized access (e.g., querying another person's account), whether it triggers anti-fraud rules, or whether it exceeds the scope of business parameters. Finally, the system "obtains a compliance check result," which is a clear judgment (e.g., "compliant" or "non-compliant") to guide subsequent execution processes.

[0155] Then, when the compliance verification result indicates that the target intent and user information are compliant, meaning the verification determines that the request is secure, compliant, and within the user's authorized scope, the system is authorized to "execute data based on the target intent." This "data execution" is a true backend business process, such as calling the backend API bound in the intent dictionary (e.g., performing a transfer, modifying information, or registering) or performing a RAG search. After the backend returns the "execution result" (e.g., "operation successful," "balance information," or "API error code"), the system then "generates output data based on the execution result." This "output data" is the final response that converts the technical "execution result" into natural language (e.g., "transfer successful") and feeds it back to the user.

[0156] Furthermore, when the compliance check results indicate that the target intent and user information are non-compliant, such as when the system identifies an "unauthorized request" (e.g., querying another person's balance in "Example 2"), the system will skip the "data execution" step and directly "generate output data for rejection notification." This "output data" is a special, non-executive feedback that clearly informs the user that the request has been rejected (e.g., "According to regulations, it is not possible to query another person's account information"), and may include a compliance explanation or guide the user to contact human assistance. This ensures that the system does not perform any illegal or high-risk operations, and simultaneously records the event in the audit log.

[0157] Through steps 1001 to 1003 above, a closed-loop system of "review before execution" for secure execution and compliance control is constructed. This system achieves real-time risk filtering for all business requests by inserting a mandatory "compliance check" after the "target intent" is confirmed and before actual "data execution." This mechanism intelligently routes execution paths based on the "compliance check results": for "compliant information," the system executes normally and provides feedback; for "non-compliant information," the system "automatically refuses execution" and generates a "rejection prompt." This design effectively prevents users from unauthorized access to sensitive information or performing high-risk operations, ensuring the "security, controllability, and traceability" of the business processing process, and greatly improving the compliance and security of the intelligent customer service system when handling sensitive businesses (such as finance and healthcare).

[0158] also, Through steps 701 to 703 above, an adaptive decision-making and execution closed loop based on "intent confidence" is constructed. It achieves intelligent process routing through the core mechanism of "comparison between intent confidence and pre-set confidence thresholds": for requests with high "intent confidence," the system efficiently "treats them as target intents" and directly "executes data"; while for requests with low "intent confidence," the system safely switches to "clarification inquiry" mode, ensuring that it only proceeds to target intent execution after obtaining a clear "response information" from the user and "determining the target intent." This design effectively avoids erroneous execution caused by misunderstandings, ambiguities, or conflicting inputs, significantly improving the accuracy, robustness, and business security of multimodal interaction.

[0159] Reference Figure 11 This is a schematic diagram of an intent dictionary provided in an embodiment of this application. Figure 11 As shown, taking the "payment processing" "business intent" as an example, it clearly defines the "key information" (such as "payment slip, payment method") necessary to execute this intent. Simultaneously, it defines "preceding dependency information," meaning that "registration processing" must be completed before executing "payment processing"; this rule is the technical basis for triggering the "preceding business suspension and resumption" mechanism. Finally, the dictionary also binds to the "backend interface" (such as "pay") used for "data execution," providing clear guidance for the final API call.

[0160] Reference Figure 12 This is a schematic diagram illustrating an intent confidence level provided in an embodiment of this application. For example... Figure 12The example shown illustrates two candidate intentions: "appointment booking" and "querying appointment history." It lists the "intention probability" (P_model, 0.75 and 0.7 respectively) and "multimodal quality score" (0.9 and 0.85 respectively) output by the intention recognition model. Although both P_model and Q_k are high, the explicit cross-modal conflict (such as voice vs. image) significantly lowers the final "intention confidence" (Conf(I_j)) to 0.68 and 0.66. These two values, below the preset confidence threshold (e.g., 0.75), are the direct technical basis for triggering subsequent clarifying inquiries ("Do you prefer... or...").

[0161] In the first example provided in this application, the steps of the method are fully demonstrated in a medical scenario. First, the system simultaneously receives data from two modalities: one is "voice data" (the user says, "Please make an appointment for my internal medicine appointment tomorrow morning"), and the other is "visual data" (a "prescription photo" uploaded by the user). The system preprocesses and performs quality assessment on both types of data (e.g., ...). Figure 12 Q_k is shown in the figure.

[0162] Next, the system first processes the two modalities of data and "determines...at least one candidate intent and its corresponding probability." In this embodiment, the intent classification model generates two sets of "candidate intents": {appointing for an outpatient appointment, querying historical registration information}. Subsequently, the system performs "calculation of...multimodal consistency index based on intent probability." Since speech and image clearly point to two different intents, the "probability difference" calculation result is high, i.e., "cross-modal consistency verification reveals a conflict between speech and image information."

[0163] Subsequently, the system executes "based on multimodal consistency index" (high conflict) and "multimodal quality score" (e.g.) Figure 12 The final "intent confidence" (0.68 and 0.66 in Table 2) is calculated using the "round conflict index" (0.9 and 0.85 in the table) and (if present) the "round conflict index". The confidence (0.68 or 0.66) calculated afterward is "below the threshold θ_accept=0.75".

[0164] Since the condition of "intent confidence level less than the preset confidence threshold" is met, the system executes the clarification path. Specifically, the system "conducts a clarification reason analysis on the candidate intents," and "the analysis results indicate that the candidate intents conflict." Therefore, the system "generates a clarification inquiry based on multiple candidate intents and selection template prompts": "Would you like to schedule an appointment for tomorrow's outpatient visit, or would you like to view your previous registration records?" Finally, the system obtains the user's "response information" (e.g., the user selects to schedule an appointment) and "determines the target intent based on the response information" before continuing the subsequent appointment process.

[0165] The "technical effect" of this embodiment proves the value of the solution in this application: by applying the method defined in this set of claims, the system can promptly detect the conflict when there is ambiguity and conflict in the multimodal input (by calculating the consistency index and the confidence of intent), and automatically trigger clarification (by comparing the threshold) instead of erroneous execution, thereby effectively "ensuring the accuracy and security of medical registration operations".

[0166] The second example provided in this application demonstrates in detail how security controls are implemented during the execution phase after the intent is confirmed. In this scenario, the system receives a single-modal (text) input from the user: "Please check Zhang San's bank card balance." The system first identifies a high-confidence "candidate intent"—namely, "account inquiry." Because this intent is clear, its "intent confidence" meets the condition of "not less than a preset confidence threshold," therefore the system "uses the candidate intent as the target intent."

[0167] Next, before "performing data execution," the system must first perform a compliance check "based on the target intent" (account query) and the user's information (querying "Zhang San's" balance). In this embodiment, the "security compliance check" mechanism, after obtaining business rules through RAG enhancement, determines that the request (querying third-party information) is an "unauthorized" "prohibited operation." Therefore, the "compliance check result" is determined to be "non-compliant information."

[0168] Because the "compliance inspection result" was "non-compliant," the system generated output data for a rejection message. Specifically, the system "automatically refused to execute" and instead directly generated a "rejection message": "According to regulations, it is impossible to query other people's account information..." This "rejection message" is the defined "output data," and the system also records this event in the "security audit log."

[0169] This second embodiment demonstrates how, after identifying the "target intent," the solution does not blindly execute it, but effectively intercepts unauthorized and high-risk requests through a mandatory "compliance check" step, thereby achieving the invention's objective of "ensuring the compliance and security of the system."

[0170] The third example provided in this application demonstrates the calculation and application of the "round conflict index," particularly the "preceding dependency" processing logic. When a user inputs the "candidate intent" of "credit card limit adjustment," the system immediately executes "based on prior dependency information" (i.e., the "intent dictionary"), "confirms...preceding dependency intent" (i.e., "credit card activation"), and checks its "completion status." In this embodiment, the system finds that the "completion status" of the "preceding dependency intent" of "credit card activation" is "not yet completed."

[0171] This "preceding state result" will cause the system to generate a high-severity "round conflict indicator." This high "round conflict indicator," when substituted into subsequent calculations, will significantly reduce the final "intent confidence." Therefore, the system triggers a low-confidence path, deciding not to immediately execute the "credit card limit adjustment," but instead to initiate a "clarification inquiry."

[0172] At this point, the system first performs a "clarification analysis of the candidate intent," and the resulting "analysis result" clearly indicates that "the candidate intent lacks priority" (i.e., the preceding business is not completed). Instead of a simple query, the system executes a "preceding business suspension and resumption" mechanism: it "suspends 'credit card limit adjustment'" and "automatically switches to the preceding business 'credit card activation'." Then, the system "generates query information," but this query information is specifically for the newly activated preceding task, such as "Please enter the activation credit card number."

[0173] After the user completes the preliminary step of "credit card activation," the system confirms that the "preceding status result" is "completed," the original "round conflict indicator" disappears, and the "intent confidence level" increases. The system then "resumes the suspended intent 'credit card limit adjustment'" and "executes data based on the target intent." This embodiment uses this "suspend-switch-resume" closed loop to ensure a reasonable order of business operations and avoids operation failures due to unmet business dependencies.

[0174] This application also provides a multimodal query data processing apparatus, which can implement the above-described multimodal query data processing method, see reference. Figure 13 The device 1300 includes: The input data acquisition module 1310 is used to acquire the multimodal data of the input multimodal large language model and calculate the multimodal quality score of the multimodal data. The consistency index calculation module 1320 is used to determine at least one candidate intent and the corresponding intent probability corresponding to the multimodal data, and to calculate the multimodal consistency index of the multimodal data based on the intent probability. The conflict index calculation module 1330 is used to compare candidate intentions with historical confirmed intentions to obtain the round conflict index. The confidence calculation module 1340 is used to calculate the intent confidence corresponding to the multimodal data based on the multimodal consistency index, multimodal quality score and round conflict index. The output module 1350 is used to generate corresponding output data in a multimodal large language model based on the comparison relationship between intent confidence and a preset confidence threshold and candidate intents.

[0175] In some embodiments, the input data acquisition module 1310 is further configured to: The corresponding text quality is obtained based on at least one of the proportion of invalid characters in the text data and the normalization of spell correction edit distance. The corresponding speech quality is calculated based on at least one of the average confidence score, word error rate, and signal-to-noise ratio corresponding to the speech data. The corresponding visual quality is calculated based on at least one of the average confidence level and sharpness corresponding to the visual data. A multimodal quality score is obtained by weighted summation based on at least one of text quality, speech quality, and visual quality.

[0176] In some embodiments, the consistency index calculation module 1320 is further configured to: Each modality of data is processed by text encoding to obtain a text encoding vector; Each text encoding vector is input into a pre-trained intent recognition model for data processing to obtain the initial candidate intent corresponding to each modality of data and the intent probability corresponding to each initial candidate intent; Select the initial candidate intents with high intent probabilities from multiple initial candidate intents as candidate intents.

[0177] In some embodiments, the consistency index calculation module 1320 is further configured to: Calculate the probability difference between the candidate intent probabilities corresponding to the two modalities of data; The multimodal consistency index is obtained by averaging all probability differences.

[0178] In some embodiments, the conflict index calculation module 1330 is further configured to: Each candidate intent is compared with the historical confirmed intent to obtain the mutual exclusion comparison result for each candidate intent. Based on the preceding dependency information, the preceding dependency intent corresponding to each candidate intent is identified, and based on the completion status of the corresponding preceding dependency intent, the preceding status result of each candidate intent is obtained. Based on the mutual exclusion comparison results and / or the previous state results, the round conflict index corresponding to each candidate intent is obtained.

[0179] In some embodiments, the confidence calculation module 1340 is further configured to: Based on the round-based conflict index and multimodal consistency index corresponding to the candidate intent, a weighted summation is performed to obtain the comprehensive conflict metric corresponding to the candidate intent. Based on the comprehensive conflict metric, multimodal quality score, and intent probability, a weighted summation is performed to obtain the intent confidence corresponding to the candidate intent.

[0180] In some embodiments, the output module 1350 is further configured to: When the confidence level of the intent is less than the preset confidence threshold, the system generates the corresponding inquiry information to clarify the question to the user based on the candidate intent, obtains the response information entered by the user based on the inquiry information, and then determines the target intent based on the response information. When the confidence level of the intent is not less than the preset confidence threshold, the candidate intent is used as the target intent; Data execution is performed based on the target intent, and output data is generated based on the execution results.

[0181] In some embodiments, the output module 1350 is further configured to: A clarification analysis of the candidate intentions was conducted to obtain the analysis results. When the analysis results represent conflicting candidate intentions, based on multiple candidate intentions and selection template prompts, query information is generated to clarify the selection for the user. When the analysis results indicate a lack of priority in the candidate intent, query information is generated based on multiple candidate intents and selection template prompts.

[0182] In some embodiments, the output module 1350 is further configured to: Compliance checks are performed based on the target intent and the user's information to obtain the compliance check results. When the compliance inspection results indicate that the target intent and user information are compliant, data execution is performed based on the target intent, and output data is generated based on the execution results. When the compliance inspection results indicate that the target intent and user information are non-compliant, output data for rejection prompts is generated.

[0183] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the multimodal query data processing device is basically the same as the specific implementation of the multimodal query data processing method described above, and will not be repeated here.

[0184] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the multimodal query data processing method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0185] Please see Figure 14 , Figure 14 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1402 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1402 and is called and executed by the processor 1401 to execute the multimodal query data processing method of the embodiments of this application. The input / output interface 1403 is used to implement information input and output; The communication interface 1404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1405 transmits information between various components of the device (e.g., processor 1401, memory 1402, input / output interface 1403, and communication interface 1404); The processor 1401, memory 1402, input / output interface 1403 and communication interface 1404 are connected to each other within the device via bus 1405.

[0186] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described multimodal query data processing method.

[0187] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0188] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0189] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0190] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0192] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0193] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0195] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for processing multimodal query data, characterized in that, The method includes: Acquire multimodal data from the input multimodal large language model and calculate the multimodal quality score of the multimodal data; Determine at least one candidate intent and its corresponding intent probability corresponding to the multimodal data, and calculate the multimodal consistency index of the multimodal data based on the intent probability; Based on the comparison between the candidate intents and the historical confirmed intents, a round conflict index is obtained. Based on the multimodal consistency index, the multimodal quality score, and the round conflict index, the intent confidence level corresponding to the multimodal data is calculated. In the multimodal large language model, based on the comparison between the intent confidence and the preset confidence threshold and the candidate intent, corresponding output data is generated.

2. The multimodal query data processing method according to claim 1, characterized in that, The multimodal data includes at least one of text data, speech data, and visual data, and the calculation of the multimodal quality score of the multimodal data includes: The corresponding text quality is obtained based on at least one of the proportion of invalid characters corresponding to the text data and the spell correction edit distance normalization. Based on at least one of the average confidence level, word error rate, and signal-to-noise ratio corresponding to the speech data, the corresponding speech quality is calculated. Based on at least one of the average confidence level and sharpness corresponding to the visual data, the corresponding visual quality is calculated. The multimodal quality score is obtained by performing a weighted summation based on at least one of the text quality, the speech quality, and the visual quality.

3. The multimodal query data processing method according to claim 1, characterized in that, The multimodal data includes at least one modality, and determining at least one candidate intent and its corresponding intent probability corresponding to the multimodal data includes: Each modal data is processed by text encoding to obtain a text encoding vector; Each of the text encoding vectors is input into a pre-trained intent recognition model for data processing to obtain the initial candidate intent corresponding to each modality data and the intent probability corresponding to each initial candidate intent; Select a plurality of initial candidate intentions with high probability from a plurality of initial candidate intentions as the candidate intentions.

4. The multimodal query data processing method according to claim 1, characterized in that, When the multimodal data includes multiple modalities, the multimodal consistency index of the multimodal data calculated based on the intent probability includes: Calculate the probability difference between the candidate intent probabilities corresponding to each of the two modal data; The multimodal consistency index is obtained by averaging all the probability differences.

5. The multimodal query data processing method according to claim 1, characterized in that, The candidate intents are multiple, and the comparison process based on the candidate intents and historically confirmed intents yields a round-based conflict index, including: Each candidate intent is compared with the historical confirmed intent to obtain a mutual exclusion comparison result for each candidate intent. Based on the preceding dependency information, the preceding dependency intent corresponding to each candidate intent is identified, and based on the completion status of the corresponding preceding dependency intent, the preceding status result of each candidate intent is obtained. Based on the mutual exclusion comparison results and / or the preceding state results, the round conflict index corresponding to each candidate intent is obtained.

6. The multimodal query data processing method according to claim 1, characterized in that, The intention confidence level corresponding to the multimodal data is calculated based on the multimodal consistency index, the multimodal quality score, and the round conflict index, including: Based on the round-based conflict index and the multimodal consistency index corresponding to the candidate intent, a weighted summation process is performed to obtain the comprehensive conflict metric corresponding to the candidate intent. Based on the comprehensive conflict metric, the multimodal quality score, and the intent probability, a weighted summation is performed to obtain the intent confidence level corresponding to the candidate intent.

7. The multimodal query data processing method according to claim 1, characterized in that, The process of generating corresponding output data based on the comparison between the intent confidence level and a preset confidence threshold, and the candidate intent, includes: When the confidence level of the intent is less than the preset confidence threshold, the system generates inquiry information to clarify the question to the user based on the candidate intent, obtains the response information input by the user based on the inquiry information, and then determines the target intent based on the response information. When the confidence level of the intent is not less than the preset confidence threshold, the candidate intent is taken as the target intent; Data execution is performed based on the stated target intent, and the output data is generated based on the execution results.

8. The multimodal query data processing method according to claim 7, characterized in that, The step of generating inquiry information for clarifying questions to the user based on the candidate intent includes: A clarification analysis was performed on the candidate intentions to obtain the analysis results; When the analysis results indicate a conflict between the candidate intentions, based on multiple candidate intentions and selection template prompts, query information is generated to clarify the selection for the user. When the analysis results indicate that the candidate intent has a priority deficiency, the incomplete preceding dependent intent corresponding to the candidate intent is determined based on the preceding dependency information. The candidate intent is marked as suspended and recorded as an intent to be resumed in the state machine; Based on the preceding dependency intent, the query information is generated to guide the user to complete the preceding dependency intent.

9. The multimodal query data processing method according to claim 8, characterized in that, The method further includes: Obtain the state update result after the user completes the preceding dependency intent based on the query information; When the completion status of the preceding dependent intent is detected to have changed to "completed", the intent to be restored is extracted; Restore the execution context of the intent to be restored, and perform data execution on the intent to be restored as the target intent.

10. The multimodal query data processing method according to claim 7, characterized in that, The step of performing data execution based on the target intent and generating the output data based on the execution result includes: A compliance check is performed based on the stated objective intent and the user's information to obtain the compliance check result. When the compliance verification result indicates that the target intent and the user information are compliant information, data execution is performed based on the target intent, and the output data is generated based on the execution result; When the compliance check result indicates that the target intent and the user information are non-compliant, the output data for the rejection prompt is generated.

11. A multimodal query data processing device, characterized in that, The device includes: The input data acquisition module is used to acquire the multimodal data input to the multimodal large language model and calculate the multimodal quality score of the multimodal data. The consistency index calculation module is used to determine at least one candidate intent and the corresponding intent probability corresponding to the multimodal data, and to calculate the multimodal consistency index of the multimodal data based on the intent probability. The conflict index calculation module is used to compare the candidate intent with the historical confirmed intent to obtain the round conflict index. The confidence calculation module is used to calculate the intent confidence corresponding to the multimodal data based on the multimodal consistency index, the multimodal quality score, and the round conflict index. The output module is used to generate corresponding output data based on the comparison between the intent confidence and a preset confidence threshold and the candidate intent in the multimodal large language model.

12. An electronic device, characterized in that, The system includes a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the multimodal query data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal query data processing method according to any one of claims 1 to 11.

Images