Multi-modal large model multi-turn dialogue routing method and system

Abstract

This invention discloses a multimodal large-scale model multi-turn dialogue routing method and system, relating to the field of artificial intelligence technology. The method includes: establishing semantic links for entities, relationships, and context dependencies to form cross-modal semantic association data; performing task phase analysis, risk level assessment, and context preservation prediction on the dialogue state vector; generating anonymized semantic summaries and sending them to a server; constructing a candidate inference path set; determining the target inference path based on the evaluation results; evaluating the quality of the inference results; and generating performance feedback data. By constructing a cross-modal semantic graph, combined with dialogue situation tracking, multimodal information compression and anonymization processing, and server-side robust inference path search, path evaluation, and backoff mechanisms, the invention achieves efficient collaborative inference, cross-turn context preservation, improved inference result reliability, and privacy and security protection for multimodal large-scale models in multi-turn dialogues.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal large-model multi-turn dialogue routing method and system. Background Technology

[0002] With the development of multimodal large language models, artificial intelligence systems are now able to simultaneously process multiple input modalities such as text, images, audio, and structured data, and are gradually being applied to multi-turn dialogue scenarios to achieve continuous reasoning and interactive processing of complex tasks. However, in practical applications, due to the differences in the structure and semantic representation of multimodal data, as well as the complex contextual relationships in multi-turn dialogues, existing technologies still have shortcomings in multimodal collaborative reasoning.

[0003] On the one hand, existing multimodal processing methods typically use single vector representations or simple concatenation for fusion, making it difficult to establish correspondences between cross-modal entities and multi-turn reference relationships, resulting in poor consistency of multimodal information in multi-turn dialogues. On the other hand, user intent changes dynamically with each turn in multi-turn dialogues, and different stages have different requirements for model capabilities. However, existing routing methods are mostly static strategies, making it difficult to dynamically adjust the model call path according to the dialogue state, which can easily lead to inference mismatch or result deviation.

[0004] Furthermore, in multi-model collaborative reasoning, existing technologies struggle to flexibly combine different models to form effective reasoning chains according to task requirements, and lack dynamic adjustment and rollback mechanisms for the reasoning process. Moreover, they underutilize routing decisions and historical information in multi-turn dialogues, resulting in limited system stability and adaptability in complex scenarios. Therefore, a dynamic routing method for multi-turn dialogues oriented towards multimodal large language models is urgently needed to achieve multimodal semantic unification, dialogue state awareness, and adaptive optimization of reasoning paths. Summary of the Invention

[0005] This application provides a multimodal large-model multi-turn dialogue routing method and system. By employing cross-modal semantic association construction, dynamic dialogue state tracking, edge privacy compression processing, and path evaluation-based adaptive selection mechanism for inference links, it achieves dynamic adjustment and stable execution of multi-model collaborative inference paths, thereby improving the technical effects of context continuity, inference accuracy, and system operating efficiency in multi-turn dialogue scenarios.

[0006] This application provides a multimodal large-scale model multi-turn dialogue routing method and system, including: constructing a cross-modal semantic graph; generating a unified representation for each modality based on the modality parsing results of text, images, audio, and structured data; establishing semantic links for entities, relationships, and context dependencies to form cross-modal semantic association data; generating a dialogue state vector based on the modality distribution, historical turn states, and intent migration signals of user input through dialogue situation tracking; performing task stage analysis, risk level assessment, and context preservation prediction on the dialogue state vector; performing modality decoupling and information compression processing on the input data on the client side; injecting differential privacy noise into the compressed representation vector to generate an anonymized semantic summary and sending it to the server; constructing a candidate inference path set from the multimodal model resource pool based on the anonymized semantic summary and the dialogue state vector on the server side, wherein each candidate inference path is formed by combining multiple model units or tool units; performing path evaluation processing on the candidate inference path set, wherein the path evaluation processing includes at least consistency evaluation, context matching degree evaluation, and execution cost evaluation; and determining the target inference path based on the evaluation results. Multi-model collaborative reasoning is performed according to the target reasoning path, and the reasoning results are monitored in real time during the execution process. When inconsistent results, low matching degree or execution abnormality are detected, the execution path is adjusted or rolled back. The inference results are evaluated for quality, performance feedback data is generated, and the dialogue state, context information and path selection strategy are updated based on the performance feedback data. At the same time, the relevant data is written to the dialogue routing log for subsequent dialogue routing optimization. Attached Figure Description

[0007] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0008] Figure 1 This is a flowchart illustrating the multimodal large-model multi-turn dialogue routing method provided in an embodiment of this application. Detailed Implementation

[0009] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below.

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. The terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only.

[0012] This application provides a multimodal large-model multi-turn dialogue routing method, such as... Figure 1 As shown, the method includes: S1: Construct a cross-modal semantic graph. Based on the modal parsing results of text, images, audio and structured data, generate a unified representation for each modality, and establish semantic links of entities, relationships and context dependencies to form cross-modal semantic association data.

[0013] Specifically, firstly, modal parsing processing is performed on the multimodal data input by the user. This multimodal data includes text data, image data, audio data, and structured data. For text data, a language model is used for semantic encoding, extracting word vectors or sentence vectors. For image data, a visual feature extraction model is used to obtain feature vectors of target regions in the image and corresponding region label information. For audio data, an acoustic model is used to extract text information and voiceprint features corresponding to the speech content. For structured data, corresponding field semantic representations are generated based on its field structure. Through the above processing, modal feature datasets corresponding to each modality are obtained. Secondly, unified representation processing is performed on the modal feature datasets. Specifically, features from different modalities are mapped to a unified semantic representation space, enabling text entities, image regions, audio semantic segments, and structured fields to be expressed in a unified vector form, thereby achieving comparability and relevance across modalities. Further, entity extraction processing is performed based on the unified representation to identify entity nodes with independent semantic meaning from the features of each modality. The entity nodes include, but are not limited to, key nouns in text, target object regions in images, semantic segments in audio, and key fields in structured data. The system assigns a unique identifier to each entity node and constructs an entity node set. Based on this, cross-modal alignment processing is performed on entity nodes in different modalities. Specifically, through semantic similarity calculation, label matching, or context association analysis, entity nodes representing the same object or concept in different modalities are associated to establish cross-modal entity mapping relationships. Subsequently, graph relationship edges are constructed based on the semantic relationships between entity nodes. These relationships include descriptive relationships, referencing relationships, attribute relationships, temporal relationships, and contextual dependencies, used to characterize the semantic connections between entities and their referencing logic in multi-turn dialogues. Finally, combining historical dialogue turn information, contextual dependency links are established for entity nodes in the current turn, associating the current semantic structure with the historical semantic structure to form cross-turn semantic connection relationships. Through the above processing, a cross-modal semantic graph is constructed, and cross-modal semantic association data is output for subsequent dialogue state analysis and inference path construction.

[0014] S2: Based on the modal distribution of user input, historical round states, and intent transition signals, a dialogue state vector is generated through dialogue situation tracking processing, and task phase analysis, risk level assessment, and context preservation prediction are performed on the dialogue state vector.

[0015] Specifically, firstly, modal distribution information corresponding to the current round of user input is obtained. Specifically, based on the proportion of each modality type in the input data, the weight distribution of text, images, audio, and structured data in the current task is determined, forming modal distribution feature data. Simultaneously, historical round state information is extracted from the context self-holding buffer. This historical round state information includes task stage markers of historical dialogues, identified entity nodes, inference path records, and contextual dependencies. Secondly, the current round input and historical round data are jointly analyzed to extract intent migration signals. Specifically, by comparing the differences between the current round semantic representation and the historical task semantic features, it is identified whether the user intent has changed, and intent migration feature data is generated. When the difference exceeds a preset threshold, intent migration is determined to have occurred, and the migration direction and magnitude are recorded. Based on this, the modal distribution feature data, historical round state information, and intent migration feature data are fused to construct a dialogue situation feature set. This dialogue situation feature set serves as input to generate a dialogue state vector, which represents the overall state of the current dialogue under multimodal and multi-round conditions. Further, task stage analysis is performed based on the dialogue state vector. Specifically, based on preset stage division rules, the current dialogue is divided into at least one of the following stages: understanding, clarification, processing, or result generation, and a corresponding stage identifier is output. Simultaneously, a risk level assessment is performed on the dialogue state vector. The risk level is determined based on at least one of the following indicators: modal information conflict degree, semantic uncertainty, historical reasoning error records, or task sensitivity, and the assessment result is mapped to a preset risk level. Furthermore, context retention prediction is performed on the dialogue state vector. Specifically, a context retention index is calculated based on the degree of dependence of the current input on historical semantics, cross-round reference frequency, and context load, used to measure the strength of the current dialogue's dependence on historical information. Finally, the task stage analysis results, risk level assessment results, and context retention index are integrated to output the dialogue state vector, which serves as the input basis for subsequent reasoning path construction and routing decisions.

[0016] S3. At the edge, perform modal decoupling and information compression processing on the input data, encode each modal input into a representation vector, and compress the representation vector to obtain a compressed representation vector; inject differential privacy noise into the compressed representation vector to generate anonymized semantic summary and send it to the server.

[0017] Specifically, firstly, the device acquires multimodal data input by the user and performs modal decoupling processing on the multimodal data. Specifically, the input data is divided into text data, image data, audio data, and structured data according to data type, and each is processed independently to avoid information coupling and propagation between different modalities. Then, each modal input is encoded and converted into a representation vector in a unified semantic space. After obtaining the representation vector, information compression processing is performed on the representation vector. Specifically, for text data, key semantic segments or keywords are extracted to generate a compact semantic representation; for image data, key region features containing target entities are extracted and irrelevant region information is removed; for audio data, effective speech segments are extracted and corresponding semantic representations are generated; for structured data, field information relevant to the current task is extracted. Based on this, the representation vector is compressed through redundant information pruning, semantic quantization, and feature sparsity processing to obtain a compressed representation vector, thereby reducing data redundancy and minimizing the exposure of sensitive information. Furthermore, differential privacy noise is injected into the compressed representation vector. Specifically, based on preset privacy protection parameters, random perturbations are added to the compressed representation vector to limit the influence of individual input data on the output result, thereby reducing the possibility of inferring the original data from the semantic representation. The noise intensity can be adaptively adjusted according to the data sensitivity. Subsequently, the compressed representation vector after compression and differential privacy processing is structurally encapsulated to generate anonymized semantic summary data. The anonymized semantic summary does not contain the original modality data, but only retains the semantic information used for task processing. Finally, the anonymized semantic summary is sent to the server side through a communication interface for subsequent candidate inference path construction and routing decision processing.

[0018] S4: The server side constructs a set of candidate inference paths from the multimodal model resource pool based on the anonymized semantic summary and the dialogue state vector, wherein each candidate inference path is formed by combining multiple model units or tool units.

[0019] Specifically, first, the server receives an anonymized semantic summary from the client, which includes compressed representation vectors, modality weight distribution, round-reference indexes, and potential task labels. Simultaneously, the server receives a dialogue state vector generated by a dialogue situation tracker, reflecting the current dialogue stage, risk level, context preservation, and user intent transition. Then, based on the semantic features in the anonymized semantic summary and the dialogue state vector, the server performs preliminary screening of available model units and tool units in the multimodal model resource pool. The model resource pool includes, but is not limited to, text understanding models, image recognition models, speech recognition models, structured data reasoning models, and various tool interface modules. By matching task requirements with the capability labels, historical performance indicators, and running status of each model unit, a set of candidate units capable of participating in the current inference task is selected. Next, the server constructs a set of candidate inference paths based on the selected candidate units. Each candidate inference path is formed by combining several model units or tool units, and the path order can be arranged according to model input / output compatibility, task stage requirements, and context dependencies. Specifically, the following combination strategies can be adopted: Sequential combination: Multiple model units are called sequentially, and the output results are used as input for the next unit to gradually complete the task reasoning; Parallel combination: Multiple model units or tool units are called simultaneously to handle different modalities or different subtasks, and then integrated through a fusion module; Hybrid combination: Based on sequential and parallel combinations, some model units are dynamically selected for skipping or backtracking to adapt to changes in user intent in multi-turn dialogues. During the construction of candidate paths, the server can also prioritize paths or mark potential conflict points based on task stage information and risk assessment indicators in the dialogue state vector, so as to facilitate subsequent path selection and risk control. Through the above steps, the server finally generates a set containing multiple candidate reasoning paths, providing input for subsequent multi-model reasoning routing decisions, enabling the system to flexibly select the optimal path to perform multi-modal reasoning based on the current dialogue state and task requirements.

[0020] S5. Perform path evaluation processing on the candidate inference path set. The path evaluation processing includes at least consistency evaluation, context matching degree evaluation and execution cost evaluation, and determine the target inference path based on the evaluation results.

[0021] Specifically, firstly, the server performs a consistency evaluation on each candidate inference path. This evaluation determines the degree of semantic, modal, and task-stage coordination among the outputs of each model unit or tool unit within the path. Based on cross-modal semantic graphs, round-reference indices, and task-stage information, the server calculates the semantic consistency score for each node in the path, ensuring that the model output matches the current dialogue context, user intent, and historical round information. Secondly, the server evaluates the context matching degree of the candidate paths. This evaluation compares the intermediate representations generated by each unit in the path with contextual features in the dialogue state vector, including user intent transition signals, completed task states, and modal reference relationships, assessing the path's ability to maintain context across multiple rounds of dialogue. Paths with high context matching degrees indicate that they can continue inference in subsequent rounds, reducing information loss or semantic bias. Thirdly, the server evaluates the execution overhead of the candidate paths. Execution overhead includes model unit computational resource consumption, call latency, network transmission costs, and potential tool call overhead. By quantifying the computation and resource consumption of each path, the server can prioritize more efficient paths while ensuring inference quality, reducing system load and response latency. After completing consistency assessment, context matching assessment, and execution overhead assessment, the server will synthesize various assessment indicators to generate a total score for each candidate path, and select the target inference path based on the total score and preset constraints (such as risk thresholds, time limits, and cost budgets). This target path will serve as the execution link for the final multi-model inference, used to generate the output results of the current round of dialogue, and provide a foundation for context maintenance and multi-modal inference in subsequent rounds. Through the above path evaluation process, this application can achieve the following technical effects in a multi-model, multi-round dialogue environment: ensuring semantic consistency: the outputs of each model unit in the path are coordinated, avoiding cross-modal conflicts or reference errors; improving context continuity: path selection fully considers the historical round states and user intent evolution, realizing continuous inference in multi-round dialogues; optimizing system resource utilization: under the premise of meeting quality requirements, paths with lower execution overhead are prioritized to improve inference efficiency and system response speed.

[0022] S6. Perform multi-model collaborative reasoning according to the target reasoning path, and monitor the reasoning results in real time during the execution process. When inconsistent results, low matching degree or execution abnormality are detected, the execution path is adjusted or rolled back.

[0023] Specifically, the server performs a multi-dimensional performance evaluation of the output results of the target path. This performance evaluation includes, but is not limited to, the following: Semantic accuracy evaluation: by comparing the output results with entities, relationships, and contextual dependencies in the cross-modal semantic graph, the matching degree between the output results and the expected semantics is calculated; Round coherence evaluation: detecting the logical connection between the output results and the content of historical rounds of dialogue to determine whether the contextual consistency of multi-round dialogue is maintained; Modal consistency evaluation: analyzing the semantic correspondence between text, images, audio, or structured data to identify cross-modal conflicts or deviations; Execution resource and latency evaluation: recording the computing resources consumed, latency, and tool call overhead of the target path execution to provide a quantitative basis for subsequent path optimization. After obtaining the above evaluation results, the server generates a route optimization strategy according to a preset algorithm. This strategy dynamically adjusts candidate model paths, context-inherent buffers, and multi-turn dialogue states. This includes updating modal references, turn dependency chains, and key semantic nodes recorded in the context-inherent buffer to ensure continuous inference in subsequent turns; readjusting candidate path weights and prioritizing high-performance, low-conflict paths based on output quality feedback; triggering path fallback or link retry mechanisms to improve output reliability in the event of semantic deviations or modal conflicts; and dynamically adjusting the multi-model call order or tool combination based on resource consumption assessment to optimize system response speed and operating efficiency. Through the above output quality feedback and routing optimization processing, this application achieves the following technical effects: improved inference reliability: real-time evaluation of output quality, timely correction of potential errors, and ensuring semantic coherence and cross-modal consistency in multi-turn dialogues; support for dynamic adaptive routing: adjusting path strategies based on performance feedback and context changes, making multi-model collaborative inference more efficient and flexible; and continuous optimization and resource management: adjusting model call order and tool combination through routing optimization strategies to save computational resources and reduce response latency. In summary, this application achieves dynamic adaptive dialogue reasoning in multi-turn, multi-modal, and multi-model environments through real-time quality feedback and routing optimization, thereby improving the system's accuracy, continuity, and execution efficiency.

[0024] S7. Evaluate the quality of the inference results, generate performance feedback data, and update the dialogue state, context information, and path selection strategy based on the performance feedback data. At the same time, write the relevant data into the dialogue routing log for subsequent dialogue routing optimization.

[0025] Specifically, the quality assessment includes the following: semantic accuracy detection, analyzing the degree of matching between the reasoning result and entities, relations, and context dependencies in the cross-modal semantic graph, and quantifying the consistency between the output semantics and the expected semantics; turn coherence detection, judging the logical connection between the current output result and the dialogue state and key semantic nodes of the previous turns, and evaluating the context preservation degree of multi-turn dialogue; modal consistency detection, performing cross-modal comparisons on the output results of text, images, audio, and structured data, and identifying potential conflicts or inconsistencies; performance and resource assessment, recording the computing resources consumed in executing the target path, the latency and overhead of calling tools, for subsequent path selection optimization. Based on the above quality assessment results, the system generates performance feedback data, including but not limited to: semantic score, modal consistency score, and turn coherence score of the output result; computing resources consumed in executing the current path, latency indicators, and tool call records; confidence assessment, risk prediction, and uncertainty indicators of path selection. Subsequently, based on the performance feedback data, the following operations are performed: Dialogue state update: updating the dialogue state vector for the current round, including user intent, task stage, and risk level; Context information maintenance: adjusting the state of modal references, round dependency chains, and key semantic nodes in the context self-sustaining buffer to support continuous reasoning in multi-round dialogues; Path selection strategy optimization: adjusting the weights of candidate paths based on feedback data, prioritizing high-quality, low-conflict, and low-resource-cost paths, and triggering fallback, retry, or link switching mechanisms when necessary. Furthermore, the aforementioned dialogue state, context information, path selection strategy, and performance feedback data are written to the dialogue routing log, which serves as the foundational data for future multi-round dialogue routing optimization, model training, and continuous learning.

[0026] Furthermore, claim 1 also includes: The cross-modal semantic graph is established through entity extraction, cross-modal entity alignment, semantic relationship construction, and round-based reference links to maintain semantic consistency and reference coherence among different modalities in multi-round dialogues.

[0027] Specifically, the construction process of the cross-modal semantic graph includes steps such as entity extraction, cross-modal entity alignment, semantic relationship construction, and round-based reference link establishment. Specifically, the system first extracts entities from input information from different modalities, such as text, images, audio, and structured data, identifying semantically meaningful objects or concept nodes. Then, through cross-modal entity alignment, corresponding entities in different modalities are matched and associated, enabling identical or related entities to form corresponding relationships in a unified semantic space, thereby achieving consistent representation among multimodal entities. Based on this, the system further constructs semantic relationships, including logical relationships, functional relationships, and contextual dependencies between entities, to describe the interactions and semantic connections between entities. Simultaneously, the system establishes reference relationships between entities in the current round and related entities in previous rounds through a round-based reference link mechanism, enabling continuous tracking of contextual information in multi-turn dialogues. The cross-modal semantic graph constructed through the above steps can maintain semantic consistency between different modalities, ensuring that the reference relationships between modalities are clear and the semantic information is coherent during multi-turn dialogues, providing a unified semantic foundation for subsequent dialogue state tracking, model routing, and inference path selection.

[0028] Furthermore, claim 1 also includes: The dialogue situation tracking process includes: intent migration detection, task stage determination, context load monitoring, and risk factor estimation, which are used to generate the dialogue state vector.

[0029] Specifically, the dialogue situation tracking processing is used to generate a dialogue state vector based on user input and historical dialogue information to support context management and task scheduling in multi-turn dialogues. Specifically, the system first analyzes the semantic differences between the user's input in the current turn and the previous turn through the intent migration detection module, identifying whether the user's intent has changed, thereby capturing the dynamic evolution characteristics of user needs. Subsequently, the task stage determination module determines the task stage of the current dialogue turn based on the dialogue progress and task attributes, such as distinguishing between the information gathering stage, clarification stage, or result generation stage, to assist in the phased optimization of the inference path. The context load monitoring module is used to evaluate the complexity of historical dialogue information and the density of context references, determining the amount of context information the dialogue system needs to maintain in the current state, thereby dynamically adjusting caching and inference strategies. The risk factor estimation module assesses the probability of potential errors or anomalies based on the task stage, changes in user intent, and the uncertainty of multimodal inputs, providing a basis for subsequent path selection and risk control. Through the above processing, the system can generate a dialogue state vector reflecting the current multi-turn dialogue situation, accurately representing the semantic environment, task stage, and potential risks of the current turn, providing basic data support for cross-modal model routing and inference link optimization.

[0030] Furthermore, claim 1 also includes: The information compression process includes modal decoupling coding, redundant information pruning, semantic quantization and feature sparsification, and differential privacy noise injection is performed after compression.

[0031] Specifically, the information compression process preprocesses multimodal input data on the edge to reduce transmission overhead and protect user privacy. Specifically, the system first independently encodes information from different modalities, such as text, images, audio, and structured data, using modal decoupling coding. The feature representations of each modality are generated in a decoupled vector space to avoid intermodal interference. Subsequently, a redundant information pruning module prunes unnecessary or low-contribution features based on feature importance or information entropy, reducing the dimensionality of the representation vector and removing redundant information. Next, a semantic quantization module converts the high-precision representation vector into a low-bit quantization representation while preserving the main semantic information, further reducing data storage and transmission load. Feature sparsification is used to set insignificant features in the vector to zero, thereby enhancing the sparsity and compressibility of the representation. After completing the above compression processing, the system injects differential privacy noise into the representation vector to generate anonymized semantic summaries, ensuring that sensitive user information is not leaked during upload to the server. Through the above processing, the edge side achieves both efficient data compression and modal decoupling, while also protecting information privacy, providing reliable and secure input data for cross-modal model inference on the server side.

[0032] Furthermore, claim 1 also includes: The anonymized semantic summary includes a compressed representation vector, modality weight distribution, task label information, round reference index, and cross-modality consistency evaluation index; Specifically, the anonymized semantic summary is generated after information compression and differential privacy processing on the client side, and is used to securely transmit multimodal information to the server side. Specifically, the anonymized semantic summary includes the following: Compressed representation vector: A low-dimensional representation vector of each modality after decoupling encoding, redundancy pruning, semantic quantization, and feature sparsification, retaining the main semantic information while reducing data volume. Modality weight distribution: Records the weight of different modalities' contribution to the overall semantics in the current round, used by the server side for modality priority scheduling and fusion during multi-model inference. Task label information: Labels the potential categories of the current user intent or inference task, providing a reference for the server side to select appropriate model units or tool units. Round reference index: Contains the semantic reference relationship between the current round of dialogue and previous rounds, used to maintain contextual continuity and cross-round semantic association in multi-round dialogues. Cross-modal consistency evaluation index: Quantitatively evaluates the semantic consistency between the representations of each modality, providing a reference for the server side's candidate path construction and inference decision-making. By combining the structured information described above, the anonymized semantic summary achieves secure transmission and information integrity of cross-modal semantics while ensuring user privacy, providing reliable input for candidate reasoning path generation and multi-model collaborative reasoning on the server side.

[0033] Furthermore, claim 1 also includes: In the process of constructing the candidate inference path, at least one of the following criteria is used as the path selection basis: inference accuracy, cross-modal consistency, execution latency, resource consumption, and risk level. Specifically, the candidate inference path is constructed on the server side based on the received anonymized semantic summary and dialogue state vector, generating multiple possible inference paths from the multimodal model resource pool. Each candidate path is composed of several model units or tool units combined sequentially. To improve the efficiency and reliability of multi-model inference, this application introduces a path screening mechanism during the candidate path construction process, that is, filtering paths according to at least one of the following indicators: Inference accuracy: Evaluating the accuracy and credibility of candidate paths in completing the current task or answering user questions, prioritizing model combinations that can provide high-quality output. Cross-modal consistency: Measuring the consistency of different modal information in semantic representation and inference results in the path, ensuring stable multimodal fusion effect and natural semantic connection. Execution latency: Considering the time required for path execution, avoiding the selection of paths with high latency or that may lead to slow dialogue response, improving the real-time performance of multi-turn dialogue. Resource consumption: Analyzing the occupation of computing resources, storage, and network bandwidth during path execution, prioritizing model combinations with efficient resource utilization. Risk level: Assessing potential risks such as errors, modal conflicts, tool call failures, or uncertainties that may occur during path execution, in order to reduce the probability of dialogue interruption or output errors. Through the above path selection, this application can dynamically select the optimal candidate inference path from the multimodal large model resource pool, providing high-quality input for subsequent path evaluation and final inference decision, and ensuring the accuracy, coherence and security of multi-turn dialogue.

[0034] Furthermore, claim 1 also includes: The path evaluation process includes: reasoning result correctness evaluation, multi-round context coherence evaluation, modal conflict detection, execution latency and resource overhead evaluation, and a path score value is generated based on the above evaluation results for path selection; Specifically, the path evaluation process involves a multi-dimensional evaluation of the constructed candidate inference path set on the server side to determine the optimal inference path. Specifically, the path evaluation process includes the following aspects: Inference result correctness evaluation: The accuracy of the output generated by the candidate path is checked to determine whether it meets the current task requirements or the user's query intent. Multi-turn contextual coherence evaluation: The semantic connection between the path output in the current turn and historical dialogue turns is analyzed to ensure consistent information references across turns and maintain the continuity and coherence of multi-turn dialogues. Modal conflict detection: Semantic conflicts or inconsistencies that may exist between different modal information in the path, such as referential conflicts between text, images, or audio, thereby avoiding erroneous or contradictory inference results. Execution latency and resource overhead evaluation: The response time of the candidate path during execution and its consumption of computing resources, storage, and network bandwidth are evaluated to select an efficient and real-time responsive path. Based on the above multi-dimensional evaluation indicators, a corresponding path score is generated for each candidate path. The path score comprehensively reflects factors such as correctness, coherence, consistency, and resource efficiency. The server then sorts and filters the candidate paths based on the score to determine the final target inference path, which is used to perform multimodal inference and generate dialogue responses, thereby improving the accuracy, stability and real-time performance of multi-turn dialogues.

[0035] Furthermore, claim 1 also includes: The path evaluation process includes: inference result correctness evaluation, multi-round context coherence evaluation, modal conflict detection, execution latency and resource overhead evaluation, and a path score value is generated based on the above evaluation results for path selection.

[0036] Specifically, the path evaluation process is used to comprehensively evaluate the candidate inference paths constructed on the server side in order to select the optimal inference path. Specifically, the path evaluation process includes the following steps: Inference result correctness evaluation: The accuracy of the inference output generated by the candidate path is checked to determine whether the output meets the requirements of the current task or the user's input intent, thereby ensuring the reliability of the inference result. Multi-turn contextual coherence evaluation: The semantic connection between the candidate path and previous turns of the dialogue is analyzed, including entity references, intent continuation, and contextual information consistency, to ensure the continuity and logical coherence of multi-turn dialogues. Modal conflict detection: Potential conflicts or inconsistencies between different modal information in the candidate path are detected, such as referential or semantic contradictions between text, images, audio, and structured data, thereby avoiding the generation of incorrect or contradictory inference results. Execution latency and resource overhead evaluation: The response time, computational resource consumption, storage consumption, and network bandwidth usage of the candidate path in actual execution are evaluated to select an efficient and real-time responsive path. Based on the aforementioned evaluation metrics, a path score is generated for each candidate path. This path score comprehensively reflects factors such as correctness, contextual coherence, cross-modal consistency, and execution efficiency. The server sorts and filters candidate paths according to their path scores to determine the target inference path, which is then used for subsequent multimodal inference and multi-turn dialogue generation, thereby improving the accuracy, stability, and response efficiency of the system's inference.

[0037] Furthermore, claim 1 also includes: Path rollback or adjustment is triggered when any of the following conditions are detected: inference results are inconsistent with cross-modal semantic graphs, context matching degree is lower than a preset threshold, inference result uncertainty exceeds a preset range, or execution cost exceeds a preset budget.

[0038] Specifically, to ensure the reliability and robustness of multimodal, multi-turn dialogue reasoning, the system incorporates path rollback and adjustment mechanisms when executing the target reasoning path. Path rollback or adjustment is triggered when any abnormal condition is detected. These abnormal conditions include: 1. Inconsistency between the reasoning result and the cross-modal semantic graph: If the reasoning output generated by the candidate path conflicts with existing semantic links in the cross-modal semantic graph at the entity, relation, or semantic level, it indicates that the path may produce erroneous reasoning, triggering a rollback mechanism to avoid erroneous output. 2. Context matching degree below a preset threshold: By evaluating the matching degree between the candidate path and the current dialogue state vector and historical turn contexts, if the matching degree is below a preset threshold, the system determines that the path cannot maintain the coherence of multi-turn dialogue and performs path adjustment. 3. Uncertainty of the reasoning result exceeding a preset range: The system measures the confidence interval or uncertainty index of the candidate path output using a behavior predictor or uncertainty assessment model. If the uncertainty exceeds a set range, it indicates insufficient reliability of the reasoning result, requiring rollback to a high-reliability model or triggering a multi-model consensus mechanism. Execution cost exceeds preset budget: When a candidate path exceeds preset limits in terms of computing resources, latency, or other execution overhead, the system will suspend the execution of the current path and select another path that meets cost constraints to ensure system response efficiency and resource controllability. Through the above mechanism, the server can dynamically adjust the inference path based on real-time evaluation results, including switching to a high-reliability model, performing cross-modal conflict resolution, pruning and retrying content, calling the toolchain, or triggering multi-model consensus, thereby ensuring accurate inference results, coherent context, and efficient execution in multimodal multi-turn dialogues, and improving the overall robustness and adaptability of the system.

[0039] Furthermore, claim 1 also includes: The path rollback or adjustment includes at least one of the following: switching to a highly reliable model, performing cross-modal conflict resolution, retrying after pruning the input content, or calling a toolchain or multi-model collaborative processing; and recording modal reference relationships, round dependencies, and incomplete task states through a context-self-holding buffer structure to support multi-turn dialogue continuity.

[0040] Specifically, to ensure the continuity and robustness of multimodal multi-turn dialogue, when path rollback or adjustment is triggered, the rollback or adjustment operation includes at least one of the following: switching to a highly reliable model to improve the credibility of the inference results; performing cross-modal conflict resolution to resolve possible semantic or reference conflicts between different modalities; pruning the input content and retrying to reduce the inference risk caused by complex input; calling the toolchain to utilize external functional modules to assist in inference; or achieving joint inference of multiple models through a multi-model collaborative processing mechanism to improve the overall output quality. Simultaneously, the system utilizes a context-self-maintaining buffer structure to record key information related to inference, including reference relationships between modalities, turn dependencies, and the status of unfinished tasks. By maintaining the above information, the system can maintain contextual continuity and task state coherence during multi-turn dialogue, thereby supporting adaptive adjustment and highly reliable output across turns of inference, improving the overall robustness, accuracy, and user experience of the multimodal large-model multi-turn dialogue system.

[0041] Example 2 Based on the multimodal large model multi-turn dialogue routing method in the foregoing embodiments, and using the same inventive concept, this application also provides a multimodal large model multi-turn dialogue routing system, the system comprising: The cross-modal semantic graph construction module is used to construct cross-modal semantic graphs. Based on the modal parsing results of text, images, audio and structured data, it generates a unified representation for each modality and establishes semantic links of entities, relations and context dependencies to form cross-modal semantic association data. The dialogue situation tracking module is used to generate a dialogue state vector based on the modal distribution, historical turn states, and intent transition signals of user input through dialogue situation tracking processing, and to perform task stage analysis, risk level assessment, and context preservation degree prediction on the dialogue state vector. The edge-side compression and privacy processing module is used to perform modal decoupling and information compression processing on the input data at the edge, encode each modal input into a representation vector, and compress the representation vector to obtain a compressed representation vector; inject differential privacy noise into the compressed representation vector to generate an anonymized semantic summary and send it to the server; The path construction and routing decision module is used by the server to construct a set of candidate inference paths from the multimodal model resource pool based on the anonymized semantic summary and the dialogue state vector, wherein each candidate inference path is formed by combining multiple model units or tool units. The path evaluation and adjustment module is used to perform path evaluation processing on the candidate inference path set. The path evaluation processing includes at least consistency evaluation, context matching degree evaluation and execution cost evaluation, and determines the target inference path based on the evaluation results. The context state management module is used to perform multi-model collaborative reasoning according to the target reasoning path, and to monitor the reasoning results in real time during the execution process. When inconsistent results, low matching degree or execution abnormality are detected, the execution path is adjusted or rolled back. The routing log module is used to evaluate the quality of the inference results, generate performance feedback data, and update the dialogue state, context information, and path selection strategy based on the performance feedback data. At the same time, it writes relevant data into the dialogue routing log for subsequent dialogue routing optimization.

[0042] This application achieves efficient collaborative reasoning and adaptive dynamic routing of multimodal large models in multi-turn dialogues through a comprehensive technical solution that includes constructing a cross-modal semantic graph, dialogue situation tracking, edge privacy protection, server-side robust path planning, and multi-turn feedback optimization. First, by using a cross-modal semantic graph, this application can unify the representation of multimodal inputs such as text, images, audio, and structured data, establishing semantic links for entities, relationships, and context dependencies. This achieves semantic alignment and reference consistency between different modalities, effectively solving the problems of difficulty in unifying multimodal features and unstable semantic connections across turns in existing technologies. Second, by employing a dialogue situation tracker, a dialogue state vector is generated based on the user input modality distribution, historical turn states, and intent transition signals. This captures the phased evolution of user intent and changes in task complexity, thereby enabling dynamic prediction and evaluation of task stages, risk levels, and context preservation in multi-turn dialogues, improving the matching degree of model calls and the reliability of the inference chain. Furthermore, this application performs modal decoupling, information compression, and differential privacy processing on the client side, uploading only anonymized semantic summaries to the server. This not only reduces data transmission volume and computational overhead but also effectively protects user data privacy and prevents the leakage of sensitive information. On the server side, by constructing a candidate inference path set and combining Bayesian robust policy search and a risk-aware reward function, a cross-model, cross-tool path selection, jumping, and backtracking mechanism is implemented. This enables multi-model collaborative inference to quickly self-recover when faced with path conflicts, modal inconsistencies, or tool call failures, significantly improving system stability and inference accuracy. In addition, this application generates performance feedback data by evaluating the quality of inference results and updates the dialogue state, context information, and path selection strategy based on the feedback data, while simultaneously recording it in the dialogue routing log. This achieves adaptive optimization and continuous learning capabilities for multi-turn dialogues. This solution not only improves the semantic continuity and cross-modal consistency of multi-turn dialogues but also optimizes computational resource utilization and inference efficiency, providing reliable assurance for long-term operation and multi-modal large-model applications in complex business scenarios. In summary, this application can effectively solve the technical bottlenecks of existing technologies in multimodal fusion, dynamic task adaptation, privacy protection, and multi-turn dialogue continuity, and has significant application value and promotion prospects.

[0043] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A multimodal large-model multi-turn dialogue routing method, characterized in that, The method includes: S1. Construct a cross-modal semantic graph. Based on the modal parsing results of text, images, audio and structured data, generate a unified representation for each modality and establish semantic links of entities, relations and context dependencies to form cross-modal semantic association data. S2. Based on the modal distribution, historical round states, and intent transition signals of user input, a dialogue state vector is generated through dialogue situation tracking processing, and task phase analysis, risk level assessment, and context preservation prediction are performed on the dialogue state vector. S3. At the edge, perform modal decoupling and information compression processing on the input data, encode each modal input into a representation vector, and compress the representation vector to obtain a compressed representation vector; inject differential privacy noise into the compressed representation vector to generate an anonymized semantic summary and send it to the server; S4. The server side constructs a set of candidate reasoning paths from the multimodal model resource pool based on the anonymized semantic summary and the dialogue state vector, wherein each candidate reasoning path is formed by combining multiple model units or tool units. S5. Perform path evaluation processing on the candidate inference path set. The path evaluation processing includes at least consistency evaluation, context matching degree evaluation and execution cost evaluation, and determine the target inference path based on the evaluation results. S6. Perform multi-model collaborative reasoning according to the target reasoning path, and monitor the reasoning results in real time during the execution process. When inconsistent results, low matching degree or execution abnormality are detected, the execution path is adjusted or rolled back. S7. Evaluate the quality of the inference results, generate performance feedback data, and update the dialogue state, context information, and path selection strategy based on the performance feedback data. At the same time, write the relevant data into the dialogue routing log for subsequent dialogue routing optimization.

2. The multimodal large-model multi-turn dialogue routing method as described in claim 1, characterized in that, The cross-modal semantic graph is established through entity extraction, cross-modal entity alignment, semantic relationship construction, and round-based reference links to maintain semantic consistency and reference coherence among different modalities in multi-round dialogues.

3. The multimodal large-model multi-turn dialogue routing method as described in claim 1, characterized in that, The dialogue situation tracking process includes: intent migration detection, task stage determination, context load monitoring, and risk factor estimation, which are used to generate the dialogue state vector.

4. The multimodal large-model multi-turn dialogue routing method as described in claim 1, characterized in that, The information compression process includes modal decoupling coding, redundant information pruning, semantic quantization and feature sparsification, and differential privacy noise injection is performed after compression.

5. The multimodal large-model multi-turn dialogue routing method as described in claim 1, characterized in that, The anonymized semantic summary includes a compressed representation vector, modality weight distribution, task label information, round reference index, and cross-modal consistency evaluation index.

6. The multimodal large-model multi-turn dialogue routing method as described in claim 1, characterized in that, In the process of constructing the candidate inference path, at least one of the following criteria is used as the path selection basis: inference accuracy, cross-modal consistency, execution latency, resource consumption, and risk level.

7. The multimodal large-model multi-turn dialogue routing method as described in claim 1, characterized in that, The path evaluation process includes: inference result correctness evaluation, multi-round context coherence evaluation, modal conflict detection, execution latency and resource overhead evaluation, and a path score value is generated based on the above evaluation results for path selection.

8. The multimodal large-model multi-turn dialogue routing method as described in claim 1, characterized in that, It also includes triggering path rollback or adjustment when any of the following conditions are detected: inference results are inconsistent with cross-modal semantic graphs, context matching degree is lower than a preset threshold, inference result uncertainty exceeds a preset range, and execution cost exceeds a preset budget.

9. The multimodal large-model multi-turn dialogue routing method as described in claim 8, characterized in that, The path rollback or adjustment includes at least one of the following: switching to a highly reliable model, performing cross-modal conflict resolution, retrying after pruning the input content, or calling a toolchain or multi-model collaborative processing. It also uses a context-independent buffer structure to record modal references, round dependencies, and unfinished task states to support multi-turn dialogue continuity.

10. A multimodal large-scale multi-turn dialogue routing system, characterized in that, The system is used for implementing the multimodal large-model multi-turn dialogue routing method according to any one of claims 1 to 9, wherein the system comprises: The cross-modal semantic graph construction module is used to construct cross-modal semantic graphs. Based on the modal parsing results of text, images, audio and structured data, it generates a unified representation for each modality and establishes semantic links of entities, relations and context dependencies to form cross-modal semantic association data. The dialogue situation tracking module is used to generate a dialogue state vector based on the modal distribution, historical turn states, and intent transition signals of user input through dialogue situation tracking processing, and to perform task stage analysis, risk level assessment, and context preservation degree prediction on the dialogue state vector. The edge-side compression and privacy processing module is used to perform modal decoupling and information compression processing on the input data at the edge, encode each modal input into a representation vector, and compress the representation vector to obtain a compressed representation vector; inject differential privacy noise into the compressed representation vector to generate an anonymized semantic summary and send it to the server; The path construction and routing decision module is used by the server to construct a set of candidate inference paths from the multimodal model resource pool based on the anonymized semantic summary and the dialogue state vector, wherein each candidate inference path is formed by combining multiple model units or tool units. The path evaluation and adjustment module is used to perform path evaluation processing on the candidate inference path set. The path evaluation processing includes at least consistency evaluation, context matching degree evaluation and execution cost evaluation, and determines the target inference path based on the evaluation results. The context state management module is used to perform multi-model collaborative reasoning according to the target reasoning path, and to monitor the reasoning results in real time during the execution process. When inconsistent results, low matching degree or execution abnormality are detected, the execution path is adjusted or rolled back. The routing log module is used to evaluate the quality of the inference results, generate performance feedback data, and update the dialogue state, context information, and path selection strategy based on the performance feedback data. At the same time, it writes relevant data into the dialogue routing log for subsequent dialogue routing optimization.

Images