A multimodal heterogeneous model retrieval enhancement method and system

Abstract

The application provides a multimodal heterogeneous model retrieval enhancement method and system, comprising: based on user multimodal query, constructing knowledge and application example bilingual corpus, designing joint retrieval mechanism to get result set; through text, image, audio special processing channel and segmented trapezoidal topology structure Spiking neural network, mapping scheduling to get feature representation; constructing three-level cascade architecture of basic model, advanced model and human expert, combining with deferred and waiver decision mechanism, getting decision path and answer candidate set; using Hamilton graph network to represent multimodal relationship, using gradient-free descent method to quickly train and optimize model parameters; through cross-modal semantic alignment and dynamic retrieval window adjustment, getting enhanced retrieval result; through context perception sorting and retrieval enhancement reasoning, getting high-quality response. The application improves the multimodal information retrieval processing efficiency and heterogeneous model reasoning response quality.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for enhancing the retrieval of multimodal heterogeneous models, applicable to retrieval enhancement and generation systems for large language models and multimodal large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) and multimodal large models have become core technologies for natural language processing and multimedia content understanding. These models can process information in various forms such as text, images, and audio, but they still face challenges in practical applications, such as knowledge timeliness, reasoning ability, and processing of multi-source heterogeneous data.

[0003] Common techniques in the current field include basic retrieval augmented generation (RAG) and single-modal expert models. RAG technology enhances model output by retrieving external knowledge bases during inference, but its retrieval mechanism is usually limited to single-modal data and lacks a deep understanding of the retrieved content. While expert models can demonstrate excellent performance in specific domains, they often cannot efficiently integrate multimodal information for comprehensive reasoning.

[0004] Existing closest-to-the-source techniques employ a simple multimodal retrieval enhancement framework, storing and retrieving multimodal content through a vector database. This technique maps data from different modalities to a shared feature space based on an embedding model and uses similarity calculations for retrieval. During inference, the system retrieves relevant information based on the query content and provides the retrieval results as context to a larger model for answer generation.

[0005] However, this technology has obvious technical shortcomings: First, it lacks a principled modular architecture, making it impossible to effectively integrate application-aware reasoning into the retrieval pipeline; second, the retrieval strategy lacks adaptability, failing to dynamically adjust the model level used based on task complexity; third, it has limited preprocessing capabilities for heterogeneous data sources (especially audio and images), making it unable to fully extract and fuse multimodal information; and finally, the training methods of existing technologies have high computational complexity, making it difficult to quickly adapt to new domain knowledge. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for enhancing multimodal heterogeneous model retrieval, so as to solve the above-mentioned problems existing in the prior art.

[0007] To achieve the above objectives, this invention provides a method for enhancing multimodal heterogeneous model retrieval, comprising the following steps:

[0008] Based on user multimodal query input, a joint retrieval result set is obtained by constructing a dual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism.

[0009] The joint retrieval result set is mapped and scheduled using dedicated processing channels for text, image, and audio data and a Spiking neural network architecture with a segmented trapezoidal topology to obtain multimodal feature representations.

[0010] Based on the multimodal feature representation, by constructing a three-level cascaded architecture including a basic model, a high-level model, and human experts, and a dual decision-making mechanism of deferred and abstention, decision path information and a set of candidate answers are obtained.

[0011] For the decision path information and the answer candidate set, the multimodal information relationship is represented by a Hamiltonian graph network and a fast training method without gradient descent is used to obtain optimized model parameters;

[0012] Based on the optimized model parameters, multimodal enhanced retrieval results are obtained through a cross-modal semantic alignment algorithm and a dynamic retrieval window adjustment strategy based on task complexity.

[0013] The multimodal enhanced retrieval results are used to obtain high-quality final response results through a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism.

[0014] Preferably, the step of obtaining a joint retrieval result set based on user multimodal query input by constructing a dual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism includes:

[0015] The user's multimodal query input is subjected to a unified data interface protocol and standardized preprocessing to obtain a standardized multimodal query representation;

[0016] Based on the standardized multimodal query representation, a domain knowledge corpus index is obtained through multimodal knowledge extraction, structured storage, and vector index construction techniques using entity recognition and semantic analysis.

[0017] The standardized multimodal query representation is used to obtain an application example corpus index through task-oriented application example collection, annotation, and knowledge alignment processing.

[0018] Based on the domain knowledge corpus index and the application example corpus index, the joint retrieval result set is obtained through a weighted fusion retrieval algorithm and a relevance scoring mechanism.

[0019] Preferably, the joint retrieval result set is mapped and scheduled using dedicated processing channels for text, image, and audio data, and a Spiking neural network architecture with a segmented trapezoidal topology, to obtain multimodal feature representations, including:

[0020] The text content in the joint retrieval result set is processed by word segmentation, normalization, entity recognition, and semantic analysis to obtain a text feature vector set;

[0021] The image content in the joint retrieval result set is processed by image segmentation, enhancement, and target detection, and then mapped using a multi-level segmented trapezoidal topology Spiking neural network architecture to obtain an image feature vector set.

[0022] The audio content in the joint retrieval result set is processed by audio segmentation, noise reduction, and feature extraction, followed by temporal processing using a Spiking neural network to obtain an audio feature vector set.

[0023] Based on the text feature vector set, the image feature vector set, and the audio feature vector set, the multimodal feature representation is obtained through a cross-modal attention mechanism and a feature alignment algorithm.

[0024] Preferably, based on the multimodal feature representation, by constructing a three-level cascaded architecture including a base model, a high-level model, and human experts, and a dual decision-making mechanism of deferred and abstention, decision path information and a candidate set of answers are obtained, including:

[0025] The multimodal feature representation is used for preliminary reasoning and answer generation through a lightweight basic language model to obtain a preliminary answer and confidence score;

[0026] Based on the preliminary answer and the confidence score, the model level selection decision is obtained through confidence threshold comparison and task complexity evaluation algorithm;

[0027] For tasks that trigger deferred responses, deep reasoning is performed using a large-scale high-level language model with a parameter size larger than that of the base model, based on the multimodal feature representation, to obtain advanced answers and new confidence scores.

[0028] Based on the final answer and the new confidence score, a decision on human intervention is obtained through an abstention decision algorithm and uncertainty assessment.

[0029] Based on the model-level selection decision and the human intervention decision, combined with the preliminary answer and the final answer, the decision path information and answer candidate set are obtained through decision path tracking and answer quality evaluation mechanisms.

[0030] Preferably, the optimized model parameters are obtained by representing the multimodal information relationships between the decision path information and the answer candidate set using a Hamiltonian graph network and employing a fast training method without gradient descent, including:

[0031] Based on the decision path information and the answer candidate set, an initial Hamiltonian graph structure representing the multimodal information relationship is obtained through multimodal adaptation entity-relation extraction and graph construction algorithms;

[0032] For the initial Hamiltonian graph structure, the Hamiltonian graph network model to be trained is obtained through Hamiltonian graph network architecture design and parameter initialization strategy;

[0033] For the Hamiltonian graph network model to be trained, the model state sequence during the training process is obtained through the definition of energy function, direct state update and gradient-free optimization algorithm;

[0034] Based on the model state sequence during the training process, the optimized model parameters are obtained through convergence criteria and optimal state selection algorithm.

[0035] Preferably, the multimodal enhanced retrieval results obtained based on the optimized model parameters, through a cross-modal semantic alignment algorithm and a dynamic retrieval window adjustment strategy based on task complexity, include:

[0036] For the optimized model parameters, a unified cross-modal semantic representation space is obtained through shared semantic space mapping and inter-modal semantic alignment algorithms;

[0037] Based on the current user query content, combined with the unified cross-modal semantic representation space, a task complexity score is obtained through a multi-dimensional task complexity evaluation index calculation and difficulty grading mechanism, including concept complexity, relation complexity and reasoning complexity.

[0038] The task complexity is scored, and an optimized retrieval parameter configuration is obtained through an adaptive window size calculation and retrieval depth adjustment algorithm.

[0039] Based on the optimized retrieval parameter configuration and the unified cross-modal semantic representation space, the multimodal enhanced retrieval results are obtained through multi-dimensional retrieval strategies and result fusion mechanisms.

[0040] Preferably, the optimized model parameters are used to obtain a unified cross-modal semantic representation space through shared semantic space mapping and inter-modal semantic alignment algorithms, including:

[0041] The optimized model parameters are designed with a multi-branch structure, and the initial feature representations of each modality are obtained through specific encoders for text, image and audio branches.

[0042] The initial feature representations of each modality are mapped by a projection layer, normalized, and adjusted to obtain feature vectors with shared dimensions.

[0043] Based on the feature vector of the shared dimension, an alignment strategy including contrastive learning, semantic anchors and structure preservation is constructed through positive and negative samples to obtain a semantically aligned feature representation.

[0044] Cross-modal retrieval tests, semantic clustering analysis, and analogy verification are performed on the semantically aligned feature representations to obtain the unified cross-modal semantic representation space.

[0045] Preferably, based on the current user query content and combined with the unified cross-modal semantic representation space, a task complexity score is obtained through a multi-dimensional task complexity evaluation index calculation and difficulty grading mechanism, including concept complexity, relational complexity, and reasoning complexity. Based on the task complexity score, an optimized retrieval parameter configuration is obtained through an adaptive window size calculation and retrieval depth adjustment algorithm, including:

[0046] The query content of the current user is parsed and represented. Through query decomposition, multimodal processing and semantic encoding, the cross-modal semantic representation of the query is obtained.

[0047] Based on the cross-modal semantic representation of the query, a multi-dimensional complexity index is obtained through a comprehensive evaluation including concept complexity, relational complexity, reasoning complexity, and multimodal dependency.

[0048] The task complexity score is obtained by weighted fusion and nonlinear mapping of the multi-dimensional complexity index.

[0049] Based on the task complexity score, a complexity-resource mapping relationship is established, and the retrieval window size is obtained through basic window setting, task characteristic adjustment, and adaptive formula calculation.

[0050] The depth and breadth parameters of the search window size are adjusted, and the optimized search parameter configuration is obtained through modality-specific search parameter configuration.

[0051] Preferably, the multimodal enhanced retrieval results are processed using a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism to obtain high-quality final response results, including:

[0052] Based on the multimodal enhanced retrieval results and the user's multimodal query input, a query context representation is obtained through context encoding and context understanding algorithms;

[0053] Based on the query context representation and the multimodal enhanced retrieval results, a priority-ranked retrieval result is obtained through a context-aware relevance scoring and ranking algorithm.

[0054] Based on the priority-sorted retrieval results and the decision path information, a comprehensive reasoning result is obtained through the integration mechanism of retrieval knowledge and model reasoning.

[0055] The comprehensive reasoning results are used to generate high-quality final response results through response template generation, consistency checks, and quality optimization mechanisms based on accuracy and readability standards.

[0056] This invention also provides a multimodal heterogeneous model retrieval enhancement system, comprising:

[0057] The bilingual corpus retrieval module is used to obtain a joint retrieval result set based on user multimodal query input by constructing a bilingual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism.

[0058] The multimodal data preprocessing module is used to map and schedule the joint retrieval result set to obtain multimodal feature representations through dedicated processing channels for text, image and audio data and a Spiking neural network architecture with a segmented trapezoidal topology.

[0059] An adaptive cascaded decision module is used to obtain decision path information and a set of answer candidates based on the multimodal feature representation by constructing a three-level cascaded architecture including a base model, a high-level model, and human experts, as well as a dual decision-making mechanism of deferred and abstention.

[0060] The Hamiltonian graph training module is used to represent the multimodal information relationship between the decision path information and the answer candidate set through a Hamiltonian graph network and to obtain optimized model parameters using a fast training method without gradient descent.

[0061] The cross-modal alignment adjustment module is used to obtain multimodal enhanced retrieval results based on the optimized model parameters, through a cross-modal semantic alignment algorithm and a dynamic retrieval window adjustment strategy based on task complexity.

[0062] The response generation module is used to obtain high-quality final response results from the multimodal enhanced retrieval results through a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism.

[0063] The beneficial effects of this invention are:

[0064] 1. By constructing a dual-corpus RAG+ architecture, application-aware reasoning is explicitly integrated into the retrieval enhancement generation pipeline, enabling large models not only to access relevant information but also to apply it to structured, goal-oriented reasoning processes;

[0065] 2. By using an adaptive cascaded LLM decision framework, the task is adaptively delegated among multiple knowledge levels, balancing inference quality and computational resource consumption;

[0066] 3. Through multimodal data preprocessing based on Spiking neural network, efficient multimodal data preprocessing and prefusion were achieved, improving the ability to extract temporal and spatial features;

[0067] 4. A fast training method for Hamiltonian graph networks without gradient descent was adopted, which achieved efficient model training and significantly reduced computational complexity and training time.

[0068] 5. Through a dynamic search window adjustment mechanism, intelligent allocation of search resources is achieved, improving the system's adaptability and efficiency under tasks of varying complexity;

[0069] 6. By using cross-modal semantic alignment technology, effective alignment between data of different modalities is achieved, enhancing the comprehensive utilization of multimodal information and improving retrieval quality and inference accuracy. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of the multimodal heterogeneous model retrieval enhancement method provided in the embodiments of the present invention;

[0072] Figure 2 This is a flowchart of bilingual corpus retrieval provided in an embodiment of the present invention;

[0073] Figure 3 This is a flowchart of a gradient-free fast training method for Hamiltonian graph networks provided in an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram of the structure of the multimodal heterogeneous model retrieval enhancement system provided in an embodiment of the present invention. Detailed Implementation

[0075] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0076] Figure 1 This is a flowchart of the multimodal heterogeneous model retrieval enhancement method provided in an embodiment of the present invention. For example... Figure 1 As shown, the method includes the following steps:

[0077] Step S101: Based on the user's multimodal query input, a joint retrieval result set is obtained by constructing a dual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism.

[0078] In this step, the system first receives multimodal query input from the user, which may include various forms of information such as text questions, image content, and audio clips. To address this diverse input, the system constructs an innovative dual-corpus structure, consisting of two complementary parts: a knowledge corpus and an application example corpus. The knowledge corpus stores fundamental information such as factual knowledge, concept definitions, and theoretical frameworks within the domain; while the application example corpus collects specific application cases, solutions, and best practices related to the task. This dual-corpus design enables the system to not only retrieve query-related knowledge but also find solution examples for similar problems, thus achieving an organic combination of knowledge and application. A dedicated joint retrieval mechanism is also designed, capable of searching both corpora simultaneously and integrating the retrieval results through a weighted fusion algorithm, ensuring that the returned content includes both necessary background knowledge and actionable application guidance. Finally, a comprehensive joint retrieval result set is output, providing a rich information foundation for subsequent multimodal processing.

[0079] Step S102: For the joint retrieval result set, multimodal feature representations are obtained through mapping and scheduling using dedicated processing channels for text, image, and audio data and a Spiking neural network architecture with a segmented trapezoidal topology.

[0080] In this step, the joint retrieval result set obtained in step S101 undergoes in-depth multimodal data processing. Dedicated processing channels are designed for different types of data: text data is processed using techniques such as word segmentation, normalization, entity recognition, and semantic analysis; image data, after preprocessing such as segmentation, enhancement, and object detection, is processed using an innovative segmented trapezoidal topology Spiking neural network; audio data is segmented, denoised, and feature extracted, and then analyzed using the temporal processing capabilities of the Spiking neural network. This segmented trapezoidal architecture of the Spiking neural network is a key innovation of this invention. It simulates the impulse transmission mechanism of biological nervous systems, enabling efficient processing of spatiotemporal information, and is particularly suitable for capturing the spatial features of images and the temporal features of audio. Through this architecture, efficient feature extraction from multimodal data is achieved, and through cross-modal attention mechanisms and feature alignment algorithms, feature vectors from different modalities are integrated into a unified multimodal feature representation. This representation retains the key information of each modality and establishes semantic connections between modalities, providing rich and structured input for subsequent model decisions.

[0081] Step S103: Based on the multimodal feature representation, by constructing a three-level cascaded architecture including a basic model, a high-level model, and human experts, and a dual decision-making mechanism of deferred and abstention, decision path information and answer candidate set are obtained.

[0082] This step implements an adaptive cascaded decision-making framework that intelligently allocates computational resources and ensures answer quality through a three-level architecture and a dual decision-making mechanism. First, a lightweight base language model is used to perform preliminary inference on the multimodal feature representations, generating preliminary answers and corresponding confidence scores. Then, the system determines whether to defer the task to a higher-level model based on confidence threshold comparison and task complexity assessment. For complex or critical tasks, a higher-level language model with a larger parameter scale is invoked for deep inference, generating more accurate answers. The system also includes an abstention mechanism; when the model's confidence in the answer is insufficient or the task risk is too high, the system seeks intervention from human experts. This dual decision-making mechanism of deferral and abstention ensures efficient use of computational resources while maintaining answer quality, avoiding the overuse of expensive high-level models for simple tasks. Finally, the entire decision-making process is recorded, generating detailed decision path information and multiple answer candidates, providing complete decision-making basis and alternative solutions for subsequent processing.

[0083] Step S104: For the decision path information and the answer candidate set, the multimodal information relationship is represented by a Hamiltonian graph network and a fast training method without gradient descent is used to obtain optimized model parameters.

[0084] In this step, an innovative Hamiltonian graph network structure and a gradient-free descent optimization method are employed to achieve efficient representation of multimodal information relationships and rapid model training. The system first extracts key entities and concepts based on decision path information and answer candidate sets, analyzes their semantic relationships, and constructs an initial Hamiltonian graph structure. A Hamiltonian graph is a special graph structure where a path passes through each vertex exactly once; this structure is particularly suitable for representing the sequential and holistic nature of information. Based on this structure, the system designs a specialized Hamiltonian graph network, creating specialized neuron models for different types of nodes and defining an information transfer mechanism along the Hamiltonian path. Unlike traditional gradient descent methods, a gradient-free descent optimization algorithm is used. Through defining an energy function, direct state updates, and parallel state exploration, efficient model training is achieved. This method avoids complex gradient calculations, significantly reducing computational complexity, and is particularly suitable for handling rapid adaptation and transfer learning scenarios involving multimodal data. By analyzing the state sequence during training and applying convergence criteria and optimal state selection algorithms, optimized model parameters are finally obtained, providing an algorithmic foundation for subsequent cross-modal alignment and retrieval window adjustment.

[0085] Step S105: Based on the optimized model parameters, multimodal enhanced retrieval results are obtained through a cross-modal semantic alignment algorithm and a dynamic retrieval window adjustment strategy based on task complexity.

[0086] This step achieves deep alignment of cross-modal semantics and intelligent resource allocation based on task complexity. The system first constructs a unified cross-modal semantic representation space using optimized model parameters, enabling data from different modalities to be compared and manipulated within the same semantic space. This representation space is implemented through a multi-branch structure design, shared semantic space mapping, and inter-modal semantic alignment algorithms, ensuring semantic consistency across different modalities such as text, images, and audio. Simultaneously, the complexity of the current user query is assessed by analyzing multiple dimensions, including conceptual complexity, relational complexity, and inference complexity, to calculate a comprehensive task complexity score. Based on this score, the retrieval window size and retrieval depth are dynamically adjusted to allocate appropriate computational resources for tasks of varying complexity. For simple tasks, a smaller retrieval window and shallow retrieval are used; for complex tasks, the window is expanded and the retrieval depth is increased. Specific retrieval parameters are also designed for different modalities to ensure that all types of data are processed appropriately. Through this dynamic adjustment mechanism, supported by multi-dimensional retrieval strategies, the system generates high-quality multimodal enhanced retrieval results, ensuring both retrieval effectiveness and optimized resource utilization efficiency.

[0087] Step S106: For the multimodal enhanced retrieval results, a high-quality final response result is obtained through a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism.

[0088] In this step, the multimodal enhanced retrieval results are deeply understood and integrated to generate a final high-quality response. The system first constructs a rich contextual representation based on the retrieval results and the original query content. This includes analyzing the history of the entire interaction session, understanding the user intent behind the query, capturing the correlation between different modal information, and constructing a structured contextual knowledge graph. Based on this contextual representation, the system applies a context-aware relevance ranking algorithm to evaluate the relevance of the retrieval results to the query from multiple dimensions, including semantic relevance, intent satisfaction, and information completeness. A diversity guarantee mechanism is also implemented to ensure that the results cover different angles and perspectives. The ranked retrieval results are combined with decision path information. Through a retrieval-enhanced reasoning integration mechanism, the system constructs a reasoning knowledge base and applies a multi-step reasoning framework to seek the optimal balance between knowledge-driven and model-based reasoning, generating a comprehensive reasoning result that is both factually grounded and logically profound. Finally, through response generation templates, language optimization techniques, consistency checks, and quality improvement optimization, the reasoning result is transformed into a clear, accurate, and practical final response, effectively meeting user needs. The entire process ensures the high quality, readability, and practical value of the response, marking the successful completion of the multimodal heterogeneous model retrieval enhancement process.

[0089] The following is a detailed explanation of each step.

[0090] In step S101, based on the user's multimodal query input, a joint retrieval result set is obtained by constructing a dual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism.

[0091] Specifically, such as Figure 2 As shown, step S101 includes the following sub-steps:

[0092] Step S1011: Perform a unified data interface protocol and standardized preprocessing on the user's multimodal query input to obtain a standardized multimodal query representation.

[0093] In this sub-step, a unified data access layer is first established, capable of simultaneously processing different forms of query input from users, such as text, images, and audio. For text data, encoding conversion (e.g., standardizing to UTF-8), language recognition (determining the language type of the text), and basic cleaning (removing special characters, normalizing whitespace, etc.) are performed. For image data, format unification (converting to standard formats such as PNG or JPEG), resolution adjustment (scaling the image to a predefined standard size), and color space normalization (e.g., RGB normalization) are executed. For audio data, sampling rate unification (e.g., converting to a 16kHz standard sampling rate), audio format normalization (e.g., converting to WAV format), and duration normalization (truncating or padding the audio to a standard length) are performed. A unique identifier is assigned to each modality of data, and an association mapping between modality type and metadata is established, including the creation of timestamps, source tags, and data quality metrics. This mapping relationship enables the system to track and manage data of different modalities in subsequent processing. Finally, all standardized multimodal data are integrated into a unified query representation structure, which includes standardized raw data, metadata information, and descriptions of the relationships between modalities, providing a structured input foundation for subsequent corpus retrieval.

[0094] Step S1012: Based on the standardized multimodal query representation, a domain knowledge corpus index is obtained through multimodal knowledge extraction, structured storage, and vector index construction techniques using entity recognition and semantic analysis.

[0095] In this sub-step, the relevant knowledge domain scope is first determined based on the standardized query representation. The system analyzes the key concepts, entities, and topics in the query and maps them to a predefined knowledge domain classification system. This process combines keyword matching, entity linking, and semantic analysis techniques. Subsequently, relevant domain knowledge is extracted from multi-source knowledge bases, including structured databases (such as MySQL and MongoDB), semi-structured documents (such as JSON and XML), unstructured text (such as PDFs and web pages), and multimedia content libraries. Appropriate knowledge extraction techniques are used for different types of knowledge sources: SQL queries or API calls are used for structured data; parsers and mapping rules are used for semi-structured data; natural language processing techniques are used to extract entities, relations, and facts from unstructured text; and content recognition and semantic analysis techniques are used for multimedia content. The extracted knowledge is structured, organized, and stored to build knowledge graphs (entity-relationship-entity triples), semantic networks, or other knowledge representation structures suitable for multimodal content, while maintaining the relationships between different modalities of knowledge, such as links between text descriptions and related images, and correspondences between audio content and its text transcription. Finally, appropriate encoding models (such as BERT, ResNet, and WavLM) are used to convert knowledge content from different modalities into high-dimensional vector representations. Then, an index structure is created using vector databases (such as FAISS and Milvus), supporting fast approximate nearest neighbor search based on semantic similarity, which greatly improves retrieval efficiency. Through these steps, a structured domain knowledge corpus index is output, laying the foundation for subsequent joint retrieval.

[0096] Step S1013: The standardized multimodal query representation is processed through task-oriented application example collection, annotation, and knowledge alignment to obtain an application example corpus index.

[0097] In this sub-step, the implicit task type and objective are first analyzed based on the standardized multimodal query representation. The system performs task intent recognition (such as information query, question answering, content generation, etc.), task domain classification, and task complexity assessment. Through this analysis, it determines which types of application examples need to be collected. Subsequently, the system collects relevant application examples from multiple sources, including: a high-quality set of manually annotated examples (carefully created by domain experts to showcase ideal question-answer patterns), historical interaction records (successful interaction cases with users before, after screening and purification), and automatically generated synthetic examples (created through template filling or controlled generation to supplement scarce scenarios). For each application example, the complete input-processing-output flow is recorded, including query content, processing steps, and final response. The collected application examples are annotated and structured. The annotation information includes task type, domain tags, difficulty level, reasoning steps used, and knowledge points involved. This structured annotation enables the system to accurately match relevant examples based on the characteristics of the current query. Furthermore, the system aligns application examples with a knowledge corpus, analyzing the knowledge points involved in each application example and finding corresponding knowledge entries in the corpus to establish explicit reference relationships. This alignment enables the system to understand "how this application example uses these knowledge points to solve the problem," providing a paradigm for subsequent application-aware reasoning. Finally, the system constructs a vector index for application examples, similar to the construction process of a knowledge corpus index, but with a greater focus on capturing task patterns and solution structures, enabling the system to quickly retrieve application examples similar to the current query task.

[0098] Step S1014: Based on the domain knowledge corpus index and the application example corpus index, the joint retrieval result set is obtained through a weighted fusion retrieval algorithm and a relevance scoring mechanism.

[0099] In this sub-step, an innovative dual-corpus joint retrieval strategy was designed and implemented, organically combining knowledge retrieval with application example retrieval. The system first performs parallel knowledge corpus retrieval and application example corpus retrieval based on the current query content. For knowledge corpus retrieval, the query content is encoded into a query vector, and then a nearest neighbor search is performed in the knowledge base index to retrieve the K knowledge entries with the highest semantic similarity. Similarly, for application example corpus retrieval, the system encodes the query content into a task vector and finds the M application examples with the most similar task patterns in the application example index. Subsequently, a weighted fusion algorithm is applied to balance the results of knowledge retrieval and example retrieval, considering multiple factors: query explicitness (for explicit factual queries, knowledge retrieval results may be more important; for open or procedural tasks, application examples may be more valuable), the confidence level of the retrieval results (the reliability of the retrieval results is evaluated based on similarity scores), and task complexity (complex tasks may require more application example guidance). The system dynamically adjusts the weights of knowledge entries and application examples in the final result based on these factors. A relevance scoring mechanism was also implemented, calculating a comprehensive relevance score for each search term, considering multiple dimensions such as semantic relevance (the degree of matching with the query content), timeliness (the recentity of the information), authority (the reliability of the source), and diversity (avoiding redundant information). Finally, based on the comprehensive relevance score, the most relevant items were selected from knowledge entries and application examples, combined into the final joint search result set, and post-processed, including redundancy removal, content merging, and context organization, to ensure the compactness and coherence of the result set. Through this dual-corpus joint search strategy, a high-quality joint search result set was output, providing rich input for subsequent multimodal data processing.

[0100] In step S102, the joint retrieval result set is mapped and scheduled using dedicated processing channels for text, image, and audio data and a Spiking neural network architecture with a segmented trapezoidal topology to obtain multimodal feature representations.

[0101] Specifically, step S102 includes the following sub-steps:

[0102] Step S1021: Perform word segmentation, normalization, entity recognition, and semantic analysis on the text content in the joint retrieval result set to obtain a text feature vector set.

[0103] In this sub-step, the text content in the combined retrieval result set is deeply processed to extract rich text features. First, perform basic preprocessing operations, including word segmentation (dividing the text into words or sub-word units, using appropriate tokenizers such as BPE, WordPiece, etc. for different languages), part-of-speech tagging (identifying the grammatical roles of each word, such as nouns, verbs, etc.), stop-word filtering (removing common but information-poor words such as "de", "shi", etc.), and stemming (reducing words to their basic forms, e.g., "running" to "run"). These basic processing steps purify the text, removing noise and facilitating subsequent deep feature extraction. Next, perform semantic structure analysis, including syntactic parsing (identifying the grammatical structure of sentences, such as subject-verb-object relationships), dependency analysis (determining the dependency relationships between words), and semantic role labeling (identifying semantic relationships such as who did what to whom). This structured analysis helps to understand the internal organization and logical relationships of the text. Then, perform entity recognition and relationship extraction, including named entity recognition (identifying specific entities such as person names, place names, organization names, etc.), entity linking (mapping entities to standard entries in the knowledge base), and relationship extraction between entities (identifying semantic relationships between entities, such as "subordinate to", "founded in", etc.). This step enables the capture of key information points in the text and their interrelationships. Next, use pre-trained language models (such as BERT, RoBERTa, T5, etc.) to extract context-sensitive representations of the text. Specifically, input the preprocessed text into these models to obtain high-dimensional vector representations of each word or sentence, which can capture lexical semantics and context relationships. To handle long texts, adopt a sliding window technique or a hierarchical encoding strategy to ensure the capture of long-distance dependencies. Finally, through feature aggregation and dimensionality reduction techniques, integrate the various extracted features into a compact and information-rich set of text feature vectors. This may involve techniques such as weighted averaging, attention mechanisms, or principal component analysis, aiming to generate a set of feature vectors that can comprehensively represent the semantics, structure, and entity relationships of the text content. Through these steps, the text content in the combined retrieval result set is transformed into a structured set of text feature vectors, laying the foundation for subsequent multi-modal feature fusion.

[0104] Step S1022: For the image content in the combined retrieval result set, after image segmentation, enhancement, and object detection, perform mapping processing using a Spiking neural network architecture with a multi-level segmented trapezoidal topology structure to obtain an image feature vector set.

[0105] In this sub-step, the image content in the joint retrieval result set undergoes specialized processing, utilizing an innovative Spiking neural network segmented trapezoidal architecture to extract image features. First, basic image preprocessing is performed, including image resizing (unifying images of different sizes to a standard size), color space conversion (e.g., RGB to HSV, facilitating specific feature extraction), contrast enhancement (improving image clarity and detail visibility), noise removal (eliminating interfering noise in the image), and illumination normalization (reducing the impact of illumination variations on feature extraction). These preprocessing steps standardize the image, reduce interference from external factors, and improve the stability and effectiveness of subsequent feature extraction. Next, image segmentation is performed, dividing the image into meaningful regions or objects. Depending on the task requirements, different segmentation methods are employed: semantic segmentation (assigning a category label to each pixel), instance segmentation (distinguishing different object instances of the same category), or panoramic segmentation (combining semantic and instance segmentation). This segmentation allows for the separate processing of different components in the image, improving the accuracy of feature extraction. Then, the Spiking neural network is mapped onto the segmented trapezoidal architecture for image feature extraction. This innovative architecture has several key features: segmented processing, dividing image feature extraction into multiple stages, progressively extracting features from low-level features (such as edges and textures) to high-level features (such as shapes and objects); a trapezoidal structure, where the number of neurons and connection patterns in each processing stage are distributed in a trapezoidal pattern, with a large number of neurons and dense connections in the lower layers, and fewer neurons in the higher layers that process more complex features; and a Spiking neuron model, using a spiking-based neuron model to simulate the firing behavior of biological neurons. This model is more efficient in processing spatiotemporal information and can naturally handle abrupt changes and sparse features in images. In this architecture, image information is processed through a multi-layered Spiking neural network hierarchy: the first layer performs basic feature detection (such as edges and corners); the middle layers combine these basic features to form complex patterns (such as textures and partial shapes); and the higher layers integrate these patterns to identify complex objects and scenes. Due to the temporal characteristics of the Spiking neural network, it can naturally capture the dynamic visual features of image content, such as motion trends or change patterns. Finally, the extracted features are integrated and optimized to generate an image feature vector set. This includes feature selection (selecting the most discriminative features), feature aggregation (combining related features), and feature compression (reducing dimensionality while retaining key information). The final image feature vector set contains both visual content information of the image and high-level semantic information for object recognition and scene understanding. Through this segmented trapezoidal architecture based on Spiking neural networks, rich and efficient feature representations can be extracted from images, providing high-quality image feature input for subsequent multimodal fusion.

[0106] Specifically, the Spiking neural network in this invention employs a specific structure and parameter configuration. Regarding network structure, the number of neurons in the input layer is consistent with the feature dimension of the input image, ensuring complete reception of image information. The hidden layers adopt a trapezoidal structure design of 3 to 5 layers, with the number of neurons decreasing layer by layer, typically from 1024 neurons to 256 neurons. This trapezoidal structure allows the network to progressively abstract high-level features from low-level features. In terms of neuron model selection, a leak-integral firing model is used. This model simulates the membrane potential changes, threshold triggering, and reset mechanisms of biological neurons, possessing both biological interpretability and computational efficiency. The membrane potential time constant of the neurons is set to a moderate value, ensuring sensitivity to the input signal while avoiding noise interference caused by oversensitivity. The initialization of synaptic weights uses a method adapted to the depth of the neural network to ensure effective signal propagation. During training, a plasticity learning rule based on pulse timing is used to adjust the synaptic weights. This learning rule, inspired by biological neurons, can adjust the connection strength according to the firing timing relationship between preceding and subsequent neurons. The network training uses preprocessed standardized image features as input data, sets the batch size to a moderate value, and dynamically adjusts the number of training rounds according to the convergence of the network performance. It usually takes dozens of training rounds to reach a stable state.

[0107] Step S1023: The audio content in the joint retrieval result set is processed by audio segmentation, noise reduction and feature extraction, followed by temporal processing using a Spiking neural network to obtain an audio feature vector set.

[0108] In this sub-step, the audio content in the joint retrieval result set undergoes specialized processing, utilizing the temporal processing capabilities of the Spiking neural network to extract audio features. First, basic audio preprocessing is performed, including audio sampling rate unification (converting to a standard sampling rate such as 16kHz or 44.1kHz to ensure processing consistency), channel number adjustment (e.g., converting to mono or stereo, selecting an appropriate channel configuration based on task requirements), volume normalization (adjusting the amplitude of the audio signal to an appropriate range to avoid volume differences affecting feature extraction), and noise suppression (removing background noise, buzzing, and other interference to improve signal quality). These preprocessing steps ensure the quality and consistency of the audio data, creating favorable conditions for subsequent feature extraction. Next, audio segmentation is performed, dividing the continuous audio stream into meaningful units. This segmentation may be based on acoustic characteristics (such as silence detection, energy changes), semantic content (such as sentence boundaries, topic transitions), or fixed time windows. For speech content, speaker segmentation is also performed to distinguish speech segments from different speakers. This fine-grained segmentation enables more precise analysis of the temporal structure of the audio content. Next, time-frequency features of the audio are extracted, including Short-Time Fourier Transform (STFT, which converts a time-domain signal into a time-frequency representation), Mel-frequency cepstral coefficients (MFCC, which simulates human hearing frequency perception), spectrograms (visualizing the changes in sound intensity over time and frequency), and other acoustic features such as fundamental frequency (F0) and harmonic noise ratio (HNR). These features collectively describe the acoustic properties of the audio, providing a foundation for understanding audio content. Next, a Spiking neural network is used for temporal processing of the audio. Unlike traditional neural networks, the Spiking neural network simulates the pulse communication mechanism of biological neurons, making it particularly suitable for processing time-dependent signals (such as audio). The extracted time-frequency features are mapped to the input stimuli of the Spiking neurons, which generate pulse sequences based on the spatiotemporal patterns of the input stimuli. These pulse sequences are processed through a multi-layered Spiking neural network to capture the temporal dynamics of the audio signal. This Spiking neural network is particularly suitable for recognizing patterns in audio (such as speech phonemes, musical rhythms, ambient sounds, etc.) because it can naturally represent and process time-coded information. Different layers of the network process temporal features at different levels of abstraction: low layers capture basic acoustic features, mid-layers identify audio patterns, and high layers extract semantic information. Finally, the outputs of the Spiking neural network are integrated to generate an audio feature vector set. This includes statistical analysis of the pulse sequence (such as pulse rate and pulse time distribution), feature aggregation, and dimensionality optimization. The final audio feature vector set contains not only the acoustic properties of the audio but also its temporal dynamic features and potential semantic information. This temporal processing method based on the Spiking neural network enables the extraction of rich feature representations from audio, providing high-quality audio feature input for subsequent multimodal fusion.

[0109] Step S1024: Based on the text feature vector set, the image feature vector set, and the audio feature vector set, the multimodal feature representation is obtained through a cross-modal attention mechanism and a feature alignment algorithm.

[0110] In this sub-step, deep fusion of text, image, and audio features is achieved, creating a unified multimodal feature representation. First, the alignment problem of features from different modalities is addressed, a key challenge in multimodal fusion because data from different modalities have different dimensions, scales, and representations. Several techniques are employed to achieve intermodal alignment: temporal alignment, for modalities containing temporal information (such as audio and video), establishing a timeline mapping through timestamps or content synchronization points; semantic alignment, using pre-trained cross-modal models (such as CLIP, AudioCLIP, etc.) to map features from different modalities to a shared semantic space; and structural alignment, for structured content (such as objects described in text and visual objects in images), establishing mapping relationships through entity links and scene graph construction. Next, a cross-modal attention mechanism is implemented, allowing features from one modality to selectively focus on relevant information in other modalities. Specifically, query, key, and value matrices are defined for each modality, and then intermodal attention weights are calculated to generate a context vector. For example, text features can focus on relevant regions in an image or relevant moments in audio. This attention mechanism enables the capture of complex interactions between modalities, rather than simply concatenating them. Then, feature alignment algorithms are designed and applied to ensure that features from different modalities have compatible representations during the fusion process. This includes feature normalization (making features from different modalities have similar statistical properties), dimensionality transformation (adjusting feature dimensions through projection layers), and modality-specific encoding (considering the unique properties of each modality). These alignment operations are implemented using deep neural networks (such as multilayer perceptrons) or linear transformations. Next, a multi-level fusion strategy is executed to fuse multimodal features at different levels of abstraction: early fusion, fusing raw features in the early stages of feature extraction; mid-stage fusion, fusing intermediate feature representations in the intermediate processing stages; and late-stage fusion, fusing high-level representations of each modality at the decision level. These fusion strategies are adaptively selected or combined based on task characteristics and data features to achieve optimal results. Finally, a unified multimodal feature representation is generated. This representation not only preserves the key information of each modality but also captures the complementarity and synergistic effects between modalities. Specifically, gating mechanisms (such as LSTM or GRU units) may be used to control the flow of information between modalities, or self-attention mechanisms may be used to integrate global contextual information. The final multimodal feature representation is a high-dimensional vector or sequence of vectors that comprehensively describes the multimodal content in the joint retrieval result set. This cross-modal attention and feature alignment fusion method integrates text, image, and audio features into a unified multimodal feature representation, providing rich and coordinated input for the subsequent adaptive cascaded LLM decision framework.

[0111] In step S103, based on the multimodal feature representation, decision path information and answer candidate set are obtained by constructing a three-level cascaded architecture including a basic model, a high-level model, and human experts, as well as a dual decision-making mechanism of deferred and abstention.

[0112] Specifically, step S103 includes the following sub-steps:

[0113] Step S1031: For the multimodal feature representation, perform preliminary reasoning and answer generation through a lightweight basic language model to obtain a preliminary answer and confidence score.

[0114] In this sub-step, a lightweight base language model is used to perform preliminary inference on multimodal features to quickly generate preliminary answers. First, an input format suitable for the base model is prepared. Base models are typically language models with low computational resource requirements (such as small Transformer models, distillation models, etc.), with far fewer parameters and lower inference costs than large models. The multimodal feature representation output in step S1024 is converted into an input format acceptable to the base model. This includes: feature mapping, mapping multimodal features to the model's input space through an adaptation layer to ensure that the feature dimensions match the model's input dimensions; sequence construction, organizing multimodal features into the sequence format required by the model, which may include special delimiters to distinguish different parts of the information; length control, truncating or segmenting features exceeding the model's maximum input length to ensure that the model's context window limit is not exceeded; and modality labeling, using special labels to distinguish information from different modalities, helping the model understand the structure and source of the input. Next, a cue template suitable for the multimodal task is designed. The cue template is key to guiding the model to generate specific types of output; it needs to clearly express the task requirements and output format. For multimodal tasks, the prompt template also needs to guide the model on how to integrate information from different modalities. Prompt templates may be dynamically generated based on task type (e.g., question answering, summarizing, classification) and specific needs, and combined with processed multimodal features to form a complete model input. Then, preliminary inference is performed through a base language model. After receiving the input, the base model generates the corresponding output sequence through forward propagation. During this process, specific decoding strategies (e.g., bundle search, greedy decoding) may be used to control the generation process, balancing speed and quality. Because the base model is relatively lightweight, this step can be completed quickly, providing timely feedback for subsequent decisions. Next, a structured preliminary answer is extracted from the model output. This includes parsing the output text, identifying key information points, and formatting the answer structure. For specific tasks, further processing of the output may be required, such as extracting specific entities, relationships, or operational steps from the generated text to ensure the structure and usability of the answer. Finally, a confidence score for the preliminary answer is calculated. This score reflects the base model's confidence in its generated answer and is an important basis for subsequent deferred decisions. Confidence scoring may be based on multiple factors: the probability distribution within the model, such as the probability value of generating each token; answer completeness, whether the answer contains all necessary information points; reasoning consistency, whether the internal logic of the answer is self-consistent; and knowledge coverage, how much relevant knowledge the answer covers. These factors are considered together to calculate a comprehensive confidence score. Through these steps, the output includes a preliminary answer and a confidence score, providing a foundation for the next step of deferred strategy decision-making. This lightweight basic model's preliminary inference ensures both response speed and provides a reference for subsequent, more complex processing.

[0115] Step S1032: Based on the preliminary answer and the confidence score, obtain the model level selection decision through confidence threshold comparison and task complexity evaluation algorithm.

[0116] In this sub-step, an intelligent deferral decision-making mechanism is implemented to determine whether the task needs to be upgraded to a higher-level model. First, a deferral threshold system is set. This is a set of dynamically adjusted threshold parameters used to determine whether deferral to a higher-level model is necessary. Multiple dimensions are considered when setting thresholds: confidence threshold, the minimum confidence score the base model output must achieve; below this threshold, it indicates the base model lacks sufficient confidence in its answer; complexity threshold, the criterion for judging task complexity; above this threshold, it indicates the task may exceed the processing capacity of the base model; sensitivity threshold, the criterion for judging task sensitivity or importance; highly sensitive tasks, even with high confidence, may require validation with a higher-level model. These thresholds may be dynamically adjusted based on the task domain, user settings, or historical performance, forming an adaptive decision-making standard. Next, the confidence score of the base model output is evaluated. The confidence score output in step S1031 is compared with the set confidence thresholds. If the confidence score is significantly higher than the threshold, it indicates the base model has high confidence in its answer and deferral may not be necessary; if the confidence score is close to or lower than the threshold, deferral to a higher-level model may need to be considered. Next, a task complexity assessment is performed. This involves analyzing multiple aspects of the query content and context: semantic complexity, the conceptual depth and abstraction of the query (queries involving advanced theories or abstract concepts are generally more complex); knowledge breadth, the range of knowledge domains involved (queries across multiple domains typically require broader knowledge integration); reasoning depth, the number of reasoning steps required to solve the problem (multi-step reasoning problems are generally more challenging); and multimodal dependency, the degree of requirement for multimodal information integration (tasks heavily reliant on multimodal information may require more powerful processing capabilities). These factors are quantified using a specialized complexity assessment algorithm to generate a comprehensive complexity score, which is then compared to a complexity threshold. Next, other deferred factors are considered. In addition to confidence and complexity, additional factors are considered: task sensitivity, the potential risks or consequences of incorrect answers, such as queries in highly sensitive domains like healthcare and law; resource efficiency, the balance between the computational cost and potential benefits of invoking advanced models, considering current resource load; response time requirements, the user's need for timely answers, where fast response may be prioritized in certain scenarios; and historical performance, the historical performance differences between the base model and advanced models on similar tasks, based on performance predictions using past data. These factors collectively constitute a comprehensive consideration for deferral decision-making. Finally, based on the above assessments, a deferral decision is made and a decision path is generated. Decision outcomes typically fall into two categories: no deferral, where the answer from the base model is deemed sufficiently reliable and will be used directly, skipping the processing of the advanced model; or deferral, where a more powerful model is deemed necessary for further processing, and the task is upgraded to an advanced model. This decision and its rationale are recorded as part of the decision path for subsequent tracking and interpretation.This comprehensive evaluation and deferral strategy intelligently allocates computing resources, ensuring that complex or critical tasks are handled by sufficiently powerful models while avoiding the overuse of expensive high-level model resources for simple tasks. The output model hierarchy selection decision provides clear guidance for subsequent processing.

[0117] Step S1033: For the task that triggers deferred execution, deep reasoning is performed based on the multimodal feature representation through a large-scale high-level language model with a parameter size larger than that of the base model to obtain a high-level answer and a new confidence score.

[0118] In this sub-step, when deferral is required, a large-scale high-level language model is invoked for deep reasoning to generate a more accurate answer. First, enhanced input suitable for high-level model processing is prepared. Compared to the base model, high-level models (such as large Transformer models, multimodal large models, etc.) typically have stronger processing capabilities and larger context windows. The multimodal feature representation from step S1024 is combined with the base model inference results from step S1031 to construct the enhanced input. This enhanced input includes: original multimodal features, preserving complete information content to ensure the high-level model can access all relevant data; the base model inference results, serving as a preliminary analysis reference, providing a starting point so the high-level model can focus on areas needing improvement; specific aspects with insufficient confidence, indicating problems that the base model has not adequately addressed, guiding the high-level model to focus on these areas; and expanded context, potentially introducing more relevant knowledge or broader background information, leveraging the high-level model's larger context window capabilities. Next, a specialized high-level prompting strategy is designed. High-level models are typically more sensitive to prompts and can understand more complex instructions. Based on task requirements and the shortcomings of the base model, structured prompts are constructed to guide the high-level model to focus on key issues. Such prompts may include: task decomposition, breaking down complex tasks into steps to guide the model's step-by-step reasoning; reasoning frameworks, providing specific reasoning structures or methods, such as "first analyze the facts, then consider possible explanations, and finally draw a conclusion"; evaluation criteria, specifying the quality standards the answer should meet, such as "ensure the answer is comprehensive, accurate, and has sufficient supporting evidence"; and example guidance, providing exemplary solutions to similar questions, demonstrating the expected answer style and depth. These carefully designed prompts fully leverage the capabilities of high-level models. Then, a large-scale high-level language model is invoked to perform deep reasoning. After receiving enhanced input and specialized prompts, the high-level model performs deeper and more comprehensive analysis and generation. Compared to the basic model, high-level models are typically able to: perform longer, multi-step reasoning, handle complex logical relationships and conditional judgments; integrate broader knowledge, accessing and applying richer internal knowledge; handle more complex multimodal relationships, gaining a deeper understanding of the interactions between different modalities; and generate more coherent and comprehensive answers, providing more detailed explanations and supporting evidence. Specific parameter settings (such as temperature, top-p values, etc.) may be used to control the generation process, balancing creativity and accuracy. Next, the output of the high-level model is extracted and structured. Advanced models typically generate richer and more complex content, requiring the extraction of key information and organization according to a predefined structure. This might include: the answer body, the core conclusion or response, directly answering the user's query; the reasoning process, the logical steps leading to the conclusion, demonstrating the thought process; evidence citations, specific knowledge points supporting the conclusion, enhancing the credibility of the answer; and an explanation of uncertainty, the model's degree of confidence in different aspects, transparently expressing potential uncertainties. This structured processing makes the output of advanced models easier to understand and evaluate.Finally, a new confidence score for the advanced answer is calculated. Similar to the base model, the reliability of the answers generated by the advanced model needs to be evaluated. However, the confidence evaluation of the advanced model may be more complex, considering more factors: generation probability (the model's probabilistic assessment of the output); knowledge support (the consistency between the answer and the retrieved knowledge); reasoning completeness (the completeness and reasonableness of the reasoning chain); multimodal integration (the effective utilization of information from different modalities); and self-evaluation (the model's evaluation of its own answer, if supported). These factors are then considered to calculate the confidence score for the advanced answer. Through these steps, when deferral is required, the powerful capabilities of the advanced model are leveraged for deep reasoning, outputting high-quality answers and corresponding confidence scores, providing a basis for subsequent abstention strategy decisions.

[0119] Step S1034: Based on the final answer and the new confidence score, a decision on human intervention is obtained through an abstention decision algorithm and uncertainty assessment.

[0120] In this sub-step, an intelligent abstention decision-making mechanism is implemented to determine whether the model-generated answer is reliable enough or requires human intervention. First, an abstention threshold system is established. This is a set of standards to determine when to abstain and seek human intervention. Thresholds are set in multiple dimensions: minimum confidence threshold, the minimum confidence level the model answer must reach; below this threshold, it indicates the model lacks sufficient confidence in the answer; consistency threshold, the degree of consistency that different models or inference paths must achieve; high inconsistency indicates the answer may be controversial; risk threshold, the acceptable level of potential error risk; high-risk tasks require higher confidence. These thresholds may be preset based on the task domain, regulatory requirements, or organizational policies, or dynamically adjusted based on historical performance and user feedback. Next, the confidence of the model answer is evaluated. The currently valid model answer is obtained; if the task has been deferred, it is the advanced model answer of step S1033; if not deferred, it is the basic model answer of step S1031. The confidence score of this answer is compared with the minimum confidence threshold. If the confidence score is significantly below the threshold, it indicates a lack of confidence in the model's answer and may require abandonment; if the confidence score is close to or above the threshold, the model's answer is likely sufficiently reliable. Then, an uncertainty analysis is performed. Even with a high confidence score, some parts of the answer may still contain high uncertainty. Analyze different parts or aspects of the answer to identify particularly uncertain elements: factual assertions—the model's certainty about specific facts, especially statements involving numbers, dates, or expertise; reasoning steps—weak links in the reasoning chain, such as inferences based on assumptions or indirect evidence; predictive content—predictions about future or unknown situations that are inherently uncertain; and boundary cases—content falling within the model's knowledge boundaries, potentially beyond the model's training scope. Mark these uncertainties and assess their impact on the overall answer quality. Next, assess the potential risks and impacts. The decision to abstain is based not only on technical reliability but also on practical impact: consequences of errors (the negative consequences of an incorrect answer, such as the potential health effects of an error in medical advice); task criticality (the importance of the task to the user or individual, as critical tasks require higher accuracy); time sensitivity (the acceptability of delayed responses (waiting for human intervention), as a quick response may be more important than a perfect one in some cases); and resource availability (the current availability of human expert resources, including the number of experts, workload, and expertise matching). These factors are incorporated into a comprehensive consideration to balance automation and security. Finally, an abstention decision is made and the decision path is recorded. Decision outcomes are typically divided into two categories: accepting the model's answer, deeming the current model answer sufficiently reliable and usable directly; and abstaining and seeking human intervention, recognizing significant uncertainty or risk in the model's answer and requiring human expert intervention. For abstention cases, an auxiliary information package is generated to help human experts quickly understand the problem: this includes the original query, the model's preliminary answer, uncertainty markers, and relevant references. This decision and its basis are recorded as part of the decision path.This comprehensive evaluation-based abstention strategy strikes a balance between automation efficiency and answer quality assurance, ensuring timely intervention by human experts when the model's capabilities are insufficient or the task risk is too high. The output, which determines whether human intervention is needed, provides crucial guidance for the final response generation.

[0121] Step S1035: Based on the model hierarchical selection decision and the manual intervention decision, combined with the preliminary answer and the final answer, the decision path information and answer candidate set are obtained through the decision path tracking and answer quality evaluation mechanism.

[0122] In this sub-step, the aforementioned decision-making process is synthesized, the complete decision path is tracked, and the final set of candidate answers is generated. First, decision path tracking and recording are performed. The decision-making processes of the preceding sub-steps are integrated into a coherent decision path record, including: initial reasoning, the reasoning process and results of the base model, recording how the base model handles multimodal features and generates preliminary answers; deferred decision, whether to defer to the advanced model and the reasons, recording the criteria used to make the deferral decision (e.g., insufficient confidence, high task complexity); advanced reasoning (if applicable), the reasoning process and results of the advanced model, recording how the advanced model improves or supplements the answers of the base model; and abstention decision, whether human intervention is needed and the reasons, recording the considerations used (e.g., high uncertainty, high-risk task) to decide to seek help from human experts. This detailed decision path record not only supports the current answer generation but also contributes to interpretability and subsequent improvements. Next, the sources of valid answers are determined based on the decision path. Based on the aforementioned decision, three scenarios are possible: 1) Using the basic model answer, the task is not deferred, and the basic model answer is accepted; in this case, the preliminary answer from step S1031 is directly adopted. 2) Using the advanced model answer, the task has been deferred, and the advanced model answer is accepted; in this case, the advanced answer from step S1033 is adopted. 3) Marked as requiring manual intervention, the task has been abandoned, and the task awaits human expert processing; in this case, a model answer may be provided as a reference, but it is explicitly marked as requiring manual confirmation. Clearly identify the currently valid answer sources as the basis for subsequent processing. Then, perform answer quality assessment. For model-generated answers (whether from the basic or advanced model), conduct a comprehensive quality assessment: completeness (whether the answer covers all aspects of the query and whether any key information is missing); coherence (whether the logical structure of the answer is clear and consistent, and whether the parts are organically connected); accuracy (whether the answer is consistent with the retrieved knowledge and whether there are any factual errors); relevance (whether the answer directly responds to the original query and whether it contains irrelevant content); multimodal integration (whether the answer effectively utilizes information from different modalities and whether there are any inconsistencies between modalities). Generate a comprehensive quality score based on these dimensions and mark areas for improvement. Next, optimize the answer and generate candidate variants. Based on the quality assessment results, several candidate variants of the answer may be generated: the original answer, directly generated by the model, retaining the complete reasoning process and details; supplementary answers, supplementing the original answer by adding missing information or explanations; simplified answers, a concise version of the core question, removing redundant content and highlighting key points; and explanatory answers, a version with added explanations of the reasoning process, making the answer more transparent and easier to understand. These candidate variants provide a rich selection space for the final response generation, allowing the most suitable version to be selected based on the specific scenario and user needs. Finally, the decision path and candidate answers are integrated to generate the final output.This output includes: a decision path summary, a concise record of key decision points and their rationale, providing a transparent view of the reasoning process; a candidate answer set, an evaluated and optimized set of answer variants, offering choices for different use cases; confidence analysis, confidence assessments of different answer components, clearly identifying high and low confidence areas; and human intervention markers (if applicable), clearly indicating the parts requiring human intervention and the reasons, facilitating efficient intervention by human experts. This comprehensive output includes both the final answer content and the process information on how to reach that answer. Through this comprehensive decision path and answer quality management, a high-quality preliminary answer candidate set can be generated while maintaining the transparency and traceability of the processing. The output model decision path and preliminary answer candidate set provide crucial input for subsequent Hamiltonian graph network training.

[0123] In step S104, the decision path information and the answer candidate set are represented by a Hamiltonian graph network to represent the multimodal information relationship and a fast training method without gradient descent is used to obtain optimized model parameters.

[0124] Specifically, such as Figure 3 As shown, step S104 includes the following sub-steps:

[0125] Step S1041: Based on the decision path information and the answer candidate set, an initial Hamiltonian graph structure representing the multimodal information relationship is obtained through multimodal adaptation entity-relation extraction and graph construction algorithms.

[0126] In this sub-step, multimodal information and model decision paths are transformed into Hamiltonian graph structures, laying the foundation for subsequent graph network training. First, entity and concept extraction is performed. The model decision paths and preliminary answer candidate sets output in step S1035 are analyzed to identify key entities and concepts. These entities may come from different modalities: text entities (such as named entities like people, organizations, locations, and times), image entities (such as identified objects, scenes, and visual features), and audio entities (such as speakers, sound events, and speech content). Simultaneously, abstract concepts such as topics, opinions, and emotions are also extracted. This extraction process is performed using specialized entity recognition techniques and concept mining algorithms to ensure the capture of key elements from the multimodal information. Next, the semantic relationships between entities are analyzed. Based on the extracted entities and concepts, various relationship types are identified: hierarchical relationships (such as classification relationships like "is a type of," "contains," etc.), attribute relationships (characteristics or states of entities), temporal relationships (the temporal order of events or states), spatial relationships (physical location relationships of entities), causal relationships (causal connections between events or states), and functional relationships (the uses or functions of entities). By combining semantic analysis, contextual understanding, and domain knowledge, these complex relationships are comprehensively captured. Then, an initial graph structure is constructed, using extracted entities and concepts as nodes and identified relationships as edges, creating an initial semantic relationship graph. In this graph, nodes have multiple attributes (such as modality type, confidence level, importance, etc.), and edges also have multiple attributes (such as relationship type, strength, origin, etc.). The initial graph may be heterogeneous, reflecting the diversity of different types of nodes and edges. Next, the graph is transformed into a Hamiltonian graph structure. A Hamiltonian graph is a special graph structure in which there exists a path (Hamiltonian path) or cycle (Hamiltonian cycle) that passes through every vertex exactly once. A specific graph transformation algorithm maps the initial semantic relationship graph to the Hamiltonian graph structure. This transformation may involve operations such as node reordering, adding virtual nodes, adjusting edge weights, and path optimization. The Hamiltonian graph structure is particularly suitable for representing the sequential and holistic nature of information and has advantages for the coherent representation of multimodal information. Finally, the graph structure is verified and optimized. The constructed Hamiltonian graph is checked to ensure it satisfies the necessary mathematical properties, and its representation of the original multimodal information is evaluated. Graph structure optimization may be performed, including redundant node merging, critical path strengthening, noise edge filtering, and structural balancing, to ensure the quality and representational capability of the graph structure.

[0127] Step S1042: For the initial Hamiltonian graph structure, obtain the Hamiltonian graph network model to be trained through Hamiltonian graph network architecture design and parameter initialization strategy.

[0128] In this sub-step, based on the Hamiltonian graph structure from the previous step, a specialized Hamiltonian graph network model is designed and constructed. First, the topology of the Hamiltonian graph network is defined. A Hamiltonian graph network is a special type of graph neural network whose network structure directly maps to the Hamiltonian graph representation of the data. Several key factors are considered when designing the network topology: node correspondence, creating a corresponding neural network node for each node (entity or concept) in the graph; edge connections, creating connections between neural network nodes based on the edges in the Hamiltonian graph; loop structure, creating closed-loop connections in the case of Hamiltonian loops; and hierarchical organization, organizing nodes into multiple processing levels to support hierarchical information processing. This topology design ensures that the network structure can fully express the characteristics of the Hamiltonian graph. Next, specialized neuron models are designed for each type of node. Due to the heterogeneity of multimodal information, different types of nodes require different processing mechanisms: text nodes use neuronal structures suitable for processing language features, such as Transformer units; image nodes use neuronal structures suitable for processing visual features, such as convolutional units; audio nodes use neuronal structures suitable for processing temporal signals, such as recurrent units; and fusion nodes use neuronal structures capable of integrating multimodal information, such as attention mechanism units. Each neuron model has specific activation functions, connection weights, and learning parameters. Next, the information transfer mechanism between nodes is designed. In Hamiltonian graph networks, information is transferred along Hamiltonian paths or loops. This transfer mechanism requires special design: a message function, defining how to convert information from source nodes into messages; an aggregation function, defining how to merge multiple input messages; an update function, defining how to update the node state based on the aggregated messages; and path constraints, ensuring that information transfer follows the sequentiality of the Hamiltonian path. This design enables the network to effectively capture the sequential dependencies and global relationships of multimodal information. Next, parameter initialization is performed. The network parameters include node weights, edge weights, and parameters of various functions. Multiple initialization strategies are employed: pre-trained model transfer, extracting parameters from relevant pre-trained models as initial values; heuristic initialization, setting initial parameters based on node type and topological location; random initialization, using appropriate random distributions to initialize other parameters; and structure-aware initialization, considering the special structure of the Hamiltonian graph when setting parameters. This initialization strategy provides a good starting point for subsequent gradient-free optimization. Finally, the network's operating mode and interface are configured, defining how the network receives input, processes information, and generates output, enabling the network to operate as a complete processing unit.

[0129] Step S1043: For the Hamiltonian graph network model to be trained, obtain the model state sequence during the training process through energy function definition, direct state update and gradient-free optimization algorithm.

[0130] In this sub-step, an innovative gradient-free descent optimization algorithm is implemented to efficiently train Hamiltonian graph network models. First, an energy function is defined. The energy function is the core of gradient-free optimization, quantifying the "goodness" of the model's current state. For Hamiltonian graph networks, the designed energy function considers multiple aspects: representation accuracy (the degree of matching between the model output and the target answer); structural consistency (the degree of consistency between the network state and the Hamiltonian graph structure); information fluency (the smoothness of information transmission along the Hamiltonian path); and multimodal coordination (the degree of integration and coordination of information from different modalities). The energy function is usually a scalar value; the lower the value, the better the model state. A weighted combination approach can be used to integrate these aspects to form a comprehensive energy function. Next, direct state update rules are designed. Unlike traditional gradient descent, gradient-free optimization directly modifies the model state without calculating gradients. Multiple state update strategies are implemented: simulated annealing, which accepts state changes with increased energy with probability to avoid getting trapped in local optima; quantum annealing, which explores the state space using the quantum tunneling effect; discrete jumps, which make large jumps in the state space to increase the exploration range; and heuristic search, which adjusts the state based on the characteristics of the problem. These strategies enable efficient exploration of the model's state space without the need to compute complex gradients. Parallel state exploration is then implemented. To further improve optimization efficiency, a parallel computing architecture is employed to explore multiple possible states simultaneously: multi-starting point exploration, starting optimization from different initial states simultaneously; branch exploration, deriving multiple mutation branches from the current state; island model, optimizing on relatively independent "islands" and periodically exchanging information; and ensemble search, maintaining a set of states and dynamically adjusting it based on the energy function. This parallel exploration significantly increases the coverage of the state space, improving the likelihood of finding the global optimum. Next, adaptive update scheduling is implemented. As the optimization process progresses, the state update strategy and parameters are dynamically adjusted: cooling scheduling, gradually reducing the probability of accepting suboptimal solutions; step size adjustment, adjusting the magnitude of state updates based on energy changes; search direction adjustment, adjusting the exploration direction based on historical search results; and dynamic setting of termination conditions, adjusting the termination criteria based on optimization progress. This adaptive scheduling allows for extensive exploration in the early stages and fine-grained convergence in the later stages. Finally, the state sequence during the optimization process is recorded and analyzed, tracking not only the final state but also the intermediate states throughout the optimization process, providing a basis for subsequent model convergence analysis and parameter selection.

[0131] The energy function in this invention is specifically defined as a weighted combination of multiple energy terms. These energy terms include an accuracy energy term, which measures the degree of matching between the model output and the target answer; a structural consistency energy term, which measures the consistency between the network state and the Hamiltonian graph structure; and a semantic coherence energy term, which measures the degree of coordination in the integration of multimodal information. These energy terms are combined using different weight coefficients, with the weight values ​​adjusted according to the characteristics of the specific task. Typically, the weight representing accuracy is higher, followed by structural consistency, and then semantic coherence. This multi-objective energy function design ensures that the optimization process simultaneously considers model performance, structural constraints, and semantic quality.

[0132] For gradient-free optimization algorithms, an improved simulated annealing method is employed. This method first initializes the current state and energy value, setting parameters such as initial temperature, termination temperature, and cooling rate. At each temperature, multiple iterations are performed: each iteration generates candidate states by randomly modifying a small subset of parameters of the current state, calculates the energy difference, and then decides whether to accept the new state based on the energy difference and the current temperature. When the energy difference is negative (i.e., the new state is better), the new state is always accepted; when the energy difference is positive, the new state is accepted with a certain probability, which decreases as the temperature decreases and as the energy difference increases. This mechanism allows the algorithm to explore a wider state space early on, while focusing more on local optimization later. After each iteration, the temperature is lowered, gradually reducing the probability of accepting suboptimal solutions.

[0133] To further improve the efficiency of searching for the global optimum, multiple independent optimization instances are run in parallel, exchanging information periodically during the optimization process. Specifically, every certain number of iterations, each instance shares the best state it has found and may adopt better results from other instances. This parallel strategy significantly increases the coverage of the state space and improves the probability of finding the global optimum.

[0134] For convergence determination, the primary criterion is that the energy change over several consecutive iterations is less than a preset threshold. If the energy change is very small over several iterations, it indicates that the optimization process has stabilized and convergence can be considered achieved. Simultaneously, a maximum number of iterations is set as a fallback termination condition to ensure the algorithm terminates within a finite time. This dual termination condition ensures optimization quality while avoiding excessive computational resource waste.

[0135] Step S1044: Based on the model state sequence during the training process, the optimized model parameters are obtained through the convergence criterion and the optimal state selection algorithm.

[0136] In this sub-step, based on the state sequence generated in the previous step, the model convergence is analyzed, and the optimal parameters are selected. First, convergence criteria are defined. A clear convergence criterion is crucial for determining when the optimization process is complete. Several convergence criteria are designed: energy stability (energy function value changes less than a preset threshold in multiple iterations); state stability (model parameters change less than a preset threshold in multiple iterations); performance stability (model performance metrics are stable on validation data); and computational budget (reaching a preset maximum number of iterations or time limit). These criteria can be combined to form a comprehensive convergence criterion mechanism that considers both optimization quality and computational efficiency. Next, the trend of energy function changes is analyzed. The energy trajectory recorded in step S1043 is analyzed in depth: global trend (direction and rate of overall energy change); local fluctuations (short-term energy fluctuations and oscillations); plateau regions (ranges where energy values ​​are relatively stable); and abrupt change points (points where energy values ​​change significantly). This analysis helps understand the dynamic characteristics of the optimization process and identify potential convergence problems (such as premature convergence or oscillations). Then, state comparison and selection are performed. Based on the model state sequence output in step S1043, a multi-dimensional comparison is performed: energy level (energy function value for each state); generalization ability (performance on validation data); stability (stability of state parameters); and complexity (complexity of model states or number of parameters). Not only is the state with the lowest energy selected, but these factors are also considered comprehensively to avoid overfitting or structurally unreasonable states. Next, fine-tuning of the parameters is performed. For the selected optimal state, local fine-tuning may be performed: sensitivity analysis to identify parameters with the greatest impact on performance; grid search for fine-grained searching within a small range of key parameters; hybrid optimization combining gradient-based and gradient-free methods for local optimization; and ensemble strategies to integrate or weighted average multiple excellent states. This fine-tuning further improves the model's performance and stability. Finally, the final optimized model parameters are generated, converting the selected optimal state into a standard model parameter format, including node weights, edge weights, function parameters, and configuration information. Metadata for the parameters is also generated, recording the optimization process, convergence characteristics, and performance metrics for subsequent use and evaluation.

[0137] In step S105, based on the optimized model parameters, multimodal enhanced retrieval results are obtained through a cross-modal semantic alignment algorithm and a dynamic retrieval window adjustment strategy based on task complexity.

[0138] Specifically, step S105 includes the following sub-steps:

[0139] Step S1051: For the optimized model parameters, a unified cross-modal semantic representation space is obtained through shared semantic space mapping and inter-modal semantic alignment algorithms.

[0140] In this sub-step, a unified cross-modal semantic space is constructed using the model parameters optimized in step S1044 to achieve semantic alignment of data from different modalities. First, a cross-modal embedding framework is designed. This framework forms the basic architecture for constructing the unified semantic space and needs to consider the characteristics of multiple modalities. A multi-branch structure is adopted: a text branch uses a Transformer-based encoder to process text data; an image branch uses a CNN- or Vision Transformer-based encoder to process image data; and an audio branch uses a time-frequency analysis and sequence model-based encoder to process audio data. Each branch is equipped with specific preprocessing modules and feature extraction layers to adapt to the characteristics of its respective modality. The optimized Hamiltonian graph network parameters are mapped to these branches to ensure that each branch can effectively capture the semantic features of its corresponding modality. Next, a shared semantic space mapping is implemented. After each modal branch extracts features independently, these features need to be mapped to the same semantic space. This mapping is achieved using multiple techniques: a projection layer, where a dedicated projection network is designed for each modality to map the modality-specific feature space to a shared dimension; normalization processing, which uniformly normalizes the feature representations of each modality, such as L2 normalization, to ensure consistent numerical ranges; dimensionality adjustment, which uses dimensionality reduction or expansion operations to ensure that the feature vectors of all modalities have the same dimension; and distribution alignment, which adjusts the feature distribution to ensure that the feature distributions of different modalities have similar statistical properties. These techniques work together to ensure that the semantic features of each modality can be meaningfully compared and manipulated in the same space. Then, an inter-modal semantic alignment algorithm is executed. After mapping different modalities to the same space, it is also necessary to ensure that semantically corresponding content is close to each other in this space. Multiple alignment strategies are implemented: contrastive learning, which uses known cross-modal correspondences (such as images and their descriptions) as positive examples and non-correspondences as negative examples, optimizing alignment through contrastive loss; semantic anchors, which use key concepts recognizable across all modalities as anchors to align the semantic space around these anchors; structure preservation, which ensures that the semantic relationship structure within each modality is preserved after mapping; and cycle consistency, which verifies and optimizes alignment consistency through inter-modal transformations and restorations. These alignment strategies ensure that semantically similar content is close in a unified space, regardless of its modality. Next, semantic space validation and adjustment are performed. The quality of the semantic space is evaluated and optimized through validation methods such as cross-modal retrieval tests, semantic clustering analysis, analogy tests, and zero-shot generalization ability. Finally, a unified cross-modal semantic representation space is output, containing encoders, projection mappings, semantic alignment parameters, and distance metrics for each modality.

[0141] Step S1052: Based on the current user query content and combined with the unified cross-modal semantic representation space, a task complexity score is obtained through a multi-dimensional task complexity evaluation index calculation and difficulty grading mechanism that includes concept complexity, relation complexity, and reasoning complexity.

[0142] In this sub-step, the complexity of the current query task is comprehensively evaluated based on the constructed cross-modal semantic space. First, query parsing and representation are performed. The current query content is mapped to the cross-modal semantic space constructed in step S1051. This process includes: query decomposition, breaking down the complex query into basic semantic units; multimodal processing, handling different modalities such as text, images, and audio in the query; semantic encoding, using appropriate modal encoders and projection mappings to generate a cross-modal semantic representation of the query; and context integration, considering the contextual information of the query, such as previous interaction history. This processing ensures a comprehensive understanding of the query intent and content, providing a foundation for complexity evaluation. Next, a multi-dimensional complexity evaluation metric is designed. Task complexity is not a single-dimensional concept; it is assessed from multiple perspectives: conceptual complexity, which refers to the level of abstraction and specialization of the concepts involved in the query, such as basic concepts, technical terms, or abstract theories; relational complexity, which refers to the complexity and depth of relationships between concepts, such as simple associations, hierarchical structures, or complex networks; reasoning complexity, which refers to the number and difficulty of reasoning steps required to solve the problem, such as direct lookup, single-step reasoning, or multi-step reasoning; knowledge breadth, which refers to the scope of knowledge domains involved, such as a single domain or multiple domains; multimodal dependency, which refers to the degree of dependence on the integration of multiple modalities of information, such as whether a single modality is sufficient or a combination of multiple modalities is necessary; and timeliness requirements, which refers to the degree of requirement for the freshness of information, such as general knowledge or the latest developments. Specific calculation methods are designed for each dimension, such as conceptual complexity calculation based on semantic networks and reasoning complexity estimation based on reasoning graphs. Then, knowledge graph-assisted analysis is performed, leveraging domain knowledge graphs to enhance the accuracy of complexity assessment. Objective measures of task complexity are provided through methods such as query mapping, path analysis, subgraph complexity, and uncertainty analysis. Next, machine learning models are implemented for prediction. By learning the features and actual complexity of historical tasks, predictive capabilities are developed. Combining feature engineering, model training, ensemble prediction, and confidence assessment, complex patterns that are difficult to express through rules are captured. Finally, various evaluation methods are integrated to generate a final task complexity score. Through weighted fusion, nonlinear mapping, hierarchical labeling, and uncertainty representation, clear guidance is provided for subsequent retrieval window adjustments.

[0143] Step S1053: Score the complexity of the task, and obtain the optimized retrieval parameter configuration through adaptive window size calculation and retrieval depth adjustment algorithm.

[0144] In this sub-step, based on the task complexity score, retrieval parameters are intelligently adjusted to optimize resource allocation and retrieval performance. First, a complexity-resource mapping relationship is established. This requires understanding the relationship between task complexity and optimal retrieval resource allocation. This mapping considers several factors: historical performance data, analyzing the historical performance of tasks with different complexities under different resource configurations; resource efficiency curves, assessing the marginal benefit of increased resource investment in improving retrieval quality; task type characteristics, the sensitivity of different task types (such as fact queries, reasoning analysis, and creative generation) to resources; and time-quality tradeoffs, the balance point between response time and retrieval quality. By analyzing these factors, a mapping model from complexity score to optimal resource allocation is established. Next, the adaptive window size is calculated. The retrieval window size refers to the scope of context considered during the retrieval process or the number of retrieval results. Based on the complexity score in step S1052, the optimal window size is dynamically calculated: A basic window size is set according to the complexity level; the higher the complexity, the larger the window. Task characteristic adjustment is performed based on the specific characteristics of the task (such as multimodal dependency, knowledge breadth, etc.). Resource constraints are considered, taking into account available computing resources and time limitations, appropriately reducing the window size when resources are scarce. An adaptive formula is used to calculate the final window size, taking into account all the above factors. This adaptive calculation ensures that the window size is suitable for task requirements and feasible within resource constraints. Then, the retrieval depth and breadth are adjusted. In addition to the window size, the retrieval depth (the fineness of the search) and breadth (the scope of the search domain) are optimized: Depth parameters, such as similarity threshold, ranking cutoff point, semantic matching accuracy, etc.; Breadth parameters, such as domain expansion range, cross-modal search ratio, knowledge graph traversal depth, etc.; a balancing strategy is employed, balancing the resource allocation of depth and breadth according to task complexity; and incremental adjustment allows for dynamic adjustment of subsequent retrieval strategies based on initial retrieval results. This multi-dimensional adjustment enables the retrieval process to more accurately adapt to task characteristics. Next, configure modality-specific search parameters. Different modalities may require different parameter configurations. Customize search parameters for each modality, such as text search parameters, image search parameters, audio search parameters, and cross-modal parameters, to ensure that all types of data are processed appropriately. Finally, generate a complete search parameter configuration, including window configuration, similarity metric, ranking strategy, and resource allocation, while also generating metadata for the parameter configuration to facilitate monitoring and optimization of the search process.

[0145] Step S1054: Based on the optimized retrieval parameter configuration and the unified cross-modal semantic representation space, the multimodal enhanced retrieval results are obtained through multi-dimensional retrieval strategies and result fusion mechanisms.

[0146] In this sub-step, enhanced retrieval operations are performed based on optimized retrieval parameter configuration and cross-modal semantic space to generate high-quality multimodal retrieval results. First, multi-strategy parallel retrieval is implemented. Instead of relying on a single retrieval method, multiple retrieval strategies are executed in parallel: semantic retrieval, calculating the semantic similarity between the query and candidate options based on the cross-modal semantic space constructed in step S1051; keyword retrieval, based on traditional inverted indexes and term frequency-inverse document frequency (TF-IDF) methods; structured retrieval, utilizing structured queries from knowledge graphs; and example retrieval, based on historical cases similar to the query. This multi-strategy parallel retrieval improves recall, ensuring that different types of relevant content can be retrieved. Appropriate resources and weights are allocated to each strategy according to the parameter configuration in step S1053. Next, cross-modal joint retrieval is performed. This system not only performs searches within each modality but also enables cross-modal joint retrieval: modal bridging, which uses a shared semantic space to retrieve content from other modalities using queries from one modality; complementary enhancement, which automatically expands to other modalities to find supplementary information when one modality is insufficient; cross-validation, which uses multimodal information to mutually verify and enhance the reliability of search results; and modality transformation, which, when necessary, transforms search content between different modalities, such as text-to-image generation or speech-to-text transcription. This cross-modal joint retrieval fully utilizes the advantages of a unified semantic space, breaking down barriers between modalities. Then, a hierarchical search depth is applied. Based on the search depth settings in step S1053, a hierarchical search strategy is implemented: broad initial screening, which first uses lightweight methods to quickly retrieve a large number of candidate options; moderate filtering, which applies more refined filtering conditions to the initial screening results to reduce the candidate set; deep analysis, which applies computationally intensive deep semantic analysis to the reduced candidate set; and precise matching, which performs precise relevance evaluation on the final candidates. This hierarchical strategy effectively controls the consumption of computational resources while ensuring search quality. Next, real-time search adjustments are performed. The retrieval process is not static; subsequent searches are dynamically adjusted based on the initial search results. Through techniques such as query rewriting, focus shifting, deep weighting, and feedback learning, the retrieval process adaptively responds to the initial findings, improving the relevance of the final results. Finally, multi-source search results are integrated, and through processes such as deduplication, consistency checks, structured organization, and association enhancement, high-quality multimodal enhanced search results are generated.

[0147] Step S1051 may further include:

[0148] The optimized model parameters are designed with a multi-branch structure, and the initial feature representations of each modality are obtained through specific encoder processing of text branch, image branch and audio branch.

[0149] In this sub-step, a multi-branch modality-specific encoding architecture was designed based on the optimized Hamiltonian graph network parameters. For the text branch, a Transformer-based encoder was employed. These encoders effectively capture long-range dependencies and contextual information in the text through self-attention mechanisms. A pre-trained language model (such as BERT, RoBERTa, or T5) might be used as a base, with its internal weights adjusted according to the optimized parameters of the Hamiltonian graph network to better suit the semantic understanding requirements of the current task. The text first undergoes preprocessing such as word segmentation and positional encoding, and then a high-dimensional semantic representation of the text is generated through a multi-layer Transformer structure. For the image branch, an encoder structure based on convolutional neural networks (CNN) or Vision Transformers was implemented. These encoders effectively extract spatial features, texture information, and visual semantics from images. A pre-trained model such as ResNet, EfficientNet, or ViT might be used as a base, with its internal structure adjusted according to the optimized parameters. After preprocessing such as normalization and enhancement, the image's visual feature representation is generated through a multi-layer feature extraction network. For the audio branch, a dedicated encoder combining time-frequency analysis and sequence modeling was designed. These encoders first convert audio into a time-frequency representation using methods such as Short-Time Fourier Transform (STFT) or Mel-spectrum mapping, then extract audio features using structures such as CNN, RNN, or Conformer. They may use pre-trained models such as Wav2Vec, HuBERT, or Whisper as a base, and adjust them according to optimized parameters. After preprocessing such as noise reduction and segmentation, the audio is processed through a multi-layer feature extraction network to generate an acoustic feature representation. This multi-branch design ensures that each modality is processed by an encoder best suited to its characteristics, thus preserving the unique information and semantic structure of each modality.

[0150] The initial feature representations of each modality are mapped by projection layer, normalized, and adjusted to obtain feature vectors with shared dimensions.

[0151] In this sub-step, the initial feature representations of each modality generated in the previous step are unified, enabling them to be compared and manipulated within the same semantic space. First, modality-specific projection layer mappings are implemented. Since the initial features of different modalities may have different dimensions and distribution characteristics, a dedicated projection network is designed for each modality. These networks typically consist of one or more fully connected layers and may also include nonlinear activation functions (such as ReLU, GELU, etc.) and attention mechanisms. The parameters of the projection layers are initialized and fine-tuned based on the optimization results of the Hamiltonian graph network, ensuring that the projection process preserves modality-specific semantic information while converging towards a shared space. Next, feature normalization is performed. Normalization is a crucial step in ensuring numerical compatibility between features of different modalities. Several normalization techniques may be employed: L2 normalization, which scales feature vectors to unit length, ensuring all modal feature vectors have the same norm; batch normalization, which adjusts the mean and variance of features to reduce internal covariate bias; layer normalization, which normalizes across feature dimensions to reduce differences between samples; and instance normalization, which normalizes for each sample individually, suitable for cases with significant differences between samples. These normalization techniques ensure that features from different modalities are consistent in numerical scale, facilitating subsequent cross-modal operations. Then, a unified adjustment of feature dimensions is performed. Features from different modalities may have different original dimensions (e.g., text may be 768-dimensional, images may be 2048-dimensional, and audio may be 512-dimensional), requiring adjustment to the same dimension for cross-modal operations. Dimensionality reduction techniques (such as PCA and t-SNE) may be used to reduce the dimensionality of high-dimensional features, or dimensionality enhancement techniques (such as padding and linear projection) may be used to increase the dimensionality of low-dimensional features. Adaptive dimensionality adjustment may also be used, dynamically allocating the dimensionality ratio of different modalities based on information content. Finally, feature distribution alignment may also be performed. Distribution matching techniques (such as minimizing the maximum mean difference and minimizing the Wasserstein distance) can be used to make the feature distributions of different modalities more similar, reducing statistical differences between modalities. Through these processes, the initial features of different modalities are transformed into shared-dimensional feature vectors with the same dimension, similar numerical range, and compatible distribution characteristics, laying the foundation for subsequent semantic alignment.

[0152] Based on the shared dimension feature vector, an alignment strategy including contrastive learning, semantic anchors, and structure preservation is constructed using positive and negative samples to obtain a semantically aligned feature representation.

[0153] In this sub-step, deep cross-modal semantic alignment is achieved, bringing content expressing the same or similar semantics in different modalities closer together in the feature space. First, positive and negative sample pairs are constructed for contrastive learning. Positive sample pairs are sample pairs from different modalities that express the same semantic content, such as images and their text descriptions, audio and its transcribed text, etc. These positive sample pairs may come from pre-labeled datasets, user interaction history, or automatically generated pairings. Negative sample pairs are unrelated cross-modal sample pairs, typically generated through random pairing or hard example mining (identifying easily confused samples). Contrastive learning is performed based on these sample pairs, using contrastive loss functions such as InfoNCE, NT-Xent, or triplet loss to maximize the similarity of positive sample pairs in the feature space while minimizing the similarity of negative sample pairs. Next, a semantic anchor strategy is implemented. Semantic anchors are concepts or entities that have clear corresponding counterparts in all modalities, such as common objects, basic actions, and universal emotions. These cross-modal shared semantic anchors are identified and extracted, serving as benchmarks for alignment. Specifically, a set of core concepts might be predefined as anchors, and then each modality encoder is trained to map these concepts to the same or similar locations in the feature space. This anchor alignment provides a stable reference framework for more complex semantic mappings. Then, a structure-preserving strategy is applied to ensure that the semantic relationship structure within a modality is preserved after cross-modal mapping. For example, if the relationship between "cat" and "dog" in text space is analogous to the relationship between "car" and "truck," then this analogy should also be preserved in the feature space of an image or audio file. Structure-preserving loss functions, such as relation consistency loss or graph structure preservation loss, might be used to ensure that the semantic topology within a modality is not distorted during cross-modal mapping. Furthermore, a cycle consistency check might be implemented, verifying the degree of semantic information preservation during cross-modal transformations through the transformation process from modality A to modality B and back to modality A. An adversarial training strategy might also be employed, using a discriminator network to distinguish features from different modalities, prompting the feature encoder to generate modality-independent semantic representations. By comprehensively applying these alignment strategies, the feature vectors of shared dimensions are further optimized into semantically aligned feature representations, so that content expressing the same semantics in different modalities is clustered in the feature space, while content with different semantics remains separated.

[0154] Cross-modal retrieval tests, semantic clustering analysis, and analogy verification are performed on the semantically aligned feature representations to obtain the unified cross-modal semantic representation space.

[0155] In this detailed step, the quality of semantic alignment is evaluated and optimized through various validation methods, ultimately determining a unified cross-modal semantic representation space. First, cross-modal retrieval testing is conducted, a direct method to verify the effectiveness of semantic alignment. A test set is constructed, containing queries and target terms from different modalities, such as using text to query related images, or using images to query related audio. Feature similarity between query terms and candidate terms is calculated, and retrieval performance metrics such as mean precision (mAP), recall @K, and normalized discount cumulative gain (NDCG) are evaluated. Good semantic alignment should result in cross-modal retrieval performance approaching or exceeding that of single-modal retrieval. Alignment parameters may be fine-tuned based on test results to improve retrieval accuracy. Next, semantic clustering analysis is performed. Samples from different modalities but expressing similar semantics are projected into the feature space, and clustering algorithms (such as K-means, DBSCAN, spectral clustering, etc.) are applied to analyze the distribution patterns of features. In ideal semantic alignment, content with the same semantics should form tight clusters, regardless of their modality; while content with different semantics should form clearly separated clusters. The clustering quality metrics, such as silhouette coefficient, adjusted Rand index, and mutual information, are evaluated to verify the semantic organization of the feature space. Then, analogy verification is performed. Analogies refer to semantic relationships like "A is to B as C is to D," which are important characteristics of the semantic space structure. A cross-modal analogy test set is constructed to verify whether these relationships are maintained in the feature space. For example, it checks whether the vector operation "King - Man + Woman ≈ Queen" in text also holds true in the image feature space. Analogy accuracy, vector direction consistency, and other metrics are calculated to evaluate the structural consistency of the feature space. Furthermore, zero-shot generalization testing may be performed to evaluate the model's ability to handle new concepts or modal combinations not seen during training. The geometric properties of the feature space, such as dimensionality validity, local linearity, and manifold structure, may also be analyzed to ensure that the feature space possesses good mathematical properties. Based on these verification results, final fine-tuning and optimization may be performed, such as adjusting alignment parameters, rebalancing modal weights, and optimizing distance metrics. Finally, a unified cross-modal semantic representation space is determined and output, which includes encoders, projection maps, semantic alignment parameters and distance metrics for each modality, providing a foundation for subsequent cross-modal tasks.

[0156] Steps S1052 and S1053 may further include:

[0157] The query content of the current user query is parsed and represented. Through query decomposition, multimodal processing and semantic encoding, a cross-modal semantic representation of the query is obtained.

[0158] In this detailed step, the user-submitted query content is deeply analyzed and transformed into representation vectors in a cross-modal semantic space. First, query decomposition is performed, breaking down complex queries into basic semantic units. For text queries, natural language processing techniques, such as dependency parsing and semantic role labeling, may be used to identify components such as subject entities, relational descriptions, attribute constraints, and operational instructions. For example, the query "Show sunset photos taken in front of XXX Tower last summer" might be decomposed into semantic units such as time ("last summer"), location ("in front of XXX Tower"), object ("sunset photos"), and operation ("show"). For multimodal queries, the relationships between different modalities also need to be analyzed, such as referential, complementary, or constraining relationships between text descriptions and example images. This decomposition allows for a more precise understanding of various aspects of the query, providing a foundation for subsequent complexity assessment. Next, multimodal processing is performed, applying appropriate processing techniques to different modalities in the query. For the text portion, a text branch encoder is used; for the image portion, an image branch encoder is used to extract visual features; and for the audio portion, an audio branch encoder is used to analyze acoustic content. It is also necessary to handle intermodal interactions, such as text referencing images ("Who is the person in this picture?") or images supplementing text (text description plus example image). Multimodal attention mechanisms or cross-modal fusion networks may be used to handle these interactions, ensuring effective integration of information from different modalities. Then, semantic encoding is performed, mapping the processed query content to the unified cross-modal semantic space constructed in step S1051. Using encoders and projection mappings for each modality, different parts of the query are converted into semantic vectors, and then semantic alignment mechanisms ensure that these vectors have a consistent representation in the same semantic space. For key concepts and entities in the query, they may also be linked to knowledge graphs or concept bases to enrich their semantic representation. Finally, these semantic vectors are integrated to generate a comprehensive cross-modal semantic representation of the query. This representation not only includes the content information of the query but also its structural information and contextual relationships, providing a comprehensive semantic foundation for subsequent complexity evaluation.

[0159] Based on the cross-modal semantic representation of the query, a multi-dimensional complexity index is obtained through a comprehensive evaluation including concept complexity, relational complexity, reasoning complexity, and multimodal dependency.

[0160] In this detailed step, the semantic representation of the query is evaluated for complexity across multiple dimensions, quantifying the task's difficulty and resource requirements. First, the conceptual complexity of the query is assessed, reflecting the level of abstraction and specialization of the concepts involved. Several methods may be used to quantify conceptual complexity: concept hierarchy analysis, based on knowledge graphs or ontology bases, calculates the hierarchical depth of concepts in the query; deeper concepts are generally more abstract or specialized. Word frequency statistics analyze the proportion of rare words or technical terms in the query, as these typically represent more complex concepts. Concept network analysis constructs a network of associations between query concepts, calculating the network's density and connectivity; complex concepts usually have richer associations. Combining these analyses generates a conceptual complexity score, reflecting the difficulty of understanding the query's concepts. Next, relational complexity is assessed, reflecting the complexity and depth of the relationships between concepts in the query. The complexity may be assessed for several relational characteristics: relational type diversity (the more relational types a query involves, such as temporal, spatial, causal, etc.), relational nesting depth (the degree of nesting or recursion in relations, e.g., "A's B's C" is more complex than "A's B"); and relational constraint strength (the number and strictness of constraints attached to relations). These relational characteristics are calculated by analyzing the semantic structure of the query to generate a relational complexity score. Then, inference complexity is evaluated, reflecting the number and difficulty of the inference steps required to answer the query. This might involve using inference graph analysis to decompose the query into basic inference steps and calculate the number of steps and dependency depth; or using historical data analysis to predict the inference requirements of the current query based on historical inference paths of similar queries. Inference uncertainty is also assessed, such as whether fuzzy logic, probabilistic reasoning, or counterfactual reasoning needs to be handled, which typically increases inference complexity. Furthermore, multimodal dependency is evaluated, reflecting the degree to which the query relies on the integration of information from multiple modalities. The analysis determines whether the query requires cross-modal understanding (e.g., text describing image content), cross-modal comparison (e.g., comparing sentiment expressions in audio and video), or cross-modal reasoning (e.g., inferring a scenario based on both images and text). Higher multimodal dependence generally leads to greater query processing complexity. Other dimensions may also be evaluated, such as knowledge breadth (the range of knowledge domains involved) and timeliness requirements (the requirement for information to be recent). Through these multi-dimensional assessments, a set of complexity metrics is generated to comprehensively characterize the processing difficulty of the query.

[0161] The task complexity score is obtained by weighted fusion and nonlinear mapping processing of the multi-dimensional complexity index.

[0162] In this detailed step, the multi-dimensional complexity metrics generated in the previous step are integrated into a single comprehensive complexity score, providing clear guidance for resource allocation. First, weighted fusion is implemented, assigning appropriate weights to the different dimensions of complexity metrics. These weights reflect the relative importance of each dimension to the overall task complexity and may be determined based on various factors: historical performance analysis, identifying which complexity dimensions are most correlated with processing difficulty and resource requirements by analyzing historical query data; expert knowledge, setting the relative importance of each dimension based on the experience and judgment of domain experts; and task characteristics, dynamically adjusting the weight allocation according to current processing capabilities and task type. For example, for tasks that emphasize reasoning ability, reasoning complexity may receive a higher weight; while for tasks focusing on multimodal fusion, multimodal dependency may be more important. Linear weighted models, analytic hierarchy process (AHP), or machine learning models may be used to determine the optimal weight configuration. Next, non-linear mapping is applied to transform the weighted fused complexity values ​​into a more meaningful score. Linear weighting may not fully capture the interaction effects between complexity dimensions; for example, a combination of high conceptual complexity and high relational complexity may produce a complexity increase exceeding simple addition. Several nonlinear mapping methods may be used: sigmoid or tanh functions map complexity values ​​to a fixed range, such as 0 to 1 or -1 to 1, facilitating standardized comparisons; piecewise functions apply different mapping rules to different complexity intervals, reflecting the phased characteristics of complexity growth; exponential or logarithmic functions capture the nonlinear characteristics of complexity growth, such as diminishing marginal returns. Machine learning models, such as random forests and neural networks, may also be used to learn the nonlinear relationship between complexity metrics and actual processing difficulty. Then, a grading system is implemented, mapping continuous complexity scores to discrete complexity levels, such as "easy," "medium," "complex," and "very complex." This grading makes complexity assessment easier to understand and apply, facilitating user decision-making. Finally, uncertainty representations, such as confidence intervals or probability distributions, may be provided to reflect the reliability and variability of the complexity assessment. Through these processing steps, multi-dimensional complexity metrics are integrated into a comprehensive task complexity score, providing both quantitative values ​​and qualitative levels, offering a scientific basis for subsequent resource allocation.

[0163] Based on the task complexity score, a complexity-resource mapping relationship is established, and the retrieval window size is obtained through basic window settings, task characteristic adjustments, and adaptive formula calculations.

[0164] In this detailed step, the task complexity score is converted into a specific search window size to achieve intelligent resource allocation. First, a complexity-resource mapping relationship is established, which is the key bridge connecting task complexity and optimal resource allocation. This mapping relationship can be constructed in several ways: historical data analysis, collecting and analyzing performance data of tasks with different complexities under different resource configurations to identify the best matching pattern; performance model construction, based on architecture and algorithm characteristics, establishing a theoretical model to predict the relationship between resource requirements and task complexity; experimental optimization, determining the optimal resource allocation for different complexity levels through controlled experiments. Special attention is paid to the resource efficiency curve to assess the marginal benefit of increased resource input on improving search quality, avoiding over-allocation of resources to areas of diminishing returns. This mapping relationship may be represented by a function model, lookup table, or decision tree, guiding the determination of appropriate resource allocation based on the complexity score. Next, a basic window is set, determining the basic size of the search window based on the task complexity level. The search window size directly affects the scope and depth of the search; more complex tasks typically require a larger window to obtain more comprehensive information. A baseline window size might be preset for each complexity level, such as a small window (e.g., 10-20 results) for simple tasks and a large window (e.g., 50-100 results or more) for complex tasks. This baseline setting provides an initial estimate of the window size, laying the foundation for subsequent fine-tuning. Then, the window is adjusted based on the specific characteristics of the task. Besides complexity level, other task characteristics also influence the optimal window size: multimodal dependence—tasks highly dependent on multimodal information may require a larger window to ensure sufficient cross-modal content is captured; knowledge breadth—tasks involving multiple domains may require a larger window to cover a wider range of knowledge sources; timeliness requirements—tasks with high demands for up-to-date information may require a special window configuration that prioritizes recent content. The baseline window size is adjusted based on these characteristics to better suit the specific task requirements. Finally, an adaptive formula is applied to calculate the final retrieval window size. This formula comprehensively considers the complexity score, the baseline window size, task characteristic adjustments, and the current resource status (e.g., computational load, memory availability). The formula may include non-linear terms reflecting the non-linear relationship between window size and complexity, or include constraints to ensure the window size remains within acceptable limits. It may also implement a dynamic adjustment mechanism, allowing for further adjustments to the window size based on the initial search results, to achieve more refined resource allocation.

[0165] The depth and breadth parameters of the search window size are adjusted, and the optimized search parameter configuration is obtained through modality-specific search parameter configuration.

[0166] In this detailed step, based on the determined search window size, the depth, breadth, and modality-specific parameters of the search are further optimized to form a complete search configuration scheme. First, the search depth parameters are adjusted. These parameters control the granularity and matching strictness of the search process. Depth parameters include: similarity threshold, which determines the minimum similarity requirement between the search term and the query; complex tasks may require lowering the threshold to obtain more potentially relevant content; ranking cutoff point, which determines how many top results are retained after ranking; complex tasks may require more results to support comprehensive analysis; and semantic matching precision, which controls the strictness of semantic matching, such as whether exact matching is required or approximate matching is allowed. Based on task complexity and window size, the precision and recall of the search are balanced, and appropriate depth parameters are set. Next, the search breadth parameters are adjusted. These parameters control the domain scope and diversity of the search. Breadth parameters include: domain expansion range, determining whether the retrieval extends to related domains; complex tasks may require broader knowledge coverage; cross-modal search ratio, determining the proportion of retrieval results from different modalities, such as the distribution of text, image, and audio results; knowledge graph traversal depth, controlling the extent to which related knowledge is explored within the knowledge graph; and diversity factor, balancing the relevance and diversity of results to avoid information redundancy. Based on task characteristics and complexity, appropriate breadth parameters are determined to ensure that retrieval results are both focused on relevant content and possess the necessary breadth and diversity. Then, modality-specific retrieval parameters are configured to optimize retrieval strategies for different types of data sources. For text retrieval, keyword weights, semantic similarity algorithms, and document length normalization parameters may be adjusted; for image retrieval, visual feature extraction levels, spatial sensitivity, and balanced weights for color and shape may be configured; for audio retrieval, temporal resolution, frequency range, and speaker recognition sensitivity may be set. Cross-modal retrieval parameters are also configured to control the weights and integration strategies between different modalities, such as the weight allocation for text-guided image retrieval. This modality-specific parameter configuration ensures that various types of data receive processing best suited to their characteristics. Finally, all parameter settings are integrated to generate a complete search parameter configuration. This configuration includes window size, depth parameters, breadth parameters, and modality-specific parameters, forming a comprehensive search strategy. Metadata for the configuration is also generated, including the basis for the parameter settings, expected results, and possible adjustment suggestions, facilitating monitoring and optimization of the search process. Through this multi-layered parameter optimization, customized search strategies can be provided for tasks of varying complexity and characteristics, achieving the optimal balance between efficiency and effectiveness.

[0167] In step S106, the multimodal enhanced retrieval results are processed using a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism to obtain high-quality final response results.

[0168] Specifically, step S106 includes the following sub-steps:

[0169] Step S1061: Based on the multimodal enhanced retrieval results and the user's multimodal query input, obtain the query context representation through context encoding and context understanding algorithms.

[0170] In this sub-step, a rich query context representation is constructed, providing a comprehensive contextual foundation for subsequent relevance ranking and inference. First, conversation history analysis is performed to understand the current query within the context of the entire interactive conversation. The system collects and processes historical interaction records between the user and the system, including previous queries, system responses, and user feedback. This historical information is processed by a temporal encoder (such as a recurrent neural network or a Transformer) to generate a contextual representation reflecting the evolution of the interaction. The system pays particular attention to pronoun resolution (e.g., the reference object of pronouns like "it" or "that"), topic continuation (the association between the current query and previous topics), and intent shift (changes or refinements in user intent). Through this conversation history analysis, the system can understand the position and meaning of the current query within the broader conversation flow, avoiding comprehension biases caused by processing queries in isolation. Next, query intent parsing is performed to gain a deeper understanding of the user's purpose and expectations in submitting the query. The system analyzes the linguistic structure, keywords, and semantic patterns of the query to identify the basic intent type (e.g., information query, action request, opinion solicitation) and specific subtypes (e.g., fact query, comparison query, causal query). The system also identifies elements such as constraints, preference expressions, and expected output formats in the query. For multimodal queries, the system analyzes how different modal contents collectively constitute or modify the query intent; for example, text poses a question while an image provides context, or audio expresses emotion while text provides specific content. This in-depth intent parsing enables the system to more accurately understand the user's true needs, rather than just the surface form of the query. Then, the system performs multimodal context integration, associating and integrating different modal information in the query with multimodal content in the search results. It analyzes complementary, reinforcing, and validating relationships between different modalities such as text, images, and audio. For example, an image in the query might show an object of interest to the user, while the text description indicates which aspects of that object the user is interested in; images in the search results might provide visual evidence, while related text provides explanation or background. Using cross-modal attention mechanisms and multimodal fusion networks, an integrated representation reflecting intermodal relationships is constructed. This integration considers not only intramodal semantics but also intermodal consistency, complementarity, and conflict, resulting in a more comprehensive contextual understanding. Next, a contextual knowledge graph is constructed, organizing entities, concepts, and relationships from query and retrieval results into a structured knowledge representation. The system identifies key entities and concepts, analyzes the semantic relationships between them, and constructs a graph reflecting the current contextual knowledge structure. This graph not only contains explicitly mentioned information but also includes implicit associations and background knowledge through reasoning extensions. The system may enhance this contextual graph using existing knowledge bases (such as general knowledge graphs or domain ontologies) to add relevant background knowledge. This structured knowledge representation enables the system to more systematically understand the semantic network of the query context, supporting subsequent reasoning and relevance judgments.Finally, a comprehensive query context representation is generated. This is a multi-layered, multi-dimensional representation structure that includes session history, query intent, multimodal relationships, and knowledge structure. This representation may take the form of vectors, graphs, or a hybrid approach, capturing both the semantic content of the context and reflecting its structural relationships. The goal is to ensure that this representation is both rich and comprehensive, containing all relevant information, and compact and efficient, facilitating subsequent processing. Through this deep context encoding and understanding, the system lays a solid foundation for subsequent relevance ranking and reasoning, enabling the final response to more accurately meet the user's actual needs.

[0171] Step S1062: Based on the query context representation and the multimodal enhanced retrieval results, obtain the priority-ranked retrieval results through a context-aware relevance scoring and ranking algorithm.

[0172] In this sub-step, the search results are evaluated for deep relevance and intelligently ranked based on the query context to ensure that the most relevant information is prioritized. First, the system designs a multi-dimensional relevance scoring framework to evaluate the relevance of the search results to the query from multiple perspectives. This framework considers several relevance dimensions: semantic relevance, evaluating the semantic matching degree of content, using a deep semantic matching model to calculate the semantic similarity between the query and the search terms; intent satisfaction, evaluating the degree to which the search terms satisfy the query intent, considering the purpose and expectations of the query; information completeness, evaluating the completeness and coverage of the information provided by the search terms, whether it covers all aspects of the query; timeliness, evaluating the recentity and current applicability of the information, especially for queries with high timeliness requirements; authority, evaluating the reliability and authority of the information source, considering the source's reputation and professional level; and multimodal coordination, evaluating the consistency and complementarity between different modalities of content. Specific scoring algorithms are designed for each dimension, and the weights of each dimension are dynamically adjusted according to the query characteristics to form a comprehensive scoring mechanism. Next, the system implements context-aware relevance calculation, integrating the query context representation from step S1061 into the relevance evaluation process. The system considers not only the surface content of the query but also its context: conversation-aware relevance, which considers the coherence of the search terms with the entire conversation history, not just the matching degree with the current query; intent-driven relevance, which evaluates the relevance of search terms based on the parsed query intent to ensure that the user's real needs are met; knowledge graph-enhanced relevance, which uses a contextual knowledge graph to evaluate the degree of matching between search terms and the query's knowledge structure; and multimodal cross-relevance, which evaluates the cross-relevance of search terms across different modalities, such as the relevance between text content and query images. This context-aware relevance calculation enables the system to more comprehensively and accurately evaluate the actual value of search results, avoiding misjudgments of relevance caused by surface matching. Then, the system performs relevance ranking and result reorganization, sorting and organizing the search results based on the calculated relevance scores. The ranking process considers not only the relevance of individual search terms but also the overall quality of the result set: ensuring diversity by covering different angles and perspectives and avoiding information redundancy; optimizing complementarity by prioritizing mutually complementary search terms for more comprehensive information coverage; organizing the search results into a logically structured format, such as grouping by topic, sorting by time, or organizing by relationship; and highlighting key information by ensuring that the most crucial and direct answers to the query are given prominent positions. Multi-objective ranking algorithms may be used to balance multiple objectives such as relevance, diversity, and structure to generate optimal ranking results. Next, result validation and consistency checks are performed to ensure the internal consistency and reliability of the ranked search result set.The system checks for contradictory information, duplicate content, and information gaps in the results and processes them appropriately: contradictory information is marked by identifying and labeling contradictory viewpoints or facts in the results to provide attention for subsequent processing; redundant content is merged by identifying and merging duplicate content expressing the same information to increase information density; and information gaps are filled by identifying missing key information, which may trigger additional targeted searches to fill these gaps. This verification and checking ensures the quality and usability of the ranking results. Finally, the system outputs the priority-ranked search results, a structured collection of results ordered by relevance and accompanied by rich metadata, such as relevance scores, information types, and reliability assessments. This ranking provides a high-quality information foundation for subsequent comprehensive reasoning, enabling the prioritization of the most relevant and valuable information, thus improving the quality and efficiency of the final response.

[0173] Step S1063: Based on the priority-sorted retrieval results and the decision path information, a comprehensive reasoning result is obtained through the integration mechanism of retrieval knowledge and model reasoning.

[0174] In this sub-step, the retrieved knowledge is organically combined with the model's reasoning ability to generate a deep, comprehensive reasoning result. First, a reasoning knowledge base is constructed, transforming the sorted retrieval results into a structured reasoning knowledge foundation. Knowledge extraction and organization are performed on the retrieval results: fact extraction, identifying and extracting key facts, data, and statements from the retrieval results; relationship construction, analyzing the logical, causal, and temporal relationships between facts; evidence linking, associating facts with their sources and supporting evidence to establish a traceable chain of evidence; and uncertainty representation, marking the degree of certainty and possible conditional constraints of the information. The system organizes this structured knowledge into a reasoning knowledge base, which may employ knowledge graphs, logic rule bases, or hybrid representations. This knowledge base not only contains the retrieved explicit knowledge but also includes implicit knowledge through knowledge reasoning extensions, providing a comprehensive knowledge foundation for subsequent reasoning. Next, a multi-step reasoning framework is designed to execute complex reasoning processes based on the constructed knowledge base. Based on the complexity and nature of the query, an appropriate reasoning strategy is selected: deductive reasoning, which derives conclusions based on logical rules and known facts, is suitable for problems with clear logical relationships; inductive reasoning, which deduces general patterns or rules from specific examples, is suitable for problems requiring summarization or pattern recognition; analogical reasoning, which draws conclusions through comparison of similar situations, is suitable for novel or rare problems; causal reasoning, which analyzes causal relationships between events, is suitable for "why" type questions; and counterfactual reasoning, which considers possible outcomes under hypothetical scenarios, is suitable for "what if..." type questions. Multiple reasoning strategies can be combined to construct multi-step reasoning chains, gradually deriving complex conclusions from known facts. A reasoning monitoring mechanism is also implemented to track each step of the reasoning process, ensuring logical consistency and the effectiveness of the reasoning. Finally, a retrieval-enhanced model reasoning is implemented, combining retrieval knowledge with the reasoning capabilities of the language model. Several methods are employed to enhance model reasoning: knowledge-guided prompts, integrating key retrieved knowledge into prompts to guide the model in reasoning based on this knowledge; fact constraint generation, using retrieved facts to constrain the model's generation process, ensuring the output is consistent with known facts; reasoning chain verification, using retrieved knowledge to verify the model's reasoning steps, correcting potential logical errors or factual biases; and multi-source information integration, helping the model integrate information from different sources to form a comprehensive judgment. This enhanced reasoning may be performed using either the basic model or the advanced model from step S103, selecting the appropriate model level based on task complexity and previous deferred decisions. Next, the reasoning results are synthesized and reconciled, integrating results from multiple reasoning paths and sources.When different reasoning paths or information sources lead to different conclusions, it is necessary to reconcile the results: evidence weighting analysis to assess the strength and reliability of evidence supporting different conclusions; consistency preference to prioritize conclusions consistent with more known facts; uncertainty expression to clearly express uncertainty and possible options when a single correct conclusion cannot be determined; and expert model judgment, which may introduce more advanced models or pre-defined expert rules for judgment in complex or controversial situations. Through this synthesis and reconciliation, a consistent and comprehensive reasoning result is formed. Finally, a structured comprehensive reasoning result is generated, including the main conclusions, supporting evidence, reasoning paths, and uncertainty explanations. This ensures that the reasoning result has both a solid knowledge base and a clear logical structure, effectively answering user queries and providing necessary explanations and support. This deep integration of retrieval knowledge and model reasoning generates a high-quality response that is both factually based and reasoning-deep.

[0175] Step S1064: Based on the comprehensive reasoning result, the high-quality final response result is obtained through response generation template, consistency check, and quality optimization mechanism based on accuracy and readability standards.

[0176] In this sub-step, the comprehensive reasoning results are transformed into a clear, accurate, and practical final response to ensure effective fulfillment of user needs. First, an appropriate response generation template is selected. Based on the query type and the characteristics of the reasoning results, the most suitable response structure and style are determined. Multiple response templates are maintained to suit different types of queries and output requirements: informational templates, suitable for responses providing factual information, emphasizing clear information organization and complete factual presentation; explanatory templates, suitable for responses explaining concepts or phenomena, emphasizing logical clarity and a well-structured explanatory framework; guidance templates, suitable for responses providing operational guidelines or suggestions, emphasizing clear steps and practicality; comparison templates, suitable for responses comparing multiple options or viewpoints, emphasizing a fair and comprehensive comparative structure; and multimodal templates, suitable for responses requiring the integration of text, images, or other modalities, emphasizing coordination and complementarity between modalities. Based on query intent, the characteristics of the reasoning results, and user preferences, the most suitable template is selected and parameterized according to the specific content to ensure a natural fit between the template and the content. Next, the response content is organized and generated, filling the selected template structure with the reasoning results to form the initial response. Based on the template structure, organize the various parts of the reasoning result: Main conclusion or answer, directly addressing the user's core question and placed prominently in the response; Supporting evidence and explanations, providing facts, data, or reasoning processes to support the main conclusion, enhancing the credibility of the response; Supplementary information and context, providing background knowledge or relevant information to aid understanding, enriching the informational value of the response; Uncertainty and limitation statements, honestly expressing the degree of certainty and applicability of the conclusion, avoiding misleading the user. Using natural language generation technology, transform this organized content into fluent and coherent natural language expressions, ensuring grammatical correctness, clarity, and stylistic consistency. Then, perform consistency checks and quality optimization on the response to ensure high quality and reliability of the final output. Perform multi-level checks and optimizations: Fact consistency check, verifying the consistency between factual statements in the response and retrieved knowledge, avoiding factual errors; Logical consistency check, verifying the internal logical relationships of the response are self-consistent, avoiding contradictions or reasoning errors; Language quality optimization, improving the clarity, conciseness, and readability of the expression, including optimizing sentence structure, eliminating redundancy, and enhancing coherence; Multimodal consistency check, ensuring consistency and mutual support rather than contradiction between text and different modalities such as images. The system may use specialized quality assessment models or rule systems to perform these checks and make necessary corrections and optimizations based on the check results. Next, the system performs personalized adjustments to the response, customizing the final output according to user characteristics and preferences.Several personalized factors are considered: user knowledge level, adjusting the use and depth of technical terminology to ensure the content is neither too simple nor too complex; interaction history, maintaining consistency with previous interactions and avoiding duplication of known information; expression preferences, adjusting language style, level of detail, and formalization to align with user communication habits; device and environmental constraints, considering user device type and usage environment to optimize response format and length. This personalization makes the response more tailored to the user's specific needs and context, improving user experience and satisfaction. Finally, the final response result is generated—a fully optimized, high-quality output presented directly to the user. The final response not only includes core content but may also include meta-information such as source citations, confidence metrics, or feedback mechanisms, enhancing transparency and credibility. The final response is ensured to meet the highest standards in accuracy, clarity, relevance, and usability, effectively satisfying the user's query needs. This comprehensive response generation and optimization process transforms complex reasoning results into a user-friendly, high-quality response, marking the successful completion of the multimodal heterogeneous model retrieval enhancement process.

[0177] Figure 4 This is a schematic diagram of the structure of the multimodal heterogeneous model retrieval enhancement system provided in an embodiment of the present invention. Figure 4 As shown, the system includes:

[0178] The bilingual corpus retrieval module 801 is used to obtain a joint retrieval result set based on the user's multimodal query input by constructing a bilingual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism.

[0179] The multimodal data preprocessing module 802 is used to map and schedule the joint retrieval result set to obtain multimodal feature representations through dedicated processing channels for text, image and audio data and a Spiking neural network architecture with a segmented trapezoidal topology.

[0180] The adaptive cascaded decision module 803 is used to obtain decision path information and answer candidate set based on the multimodal feature representation by constructing a three-level cascaded architecture including a basic model, a high-level model and a human expert, and a dual decision-making mechanism of deferred and abstention.

[0181] The Hamiltonian graph training module 804 is used to represent the multimodal information relationship between the decision path information and the answer candidate set through a Hamiltonian graph network and to obtain optimized model parameters by using a fast training method without gradient descent.

[0182] The cross-modal alignment adjustment module 805 is used to obtain multimodal enhanced retrieval results based on the optimized model parameters, through a cross-modal semantic alignment algorithm and a dynamic retrieval window adjustment strategy based on task complexity.

[0183] The response generation module 806 is used to obtain a high-quality final response result from the multimodal enhanced retrieval results through a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism.

[0184] The multimodal heterogeneous model retrieval enhancement method and system of this invention achieves the following: First, by constructing a dual-corpus RAG+ architecture, application-aware reasoning is explicitly integrated into the retrieval enhancement generation pipeline. Second, through an adaptive cascaded LLM decision framework, adaptive delegation of tasks across multiple professional knowledge levels is achieved. Third, through multimodal data preprocessing based on Spiking neural networks, efficient multimodal data preprocessing and pre-fusion are realized. Fourth, through a gradient-free Hamiltonian graph network fast training method, efficient model training is achieved. Fifth, through a dynamic retrieval window adjustment mechanism, intelligent allocation of retrieval resources is realized. Sixth, through cross-modal semantic alignment technology, effective alignment between data from different modalities is achieved. In summary, this invention achieves efficient multimodal information retrieval and processing, improving the reasoning ability and response quality of heterogeneous models.

[0185] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for enhancing the retrieval of multimodal heterogeneous models, characterized in that, Includes the following steps: Based on user multimodal query input, a joint retrieval result set is obtained by constructing a dual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism. The joint retrieval result set is mapped and scheduled using dedicated processing channels for text, image, and audio data and a Spiking neural network architecture with a segmented trapezoidal topology to obtain multimodal feature representations. Based on the multimodal feature representation, by constructing a three-level cascaded architecture including a basic model, a high-level model, and human experts, and a dual decision-making mechanism of deferred and abstention, decision path information and a set of candidate answers are obtained. For the decision path information and the answer candidate set, the multimodal information relationship is represented by a Hamiltonian graph network and a fast training method without gradient descent is used to obtain optimized model parameters; Based on the optimized model parameters, multimodal enhanced retrieval results are obtained through cross-modal semantic alignment algorithms and dynamic retrieval window adjustment strategies based on task complexity. Specifically, this includes: obtaining a unified cross-modal semantic representation space for the optimized model parameters through shared semantic space mapping and inter-modal semantic alignment algorithms; obtaining a task complexity score based on the current user query content and the unified cross-modal semantic representation space through multi-dimensional task complexity evaluation index calculation and difficulty grading mechanisms, including concept complexity, relation complexity, and reasoning complexity; obtaining optimized retrieval parameter configurations for the task complexity score through adaptive window size calculation and retrieval depth adjustment algorithms; and obtaining the multimodal enhanced retrieval results based on the optimized retrieval parameter configurations and the unified cross-modal semantic representation space through multi-dimensional retrieval strategies and result fusion mechanisms. The multimodal enhanced retrieval results are used to obtain high-quality final response results through a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism.

2. The method according to claim 1, characterized in that, The method, based on user multimodal query input, constructs a dual corpus consisting of a knowledge corpus and an application example corpus, and designs a joint retrieval mechanism to obtain a joint retrieval result set, including: The user's multimodal query input is subjected to a unified data interface protocol and standardized preprocessing to obtain a standardized multimodal query representation; Based on the standardized multimodal query representation, a domain knowledge corpus index is obtained through multimodal knowledge extraction, structured storage, and vector index construction techniques using entity recognition and semantic analysis. The standardized multimodal query representation is used to obtain an application example corpus index through task-oriented application example collection, annotation, and knowledge alignment processing. Based on the domain knowledge corpus index and the application example corpus index, the joint retrieval result set is obtained through a weighted fusion retrieval algorithm and a relevance scoring mechanism.

3. The method according to claim 1, characterized in that, The joint retrieval result set is mapped and scheduled using dedicated processing channels for text, image, and audio data, and a Spiking neural network architecture with a segmented trapezoidal topology, to obtain multimodal feature representations, including: The text content in the joint retrieval result set is processed by word segmentation, normalization, entity recognition, and semantic analysis to obtain a text feature vector set; The image content in the joint retrieval result set is processed by image segmentation, enhancement, and target detection, and then mapped using a multi-level segmented trapezoidal topology Spiking neural network architecture to obtain an image feature vector set. The audio content in the joint retrieval result set is processed by audio segmentation, noise reduction, and feature extraction, followed by temporal processing using a Spiking neural network to obtain an audio feature vector set. Based on the text feature vector set, the image feature vector set, and the audio feature vector set, the multimodal feature representation is obtained through a cross-modal attention mechanism and a feature alignment algorithm.

4. The method according to claim 1, characterized in that, Based on the multimodal feature representation, a three-level cascaded architecture comprising a base model, a high-level model, and human experts, along with a dual decision-making mechanism of deferred and abstention, is constructed to obtain decision path information and a candidate set of answers, including: The multimodal feature representation is used for preliminary reasoning and answer generation through a lightweight basic language model to obtain a preliminary answer and confidence score; Based on the preliminary answer and the confidence score, the model level selection decision is obtained through confidence threshold comparison and task complexity evaluation algorithm; For tasks that trigger deferred responses, deep reasoning is performed using a large-scale high-level language model with a parameter size larger than that of the base model, based on the multimodal feature representation, to obtain advanced answers and new confidence scores. Based on the final answer and the new confidence score, a decision on human intervention is obtained through an abstention decision algorithm and uncertainty assessment. Based on the model-level selection decision and the human intervention decision, combined with the preliminary answer and the final answer, the decision path information and answer candidate set are obtained through decision path tracking and answer quality evaluation mechanisms.

5. The method according to claim 1, characterized in that, The optimized model parameters are obtained by representing the multimodal information relationships between the decision path information and the answer candidate set using a Hamiltonian graph network and a fast training method without gradient descent, including: Based on the decision path information and the answer candidate set, an initial Hamiltonian graph structure representing the multimodal information relationship is obtained through multimodal adaptation entity-relation extraction and graph construction algorithms; For the initial Hamiltonian graph structure, the Hamiltonian graph network model to be trained is obtained through Hamiltonian graph network architecture design and parameter initialization strategy; For the Hamiltonian graph network model to be trained, the model state sequence during the training process is obtained through the definition of energy function, direct state update and gradient-free optimization algorithm; Based on the model state sequence during the training process, the optimized model parameters are obtained through convergence criteria and optimal state selection algorithm.

6. The method according to claim 1, characterized in that, The optimized model parameters, through shared semantic space mapping and inter-modal semantic alignment algorithms, yield a unified cross-modal semantic representation space, including: The optimized model parameters are designed with a multi-branch structure, and the initial feature representations of each modality are obtained through specific encoders for text, image and audio branches. The initial feature representations of each modality are mapped by a projection layer, normalized, and adjusted to obtain feature vectors with shared dimensions. Based on the feature vector of the shared dimension, an alignment strategy including contrastive learning, semantic anchors and structure preservation is constructed through positive and negative samples to obtain a semantically aligned feature representation; Cross-modal retrieval tests, semantic clustering analysis, and analogy verification are performed on the semantically aligned feature representations to obtain the unified cross-modal semantic representation space.

7. The method according to claim 1, characterized in that, Based on the current user query content and combined with the unified cross-modal semantic representation space, a task complexity score is obtained through a multi-dimensional task complexity evaluation index calculation and difficulty grading mechanism, including concept complexity, relation complexity, and reasoning complexity. The task complexity is scored, and an optimized retrieval parameter configuration is obtained through adaptive window size calculation and retrieval depth adjustment algorithms, including: The current user query content is parsed and represented. Through query decomposition, multimodal processing and semantic encoding, a cross-modal semantic representation of the query is obtained. Based on the cross-modal semantic representation of the query, a multi-dimensional complexity index is obtained through a comprehensive evaluation including concept complexity, relational complexity, reasoning complexity, and multimodal dependency. The task complexity score is obtained by weighted fusion and nonlinear mapping processing of the multi-dimensional complexity index. Based on the task complexity score, a complexity-resource mapping relationship is established, and the retrieval window size is obtained through basic window setting, task characteristic adjustment, and adaptive formula calculation. The depth and breadth parameters of the search window size are adjusted, and the optimized search parameter configuration is obtained through modality-specific search parameter configuration.

8. The method according to claim 1, characterized in that, The multimodal enhanced retrieval results are processed using a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism to obtain high-quality final response results, including: Based on the multimodal enhanced retrieval results and the user's multimodal query input, a query context representation is obtained through context encoding and context understanding algorithms; Based on the query context representation and the multimodal enhanced search results, a priority-ranked search result is obtained through a context-aware relevance scoring and ranking algorithm. Based on the priority-sorted retrieval results and the decision path information, a comprehensive reasoning result is obtained through the integration mechanism of retrieval knowledge and model reasoning. The comprehensive reasoning results are used to generate high-quality final response results through response template generation, consistency checks, and quality optimization mechanisms based on accuracy and readability standards.

9. A multimodal heterogeneous model retrieval enhancement system, characterized in that, include: The bilingual corpus retrieval module is used to obtain a joint retrieval result set based on user multimodal query input by constructing a bilingual corpus consisting of a knowledge corpus and an application example corpus and designing a joint retrieval mechanism. The multimodal data preprocessing module is used to map and schedule the joint retrieval result set to obtain multimodal feature representations through dedicated processing channels for text, image and audio data and a Spiking neural network architecture with a segmented trapezoidal topology. An adaptive cascaded decision module is used to obtain decision path information and a set of answer candidates based on the multimodal feature representation by constructing a three-level cascaded architecture including a base model, a high-level model, and human experts, as well as a dual decision-making mechanism of deferred and abstention. The Hamiltonian graph training module is used to represent the multimodal information relationship between the decision path information and the answer candidate set through a Hamiltonian graph network and to obtain optimized model parameters using a fast training method without gradient descent. The cross-modal alignment adjustment module is used to obtain multimodal enhanced search results based on the optimized model parameters, through a cross-modal semantic alignment algorithm and a dynamic retrieval window adjustment strategy based on task complexity. Specifically, it includes: obtaining a unified cross-modal semantic representation space for the optimized model parameters through shared semantic space mapping and inter-modal semantic alignment algorithms; obtaining a task complexity score based on the current user query content, combined with the unified cross-modal semantic representation space, through multi-dimensional task complexity evaluation index calculation and difficulty grading mechanism including concept complexity, relation complexity, and reasoning complexity; obtaining an optimized retrieval parameter configuration based on the task complexity score through adaptive window size calculation and retrieval depth adjustment algorithms; and obtaining the multimodal enhanced search results based on the optimized retrieval parameter configuration and the unified cross-modal semantic representation space through multi-dimensional retrieval strategies and result fusion mechanisms. The response generation module is used to obtain high-quality final response results from the multimodal enhanced retrieval results through a context-aware relevance ranking algorithm and a retrieval-enhanced comprehensive reasoning mechanism.

Images