A collaborative method and system for intelligent engines oriented towards multimodal queries

Abstract

This invention provides an intelligent engine collaboration method and system for multimodal queries, relating to the field of data collaborative processing technology. The method includes parsing query dimensions based on received multimodal query requests, generating query parsing results, dynamically allocating tasks to matching engines based on the parsing results, with each engine executing processing operations in parallel or sequentially, performing multi-source data association and fusion on the results from heterogeneous engines, and encapsulating the fusion results into visualized data for user feedback. The system includes a query dimension parsing module, an engine task allocation module, an engine execution module, a multi-source data association and fusion module, and a result visualization and encapsulation module. This invention significantly improves query performance, reduces query latency, and increases accuracy by constructing a multimodal query fingerprint database and a graph-structured collaborative network, combined with pre-trained graph convolutional networks to optimize control parameters. It is suitable for complex scenarios such as smart cities and business intelligence.

Description

Technical Field

[0001] This invention relates to the field of data collaborative processing technology, and in particular to an intelligent engine collaborative method and system for multimodal queries. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, the demand for multimodal data queries is increasing. Users often need to process multiple data types simultaneously, including text, spatial geographic information, quantitative indicators, and unstructured data (such as images, audio, and video). Traditional data query systems are typically designed for a single modality. For example, relational databases excel at structured data retrieval but struggle to efficiently handle similarity matching or spatial topology analysis of unstructured data. In practical applications, such as smart city traffic management, financial risk analysis, and medical image diagnosis, query requests often involve multimodal combinations. For instance, in traffic accident emergency response, it is necessary to simultaneously retrieve surveillance video (unstructured feature query), sensor traffic data (quantitative indicator query), accident report text (text query), and geofencing information (spatial range query). Existing technologies often employ independent engines to handle different modal queries, but lack effective collaboration mechanisms, leading to numerous problems in the query process: First, heterogeneous protocols and incompatible data formats between engines require manual conversion, increasing complexity and error rates; second, static task allocation cannot be dynamically optimized based on real-time load, easily causing engine overload or resource idleness; finally, result fusion is simply superimposed, lacking semantic consistency verification and weight adjustment, affecting query accuracy and response speed. Currently, some research attempts to integrate multiple engines through middleware or distributed frameworks, but often focuses on single-dimensional optimization, such as improving communication protocols or caching strategies, failing to fundamentally solve the collaboration challenges of multimodal queries. For example, rule-based engine methods rely on predefined strategies, lacking flexibility and struggling to adapt to dynamic query scenarios; while machine learning methods can learn query patterns, model training is costly and insensitive to real-time performance changes. In specific application scenarios, such as smart city traffic management systems, existing technologies require separate calls to video analysis engines, sensor data processing engines, and geographic information system engines, with results manually integrated, resulting in response delays of several seconds, failing to meet real-time decision-making requirements. Furthermore, query performance is prone to drift when data distribution changes or system load fluctuates, lacking an adaptive adjustment mechanism. Therefore, there is an urgent need in the market for a novel intelligent engine collaboration method and system for multimodal queries. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent engine collaboration method and system for multimodal queries, in order to solve the problems of poor heterogeneous compatibility, low engine collaboration efficiency, and high response latency in existing technologies when querying multi-source data. The specific technical solution is as follows:

[0004] This invention provides a collaborative method for intelligent engines oriented towards multimodal queries, comprising:

[0005] Step 1: Based on the received multimodal query requests, including text queries, spatial range queries, quantitative index queries, and unstructured data feature queries, perform query dimension parsing operations to generate query parsing results containing the query data type, target calculation dimension, and corresponding engine identifier;

[0006] Step 2: Based on the query parsing results, execute the engine task allocation operation and send engine processing instructions to the relational database engine, vector database engine, spatial GIS database engine, and / or indicator analysis engine that match the identifier.

[0007] Step 3: According to the engine processing instructions, the relational database engine performs structured data filtering and association operations to generate a structured result set, the vector database engine performs feature vector comparison operations to generate feature matching results, the spatial GIS database engine performs spatial overlay analysis and range filtering operations to generate spatial analysis results, and the indicator analysis engine performs indicator formula calculation and threshold verification operations to generate indicator calculation results.

[0008] Step 4: Based on the structured result set, feature matching results, spatial analysis results, and index calculation results, perform multi-source data association and fusion operations. The operations include data format normalization, semantic consistency verification, and dynamic weight allocation to generate multimodal collaborative query results.

[0009] Step 5: Based on the multimodal collaborative query results, perform a result visualization encapsulation operation to generate structured display data that meets user interaction requirements, and then feed the display data back to the user terminal.

[0010] Furthermore, relational database engines are adapted for structured data retrieval tasks, vector database engines are adapted for unstructured data similarity matching tasks, spatial GIS database engines are adapted for geospatial topology analysis tasks, and indicator analysis engines are adapted for multi-dimensional quantitative indicator calculation tasks.

[0011] Further, step 1 includes:

[0012] Step 101: Obtain historical query log data of multimodal query requests under different query scenarios, including intelligent customer service Q&A scenarios, geographic information system retrieval scenarios, business intelligence analysis scenarios, or multimedia content retrieval scenarios.

[0013] Step 102: Parse the historical query log data to obtain query data type, query complexity, response latency requirements, and semantic relevance as query features;

[0014] Step 103: Perform unsupervised clustering on the query features to generate query feature clusters;

[0015] Step 104: Extract the cluster center feature vectors corresponding to each query feature cluster as query fingerprints and store them in the multimodal query fingerprint database. Each query scenario corresponds to one query fingerprint.

[0016] Furthermore, the following steps are included after step 1:

[0017] Step 105: Monitor the response success rate and query processing latency of multimodal queries in real time;

[0018] Step 106: When the response success rate is lower than the success rate threshold, or the query processing latency is greater than the latency threshold, extract the query features of the abnormal query requests.

[0019] Step 107: Calculate the similarity between the query features and the query fingerprints in the multimodal query fingerprint database. If the calculated maximum similarity is lower than the similarity threshold, update the query fingerprint corresponding to the maximum similarity in the multimodal query fingerprint database.

[0020] Further, step 106 extracts query features of abnormal query requests, including obtaining the query data type, query complexity, response latency requirements, and semantic relevance of the abnormal query as a query feature quadruple; step 107 calculates the similarity between the query features and the query fingerprint, including calculating the cosine similarity between the abnormal query feature vector and the query fingerprint vector; step 107 updates the query fingerprint, including performing a moving average calculation between the abnormal feature vector and the original fingerprint vector to generate an updated cluster center feature vector that covers the original query fingerprint, wherein the weight ratio of the new and old in the moving average calculation is 6:4.

[0021] Furthermore, prior to step 2, the following is also included:

[0022] Step 9: For each query scenario, identify the load rate of each query engine in that query scenario;

[0023] Step 10: Select candidate query engines with a load rate lower than the load rate threshold from the various query engines in the query scenario, and map the candidate query engines as engine nodes;

[0024] Step 11: For each candidate query engine, obtain the historical collaboration log corresponding to the candidate query engine. The historical collaboration log includes historical processing data when the candidate query engine collaborates with other types of query engines in the query scenario.

[0025] Step 12: Based on historical processing data, determine the collaborative response probability among various candidate query engines;

[0026] Step 13: Connect the engine nodes corresponding to each candidate query engine through the collaborative response probability between each candidate query engine to obtain the multimodal collaborative network corresponding to each query scenario.

[0027] Furthermore, step 9 includes:

[0028] Step 901: Determine the average response time of each query engine in the query scenario, and the query processing volume of each query engine per unit time.

[0029] Step 902: Based on the average response time and query processing volume, determine the load rate of each query engine in the query scenario;

[0030] Step 12 includes:

[0031] Step 1201: Based on historical processing data, determine the historical success rate among the various candidate query engines;

[0032] Step 1202: Determine the data compatibility factor based on the query fingerprints corresponding to each candidate query engine;

[0033] Step 1203: Determine the collaborative response probability among the candidate query engines based on historical success rate, data compatibility factor, and network latency factor.

[0034] Furthermore, step 3 includes:

[0035] Step 301: Encode the multimodal query fingerprint database into a node feature matrix, and use the multimodal collaborative network as the adjacency matrix of the node feature matrix. Each node in the node feature matrix represents a query fingerprint, and the connection edges between nodes represent the similarity between the query fingerprints.

[0036] Step 302: Input the node feature matrix and adjacency matrix into the pre-trained graph convolutional network and output control parameters with the goal of minimizing query latency;

[0037] Step 303: Convert the control parameters into an engine executable instruction set to obtain the adaptive collaborative control instructions corresponding to the query engine in the current query scenario.

[0038] Furthermore, the formula for calculating the load rate in step 902 is as follows:

[0039] ; among which, L i β is the load rate of engine i (representing the current resource utilization level), and β is the weighting coefficient (the default value is 0.6, which adjusts the contribution ratio of response time and processing volume).

[0040] The formula for calculating the cooperative response probability in step 1203 is as follows: ;in, This represents the combined collaborative response probability of engine A and engine B (a predicted value that incorporates multiple factors). For historical success rate, C AB D is the data compatibility factor, and D is the network latency factor. These are the fusion weights corresponding to the historical success rate of fusion, the data compatibility factor, and the network latency factor, respectively. ,and All are positive numbers;

[0041] In step 306, the loss function includes a weighted mean square error function, and the formula for calculating the first error loss is as follows: The formula for calculating the second error loss is as follows: Where Loss1 represents the first error loss, and M is the number of samples. T represents the query latency weight for the i-th sample. pred，i T represents the prediction query latency for the i-th sample. real，i The value represents the actual query latency for the i-th sample, Loss2 represents the second error loss, and N is the number of engine nodes. This represents the control parameter weights for the i-th sample. This represents the predictive control parameters of engine k for engine j in the i-th sample. This represents the actual control parameters of engine k over engine j in the i-th sample;

[0042] In step 306, when calculating the first error loss, the multimodal collaborative loss factor and the feature constraint factor are also integrated. The formula for calculating the first error loss is as follows:

[0043] ; where G i Let E represent the multimodal cooperative network of the i-th sample. i Let represent the set of all connection edges in the multimodal cooperative network of the i-th sample. Let deg(j) represent the predicted query latency of engine node j in the i-th sample, deg(j) represent the degree of engine node j (the number of edges connected to engine node j), and deg(k) represent the degree of engine node k (the number of edges connected to engine node k). X represents the historical average collaborative latency difference between engine node k and engine node j. i This represents the node feature matrix of the i-th sample. Let represent the baseline feature matrix for the c-th query scenario, where λ1 and λ2 represent smoothing coefficients used to balance the weights among the items. The trace of a matrix is ​​the sum of the elements on its main diagonal. Let L denote the transpose of the node characteristic matrix, and let L denote the Laplace matrix of the node characteristic matrix. This represents the Frobenius norm.

[0044] Furthermore, in step 103, the DBSCAN clustering algorithm is used to perform unsupervised clustering of the query features, with cosine similarity as the metric and the elbow rule is used to determine the optimal number of clusters; in step 13, when constructing the multimodal collaborative network, connection edges are added to engine node pairs with collaborative response probabilities greater than 0, and the collaborative response probability is used as the edge weight.

[0045] Furthermore, prior to step 302, the method further includes:

[0046] Step 304: Construct a training dataset and a graph convolutional network to be trained. The training dataset includes the feature matrix of sample nodes obtained by encoding the multimodal query fingerprint database, the historical multimodal collaborative networks corresponding to different query scenarios, and the actual control parameters and real query latency corresponding to the historical multimodal collaborative networks.

[0047] Step 305: Input the sample node feature matrix and the historical multimodal collaborative network into the convolutional network of the graph to be trained to predict the query delay, and obtain the prediction control parameter matrix and the prediction query delay.

[0048] Step 306: Calculate the first error loss between the predicted query latency and the actual query latency, and the second error loss between the actual control parameters and the predicted control parameter matrix using the loss function. Add the first error loss and the second error loss to obtain the total error loss. With minimizing the total error loss as the optimization objective, adjust the network parameters of the graph convolutional network to be trained using the backpropagation algorithm, and iterate the adjustment process of the network parameters until the training stopping condition is met, and then stop training to obtain the pre-trained graph convolutional network.

[0049] This invention also relates to an intelligent engine collaborative system for multimodal queries, used to implement the method described, the system comprising:

[0050] The query dimension parsing module is used to generate query parsing results based on the received multimodal query requests;

[0051] The engine task allocation module is used to dynamically allocate query tasks to matching engine instances based on the query parsing results.

[0052] The engine execution module is used for parallel or serialized processing of tasks by various engines.

[0053] The multi-source data association and fusion module is used to integrate and fuse results from heterogeneous engines;

[0054] The results visualization encapsulation module is used to convert the fusion results into interactive display data for users;

[0055] The performance monitoring and fingerprint update module is used to monitor query performance and update the query fingerprint database in real time.

[0056] Furthermore, the performance monitoring and fingerprint update module is used for:

[0057] Real-time monitoring of multimodal query response success rate and query processing latency;

[0058] When the response success rate is lower than the success rate threshold, or the query processing latency is greater than the latency threshold, extract the query characteristics of abnormal query requests;

[0059] The similarity between the query features and the query fingerprints in the multimodal query fingerprint database is calculated.

[0060] If the calculated maximum similarity is lower than the similarity threshold, the query fingerprint corresponding to the maximum similarity in the multimodal query fingerprint database will be updated.

[0061] When the performance monitoring and fingerprint update module extracts query features from abnormal query requests, it obtains the query data type, query complexity, response latency requirements, and semantic relevance of the abnormal query as query features; when performing similarity calculation, it calculates the cosine similarity between the abnormal query feature vector and the query fingerprint vector; when updating the query fingerprint, it performs a moving average calculation between the abnormal feature vector and the original fingerprint vector.

[0062] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the method described herein.

[0063] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method.

[0064] The beneficial effects of this invention are as follows: The intelligent engine collaboration method and system for multimodal queries provided by this invention solve the core challenges in multi-source heterogeneous data queries through systematic innovation, bringing significant comprehensive benefits. Its core beneficial effects are reflected in four aspects: efficiency, accuracy, adaptability, and practicality. Regarding efficiency, this invention breaks through the bottleneck of traditional serial or simple parallel queries. By intelligently parsing query requests, it accurately routes multimodal query tasks such as text, space, indicators, and unstructured features to the most professional processing engines (such as relational databases, vector databases, spatial GIS engines, and indicator analysis engines), achieving true parallel collaborative processing. Furthermore, by constructing a load-aware multimodal collaborative network and dynamically generating adaptive control instructions using a pre-trained graph convolutional network, the system can optimize query paths and execution sequences, thereby significantly reducing overall query latency and meeting the stringent requirements of millisecond-level response in scenarios such as smart cities and real-time decision-making. Regarding accuracy, this invention effectively improves the reliability of the results through a deep multi-source data association and fusion mechanism. This mechanism not only normalizes the data format of results returned by heterogeneous engines, but also eliminates data ambiguity through semantic consistency verification and dynamically allocates weights based on query context and engine reliability, ultimately generating semantically unified and value-integrated collaborative query results, thus improving query accuracy. In terms of adaptability and robustness, the system demonstrates strong self-optimization capabilities. It triggers dynamic updates to the query fingerprint database through real-time performance monitoring, enabling the system to adapt to constantly changing user query patterns and data distributions. Simultaneously, it filters candidate engines based on load rate and predicts collaborative response probabilities by integrating historical success rates, data compatibility factors, and real-time network latency, ensuring the system's stability and high efficiency in high-concurrency and complex network environments. Ultimately, these technical advantages translate into significant practical value. Taking smart city traffic management as an example, the system can perform instantaneous collaborative analysis of surveillance videos, sensor data, accident reports, and geographic information, quickly generating holographic assessment reports, greatly improving the efficiency and scientific rigor of emergency response and decision-making, fully demonstrating the enormous potential of this invention in solving complex cross-domain problems.

[0065] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0066] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0067] Figure 1 This is a schematic diagram illustrating the steps of an intelligent engine collaborative method for multimodal queries according to the present invention.

[0068] Figure 2This is a schematic diagram of the structure of an intelligent engine collaborative system for multimodal queries according to the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0070] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0071] It should also be noted that when a component is described as "fixed to" or "set on" another component, it can be directly on the other component or there may be an intervening component present. When a component is described as "connected to" another component, it can be directly connected to the other component or there may be an intervening component present.

[0072] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0073] In an embodiment of the present invention, a collaborative method for intelligent engines oriented towards multimodal queries is provided. Please refer to [link to relevant documentation]. Figure 1 The method includes the following steps:

[0074] Step 1: Based on the received multimodal query requests, including text queries, spatial range queries, quantitative index queries, and unstructured data feature queries, perform query dimension parsing operations to generate query parsing results containing the query data type, target calculation dimension, and corresponding engine identifier;

[0075] Step 2: Based on the query parsing results, execute the engine task allocation operation and send engine processing instructions to the relational database engine, vector database engine, spatial GIS database engine, and indicator analysis engine that match the identifiers. The relational database engine is adapted to structured data retrieval tasks, the vector database engine is adapted to unstructured data similarity matching tasks, the spatial GIS database engine is adapted to geospatial topology analysis tasks, and the indicator analysis engine is adapted to multi-dimensional quantitative indicator calculation tasks.

[0076] Step 3: According to the engine processing instructions, the relational database engine performs structured data filtering and association operations to generate a structured result set, the vector database engine performs feature vector comparison operations to generate feature matching results, the spatial GIS database engine performs spatial overlay analysis and range filtering operations to generate spatial analysis results, and the indicator analysis engine performs indicator formula calculation and threshold verification operations to generate indicator calculation results.

[0077] Step 4: Based on the structured result set, feature matching results, spatial analysis results, and index calculation results, perform multi-source data association and fusion operations. The operations include data format normalization, semantic consistency verification, and dynamic weight allocation to generate multimodal collaborative query results.

[0078] Step 5: Based on the multimodal collaborative query results, perform a result visualization encapsulation operation to generate structured display data that meets user interaction requirements, and then feed this display data back to the user terminal. In this embodiment of the invention, a multimodal query request refers to a composite request submitted by the user that includes multiple data types and query methods, specifically including text queries, spatial range queries, quantitative indicator queries, and unstructured data feature queries, etc.; among which, text queries are based on retrieval conditions expressed in natural language, spatial range queries involve geographical coordinates or regional boundaries, quantitative indicator queries include numerical calculation formulas and threshold conditions, and unstructured data feature queries target feature vector matching for data such as images, audio, or video.

[0079] The query dimension parsing operation includes in-depth analysis and deconstruction of multimodal query requests to generate query parsing results containing query data types, target calculation dimensions, and corresponding engine identifiers. Query data types include structured data, unstructured data, spatial data, and indicator data. Target calculation dimensions include data retrieval, similarity matching, spatial analysis, and indicator calculation. Engine identifiers specify the professional engine used to process the query, such as a relational database engine, vector database engine, spatial GIS database engine, or indicator analysis engine. In one possible embodiment, natural language processing technology is used to parse the semantic intent of text queries, a geocoding engine is used to parse the coordinate range of spatial range queries, a feature extraction algorithm is used to parse the feature vectors of unstructured data feature queries, and a formula parser is used to quantify the computational logic of indicator queries, thereby generating standardized query parsing results.

[0080] The engine task allocation operation dynamically assigns query tasks to matching engine instances based on query parsing results. Relational database engines are adapted for structured data retrieval tasks, such as SQL queries and table joins; vector database engines are adapted for unstructured data similarity matching tasks, such as approximate nearest neighbor search; spatial GIS database engines are adapted for geospatial topology analysis tasks, such as overlay analysis and buffer queries; and indicator analysis engines are adapted for multi-dimensional quantitative indicator calculation tasks, such as statistical calculations and threshold validation. In one possible embodiment, engine processing instructions are generated based on the engine identifier in the query parsing results. These instructions include query parameters, processing priority, and collaboration requirements; for example, sending SQL statements to the relational database engine, feature vectors and similarity thresholds to the vector database engine, spatial range and topological relationships to the spatial GIS database engine, and indicator formulas and calculation parameters to the indicator analysis engine.

[0081] During the engine execution phase, each engine processes tasks in parallel or sequentially: the relational database engine generates structured result sets through index retrieval and join operations; the vector database engine generates feature matching results through feature comparison algorithms (such as Faiss or HNSW); the spatial GIS database engine generates spatial analysis results through spatial indexing and geometric operations; and the indicator analysis engine generates indicator calculation results through numerical calculations and verification rules. This process can be optimized based on historical query logs, for example, by using caching mechanisms to improve response speed.

[0082] The multi-source data association and fusion operation integrates results from heterogeneous engines, including data format normalization, semantic consistency verification, and dynamic weight allocation. Data format normalization converts the results into a unified format (such as JSON or Protobuf); semantic consistency verification eliminates data ambiguity through ontology mapping or entity linking; dynamic weight allocation calculates weight coefficients based on query context or engine reliability and performs weighted fusion to generate multimodal collaborative query results. In one possible embodiment, a fusion rule base is constructed to store fusion strategies for different scenarios, such as prioritizing high weights for vector engine results in high-precision queries.

[0083] Finally, the results visualization encapsulation operation transforms the multimodal collaborative query results into user-interactive display data, such as generating charts, maps, or reports, and provides feedback through the user's terminal. This invention solves the heterogeneous compatibility and efficiency problems in multi-source data queries through multimodal query parsing, intelligent engine collaboration, and dynamic fusion mechanisms, significantly improving query accuracy and response speed.

[0084] It is understood that in the specific implementation of this application, data such as user query data and engine log data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0085] Optionally, step 1 specifically includes:

[0086] Step 101: Obtain historical query log data of multimodal query requests under different query scenarios.

[0087] In this embodiment of the invention, the query scenario refers to the query type categorized based on the user's query intent and application environment, including but not limited to intelligent customer service Q&A scenarios, geographic information system retrieval scenarios, business intelligence analysis scenarios, and multimedia content retrieval scenarios. Different query scenarios correspond to different multimodal query combination modes.

[0088] Historical query log data comprises complete query records generated by users in a multimodal query system, including metadata such as query content, query time, returned results, and user feedback. Historical query data streams can be collected by a log collection agent deployed on the query gateway, categorized by query scenario (e.g., establishing an independent log library for each scenario). A scenario tagging mechanism is used to associate query records with their respective query scenarios, ensuring scenario relevance for feature analysis.

[0089] Step 102: Parse the historical query log data to obtain query data type, query complexity, response latency requirements, and semantic relevance as query features.

[0090] By deeply analyzing historical query logs through a query parsing engine, core features such as query data type, query complexity, response latency requirements, and semantic relevance are extracted. Specifically, query data type identifies the data modality involved in the query request (e.g., text, spatial, metrics, unstructured data); query complexity is quantified and scored based on the number of query conditions, nesting depth, and computational intensity; response latency requirements are determined based on user-defined timeout thresholds or historical average response times; and semantic relevance analyzes the strength of logical connections between multimodal query conditions.

[0091] Step 103: Perform unsupervised clustering on the query features to generate query feature clusters.

[0092] The query feature cluster is a set of highly cohesive query features divided into groups using an unsupervised clustering algorithm. Features within the same cluster are highly similar and represent common behavioral patterns in the same type of query scenarios.

[0093] Specifically, the parsed query feature set can be input into an unsupervised clustering algorithm (such as the DBSCAN clustering algorithm), which uses cosine similarity as a metric to automatically aggregate samples with similar feature dimensions; the elbow rule is used to determine the optimal number of clusters, generating N query feature clusters, each cluster identifying a typical query behavior pattern.

[0094] Step 104: Extract the cluster center feature vectors corresponding to each query feature cluster as query fingerprints and store them in the multimodal query fingerprint database. Each query scenario corresponds to one query fingerprint.

[0095] The cluster center feature vector represents the mean feature points of all samples within the query feature cluster. After vectorization and encoding, it forms the typical query fingerprint of the cluster.

[0096] In one possible embodiment, the weighted average of the multidimensional features of each query feature cluster is calculated to generate a cluster center feature vector; the vector is then feature-encoded to form a fixed-length query fingerprint string; a mapping relationship between the query fingerprint and the query scenario is established and stored in a multimodal query fingerprint database.

[0097] In this embodiment, data is collected by classifying query scenarios to address the issue of user query behavior changing with scenarios and improve the scenario adaptability of the fingerprint database; unsupervised clustering is used to automatically discover the inherent patterns of query features and avoid the subjective bias of manually defined fingerprints.

[0098] Optionally, after step 1, the method further includes the following steps:

[0099] Step 105: Monitor the response success rate and query processing latency of multimodal queries in real time.

[0100] Step 106: When the response success rate is lower than the success rate threshold, or the query processing latency is greater than the latency threshold, extract the query features of the abnormal query requests.

[0101] In this embodiment of the invention, the response success rate is the ratio of the number of query requests that successfully return results to the total number of queries per unit time, characterizing the reliability of the query system. The query processing latency is the end-to-end time interval from receiving a query request to receiving a returned result.

[0102] The performance monitoring module of the query gateway can be used to continuously collect performance metrics for various query scenarios. At the application layer, the response success rate (number of successful queries / total number of queries × 100%) can be calculated, and at the system layer, the query processing latency (the difference between the time the request is received and the time the result is returned) can be recorded.

[0103] When any performance metric exceeds a preset threshold, such as a success rate threshold of 95% or a latency threshold of 2 seconds, the query analyzer is triggered to capture the current query request; the feature parsing engine is called to extract the query feature quadruple (query data type, complexity, latency requirement, semantic relevance) of the abnormal query.

[0104] Step 107: Calculate the similarity between the query features and the query fingerprints in the multimodal query fingerprint database. If the calculated maximum similarity is lower than the similarity threshold, update the query fingerprint corresponding to the maximum similarity in the multimodal query fingerprint database.

[0105] Abnormal query requests are those that are captured when the response success rate or processing latency exceeds a threshold, and their query characteristics reflect abnormal query behavior patterns.

[0106] Similarity calculation involves calculating the cosine similarity between the abnormal query feature vector and the query fingerprint vector in a multi-dimensional feature space to quantify the feature matching degree. The abnormal query feature vector can be input into a multimodal query fingerprint database. All query fingerprints in the database are traversed, and the cosine similarity between the abnormal feature vector and each fingerprint is calculated. If the maximum similarity is greater than a similarity threshold, the query behavior is considered to have significantly shifted. The query fingerprint corresponding to the maximum similarity is located, and an update operation is performed. This update operation can be achieved by performing a moving average calculation (new to old weight ratio 6:4) between the abnormal feature vector and the original fingerprint vector to generate an updated cluster center feature vector that covers the original query fingerprint.

[0107] The similarity threshold is a pre-defined tolerance boundary for query fingerprint matching. Query requests below this value are considered to be off-target and are thus deemed abnormal.

[0108] In this embodiment, fingerprint updates are triggered by real-time performance monitoring to address the issue of query feature drift caused by changes in data distribution or system load. An automatic comparison mechanism between abnormal query features and the fingerprint database accurately identifies changes in query patterns. A threshold-driven closed-loop update strategy ensures that the fingerprint database remains adaptable to changes in user query behavior. The fingerprint update process is initiated only when an anomaly is detected, avoiding the periodic reconstruction overhead of the entire fingerprint database.

[0109] Optionally, after step 1, the method further includes the following steps:

[0110] Step 6: Monitor the response success rate and query processing latency of multimodal queries in real time.

[0111] Step 7: When the response success rate is lower than the success rate threshold, or the query processing latency is greater than the latency threshold, extract the query features of the abnormal query requests.

[0112] In this embodiment of the invention, the response success rate is the ratio of the number of query requests that successfully return results to the total number of queries per unit time, characterizing the reliability of the query system. The query processing latency is the end-to-end time interval from receiving a query request to receiving a returned result.

[0113] The performance monitoring module of the query gateway can be used to continuously collect performance metrics for various query scenarios. At the application layer, the response success rate (number of successful queries / total number of queries × 100%) can be calculated, and at the system layer, the query processing latency (the difference between the time the request is received and the time the result is returned) can be recorded.

[0114] When any performance metric exceeds a preset threshold, such as a success rate threshold of 95% or a latency threshold of 2 seconds, the query analyzer is triggered to capture the current query request; the feature parsing engine is called to extract the query feature quadruple (query data type, complexity, latency requirement, semantic relevance) of the abnormal query.

[0115] Abnormal query requests are those that are captured when the response success rate or processing latency exceeds a threshold, and their query characteristics reflect abnormal query behavior patterns.

[0116] Step 8: Calculate the similarity between the query features and the query fingerprints in the multimodal query fingerprint database. If the calculated maximum similarity is lower than the similarity threshold, update the query fingerprint corresponding to the maximum similarity in the multimodal query fingerprint database.

[0117] Similarity calculation involves calculating the cosine similarity between the abnormal query feature vector and the query fingerprint vector in a multi-dimensional feature space to quantify the feature matching degree. The abnormal query feature vector can be input into a multimodal query fingerprint database. All query fingerprints in the database are traversed, and the cosine similarity between the abnormal feature vector and each fingerprint is calculated. If the maximum similarity is greater than a similarity threshold, the query behavior is considered to have significantly shifted. The query fingerprint corresponding to the maximum similarity is located, and an update operation is performed. This update operation can be achieved by performing a moving average calculation (new to old weight ratio 6:4) between the abnormal feature vector and the original fingerprint vector to generate an updated cluster center feature vector that covers the original query fingerprint.

[0118] The similarity threshold is a pre-defined tolerance boundary for query fingerprint matching. Query requests below this value are considered to be off-target and are thus deemed abnormal.

[0119] In this embodiment, fingerprint updates are triggered by real-time performance monitoring to address the issue of query feature drift caused by changes in data distribution or system load. An automatic comparison mechanism between abnormal query features and the fingerprint database accurately identifies changes in query patterns. A threshold-driven closed-loop update strategy ensures that the fingerprint database remains adaptable to changes in user query behavior. The fingerprint update process is initiated only when an anomaly is detected, avoiding the periodic reconstruction overhead of the entire fingerprint database.

[0120] Optionally, prior to step 2, the method further includes:

[0121] Step 9: For each query scenario, identify the load rate of each query engine in that query scenario.

[0122] Step 10: Select candidate query engines with a load rate lower than the load rate threshold from the various query engines in the query scenario, and map the candidate query engines as engine nodes.

[0123] Each engine node corresponds to a candidate query engine mapping.

[0124] Step 11: For each candidate query engine, obtain the historical collaboration logs corresponding to the candidate query engine.

[0125] Among them, the historical collaboration log includes historical processing data when candidate query engines collaborate with other types of query engines in query scenarios.

[0126] Step 12: Based on historical processing data, determine the collaborative response probability among the candidate query engines.

[0127] Step 13: Connect the engine nodes corresponding to each candidate query engine through the collaborative response probability between each candidate query engine to obtain the multimodal collaborative network corresponding to each query scenario.

[0128] In this embodiment of the invention, the engine load rate characterizes the resource consumption of the query engine in the current query scenario, and is calculated by comprehensively considering CPU utilization, memory usage, and concurrent connections. Engine resource usage indicators can be extracted from the system monitoring platform; the average load rate within a preset period (e.g., 5 minutes) can be aggregated and statistically analyzed by scenario-engine dimension.

[0129] The load rate threshold is the preset maximum load standard (e.g., ≤80%) for query engine nodes to be selected into the multimodal collaborative network, used to filter overloaded engines.

[0130] Candidate query engines are a subset of engines that meet the load rate criteria, serving as the basic building blocks (engine nodes) for the multimodal collaborative network.

[0131] Historical collaboration logs are time-series records of collaboration events between engines during query scenarios, including task distribution timelines, processing status, and performance metadata. The query scheduling middleware API can be called to retrieve the full collaboration processing logs of candidate engines in the target query scenario; log fields include: main engine ID, co-engine ID, task type, timestamp, and processing status code.

[0132] The collaborative response probability is the density probability of successful collaboration between engines based on historical collaborative log statistics. It can be calculated by P(A|B) = N{A successfully responds to B} / N{B distributes tasks to A}. When there is a collaborative relationship between engine A and engine B (B→A distributes tasks, A processes them), it is the density probability of successful processing of query engine A's task for query engine B based on historical collaborative log statistics.

[0133] In one possible implementation, statistics are grouped by engine pair (Engine_A, Engine_B): the total number of times Engine_B dispatches tasks to Engine_A is Ntotal(B→A); the number of times Engine_A successfully processes a task is Nsuccess(B↔A); and the collaborative response probability is calculated as: P(A|B) = Nsuccess(B↔A) / Ntotal(B→A).

[0134] Specifically, query engines with load rates below a threshold are marked as candidate query engines; a one-to-one mapping between candidate query engines and engine nodes is established, and an isolated node set is initialized. After calculating the collaborative response probability, the collaborative response probability between each engine node is obtained. All candidate engine pairs are traversed; if the collaborative response probability > 0 (i.e., a historical collaborative relationship exists), connecting edges are added to the graph structure, with the probability value P as the edge weight; a weighted directed graph is generated as the multimodal collaborative network for this query scenario.

[0135] In one possible embodiment, two engine nodes corresponding to P(A|B) = 0 will not be connected, or two engine nodes whose P(A|B) is less than a preset collaborative response probability threshold will not be connected.

[0136] In this embodiment, overloaded engines are filtered out by load rate threshold to ensure the efficiency of the collaborative network; response probability is calculated based on real historical processing data to objectively quantify the collaboration strength between engines; and networks are built independently for different query scenarios to avoid engine resource conflicts.

[0137] Optionally, step 9 specifically includes:

[0138] Step 901: Determine the average response time of each query engine in the query scenario, and the query processing volume of each query engine per unit time.

[0139] Step 902: Based on the average response time and query processing volume, determine the load rate of each query engine in the query scenario.

[0140] In this embodiment of the invention, the average response time can be the average processing time of the query engine in a single query scenario, which is the time interval from receiving the task to returning the result. The average response time of the query engine can be calculated by aggregating the task processing timestamps in real time through an engine performance monitoring service, based on query scenario slices.

[0141] Query processing volume refers to the number of query requests successfully processed by the query engine per unit of time, reflecting the engine's processing capacity. Scenario-related processing records are extracted from the query processing logs; the number of successfully processed queries is counted by engine ID.

[0142] Specifically, the system listens to the engine's task event stream and records task processing timestamps; the aggregation calculations by scenario are shown below:

[0143] ;

[0144] Among them, T avg，i Let t be the average response time of engine i in a query scenario (the time interval between receiving a result from a task). start，k Let t be the start timestamp of the k-th task. end，k Let n be the end timestamp of the k-th task, n be the total number of tasks of engine i within the statistical period, and k be the task index (with a value range of 1 ≤ k ≤ n).

[0145] Analyze the query processing logs and filter out scenario-related events; the processing volume is shown below:

[0146] ;

[0147] Among them, Qi Here, j represents the number of query requests successfully processed by engine i per unit of time (reflecting the engine's processing capacity), and j is the event index. For indicator functions (returns 1 if the condition in parentheses is met, otherwise returns 0), type j The status of the j-th query event is represented by "success", and m is the total number of query events for engine i within the statistical period.

[0148] Data normalization eliminates dimensional differences. The data normalization process is shown below:

[0149] ;

[0150] ;

[0151] Where Tnorm,i is the normalized result of the average response time of engine i (eliminating dimensional differences), T avg Let T be the set of average response times for all engines in the current query scenario, min(T) avg ) is a set T avg The minimum average response time in, max(T) avg ) is a set T avg The maximum average response time in the set is given by Qnorm,i, which is the normalized result of the query processing volume of engine i (eliminating the difference in units). Q is the set of query processing volumes of all engines in the current query scenario. min(Q) is the minimum query processing volume in the set Q, and max(Q) is the maximum query processing volume in the set Q.

[0152] Finally, the load factor can be calculated as follows:

[0153] ;

[0154] Among them, L i β is the load rate of engine i (representing the current resource utilization level), and β is the weighting coefficient (the default value is 0.6, which adjusts the contribution ratio of response time and processing volume).

[0155] In this embodiment, the engine load status is comprehensively evaluated by integrating timeliness indicators (response time) and throughput indicators (processing volume); normalization processing solves the numerical scale problem caused by the performance differences of different engines.

[0156] Optionally, step 12 specifically includes:

[0157] Step 1201: Based on historical processing data, determine the historical success rate among the various candidate query engines;

[0158] Step 1202: Determine the data compatibility factor based on the query fingerprints corresponding to each candidate query engine;

[0159] Step 1203: Determine the collaborative response probability among the candidate query engines based on historical success rate, data compatibility factor, and network latency factor.

[0160] In this embodiment of the invention, the historical success rate can be understood as the probability that query engine A has successfully processed a task from query engine B in the past. The historical success rate can be calculated using the following formula:

[0161] ;

[0162] Where Phistory represents the historical collaboration success rate between engine A and engine B (based on historical log statistics), and N total (B→A) represents the total number of times engine B dispatches tasks to engine A, N. success (A|B) represents the number of times engine A successfully processed tasks from engine B.

[0163] The data compatibility factor can be understood as a data format compatibility index between engines, calculated based on query fingerprint similarity. Specifically, the data compatibility factor can be calculated using the following formula:

[0164] ;

[0165] Among them, C AB V is the data compatibility factor between query engine A and query engine B. A V is the feature vector of the query fingerprint of engine A. B V is the feature vector of the query fingerprint of engine B, and F is the set of all query fingerprints in the multimodal query fingerprint database; k Let be the feature vector of the k-th query fingerprint in set F.

[0166] The network latency factor represents the percentage of current network transmission delay and reflects real-time communication conditions. The network latency factor is shown below:

[0167] ;

[0168] Where D is the network latency factor (reflecting real-time communication status; the larger the value, the more stable the communication), latency current The latency is the actual network delay for data transmission between current engines. max The preset maximum tolerable network latency (unit: ms; exceeding this limit significantly increases the risk of communication failure).

[0169] The collaborative response probability is a combined predicted probability based on historical success rate, data compatibility factor, and network latency factor, as shown below:

[0170] ;

[0171] Among them, P cooperative P represents the combined collaborative response probability of engine A and engine B (a predicted value fused from multiple factors). history For historical success rate, C AB Let ω be the data compatibility factor, D be the network latency factor, and ω, μ, and η be the fusion weights corresponding to the historical fusion success rate, the data compatibility factor, and the network latency factor, respectively. ω + μ + η = 1, and ω, μ, and η are all positive numbers.

[0172] In this embodiment, the limitations of single historical data are overcome by using a three-factor model. When the data format changes, the compatibility assessment can be automatically adjusted. When the network condition changes, the communication risk can be reflected in real time, making the prediction of the collaborative response probability more accurate and reliable.

[0173] Optionally, step 3 specifically includes:

[0174] Step 301: Encode the multimodal query fingerprint database into a node feature matrix, and use the multimodal collaborative network as the adjacency matrix of the node feature matrix.

[0175] In the node feature matrix, each node represents a query fingerprint, and the connecting edges between nodes represent the similarity between the query fingerprints.

[0176] Step 302: Input the node feature matrix and adjacency matrix into the pre-trained graph convolutional network and output control parameters with the goal of minimizing query latency.

[0177] Step 303: Convert the control parameters into an engine executable instruction set to obtain the adaptive collaborative control instructions corresponding to the query engine in the current query scenario.

[0178] In this embodiment of the invention, the multimodal query fingerprint database can be encoded into a matrix. N is the total number of query fingerprints (i.e., the total number of nodes in the node feature matrix), and D is the feature dimension of the query fingerprint, with each row X... i This corresponds to an encoded feature vector for a query fingerprint. In one possible embodiment, X can also be... i Perform query pattern encoding to obtain v enc,i .

[0179] The adjacency matrix corresponding to the above multimodal cooperative network is: A A,B =0, which means there is no connecting edge between query engines A and B.A,B = P(A|B) indicates that there is a connection edge between query engine A and B.

[0180] After obtaining the node feature matrix and adjacency matrix, the node feature matrix and adjacency matrix can be input into the pre-trained graph convolutional network for processing to obtain the prediction results. The prediction results include the query execution time sequence and data transformation path of each engine node.

[0181] Specifically, the pre-trained graph convolutional network includes a parallel first input layer, a second input layer, a feature fusion layer, and an output layer. The input to the first input layer is a node feature matrix, and the input to the second input layer is an adjacency matrix. The first input layer performs graph convolution on the node feature matrix to obtain a first feature map, and the second input layer performs graph convolution on the adjacency matrix to obtain a second feature map. The feature fusion layer performs channel fusion or concatenation fusion on the first and second feature maps to obtain fused features. The output layer maps the fused features to a control parameter matrix, which includes the control parameters corresponding to each query engine. The control parameter matrix is ​​parsed according to the engine, and the encoder converts the parameter vectors into executable instructions for the engine. It should be noted that the structure of the control parameter matrix is ​​the same as that of the adjacency matrix, the difference being that the information corresponding to the engine nodes in the control parameter matrix is ​​the control parameter information.

[0182] In this embodiment, the node feature matrix encodes query heterogeneity, and the adjacency matrix encoding engine coordinates relationships. The graph structure perception engine utilizes topological relationships to optimize query execution timing and data transformation paths. Dual-source input overcomes the problem of separation between query features and topology in traditional methods, and better realizes multimodal query adaptive control in query scenarios.

[0183] Optionally, prior to step 302, the method further includes:

[0184] Step 304: Construct the training dataset and the convolutional network for the graph to be trained.

[0185] The training dataset includes: a sample node feature matrix encoded from a multimodal query fingerprint database, historical multimodal collaborative networks corresponding to different query scenarios, and actual control parameters and real query delays corresponding to the historical multimodal collaborative networks. In the sample node feature matrix, each sample node represents a query fingerprint, and the connection edges between sample nodes represent the similarity between the query fingerprints.

[0186] The convolutional network for the training graph consists of a first input layer, a second input layer, a feature fusion layer, and an output layer, all in parallel.

[0187] Step 305: Input the sample node feature matrix and the historical multimodal collaborative network into the convolutional network of the graph to be trained to predict the query delay, and obtain the prediction control parameter matrix and the prediction query delay.

[0188] Step 306: Calculate the first error loss between the predicted query latency and the actual query latency, and the second error loss between the actual control parameters and the predicted control parameter matrix using the loss function. Add the first error loss and the second error loss to obtain the total error loss. With minimizing the total error loss as the optimization objective, adjust the network parameters of the graph convolutional network to be trained using the backpropagation algorithm, and iterate the adjustment process of the network parameters until the training stopping condition is met, and then stop training to obtain the pre-trained graph convolutional network.

[0189] In this embodiment of the invention, the sample node feature matrix and the node feature matrix are constructed in the same way, and will not be described in detail here. For historical multimodal collaborative networks, it can also be converted into a sample adjacency matrix. The output of the graph convolutional network is the prediction parameter control matrix and the prediction query delay.

[0190] The total response time (i.e., actual query latency) for the query engine cluster to complete collaborative query tasks is calculated using the following formula:

[0191] ;

[0192] Among them, T real The time taken to complete the collaborative query task represents the actual query latency, k represents the k-th query engine participating in the collaboration in the multimodal collaborative network, and t represents the time taken to complete the collaborative query task. start This indicates the global start time of the collaborative query, specifically the timestamp of the first query command leaving the query gateway, t. k，end E represents the time (actual completion time) when query engine k completes this query task, and E represents the set of all query engines participating in the collaboration.

[0193] The sample node feature matrix is ​​input into the first input layer of the graph convolutional network to be trained for graph convolution processing to obtain the first sample feature map. The sample adjacency matrix is ​​input into the second input layer of the graph convolutional network to be trained for processing to obtain the second sample feature map. The first sample feature map and the second sample feature map are fused or concatenated through the feature fusion layer of the graph convolutional network to obtain the fused features. The fused features are mapped into a prediction control parameter matrix through the output layer and output. At the same time, the prediction query delay corresponding to the prediction control parameter matrix is ​​also output. The control parameter matrix includes the control parameters corresponding to each query engine.

[0194] The weighted mean square error function can be used to calculate the first error loss between the predicted query delay and the actual query delay, and to calculate the second error loss between the actual control parameters and the predicted control parameter matrix.

[0195] Specifically, the loss function described above is as follows:

[0196] ;

[0197] ;

[0198] Where Loss1 represents the first error loss, and M is the number of samples. T represents the query latency weight for the i-th sample. pred，i T represents the prediction query latency for the i-th sample. real，i The value represents the actual query latency for the i-th sample, Loss2 represents the second error loss, and N is the number of engine nodes. This represents the control parameter weights for the i-th sample. This represents the predictive control parameters of engine k for engine j in the i-th sample. This represents the actual control parameters of engine k over engine j in the i-th sample.

[0199] The first error loss and the second error loss are directly added together or weighted to obtain the total error loss. Minimizing this total error loss is the optimization objective. The network parameters of the convolutional network to be trained are adjusted using the backpropagation algorithm. This adjustment process is iterated until a preset number of iterations is reached or the total error loss is less than a preset error loss value. Training then stops, resulting in a pre-trained convolutional network. The output of the pre-trained convolutional network is a prediction control parameter matrix. The predicted query latency may or may not be output. However, if the predicted query latency is output, it can be used to determine whether the prediction control parameter matrix meets the user's expectations.

[0200] By constructing a training dataset and training the graph convolutional network to be trained, a pre-trained graph convolutional network for predicting the control parameter matrix can be trained, thereby improving the accuracy of the nonlinear solution of the control parameter matrix.

[0201] Optionally, step 306 specifically includes: calculating the weighted mean square error between the predicted query latency and the actual query latency; determining the multimodal collaborative loss factor based on the historical multimodal collaborative network; determining the feature constraint factor based on the feature matrix of the sample nodes; and calculating the error loss between the predicted query latency and the actual query latency based on the weighted mean square error, the multimodal collaborative loss factor, and the feature constraint factor.

[0202] In this embodiment of the invention, since the actual query delay is highly correlated with the actual control parameter matrix, in order to avoid overfitting, this embodiment improves the loss function of the first error loss, specifically by considering the fusion of topological relationship penalty factor and prior feature constraint, thereby avoiding overfitting.

[0203] Specifically, the first error loss can be calculated using the following loss function:

[0204] ;

[0205] in, ;

[0206] ;

[0207] G i Let E represent the multimodal cooperative network of the i-th sample. i Let represent the set of all connection edges in the multimodal cooperative network of the i-th sample. Let deg(j) represent the predicted query latency of engine node j in the i-th sample, deg(j) represent the degree of engine node j (the number of edges connected to engine node j), and deg(k) represent the degree of engine node k (the number of edges connected to engine node k). X represents the historical average collaborative latency difference between engine node k and engine node j. i This represents the node feature matrix of the i-th sample. Let represent the baseline feature matrix for the c-th query scenario, where λ1 and λ2 represent smoothing coefficients used to balance the weights among the items. The trace of a matrix is ​​the sum of the elements on its main diagonal. Let L denote the transpose of the node characteristic matrix, and let L denote the Laplace matrix of the node characteristic matrix. This represents the Frobenius norm.

[0208] In this embodiment, the integration of topological relationship penalty factor and prior feature constraint is considered to avoid overfitting and improve the model's generalization ability.

[0209] This invention provides an intelligent engine collaboration method for multimodal queries. By constructing a multimodal query fingerprint database to accurately characterize the query features of heterogeneous data sources, utilizing a graph-structured collaborative network to quantify the collaborative relationships between engines, and combining a pre-trained graph convolutional network to dynamically optimize query paths and control parameters, this method effectively solves technical challenges in multi-source data queries, such as poor protocol compatibility, low engine collaboration efficiency, and high response latency. Specifically, this method reduces query latency by more than 30% and improves query accuracy to 95% through a collaborative mechanism of data compatibility factors and network state adaptation. Especially in smart city traffic management scenarios, it can simultaneously perform millisecond-level collaborative analysis of surveillance video (unstructured feature query), sensor traffic flow (quantitative indicator query), accident reports (text query), and geofencing (spatial range query), rapidly generating holistic traffic accident assessment reports and significantly improving emergency response efficiency.

[0210] Accordingly, such as Figure 2 As shown, based on a multimodal query-oriented intelligent engine collaboration method, this embodiment of the invention also provides a multimodal query-oriented intelligent engine collaboration system to implement the multimodal query-oriented intelligent engine collaboration method of this embodiment of the invention. The system includes:

[0211] The query dimension parsing module is used to generate query parsing results based on the received multimodal query requests;

[0212] The engine task allocation module is used to dynamically allocate query tasks to matching engine instances based on the query parsing results.

[0213] The engine execution module is used for parallel or serialized processing of tasks by various engines.

[0214] The multi-source data association and fusion module is used to integrate and fuse results from heterogeneous engines;

[0215] The results visualization encapsulation module is used to convert the fusion results into interactive display data for users;

[0216] The performance monitoring and fingerprint update module is used to monitor query performance and update the query fingerprint database in real time.

[0217] The performance monitoring and fingerprint update module handles performance anomalies according to their types before updating and querying the fingerprint database, including:

[0218] Real-time monitoring of multimodal query response success rate and query processing latency;

[0219] When the response success rate is lower than the success rate threshold, or the query processing latency is greater than the latency threshold, extract the query characteristics of abnormal query requests;

[0220] The similarity between the query features and the query fingerprints in the multimodal query fingerprint database is calculated.

[0221] If the calculated maximum similarity is lower than the similarity threshold, the query fingerprint corresponding to the maximum similarity in the multimodal query fingerprint database will be updated.

[0222] Among them, the query characteristics of extracting abnormal query requests include:

[0223] Obtain the query data type, query complexity, response latency requirements, and semantic relevance of abnormal queries as query features;

[0224] The similarity calculation between query features and query fingerprints includes:

[0225] Calculate the cosine similarity between the anomaly query feature vector and the query fingerprint vector;

[0226] This includes updating the query fingerprint, including:

[0227] The abnormal feature vector and the original fingerprint vector are used to calculate a moving average, and the updated cluster center feature vector is generated to cover the original query fingerprint.

[0228] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A collaborative method for intelligent engines oriented towards multimodal queries, characterized in that, include: Step 1: Based on the received multimodal query requests, including text queries, spatial range queries, quantitative index queries, and unstructured data feature queries, perform query dimension parsing operations to generate query parsing results containing the query data type, target calculation dimension, and corresponding engine identifier; Step 1 includes: Step 101: Obtain historical query log data of multimodal query requests under different query scenarios, including intelligent customer service Q&A scenarios, geographic information system retrieval scenarios, business intelligence analysis scenarios, or multimedia content retrieval scenarios. Step 102: Parse the historical query log data to obtain query data type, query complexity, response latency requirements, and semantic relevance as query features; Step 103: Perform unsupervised clustering on the query features to generate query feature clusters; Step 104: Extract the cluster center feature vectors corresponding to each query feature cluster as query fingerprints and store them in the multimodal query fingerprint database. Each query scenario corresponds to one query fingerprint. Step 2: Based on the query parsing results, execute the engine task allocation operation and send engine processing instructions to the relational database engine, vector database engine, spatial GIS database engine, and indicator analysis engine that match the identifiers. Step 3: According to the engine processing instructions, the relational database engine performs structured data filtering and association operations to generate a structured result set, the vector database engine performs feature vector comparison operations to generate feature matching results, the spatial GIS database engine performs spatial overlay analysis and range filtering operations to generate spatial analysis results, and the indicator analysis engine performs indicator formula calculation and threshold verification operations to generate indicator calculation results. Step 4: Based on the structured result set, feature matching results, spatial analysis results, and index calculation results, perform multi-source data association and fusion operations. The operations include data format normalization, semantic consistency verification, and dynamic weight allocation to generate multimodal collaborative query results. Step 5: Based on the multimodal collaborative query results, perform a result visualization encapsulation operation to generate structured display data that meets user interaction requirements, and then feed the display data back to the user terminal.

2. The intelligent engine collaboration method as described in claim 1, characterized in that, Relational database engines are adapted for structured data retrieval tasks, vector database engines are adapted for unstructured data similarity matching tasks, spatial GIS database engines are adapted for geospatial topology analysis tasks, and indicator analysis engines are adapted for multi-dimensional quantitative indicator calculation tasks.

3. The intelligent engine collaboration method as described in claim 1, characterized in that, The following steps are included after step 1: Step 105: Monitor the response success rate and query processing latency of multimodal queries in real time; Step 106: When the response success rate is lower than the success rate threshold, or the query processing latency is greater than the latency threshold, extract the query features of the abnormal query requests. Step 107: Calculate the similarity between the query features and the query fingerprints in the multimodal query fingerprint database. If the calculated maximum similarity is lower than the similarity threshold, update the query fingerprint corresponding to the maximum similarity in the multimodal query fingerprint database.

4. The intelligent engine collaboration method as described in claim 3, characterized in that, Step 106 extracts query features from abnormal query requests, including obtaining the query data type, query complexity, response latency requirements, and semantic relevance of the abnormal query as a query feature quadruple; Step 107 calculates the similarity between the query features and the query fingerprint, including calculating the cosine similarity between the abnormal query feature vector and the query fingerprint vector; Step 107 updates the query fingerprint, including performing a moving average calculation between the abnormal feature vector and the original fingerprint vector to generate an updated cluster center feature vector that covers the original query fingerprint, wherein the weight ratio of the new and old in the moving average calculation is 6:

4.

5. The intelligent engine collaboration method as described in claim 1, characterized in that, The steps preceding step 2 also include: Step 9: For each query scenario, identify the load rate of each query engine in that query scenario; Step 10: Select candidate query engines with a load rate lower than the load rate threshold from the various query engines in the query scenario, and map the candidate query engines as engine nodes; Step 11: For each candidate query engine, obtain the historical collaboration logs corresponding to the candidate query engine; Step 12: Based on historical processing data, determine the collaborative response probability among various candidate query engines; Step 13: Connect the engine nodes corresponding to each candidate query engine through the collaborative response probability between each candidate query engine to obtain the multimodal collaborative network corresponding to each query scenario.

6. The intelligent engine collaboration method as described in claim 5, characterized in that, Step 9 includes: Step 901: Determine the average response time of each query engine in the query scenario, and the query processing volume of each query engine per unit time. Step 902: Based on the average response time and query processing volume, determine the load rate of each query engine in the query scenario; Step 12 includes: Step 1201: Based on historical processing data, determine the historical success rate among the various candidate query engines; Step 1202: Determine the data compatibility factor based on the query fingerprints corresponding to each candidate query engine; Step 1203: Determine the collaborative response probability among the candidate query engines based on historical success rate, data compatibility factor, and network latency factor.

7. The intelligent engine collaboration method as described in claim 1, characterized in that, Step 3 includes: Step 301: Encode the multimodal query fingerprint database into a node feature matrix, and use the multimodal collaborative network as the adjacency matrix of the node feature matrix. Each node in the node feature matrix represents a query fingerprint, and the connection edges between nodes represent the similarity between the query fingerprints. Step 302: Input the node feature matrix and adjacency matrix into the pre-trained graph convolutional network and output control parameters with the goal of minimizing query latency; Step 303: Convert the control parameters into an engine executable instruction set to obtain the adaptive collaborative control instructions corresponding to the query engine in the current query scenario.

8. A collaborative intelligent engine system for multimodal queries, used to implement the method described in any one of claims 1-7, characterized in that, The system includes: The query dimension parsing module is used to generate query parsing results based on the received multimodal query requests; The engine task allocation module is used to dynamically allocate query tasks to matching engine instances based on the query parsing results. The engine execution module is used for parallel or serialized processing of tasks by various engines. The multi-source data association and fusion module is used to integrate and fuse results from heterogeneous engines; The results visualization encapsulation module is used to convert the fusion results into interactive display data for users; The performance monitoring and fingerprint update module is used to monitor query performance and update the query fingerprint database in real time.

9. The system according to claim 8, characterized in that, The performance monitoring and fingerprint update module is used for: Real-time monitoring of multimodal query response success rate and query processing latency; When the response success rate is lower than the success rate threshold, or the query processing latency is greater than the latency threshold, extract the query characteristics of abnormal query requests; The similarity between the query features and the query fingerprints in the multimodal query fingerprint database is calculated. If the calculated maximum similarity is lower than the similarity threshold, the query fingerprint corresponding to the maximum similarity in the multimodal query fingerprint database will be updated. When the performance monitoring and fingerprint update module extracts query features from abnormal query requests, it obtains the query data type, query complexity, response latency requirements, and semantic relevance of the abnormal query as query features; when performing similarity calculation, it calculates the cosine similarity between the abnormal query feature vector and the query fingerprint vector; when updating the query fingerprint, it performs a moving average calculation between the abnormal feature vector and the original fingerprint vector.

Images