An efficiency self-adaptive optimization method and system for data retrieval

Abstract

The application discloses an efficiency adaptive optimization method and system for data retrieval, and relates to the technical field of data management.The efficiency adaptive optimization system for data retrieval comprises a mode acquisition module, a task evaluation module, a cost prediction module, a strategy selection module, a consensus retrieval module, an execution decision module, a cache management module and a model updating module.The application introduces a task evaluation model based on a convolutional neural network to accurately evaluate the uncertainty level of a retrieval task and provide risk perception support for strategy selection and execution mode.The application uses a query cost prediction model to predict resource consumption, execution time delay and path complexity under multiple standard retrieval strategies, and outputs a confidence score to realize forward-looking evaluation of the execution effect of the strategy.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to an efficiency adaptive optimization method and system for data retrieval. Background Technology

[0002] With the development of big data and artificial intelligence technologies, information retrieval systems are widely used in various fields such as search engines, intelligent question answering, knowledge graphs, and database management. Existing information retrieval technologies typically rely on fixed index structures and preset search paths. When executing a retrieval task, they match data items based on the user's query and return relevant results. Some systems further combine similarity calculations, ranking algorithms, and simple caching strategies to improve result relevance and response speed. In standardized scenarios, these solutions can meet basic data access needs.

[0003] However, with the diversification of data types, the dynamism of user behavior, and the increasing complexity of retrieval tasks, traditional retrieval systems face problems such as declining response efficiency, inefficient resource scheduling, and unreasonable execution path selection. On the one hand, existing systems lack in-depth modeling of historical access behavior, making it impossible to effectively mine user access patterns to guide path selection for the current task. On the other hand, retrieval path configurations are often statically set, making it difficult to adaptively adjust according to task uncertainties or system states, leading to resource waste or query delays. Furthermore, the lack of a parallel scheduling mechanism based on task dependencies during multi-stage or multi-link task execution also limits further improvements in overall system efficiency.

[0004] How to adaptively select search paths and dynamically optimize task execution based on historical access patterns and task characteristics, when faced with complex task structures, dynamically changing resource status, and high uncertainty in query feedback, is a technical challenge that remains to be solved. Summary of the Invention

[0005] This invention proposes an adaptive optimization method that combines historical access characteristics and task structure information to dynamically select the optimal execution path based on the uncertainty characteristics of the retrieval task, and implements task partitioning and parallel scheduling during the execution process, so as to improve the overall response performance and resource utilization efficiency of the information retrieval system in complex dynamic environments.

[0006] An efficiency adaptive optimization method for data retrieval includes:

[0007] The system acquires historical retrieval records, performs feature extraction and pattern modeling, generates an access pattern model, and calculates the trajectory index of the access pattern model based on the data features of the historical retrieval records.

[0008] After receiving the retrieval task, the retrieval task is analyzed based on the task evaluation model to obtain the uncertainty level; based on the query cost prediction model and combined with the uncertainty level, the cost prediction results and corresponding confidence scores of the retrieval task under different standard retrieval strategies are obtained.

[0009] Based on the access pattern model and the corresponding trajectory index, cost prediction results and corresponding confidence scores, the target retrieval path used by the retrieval task is adaptively selected to obtain the strategy selection results.

[0010] After obtaining the strategy selection results, if the combination of uncertainty level and confidence score meets the risk decision-making conditions, then the retrieval is performed through cross-modal consensus verification.

[0011] Otherwise, a data index structure is selected from the dynamic index structure library based on the strategy selection result; based on the strategy selection result and the data index structure, the retrieval process corresponding to the retrieval task is divided into task stages, and the dependencies between task stages are analyzed. Based on the uncertainty level, confidence score, dependencies and the obtained system computing resource status, a parallel execution decision is generated.

[0012] Execute retrieval tasks and collect cache hit information. Based on the access pattern model, corresponding trajectory index and cache hit information, selectively cache the retrieval tasks, update the dynamic index structure library, and adjust the cache eviction policy.

[0013] Output the search results and collect response data during the execution process to update the task evaluation model and query cost prediction model.

[0014] As a preferred technical solution of the present invention, the retrieval through cross-modal consensus verification includes: activating multiple heterogeneous retrieval paths in parallel, wherein the heterogeneous retrieval paths use different data index structures, similarity measurement criteria and data modalities to perform retrieval operations on the retrieval task, and respectively obtain corresponding candidate result sets; performing cross-validation and consensus calculation on the candidate result sets to obtain retrieval results.

[0015] As a preferred technical solution of the present invention, the cross-validation and consensus calculation of the candidate result set includes: comparing the intersection, semantic similarity and structural similarity between each candidate result set based on the preset consensus calculation criteria, obtaining a subset of consensus results and using it as the retrieval result; or clustering the candidate result set, filtering the candidate result set according to the characteristics of the clustering results, and performing semantic fusion on the filtered candidate result set to generate the retrieval result.

[0016] As a preferred technical solution of the present invention, the generation of the access pattern model includes: extracting target fields, query semantic structure and time distribution features from historical retrieval records, constructing an access pattern model representing the regularity of query behavior; and calculating the trajectory index based on the corresponding data volume, query feedback and query response time sequence distribution in the historical retrieval records.

[0017] As a preferred technical solution of the present invention, the acquisition of uncertainty level includes: analyzing the structural features, query semantic features and historical feedback features of the retrieval task through a task evaluation model built on a convolutional neural network, and calculating the uncertainty level based on the comprehensive evaluation results of the three types of features; the task evaluation model is obtained by supervised training with the uncertainty level corresponding to the execution results of historical retrieval tasks as the training target.

[0018] As a preferred embodiment of the present invention, the step of obtaining the cost prediction results and corresponding confidence scores of the retrieval task under different standard retrieval strategies includes: analyzing the structure and target fields of the retrieval task using a query cost prediction model built based on a convolutional neural network, and combining the uncertainty level as a joint input; jointly evaluating the query cost of each standard retrieval strategy for multiple standard retrieval strategies; outputting the cost prediction results and confidence scores corresponding to each standard retrieval strategy respectively, wherein the confidence score is generated based on the confidence estimate of the cost prediction results output by the query cost prediction model, and the cost prediction results include resource consumption estimation, execution latency estimation, or path complexity assessment; the query cost prediction model is trained using supervised learning with the query cost of historical retrieval tasks under each standard retrieval strategy as the training target.

[0019] As a preferred technical solution of the present invention, obtaining the strategy selection result includes: comprehensively evaluating different retrieval path configuration schemes based on the query behavior preference information provided by the access pattern model and the access stability represented by the corresponding trajectory index, combined with the query cost prediction results and confidence scores corresponding to each standard retrieval strategy; based on the comprehensive evaluation result, constructing or selecting the target retrieval path with the highest matching degree with the current retrieval task from the preset retrieval path generation rules, as the strategy selection result.

[0020] As a preferred embodiment of the present invention, the generation of parallel execution decisions includes: dividing the retrieval task into multiple task segments to be executed based on the strategy selection result and the data index structure; constructing a task dependency graph, analyzing the data input-output relationship between each task segment, identifying serial dependency paths and parallel task sets, and obtaining dependency relationships; and combining the uncertainty level of the current retrieval task, the confidence score of the strategy selection result, and the obtained system computing resource status to determine the parallel execution granularity, grouping method, and scheduling order of the task segments, thereby generating parallel execution decisions.

[0021] As a preferred technical solution of the present invention, the selective caching of retrieval tasks, updating of the dynamic index structure library, and adjustment of cache eviction strategy include: evaluating the probability of result reuse of the current retrieval task based on the access pattern model and the corresponding trajectory index, and determining whether to write the retrieval result into the cache in combination with cache hit information to achieve selective caching; after executing the retrieval task, updating the status index of the corresponding index structure in the dynamic index structure library according to the access frequency, hit performance and feedback results of the data index structure used; and adjusting the parameter configuration of the cache eviction strategy according to the evolution trend of cache hit information and access pattern model to dynamically control the cache content retention period and eviction priority.

[0022] An efficiency adaptive optimization system for data retrieval includes:

[0023] Pattern acquisition module: Acquires historical retrieval records, performs feature extraction and pattern modeling, generates access pattern models, and calculates the trajectory index of the access pattern models based on the data features of historical retrieval records;

[0024] Task evaluation module: After receiving a retrieval task, it analyzes the retrieval task based on the task evaluation model to obtain the uncertainty level;

[0025] Cost prediction module: Based on the query cost prediction model and combined with the uncertainty level, it obtains the cost prediction results and corresponding confidence scores of the retrieval task under different standard retrieval strategies;

[0026] Strategy selection module: Based on the access pattern model and corresponding trajectory index, cost prediction results and corresponding confidence scores, it adaptively selects the target retrieval path used by the retrieval task to obtain the strategy selection result;

[0027] Consensus retrieval module: After obtaining the strategy selection results, if the combination of uncertainty level and confidence score meets the risk decision-making conditions, then the retrieval is performed through cross-modal consensus verification;

[0028] Execution decision module: Selects the data index structure, divides the retrieval process corresponding to the retrieval task into task stages, analyzes the dependencies between task stages, and generates parallel execution decisions;

[0029] Cache management module: Executes retrieval tasks and collects cache hit information. Based on the access pattern model, corresponding trajectory index and cache hit information, it selectively caches retrieval tasks, updates the dynamic index structure library, and adjusts the cache eviction policy.

[0030] Model update module: Outputs search results and collects response data during execution to update the task evaluation model and query cost prediction model.

[0031] The present invention has the following advantages:

[0032] This invention effectively captures the regularity and stability of historical retrieval behavior by constructing an access pattern model and calculating a trajectory index, making retrieval path selection more targeted and improving the accuracy of strategy matching.

[0033] This invention introduces a task evaluation model based on convolutional neural networks to accurately assess the uncertainty level of retrieval tasks, providing risk awareness support for strategy selection and execution methods, and enhancing robustness in complex query scenarios. By using a query cost prediction model, it predicts resource consumption, execution latency, and path complexity under multiple standard retrieval strategies and outputs confidence scores, thereby achieving a forward-looking evaluation of the strategy execution effect and optimizing the strategy selection process.

[0034] This invention achieves adaptive selection of retrieval paths by integrating access pattern models, cost prediction results, and uncertainty levels, avoiding resource waste and performance bottlenecks caused by static strategies and improving retrieval efficiency. Through a cross-modal consensus verification mechanism, multiple heterogeneous retrieval paths are triggered in parallel under high uncertainty conditions, and cross-validation and semantic fusion are performed to ensure the accuracy and consistency of the final retrieval results.

[0035] This invention generates granular and flexible parallel execution decisions by constructing a task dependency graph and combining it with computing resource status, enabling efficient parallel processing of complex retrieval tasks in a multi-core environment and shortening the overall execution time. Through the update and maintenance mechanism of the dynamic index structure library, combined with access frequency and hit feedback, the data index structure is continuously optimized.

[0036] This invention combines trajectory index and cache hit information to implement selective caching and eviction strategies, which dynamically adjusts the cache hit rate and extends the residence time of high-value data, thereby improving overall response efficiency. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only schematic diagrams of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the structure of an efficiency adaptive optimization system for data retrieval used in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0040] Example 1: An efficiency adaptive optimization method for data retrieval, comprising the following steps:

[0041] Step S1: Obtain historical retrieval records, perform feature extraction and pattern modeling, generate an access pattern model, and calculate the trajectory index of the access pattern model based on the data features of historical retrieval records;

[0042] The generated access pattern model includes: extracting target fields, query semantic structure and time distribution features from historical retrieval records to construct an access pattern model representing the regularity of query behavior; and calculating the trajectory index based on the corresponding data volume, query feedback and query response time distribution in historical retrieval records.

[0043] In one embodiment of the present invention, historical retrieval records refer to structured query execution logs that are automatically collected and stored by the system during long-term operation. The data originates from retrieval tasks submitted by users on the front end or within the system. Each record includes at least: the original query statement, the target field set, the query logic structure (such as Boolean logic, nested structure), the query timestamp, the index type used in the execution, the number of hit results, the response time, and user feedback information (such as the result click-through rate, whether the query was modified again afterward, etc.).

[0044] In the feature extraction process, the original query statement is first parsed to extract structured field information, and the semantic structure of the query is analyzed, including the logical relationships between fields (e.g., the depth of AND / OR nesting) and semantic patterns such as the types of operators used (range, fuzzy, equality). Simultaneously, the distribution characteristics of the query over time are extracted, including the frequency, periodicity, and time-series concentration of query requests, in order to subsequently model the temporal tendency of query behavior.

[0045] Based on the extracted structural, semantic, and temporal features, an access pattern model is constructed. An access pattern model is a high-level abstract structure used to express the patterns of user query behavior. Essentially, it is a set of models representing clustering patterns of query behavior. For example, a Hidden Markov Model (HMM) can be used to model the structural and semantic combinations of user queries, along with labels such as frequency, temporal density, and confidence level of field combinations.

[0046] The trajectory index is a quantitative indicator used to characterize the stability and trajectory evolution trend of query behavior, and is one of the innovative parameters proposed in this invention. Its calculation is based on: the similarity of query paths for similar query tasks within a certain time window (such as field combination similarity and index usage overlap), the degree of semantic evolution (keyword change rate), and the volatility of query results (variance of the number of hits and response latency). The trajectory index ranges from [0,1]. A higher value indicates that the query type has historically exhibited a stable and highly repetitive behavioral trajectory, making it more suitable for using historically optimal strategies; a lower value indicates that the query type exhibits divergent or dynamically changing behavioral characteristics, making it suitable for dynamic evaluation.

[0047] For example, if a certain type of query frequently appears within a certain time period and the field structure is stable (such as fixed filtering by region + sales amount), its hit results are stable and the response time fluctuates little, then the trajectory index is close to 1; conversely, if a certain type of query field combination changes frequently, the keyword semantics are ambiguous, and the number of hits is unstable, then the trajectory index is close to 0.

[0048] The access pattern model and trajectory index will play a key role in the subsequent strategy selection (step S4) and selective caching (step S7) processes, and will be used to determine whether the retrieval task belongs to a reusable path, a high-confidence pattern, or needs to enter the risk analysis and cross-modal consensus process.

[0049] Step S2: After receiving the retrieval task, analyze the retrieval task based on the task evaluation model to obtain the uncertainty level;

[0050] The acquisition of the uncertainty level includes: analyzing the structural features, query semantic features, and historical feedback features of the retrieval task using a task evaluation model built on a convolutional neural network, and calculating the uncertainty level based on the comprehensive evaluation results of the three types of features; the task evaluation model is trained under supervised conditions using the uncertainty levels corresponding to the execution results of historical retrieval tasks as the training objective. The structural features include the number of query fields and the complexity of logical relationships between fields; the query semantic features include keyword ambiguity and semantic ambiguity; and the historical feedback features include the hit rate and result volatility related to similar queries.

[0051] In one embodiment of the present invention, structural features refer to the composition and expression complexity of the current retrieval task at the field level, specifically including the number of fields involved in the query, field types (such as numeric, enumerated, and text types), the number of Boolean logic nesting levels between fields, and whether subqueries exist. By constructing a structural vector representation, the above dimensions are mapped to fixed-length tensors for model input, i.e., structural feature tensors.

[0052] Query semantic features refer to the degree of uncertainty at the linguistic level in the current retrieval task, including whether keywords are ambiguous (e.g., "apple" refers to a fruit or a company), semantic ambiguity between keywords (e.g., "recent" or "higher"), and unspecified conditions in natural language queries. Semantic ambiguity is assessed by embedding the query phrase using a language model (e.g., a pre-trained BERT sub-model) and then evaluating its semantic clarity through similarity distribution, thereby obtaining an ambiguity score, i.e., a semantic feature vector.

[0053] Historical feedback features are derived from the performance of retrieval tasks with similar semantic structures to the current task in the historical retrieval record database. These features mainly include: the average hit rate of similar queries in history, the degree of fluctuation in the number of response results (such as standard deviation or coefficient of variation), and user behavior feedback (such as whether there are duplicate queries or whether queries are quickly switched). These indicators together characterize the "output stability" of this type of query in history, i.e., feedback statistics.

[0054] The task evaluation model employs a convolutional neural network (CNN) architecture. The input is a multi-channel input tensor composed of the aforementioned structural feature tensor, semantic feature vectors, and feedback statistics. After multiple convolutional and pooling operations, latent combined features are extracted. Finally, a normalized uncertainty score, ranging from [0,1], is output through a fully connected layer. This score is categorized into multiple uncertainty levels (e.g., low, medium, relatively high, high) and serves as a key basis for path selection and consensus judgment in subsequent decision-making.

[0055] The model training phase employs supervised learning, with training data derived from a large number of completed historical retrieval task samples. These samples are labeled with their corresponding uncertainty levels based on performance characteristics (such as the deviation between predicted and actual costs, the stability of hit results, and user satisfaction) by a rule-based model or expert system. The training objective is to minimize the cross-entropy loss function between the model's predicted uncertainty and the true labels.

[0056] Tasks with high uncertainty levels will enter the cross-modal consensus verification path in subsequent processes to ensure the accuracy and consistency of retrieval results; tasks with low uncertainty levels will be given priority to enter the standard retrieval strategy path to improve execution efficiency.

[0057] For example, if a retrieval task contains multiple nested queries, has semantically ambiguous keywords, and is highly similar to historically high-volatility tasks, its uncertainty level will be assessed as "high"; while tasks with simple structure, clear semantics, and stable historical performance will be assessed as "low".

[0058] Step S3: Based on the query cost prediction model and combined with the uncertainty level, obtain the cost prediction results and corresponding confidence scores of the retrieval task under different standard retrieval strategies;

[0059] The process of obtaining cost prediction results and corresponding confidence scores for retrieval tasks under different standard retrieval strategies includes: using a query cost prediction model built on a convolutional neural network, analyzing the structure and target fields of the retrieval task, and combining uncertainty level as a joint input, jointly evaluating the query cost of each standard retrieval strategy; outputting the cost prediction results and confidence scores for each standard retrieval strategy respectively, wherein the confidence score is generated based on the confidence estimate of the cost prediction results output by the query cost prediction model, and the cost prediction results include resource consumption estimation, execution latency estimation, or path complexity assessment; the query cost prediction model is trained using supervised learning with the query cost of historical retrieval tasks under each standard retrieval strategy as the training target.

[0060] In one embodiment of the present invention, the standard retrieval strategy refers to a variety of predefined typical retrieval execution schemes, including but not limited to: sequential scan (full table scan), inverted index retrieval, multidimensional index retrieval (such as R-tree, KD-tree), vector index retrieval (such as high-dimensional retrieval based on ANN algorithm), graph index query, and distributed joint query strategy, etc. Each strategy has different performance characteristics under different types of retrieval tasks, therefore, independent cost modeling is required for each type of strategy.

[0061] The query cost prediction model is one of the core components proposed in this invention for evaluating the cost of retrieval strategies. Essentially, it is a convolutional neural network model with multi-task output capabilities. The input layer receives structural features (number of fields, field types, logical complexity, etc.), target field attributes (data type, distribution characteristics), and the uncertainty level score obtained in step S2. Through multi-layer convolutional operations, it extracts latent semantic and structural interaction features, outputting query cost estimates for multiple retrieval strategies respectively.

[0062] The cost prediction results include the following three core indicators: resource consumption estimation, including estimated CPU usage (such as core time), memory usage, and number of I / O operations; execution latency estimation, which refers to the expected response time to complete the current task under this strategy, in milliseconds or seconds; and path complexity assessment, which represents the structural complexity of the task execution path, and comprehensively considers indicators such as query plan length, number of index access layers, and number of connection nodes to form a complexity score.

[0063] The confidence score is used to represent the reliability of the model's cost prediction results for each policy. Its calculation is based on an output layer confidence estimation mechanism, such as Bayesian confidence estimation. The confidence score typically ranges from [0,1], with values ​​closer to 1 indicating more reliable cost prediction results. This score will be used as a weighting factor in subsequent policy path selection.

[0064] The model training data comes from recorded historical retrieval task execution logs. These logs contain the actual performance of each retrieval task under multiple strategies, such as actual response time and resource usage information. Supervised learning is employed during training, using the actual cost corresponding to each strategy as a label to minimize the loss function (such as mean squared error, MSE, or weighted L1 / L2 loss) between the predicted and actual values. To enhance generalization ability, data augmentation mechanisms and regularization terms are introduced during training.

[0065] For example, for a query task with moderate query field complexity and an uncertainty level of 0.7, the model outputs the following results: Strategy A (inverted index) has an estimated cost of 85ms and a confidence level of 0.92; Strategy B (multidimensional index) has an estimated cost of 130ms and a confidence level of 0.78; Strategy C (full table scan) has an estimated cost of 500ms and a confidence level of 0.99. These results are then used to select the appropriate strategy.

[0066] Step S4: Based on the access pattern model and the corresponding trajectory index, cost prediction results and corresponding confidence scores, adaptively select the target retrieval path used by the retrieval task to obtain the strategy selection result;

[0067] The strategy selection result includes: comprehensively evaluating different retrieval path configuration schemes based on the query behavior preference information provided by the access pattern model and the access stability represented by the corresponding trajectory index, combined with the query cost prediction results and confidence scores of each standard retrieval strategy; and constructing or selecting the target retrieval path with the highest matching degree to the current retrieval task from the preset retrieval path generation rules as the strategy selection result. The comprehensive evaluation includes weighted calculation of the frequency of occurrence and time distribution characteristics of the query targets covered by the path in history, the resource consumption and response latency reflected by the cost prediction value, and the reliability of the strategy prediction reflected by the confidence score.

[0068] In one embodiment of the present invention, the target retrieval path refers to the optimal combination of execution paths selected in the strategy configuration space based on the characteristics of the current retrieval task. This path not only determines the data index structure, execution method, and filtering order used, but also affects the concurrency granularity of the task and subsequent caching strategies, and is the core decision output of the entire retrieval optimization process.

[0069] The query behavior preference information recorded in the access pattern model includes the field combinations that users or the system preferentially use under specific task categories, the commonly used query time periods, and the historical success rate of path configurations. The trajectory index, as a quantitative indicator of access behavior stability, will be used to assess whether the query matches the stable query patterns in history, thereby providing a behavioral similarity score for path selection.

[0070] Different retrieval path configuration schemes refer to multiple execution paths constructed for the current task. Each scheme includes, but is not limited to, the following structural parameters: the type of index used (such as inverted index, multidimensional index, graph index, etc.), the order of sorting and filtering execution, whether to enable cache preheating, and whether to call heterogeneous modal data sources, etc. Each scheme outputs the corresponding resource consumption, response latency, and path complexity score through the query cost prediction model obtained in step S3, along with a confidence score.

[0071] A weighted comprehensive evaluation function is used to score the candidate path schemes. An example of the scoring function is as follows: Score = α × stability score + β × prediction cost inverse value + γ × policy confidence + δ × historical frequency weight;

[0072] The stability score is derived from the trajectory index, the predicted cost inverse value is a comprehensive inverse quantification of response latency and resource consumption, the policy confidence is taken from the confidence score output by the model, and the historical frequency weight refers to the number of times and success rate of this path configuration in similar tasks. The weighting coefficients (α, β, γ, δ) are dynamically adjusted through the policy tuning module to adapt to the needs of different task objectives (such as emphasizing performance or reliability).

[0073] The pre-defined retrieval path generation rules refer to a path combination strategy library designed by an expert team during the initial modeling phase. This rule set can generate a list of available configurations based on factors such as field type, index applicability, and task complexity segmentation. Within this rule set, based on the scoring results of the aforementioned weighted comprehensive evaluation function, a path scheme that best matches the characteristics of the current retrieval task is constructed or selected, and the output is the final strategy selection result.

[0074] For example, for a query task with a trajectory index of 0.86, the access pattern model shows that it was executed with an inverted index as frequently as 82% of the time in history, while the query cost prediction model shows that the inverted index strategy has the lowest resource consumption and a confidence level of 0.93. The inverted path is selected first among the candidate paths, and a "delayed sorting" strategy is added to organize the results to form the final execution path.

[0075] Step S5: After obtaining the strategy selection results, if the combination of uncertainty level and confidence score meets the risk decision-making conditions, then the retrieval is performed through cross-modal consensus verification;

[0076] The retrieval via cross-modal consensus verification includes: activating multiple heterogeneous retrieval paths in parallel, wherein each heterogeneous retrieval path uses different data index structures, similarity measurement criteria, and data modalities to perform retrieval operations on the retrieval task, thereby obtaining corresponding candidate result sets; and performing cross-validation and consensus calculation on the candidate result sets to obtain the retrieval results.

[0077] The process of cross-validating and calculating consensus on the candidate result set includes: comparing the intersection, semantic similarity, and structural similarity between each candidate result set based on a preset consensus calculation criterion, obtaining a subset of consensus results and using it as the retrieval result; or clustering the candidate result set, filtering the candidate result set according to the characteristics of the clustering results, and performing semantic fusion on the filtered candidate result set to generate retrieval results.

[0078] In one embodiment of the present invention, if the uncertainty level of the retrieval task is high and the confidence score output by the corresponding query cost prediction model is below a preset risk threshold (e.g., <0.7), a cross-modal consensus verification mechanism is triggered to improve the reliability and accuracy of the retrieval results. This process is specifically designed for tasks with high semantic ambiguity, complex query structures, and large historical feedback fluctuations, avoiding error accumulation or query bias caused by using a single path.

[0079] "Cross-modal consensus verification" is a multi-path, multi-dimensional verification mechanism proposed in this invention. Its core idea is to activate multiple "heterogeneous retrieval paths" in parallel, execute the same query task from different data modalities, data index structures and similarity measurement criteria, and cross-validate and aggregate the candidate result sets output by each path, thereby obtaining highly reliable retrieval results with consistency and consensus in multiple information spaces.

[0080] Heterogeneous retrieval paths include, but are not limited to, the following combinations: heterogeneous index structure dimensions, such as inverted indexes, vector indexes, graph indexes, and hybrid indexes; heterogeneous modality dimensions, such as structured table data, text content, image annotations, and audio-to-text conversion; and heterogeneous similarity measurement criteria, such as keyword matching, semantic embedding (BERT vector) similarity, edit distance, and graph path hop count.

[0081] For example, for a fuzzy query "find customers with recent high sales in the Beijing area", structured field retrieval (region = Beijing + sales amount > threshold), natural language search (such as the keyword "high sales activity" in customer behavior description documents), and entity relationship chain path query in knowledge graph can be activated in parallel.

[0082] The cross-validation process for the candidate result set consists of two main stages:

[0083] Consensus calculation phase: Calculate intersection to identify the precise intersection of entities or records in the results returned by different paths; calculate semantic similarity based on context embedding, sentence vectors, and word vector models to calculate the semantic similarity between candidate result sets; calculate structural similarity, including but not limited to the level of entities in the graph structure, node type distribution, field value patterns, etc.

[0084] Semantic fusion stage (optional): If the number of consensus result subsets after cross-validation is lower than the confidence threshold, cluster analysis is performed on the candidate result set; the clustering criteria include semantic embedding distance, label co-occurrence frequency, and index path overlap; semantic fusion operations are performed on the candidate result set represented by each cluster center, such as aggregated description and confidence merging scoring, and finally, fused retrieval results are generated.

[0085] Key parameters in the consensus retrieval module, such as consensus threshold, lower limit of intersection ratio, minimum semantic distance, and cluster radius, are all obtained through optimization training using historical data to adapt to the complexity levels of tasks in different scenarios.

[0086] The final search results will be sorted based on consensus scores, prioritizing the return of results with high consensus and broad path coverage, thus ensuring that consistent, robust, and semantically accurate results are still output when dealing with tasks with high uncertainty levels.

[0087] Step S6: Otherwise, select a data index structure from the dynamic index structure library based on the strategy selection result; based on the strategy selection result and the data index structure, divide the retrieval process corresponding to the retrieval task into task segments, analyze the dependencies between task segments, and generate a parallel execution decision based on the uncertainty level, confidence score, dependencies and the obtained system computing resource status.

[0088] The process of generating parallel execution decisions includes: dividing the retrieval task into multiple task segments to be executed based on the strategy selection results and data index structure; constructing a task dependency graph, analyzing the data input-output relationships between each task segment, identifying serial dependency paths and parallel task sets, and obtaining dependency relationships; and combining the uncertainty level of the current retrieval task, the confidence score of the strategy selection results, and the obtained system computing resource status to determine the parallel execution granularity, grouping method, and scheduling order of the task segments, thereby generating parallel execution decisions.

[0089] Based on the task dependency structure, the set of parallelizable tasks in the retrieval process is determined. Furthermore, by combining the uncertainty level of the retrieval tasks with the confidence score of the selected strategy, the granularity and execution order of parallel execution are dynamically adjusted: when the uncertainty level is high or the confidence score is low, a more conservative task partitioning strategy is adopted to limit the degree of parallelism and prioritize the execution of critical dependent links; when the system computing resources show that there is sufficient idle time, the number of parallelizable tasks is dynamically increased to improve execution efficiency.

[0090] In one embodiment of the present invention, the dynamic index structure library refers to a dynamic management module for managing multiple types of data index structures. This library contains metadata about various index structures (such as index type, applicable fields, creation time, access frequency, resource consumption, etc.) and real-time status indicators (such as current hit rate, response latency trend, and feedback confidence). Based on the target path requirements specified in the strategy selection results, the most suitable data index structure is selected from this structure library to participate in task execution.

[0091] The system computing resource status refers to a set of multi-dimensional dynamic indicators used to characterize the availability of computing resources in a system at a specific point in time or period. These indicators mainly include CPU, memory, I / O bandwidth, scheduling window, and resource conflict degree. These status indicators guide the parallel partitioning strategy for retrieval tasks, task scheduling priorities, and system cache updates, and are one of the core input parameters of the task scheduling and execution control mechanism of this invention.

[0092] During the task division process, based on the selected path strategy and index structure, and combined with the query logic structure, the entire retrieval task is semantically decomposed into multiple task stages with clearly defined data inputs and outputs. Each task stage is represented by one of the following operation types: field filtering, range filtering, fuzzy matching, result sorting, pagination truncation, aggregation calculation, result projection, etc.

[0093] Construct a task dependency graph to represent the data flow relationships between task stages in a directed graph form. Each node in the graph represents a task stage, and edges represent data dependencies. For example, field filtering must be performed before sorting; pagination truncates intermediate results after sorting, etc. The structure of the task dependency graph determines whether tasks can be executed in parallel.

[0094] Based on task dependency graph analysis, sequential dependency paths and parallel task sets are identified. Parallel task sets refer to tasks that are independent of each other and are scheduled for execution simultaneously, provided that system resources permit, thereby accelerating the overall processing speed.

[0095] When generating parallel execution plans, an adjustable granularity strategy is adopted to control the parallel granularity (such as task slice size and number of concurrent threads) and scheduling priority:

[0096] When the uncertainty level is high or the confidence score is low, a conservative strategy is implemented, the parallelism is limited, critical path nodes are executed first, and an intermediate result verification mechanism is set up to prevent errors from being amplified.

[0097] When system resources are readily available and task uncertainty is low, an aggressive strategy is implemented to maximize the scheduling of task nodes in the parallel task set, thereby improving resource utilization and response efficiency.

[0098] For example, if a query task involves field filtering, fuzzy matching, and result aggregation, and the resource status is good and the task confidence is high, then two parallel nodes, field filtering and fuzzy matching, will be scheduled simultaneously, and the aggregation task will start after both are completed; if the confidence is low, the results after field filtering will be sampled and verified before deciding whether to trigger aggregation.

[0099] Step S7: Execute the retrieval task and collect cache hit information. Based on the access pattern model, the corresponding trajectory index and cache hit information, selectively cache the retrieval task, update the dynamic index structure library and adjust the cache eviction policy.

[0100] The selective caching of retrieval tasks, updating of the dynamic index structure library, and adjustment of cache eviction strategies include: evaluating the probability of result reuse for the current retrieval task based on the access pattern model and corresponding trajectory index, and determining whether to write the retrieval results into the cache based on cache hit information to achieve selective caching; after executing the retrieval task, updating the status indicators of the corresponding index structure in the dynamic index structure library based on the access frequency, hit performance, and feedback results of the data index structure used; and adjusting the parameter configuration of the cache eviction strategy based on the evolution trend of cache hit information and access pattern model to dynamically control the cache content retention period and eviction priority.

[0101] In one embodiment of the present invention, cache hit information refers to the status feedback information on whether some or all of the expected query results can be hit from the cache when the current retrieval task is executed. It typically includes: cache hit flag (Hit / Miss), hit rate (number of hits / number of requested items), and hit data source layer (first-level cache, second-level cache, etc.).

[0102] The access pattern model provides stability analysis results for task behavior, and the trajectory index indicates whether this type of task has exhibited high-frequency reuse characteristics in history. The reuse probability score is determined jointly based on the trajectory index and the current task cache hit status, and a decision is made accordingly on whether to write the task's result to the cache.

[0103] Selective caching refers to not caching all completed search results, but conditionally writing only those results with reusable value. Its main purpose is to improve cache resource utilization and reduce invalid storage overhead. The write conditions mainly include: the trajectory index is greater than a set threshold (e.g., 0.7); the hit rate is in the low to medium range (indicating the data is not hot but reusable); and the system's free cache space meets the minimum write threshold.

[0104] The basic structure of a cache item includes: query feature summary (such as field combination hash, fuzzy label), result summary (such as number of results, response size), cache expiration time (TTL), cache hit label, last access time, etc.

[0105] The index structure status metrics in the dynamic index structure library will be updated in real time after task execution, including but not limited to: usage frequency (number of times used in the last N tasks), recent hit rate (the proportion of the corresponding index structure that hit the cache), average response time (the average time required to execute a query under this index structure), and user feedback rating (implicit or explicit evaluation data from historical interactions). Based on the above metrics, the priority ranking of the index structure, whether to participate in the next round of task scheduling, and whether to recommend merging or removing it will be adjusted.

[0106] The cache eviction policy is the core mechanism for controlling the lifecycle of cached content. Its configuration parameters can be dynamically adjusted, including: retention period, which determines the length of time a cached item can live; eviction priority rules, which are set based on access frequency, trajectory index, system pressure, and other dimensions; and dynamic migration threshold for hot and cold zones, which is used to promote highly reusable items to high-priority cache areas and transfer low-access items to the eviction queue.

[0107] For example: if a task has a hit rate of 0.2 and a trajectory index of 0.85, and it is predicted that more than 3 similar task requests will occur within the next 7 days, then the result is written to the cache and given a longer retention period (such as 72 hours); if another task has a high hit rate but a low trajectory index (such as 0.3), then no cache writing is performed.

[0108] Step S8: Output the search results, collect response data during the execution process, and update the task evaluation model and query cost prediction model.

[0109] In one embodiment of the present invention, the final retrieval results are first output to the request source in a structured or semi-structured form. The output content includes, but is not limited to: the result set data ontology, the number of hit records, response time statistics, the strategy path identifier used, and cache status description. Dynamic response data during this execution process is automatically recorded as input samples for model training and updates.

[0110] Response data refers to multi-dimensional performance feedback information collected during the actual execution of the retrieval task, specifically including: execution performance data (such as actual response time, CPU usage, memory usage, and number of IO calls), strategy execution data (such as the selected path type, the index structure used, the execution order of task stages, and the parallelism setting), user behavior data (such as whether the user clicked on the result, whether they initiated a second query, or whether they terminated the query), and result performance data (number of hits, hit density (hit / full database ratio), and semantic relevance score).

[0111] The response data will be used to iteratively update two key models:

[0112] The task evaluation model is updated (for S2). This model is a classification model built on a convolutional neural network (CNN). Its inputs are structural features, query semantic features, and historical feedback features, and its output is the uncertainty level. During the update phase, the deviation between the actual response data for each task and its original predicted uncertainty level is calculated. Supervision labels are constructed based on "result volatility," "user feedback behavior," and "strategy deviation," forming a new training sample set. Online incremental training or periodic batch training is used, and the prediction accuracy of the uncertainty level is optimized based on the cross-entropy loss function. For example, if a task initially predicted as having "low uncertainty" ultimately exhibits significant result volatility and repeated user query modifications, the model will automatically increase the uncertainty level score for such samples and adjust parameters to avoid similar misjudgments in the future.

[0113] The query cost prediction model is updated (for S3). This model is a regression multi-output model based on a CNN architecture, predicting the response time, resource consumption, and path complexity of multiple retrieval strategies. Its training objective is to minimize the error between the predicted cost and the actual execution cost. The loss function typically includes weighted mean squared error (WMSE) and a confidence bias penalty term. After execution, the actual collected execution latency, CPU usage, and path complexity data are used as "true labels" and compared with the cost prediction results output by the model in S3. Residuals are calculated and used to form backtracking samples. To improve the reliability of the confidence score, a new confidence estimation model is constructed using the residual distribution, and the original scoring mechanism is modified.

[0114] During the update process, the prediction deviation trajectory of a specific policy under a specific type of task is recorded to form a deviation weight table, which is used to dynamically adjust the credibility of the prediction output of different policies. Through the above feedback-driven update mechanism, a closed-loop optimization system is constructed, enabling the model to continuously learn in real-world scenarios and improve its discrimination accuracy and policy evaluation capabilities for future tasks.

[0115] For example, if a query with a complex structure but clear semantics is found to have had its resource consumption frequently underestimated in the past, the model will increase the weight of such structural features in the query cost prediction model during subsequent training to improve prediction accuracy.

[0116] Example 2, an efficiency adaptive optimization system for data retrieval, see [link to example]. Figure 1 As shown, it includes the following modules:

[0117] Pattern acquisition module: Acquires historical retrieval records, performs feature extraction and pattern modeling, generates access pattern models, and calculates the trajectory index of the access pattern models based on the data features of historical retrieval records;

[0118] Task evaluation module: After receiving a retrieval task, it analyzes the retrieval task based on the task evaluation model to obtain the uncertainty level;

[0119] Cost prediction module: Based on the query cost prediction model and combined with the uncertainty level, it obtains the cost prediction results and corresponding confidence scores of the retrieval task under different standard retrieval strategies;

[0120] Strategy selection module: Based on the access pattern model and corresponding trajectory index, cost prediction results and corresponding confidence scores, it adaptively selects the target retrieval path used by the retrieval task to obtain the strategy selection result;

[0121] Consensus retrieval module: After obtaining the strategy selection results, if the combination of uncertainty level and confidence score meets the risk decision-making conditions, then the retrieval is performed through cross-modal consensus verification;

[0122] Execution decision module: Selects the data index structure, divides the retrieval process corresponding to the retrieval task into task stages, analyzes the dependencies between task stages, and generates parallel execution decisions;

[0123] Cache management module: Executes retrieval tasks and collects cache hit information. Based on the access pattern model, corresponding trajectory index and cache hit information, it selectively caches retrieval tasks, updates the dynamic index structure library, and adjusts the cache eviction policy.

[0124] Model update module: Outputs search results and collects response data during execution to update the task evaluation model and query cost prediction model.

[0125] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An efficiency adaptive optimization method for data retrieval, characterized in that, include: The system acquires historical retrieval records, performs feature extraction and pattern modeling, generates an access pattern model, and calculates the trajectory index of the access pattern model based on the data features of the historical retrieval records. After receiving the retrieval task, the retrieval task is analyzed based on the task evaluation model to obtain the uncertainty level; Based on the query cost prediction model and combined with the uncertainty level, the cost prediction results and corresponding confidence scores of the retrieval task under different standard retrieval strategies are obtained. Based on the access pattern model and the corresponding trajectory index, cost prediction results and corresponding confidence scores, the target retrieval path used by the retrieval task is adaptively selected to obtain the strategy selection results. After obtaining the strategy selection results, if the combination of uncertainty level and confidence score meets the risk decision-making conditions, then the retrieval is performed through cross-modal consensus verification. Otherwise, select a data index structure from the dynamic index structure library based on the strategy selection result; Based on the strategy selection results and data index structure, the retrieval process corresponding to the retrieval task is divided into task stages, and the dependencies between task stages are analyzed. Based on the uncertainty level, confidence score, dependencies and the obtained system computing resource status, a parallel execution decision is generated. Execute retrieval tasks and collect cache hit information. Based on the access pattern model, corresponding trajectory index and cache hit information, selectively cache the retrieval tasks, update the dynamic index structure library, and adjust the cache eviction policy. Output the search results and collect response data during the execution process to update the task evaluation model and query cost prediction model.

2. The efficiency adaptive optimization method for data retrieval according to claim 1, characterized in that, The retrieval via cross-modal consensus verification includes: activating multiple heterogeneous retrieval paths in parallel, wherein each heterogeneous retrieval path uses different data index structures, similarity measurement criteria, and data modalities to perform retrieval operations on the retrieval task, thereby obtaining corresponding candidate result sets; and performing cross-validation and consensus calculation on the candidate result sets to obtain the retrieval results.

3. The efficiency adaptive optimization method for data retrieval according to claim 2, characterized in that, The process of cross-validating and calculating consensus on the candidate result set includes: comparing the intersection, semantic similarity, and structural similarity between each candidate result set based on a preset consensus calculation criterion, obtaining a subset of consensus results and using it as the retrieval result; or clustering the candidate result set, filtering the candidate result set according to the characteristics of the clustering results, and performing semantic fusion on the filtered candidate result set to generate retrieval results.

4. The efficiency adaptive optimization method for data retrieval according to claim 1, characterized in that, The generated access pattern model includes: extracting target fields, query semantic structure and time distribution features from historical retrieval records to construct an access pattern model representing the regularity of query behavior; and calculating the trajectory index based on the corresponding data volume, query feedback and query response time distribution in historical retrieval records.

5. The efficiency adaptive optimization method for data retrieval according to claim 1, characterized in that, The acquisition of the uncertainty level includes: analyzing the structural features, query semantic features, and historical feedback features of the retrieval task through a task evaluation model built on a convolutional neural network, and calculating the uncertainty level based on the comprehensive evaluation results of the three types of features; the task evaluation model is obtained through supervised training with the uncertainty level corresponding to the execution results of historical retrieval tasks as the training target.

6. The efficiency adaptive optimization method for data retrieval according to claim 1, characterized in that, The process of obtaining cost prediction results and corresponding confidence scores for retrieval tasks under different standard retrieval strategies includes: using a query cost prediction model built on a convolutional neural network, analyzing the structure and target fields of the retrieval task, and combining uncertainty level as a joint input, jointly evaluating the query cost of each standard retrieval strategy; outputting the cost prediction results and confidence scores for each standard retrieval strategy respectively, wherein the confidence score is generated based on the confidence estimate of the cost prediction results output by the query cost prediction model, and the cost prediction results include resource consumption estimation, execution latency estimation, or path complexity assessment; the query cost prediction model is trained using supervised learning with the query cost of historical retrieval tasks under each standard retrieval strategy as the training target.

7. The efficiency adaptive optimization method for data retrieval according to claim 1, characterized in that, The strategy selection result includes: comprehensively evaluating different retrieval path configuration schemes based on the query behavior preference information provided by the access pattern model and the access stability represented by the corresponding trajectory index, combined with the query cost prediction results and confidence scores corresponding to each standard retrieval strategy; based on the comprehensive evaluation result, constructing or selecting the target retrieval path with the highest matching degree with the current retrieval task from the preset retrieval path generation rules, as the strategy selection result.

8. The efficiency adaptive optimization method for data retrieval according to claim 1, characterized in that, The process of generating parallel execution decisions includes: dividing the retrieval task into multiple task segments to be executed based on the strategy selection results and data index structure; constructing a task dependency graph, analyzing the data input-output relationships between each task segment, identifying serial dependency paths and parallel task sets, and obtaining dependency relationships; and combining the uncertainty level of the current retrieval task, the confidence score of the strategy selection results, and the obtained system computing resource status to determine the parallel execution granularity, grouping method, and scheduling order of the task segments, thereby generating parallel execution decisions.

9. The efficiency adaptive optimization method for data retrieval according to claim 1, characterized in that, The selective caching of retrieval tasks, updating of the dynamic index structure library, and adjustment of cache eviction strategies include: evaluating the probability of result reuse for the current retrieval task based on the access pattern model and corresponding trajectory index, and determining whether to write the retrieval results into the cache based on cache hit information to achieve selective caching; after executing the retrieval task, updating the status indicators of the corresponding index structure in the dynamic index structure library based on the access frequency, hit performance, and feedback results of the data index structure used; and adjusting the parameter configuration of the cache eviction strategy based on the evolution trend of cache hit information and access pattern model to dynamically control the cache content retention period and eviction priority.

10. An efficiency adaptive optimization system for data retrieval, characterized in that, An efficiency adaptive optimization method for data retrieval as described in any one of claims 1 to 9 comprises: Pattern acquisition module: Acquires historical retrieval records, performs feature extraction and pattern modeling, generates access pattern models, and calculates the trajectory index of the access pattern models based on the data features of historical retrieval records; Task evaluation module: After receiving a retrieval task, it analyzes the retrieval task based on the task evaluation model to obtain the uncertainty level; Cost prediction module: Based on the query cost prediction model and combined with the uncertainty level, it obtains the cost prediction results and corresponding confidence scores of the retrieval task under different standard retrieval strategies; Strategy selection module: Based on the access pattern model and corresponding trajectory index, cost prediction results and corresponding confidence scores, it adaptively selects the target retrieval path used by the retrieval task to obtain the strategy selection result; Consensus retrieval module: After obtaining the strategy selection results, if the combination of uncertainty level and confidence score meets the risk decision-making conditions, then the retrieval is performed through cross-modal consensus verification; Execution decision module: Selects the data index structure, divides the retrieval process corresponding to the retrieval task into task stages, analyzes the dependencies between task stages, and generates parallel execution decisions; Cache management module: Executes retrieval tasks and collects cache hit information. Based on the access pattern model, corresponding trajectory index and cache hit information, it selectively caches retrieval tasks, updates the dynamic index structure library, and adjusts the cache eviction policy. Model update module: Outputs search results and collects response data during execution to update the task evaluation model and query cost prediction model.

Images