Query optimization system based on weight calculation

Abstract

The invention discloses a query optimization system based on weight calculation, and the system comprises a weight sensing data statistics module which is responsible for collecting data, extracting a statistical characteristic spectrum, and storing the statistical characteristic spectrum in a weight warehouse; the weight construction and normalization module receives the data collected by the weight sensing module, generates a three-layer weight model and a multi-dimensional weight vector, receives the adjustment of other modules on the weight and processes the adjustment in real time; the parallel query scheduling module uses a multi-dimensional weight vector to schedule and query multiple threads to carry out actual query work, and sends a weight adjustment suggestion to the weight construction and normalization module when an exception occurs; the weight feedback and governance module evaluates the use condition of a multi-dimensional weight vector, outputs a report and a suggestion to a user, and provides popularity data and abnormal data to the weight construction and normalization module for historical performance analysis and rollback processing; the security and multi-tenant control module provides a weight template and a risk adjustment coefficient to the weight construction and normalization module, and adjusts a three-layer weight model in a risk suppression dimension. According to the method, the dynamic allocation and adjustment of the query priority and the operator weight can be realized based on the three-layer weight calculation model.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a query optimization system based on weight calculation. Background Technology

[0002] Weight refers to the degree of importance of a factor or indicator relative to a certain thing, i.e., relative importance. Relevance is the degree of matching between data (usually documents) and the query request submitted by the user. After receiving a query request, the query system should return relevant data retrieval results and sort them according to relevance, with more relevant documents displayed first. Relevance is a complex and subjective concept. To make subjective relevance calculable and measurable, a continuous, multi-dimensional set of weight parameters can be used to comprehensively calculate the relevance of data retrieval results.

[0003] Currently, most mainstream search engines employ query relevance scoring models based on inverted indexes, using similarity algorithms such as TF / IDF or BM25 to calculate text relevance. TF / IDF, short for "Term Frequency / Inverse Document Frequency," is a statistical method used to evaluate how important a word is to a document in a document library. Term frequency measures the frequency of a query term appearing in a specific document, while inverse document frequency measures the frequency of the query term appearing in all documents. Intuitively, the more times a word appears in a document but the fewer times it appears in all documents, the higher its weight. By comprehensively calculating the weights of all words in the query, the relevance between the document and the query can be determined. BM25 is an improvement on TF / IDF, introducing weight parameters such as document length and term frequency saturation, addressing the issues of unfairness between long and short documents and keyword stuffing.

[0004] The methods described above are static and local, focusing on word weights and lacking consideration for the global context. With the development of query systems and the increasing complexity of business scenarios, query requests are becoming more diverse in terms of logical structure, filtering conditions, sorting strategies, and aggregation operations. Traditional static query planning methods struggle to balance real-time performance and accuracy. In complex business scenarios, queries not only involve keyword matching but also Boolean operations, aggregation statistics, personalized sorting, geographic scope, multimodal vector recall, and other query types (hereinafter referred to as operators). Mainstream traditional engines either lack these capabilities or break them down into multiple modules, lacking a unified weight control center, leading to the following problems:

[0005] Problem 1: Lagging data statistics and inflexible weight parameters. The field statistics of static query planning methods are not synchronized with index updates, causing weight parameters to lag behind the latest data distribution and making it difficult to adapt to changes in business semantics. The weights related to complex operators are usually manually configured by engineers, lacking a data-driven adaptive mechanism, making it difficult to track changes in field distribution and user behavior.

[0006] Problem 2: Inconsistent parallelism. In concurrent scenarios, the lack of a unified weight context passing mechanism between parallel retrieval threads leads to statistical bias in the scoring results.

[0007] Issue 3: Multi-source query conflicts and lack of cross-modal expansion. Cross-index and cross-data source queries lack a unified weight coordination mechanism, making collaborative optimization of multi-modal queries impossible and easily leading to resource contention and hotspot failures. Summary of the Invention

[0008] Purpose of the invention: This invention provides a query optimization system based on weight calculation, which can dynamically allocate and adjust query priority and operator weight based on a three-layer weight calculation model.

[0009] Technical Solution: The present invention discloses a query optimization system based on weight calculation, comprising: a weight-aware data statistics module, a weight construction and normalization module, a parallel query scheduling module, a weight feedback and governance module, and a security and multi-tenant control module. The weight-aware data statistics module is responsible for collecting data, extracting statistical feature spectra, and storing them in a weight warehouse. The weight construction and normalization module receives data collected by the weight-aware module and generates a three-layer weight model and a multi-dimensional weight vector, and receives and processes weight adjustments from other modules in real time. The parallel query scheduling module uses the multi-dimensional weight vector to schedule multi-threaded queries for actual query work, and sends weight adjustment suggestions to the weight construction and normalization module when an anomaly occurs. The weight feedback and governance module evaluates the usage of the multi-dimensional weight vector, outputs reports and suggestions to users, and provides popularity data and abnormal data to the weight construction and normalization module for historical performance analysis and rollback processing. The security and multi-tenant control module provides weight templates and risk adjustment coefficients to the weight construction and normalization module, and adjusts the three-layer weight model in the risk suppression dimension.

[0010] Furthermore, the weight-aware data statistics module is responsible for collecting data including field statistics, term statistics, segment health, latency distribution, and cache hit rate.

[0011] Furthermore, the weight-aware data statistics module is responsible for collecting data and extracting statistical feature spectra to store in the weight repository. Specifically, it traverses the index segment and collects static statistics such as field sparsity, maximum number of documents, number of valid documents, total word frequency, and document frequency. During query execution, it records dynamic indicators such as data segment health, thread queue length, cache hit rate, search result collection latency, and Explain call count. It merges static statistics and dynamic indicators to generate a statistical feature spectrum, providing input for weight construction. The feature spectrum provides configurable multiple time windows, taking into account both short-term fluctuations and long-term trends when analyzing historical performance.

[0012] Furthermore, the three-layer weight model includes a semantic weight layer, which calculates the initial weight based on field type, query intent, and keyword matching degree; a historical performance layer, which adjusts the weight using historical hit rate, click-through rate, and conversion rate; and a resource status layer, which dynamically adjusts the weight based on current CPU, memory, I / O utilization, and index load.

[0013] Furthermore, the weight construction and normalization module receives data collected by the weight awareness module and generates a three-layer weight model and a multi-dimensional weight vector. It also receives weight adjustments from other modules and processes them in real time. Specifically, for each query node, a multi-dimensional vector is generated consisting of <field-related weight, historical performance weight, resource priority weight, operator cost weight, and risk suppression weight>. The first three are output by the three-layer weight model, the operator cost is determined by the query type, and the risk suppression is determined by the user type. Taking into account the required score, query type, and field sparsity, the most suitable similarity algorithm for this query is selected from BM25, language model, Hybrid (BM25 + vector), and non-scoring methods. The combined weight calculation results are globally normalized using a custom normalization function to obtain static weight values, while retaining interpretable coefficients. During normal queries, the weights will dynamically change within a reasonable range. If the statistical feature spectrum exceeds the threshold or the query context is missing, the system will fall back to the initial safe weight template to ensure stability.

[0014] Furthermore, in the normalization calculation, the combined weight = semantic weight * k1 + historical weight * k2 + resource weight * k3, where k1~k3 are coefficients that will be adjusted by operator cost and risk suppression. During operator adjustment, the must clause will increase the semantic coefficient k1, while the filter clause will decrease k1. In risk adjustment, the semantic coefficient k1 will decrease when a high-risk user queries sensitive words, while the resource coefficient k2 will increase when a high-priority user queries.

[0015] Furthermore, the parallel query scheduling module uses a multi-dimensional weight vector to schedule multiple threads for actual query work. When an anomaly occurs, it sends weight adjustment suggestions to the weight construction and normalization module. Specifically, it uses a thread pool for parallel thread scheduling, determining the number of execution threads, the number of document indexes each thread needs to query, and the data distribution based on segment data distribution, file size, IO bandwidth, and resource priority in the weight vector to achieve load balancing. It encapsulates the weight vector, statistical features, and thread budget information, binding a unified weight context to each data slice to ensure consistent scoring among parallel threads when dynamically adjusting weights. It monitors the execution status of parallel thread slices; if a timeout / anomaly is found when collecting query results, it decides whether to roll back or terminate based on the risk suppression dimension of the slice weight. For queries with high weight but high resource consumption, if a slice times out when collecting results, that thread slice triggers a partial rollback, while other slices use progressive rate limiting to ensure query scheduling fairness and ensure that the overall query can still return partial results.

[0016] Furthermore, the weight feedback and governance module evaluates the usage of multi-dimensional weight vectors, outputs reports and suggestions to users, and provides popularity data and anomaly data to the weight construction and normalization module for historical performance analysis and rollback processing. Specifically, it generates an initial Explain report containing weight contribution, field ranking, operator hit rate, cache status, and optimization suggestions; utilizing the query caching mechanism, once the access frequency of a query exceeds a certain threshold, the query result will be cached, while recording weight usage and frequency, and calculating popularity by combining cached content and user historical performance; collecting weight usage data, ensuring that the values ​​of each dimension are approximately normally distributed, performing outlier analysis on the weight vector based on popularity, and automatically triggering a strategy to roll back to the initial safe weight template if an abnormal change is found; and automatically generating executable items such as "weight retraining suggestions," "field index optimization suggestions," and "cache expansion suggestions" in the Explain report for timeout anomalies.

[0017] Furthermore, the security and multi-tenancy control module provides weight templates and risk adjustment coefficients to the weight construction and normalization module. Specifically, the three-layer weight model is adjusted in the risk suppression dimension as follows: each tenant has an independent weight template and adjustable parameter range for risk suppression weights to prevent cross-tenant interference; penalty factors are set for fields containing sensitive attributes within the risk suppression weights to prevent exceeding the limits of sensitive data when the weights are too high; all user weight modifications, parameter adjustments, and Explain requests are recorded for easy auditing; the resource dimension in the weights is bound to tenant quotas, and the scheduler limits and queues different tenants based on global resources, query weights, and resource usage predictions to prevent abuse of cluster resources.

[0018] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) Real-time weight scheduling: With the help of statistical feature spectrum and online indicators, the weight can be automatically calibrated in seconds, making the query performance more sensitive to changes in data distribution; (2) Cross-operator collaboration: The weight vector manages full-text retrieval, Boolean filtering, vector recall and aggregation calculation at the same time, so that complex queries can obtain a unified optimization strategy; (3) Parallel consistency guarantee: The slice context encapsulation strategy ensures that parallel threads share the same weight pointer, avoiding scoring deviation; (4) Transparent and controllable weight: The Explain report provides weight contribution decomposition, abnormal alarm and tuning suggestions, so that weight changes are interpretable and support auditing and continuous improvement; (5) Safe and reliable collaboration: Weight template isolation and sensitive field processing meet the needs of multimodal expansion and the security and compliance requirements of finance and government. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0020] Figure 2 This is a flowchart of the statistical query optimization process of the government affairs Q&A platform provided in Embodiment 1 of the present invention.

[0021] Figure 3 This is a flowchart of the weighted processing of news retrieval fields provided in Embodiment 2 of the present invention.

[0022] Figure 4 This is a flowchart of the multi-tenant financial risk control process provided in Embodiment 3 of the present invention. Detailed Implementation

[0023] like Figure 1 As shown, a query optimization system based on weight calculation includes: a weight-aware data statistics module, a weight construction and normalization module, a parallel query scheduling module, a weight feedback and governance module, and a security and multi-tenant control module. The weight-aware data statistics module is responsible for collecting data, extracting statistical feature spectra, and storing them in a weight warehouse. The weight construction and normalization module receives data collected by the weight-aware module and generates a three-layer weight model and a multi-dimensional weight vector, and receives weight adjustments from other modules and processes them in real time. The parallel query scheduling module uses the multi-dimensional weight vector to schedule multi-threaded queries for actual query work, and sends weight adjustment suggestions to the weight construction and normalization module when an anomaly occurs. The weight feedback and governance module evaluates the usage of the multi-dimensional weight vector, outputs reports and suggestions to users, and provides popularity data and abnormal data to the weight construction and normalization module for historical performance analysis and rollback processing. The security and multi-tenant control module provides weight templates and risk adjustment coefficients to the weight construction and normalization module, and adjusts the three-layer weight model in the risk suppression dimension.

[0024] The weight calculation model consists of three layers: a semantic weight layer, which calculates initial weights based on indicators such as field type, query intent, and keyword matching degree; a historical performance layer, which adjusts weights using indicators such as historical hit rate, click-through rate, and conversion rate; and a resource status layer, which dynamically adjusts weights based on indicators such as current CPU, memory, I / O utilization, and index load. The model outputs a multi-dimensional dynamic weight vector, or static weight values ​​can be calculated by weighting each layer.

[0025] The weight-aware data statistics process provides data collection for the three-layer weight model. It is responsible for collecting data such as field statistics, term statistics, segment health, latency distribution, and cache hit rate, extracting statistical feature spectra and storing them in the weight repository. Field and term statistics: Traverse the index segment and collect static statistics such as field sparsity, maximum number of documents, number of valid documents, total term frequency, and document frequency. Dynamic indicator collection: Record dynamic indicators such as data segment health, thread queue length, cache hit rate, search result collection latency, and Explain call count during query execution. Feature spectra construction: Merge static statistics and dynamic indicators to generate a statistical feature spectrum, providing input for weight construction. Time window updates: The feature spectrum provides configurable multiple time windows (short-term: real-time / 5 minutes; long-term: 24 hours / 1 week), allowing analysis of historical performance to consider both short-term fluctuations and long-term trends.

[0026] The weight construction and normalization process receives data collected by the weight awareness module and generates a three-layer weight model and a multi-dimensional weight vector; it also receives weight adjustments from other modules and processes them in real time. The weight calculation process generates a weight vector based on query parsing results and statistical feature spectra, including dimensions such as basic weight, combined weight, resource weight, and risk weight, and provides normalization and failure control strategies. Weight vector generation: For each query node, a multi-dimensional vector is generated consisting of <field relevance weight, historical performance weight, resource priority weight, operator cost weight, and risk suppression weight>. The first three are output by the three-layer weight model; the operator cost is determined by the query type; and risk suppression is determined by the user type. Similarity routing: Taking into account the required score, query type, and field sparsity, the most suitable similarity algorithm for the current query is selected from BM25, language model, Hybrid (BM25 + vector), and non-scoring methods. Combined weight normalization calculation: A custom normalization function is used to globally normalize the combined weight calculation results to obtain static weight values, while retaining interpretable coefficients. A simple example of normalized calculation: Combined weight = Semantic weight * k1 + Historical weight * k2 + Resource weight * k3, where k1~k3 are coefficients that are adjusted by operator cost and risk suppression. Operator adjustment, for example, in Boolean calculations, the `must` clause increases the semantic coefficient k1, while the `filter` clause decreases k1; risk adjustment, for example, decreases the semantic coefficient k1 when a high-risk user queries for sensitive words, and increases the resource coefficient k2 when a high-priority user queries. Weight invalidation control: During normal queries, weights will dynamically change within a reasonable range. If the statistical feature spectrum exceeds a threshold (e.g., dynamic indicator delay time, static indicator word frequency) or the query context is missing (e.g., accidental deletion of historical performance data), the system reverts to the initial safe weight template to ensure stability.

[0027] The parallel query scheduling process uses a multi-dimensional weight vector to schedule multiple threads for actual query work. In case of anomalies, weight adjustment suggestions are sent to the weight construction and normalization module. For querying document indexes of millions or more, multi-threaded parallelism can significantly improve query efficiency. Thread slicing, scoring, search result collector, and IO rate limiter are configured based on the weight vector to achieve weight-driven parallel query execution scheduling. Data slicing: A thread pool is used for specific thread parallel scheduling. The number of execution threads, the number of document indexes each thread needs to query, and the data distribution are determined based on segment data distribution, file size, IO bandwidth, and resource priorities in the weight vector, achieving load balancing. For example, a four-million-document index is divided into eight parts, each containing approximately 500,000 documents, and allocated to four query threads. Weight context encapsulation: Information such as weight vectors, statistical features, and thread budgets are encapsulated and bound to a unified weight context for each data slice, ensuring scoring consistency among parallel threads during dynamic weight adjustments. Anomaly response: The execution status of parallel thread slices is monitored. If a timeout / anomaly is detected during query result collection, a decision is made on whether to rollback or terminate based on the risk suppression dimension of the slice weights. Dynamic rate limiting. For queries with high weight but high resource consumption, if a slice times out during result collection, that slice triggers a partial rollback, while other slices use progressive rate limiting to ensure the fairness of query scheduling and ensure that the overall query can still return partial results.

[0028] The weight feedback and governance process evaluates the usage of multi-dimensional weight vectors and outputs reports and suggestions to users; it provides popularity data and anomaly data to the weight construction and normalization module for historical performance analysis and rollback processing. It is responsible for Explain report generation, anomaly weight detection, policy rollback, cache coordination, and weight popularity tracking. Explain output: Generates an initial Explain report containing weight contribution, field ranking, operator hit rate, cache status, and optimization suggestions. Weight popularity tracking: Utilizes a query caching mechanism; when the access frequency of a query exceeds a certain threshold, the query result is cached, while recording weight usage and frequency, and calculating popularity based on cached content and user historical performance. Anomaly detection: Collects weight usage data; values ​​for each dimension should approximate a normal distribution. Performs outlier analysis on the weight vector based on popularity; if anomalies are detected, automatically triggers policy rollback to the initial safe weight template. For example, using "three standard deviations" analysis, the mean 'a' and standard deviation 'b' of the resource status dimension are calculated. When a weight exceeds the range of (a-3b, a+3b), it is considered an outlier and requires appropriate handling. Tuning recommendations: The Explain report automatically generates actionable suggestions such as "weight retraining recommendations," "field index optimization recommendations," and "cache expansion recommendations" for timeouts and other anomalies.

[0029] The security and multi-tenancy control process provides weight templates and risk adjustment coefficients to the weight construction and normalization modules, adjusting the three-layer weight model in the risk suppression dimension. A global resource graph is constructed, mapping indexes, caches, and network bandwidth to resource nodes. Different weight policy templates are configured for different tenants, and resource quotas are limited. Sensitive field weight penalties, audit logs, and policy isolation mechanisms are provided. Weight template isolation: Each tenant has an independent weight template and adjustable parameter range for risk suppression weights, preventing cross-tenant interference. Sensitive field penalties: Penalty factors are set for fields containing sensitive attributes within the risk suppression weight to prevent exceeding sensitive data limits when the weight is too high. Access auditing: All user weight modifications, parameter adjustments, and Explain requests are recorded for easy auditing. Resource quota binding: The resource dimension in the weights is bound to tenant quotas. The scheduler limits and queues different tenants based on global resources, query weights, and resource usage predictions to prevent abuse of cluster resources.

[0030] Example 1: Optimization of Statistical Queries on Government Q&A Platform

[0031] Figure 2 This is a flowchart illustrating the optimized statistical query process of the government affairs Q&A platform provided in Embodiment 1 of the present invention. Embodiment 1 demonstrates the complete process of statistical query processing, specifically including the following steps:

[0032] S101, The user initiates a query request;

[0033] S102, the weight construction and normalization module parses the query type, determines whether it is an aggregation statistical type, and generates the weights of the operator cost dimension;

[0034] S103, non-aggregated statistical class, the weight construction and normalization module constructs the weight vector, determines that it needs to calculate the word frequency / inverse document frequency in the traditional way, and the parallel scheduling module returns the query results according to the similarity;

[0035] S104, Statistical Class, Weighted and Normalized Similarity Routing Switched to Non-Rating Similarity;

[0036] S105, For this query, the weight construction and normalization module reduces the weight of the resource dimension and publishes the changed multidimensional weight vector;

[0037] S106, the parallel scheduling module receives changes in resource dimension weights, guides the reduction of the number of threads, reduces CPU resource usage, and opens the cache to accelerate queries;

[0038] S107, the feedback governance module generates an Explain report, indicating that this query improved the cache hit rate by 18% and reduced latency by 43% because TF / IDF calculation was skipped;

[0039] S108, The feedback governance module uses the query result collector to return query results in streaming mode.

[0040] Example 2: Weighted processing of news retrieval fields

[0041] Figure 3 This is a flowchart of the weighted processing of news retrieval fields provided in Embodiment 2 of the present invention. Embodiment 2 demonstrates the complete process of field weighted optimization in the query system, specifically including the following steps:

[0042] S201, a news retrieval scenario, where a user enters a query request for "new energy subsidy policy", and the title, abstract, geographic and tag fields need to be searched simultaneously.

[0043] S202, the weight construction and normalization module analyzes the query intent. Policy information has different weight requirements for fields such as title, summary, tags, and geographic location.

[0044] S203, Semantic weight analysis adjusts the set of semantic weight parameters, increases the weight of the title and tag fields to 0.35 and 0.25 respectively, reduces the weight of the abstract to 0.2, and applies the geographic decay function to the geographic field to reduce its weight;

[0045] S204, Historical performance analysis revealed that users pay more attention to tag relevance, so the weight of the tag field was increased by 0.05;

[0046] S205, the scheduling module executes the query plan, prioritizing the use of high-weight elements, namely titles and tags, and initially filtering out documents that meet the criteria using the inverted index;

[0047] S206, the scheduling module combines low-weighted elements, namely abstract and geographic location, for reordering to improve document accuracy;

[0048] S207, the scheduling module aggregates query results and compiles user usage statistics;

[0049] S208, the feedback governance module generates an Explain report based on the query process and user usage, indicating that the query latency was reduced by 30% and the click-through rate was increased by 20%, with the title contributing 46%, the abstract contributing 32%, the geographic field contributing 14%, and the tags contributing 8%. It also suggests creating a vector index for the tag field to further improve relevance.

[0050] Example 3: Multi-tenant Financial Risk Control Processing

[0051] Figure 4 This is a flowchart of the multi-tenant financial risk control process provided in Embodiment 3 of the present invention. Embodiment 3 demonstrates the complete process for configuring differentiated risk control for different business lines in a financial scenario, specifically including the following steps:

[0052] S301: In a multi-tenant scenario, enter a query request;

[0053] S302: The security and multi-tenant control module identifies business lines. Financial institutions need to configure differentiated weight templates for different business lines. High-risk business lines have independent weight templates, while ordinary business lines have standard templates.

[0054] S303: Configure independent weight templates for high-risk business lines, set high penalty coefficients for sensitive fields (such as ID card numbers), and restrict sensitive information output in the Explain report;

[0055] S304: The weight construction and normalization module generates a set of weight parameters and a correlation calculation method;

[0056] S305: The parallel scheduling module filters out documents that meet the criteria and returns the query results to the user.

[0057] S306: The feedback governance module generates an Explain report to de-identify sensitive field information;

[0058] S307: The security and multi-tenancy control module generates audit logs to record template isolation, weight modification, and optimization operations to meet regulatory requirements.

Claims

1. A query optimization system based on weight calculation, characterized in that, include: The module includes a weight-aware data statistics module, a weight construction and normalization module, a parallel query scheduling module, a weight feedback and governance module, and a security and multi-tenant control module. The weight-aware data statistics module is responsible for collecting data, extracting statistical feature spectra and storing them in the weight warehouse; the weight construction and normalization module receives the data collected by the weight-aware module and generates a three-layer weight model and a multi-dimensional weight vector, receives weight adjustments from other modules and processes them in real time; the parallel query scheduling module uses the multi-dimensional weight vector to schedule multi-threaded queries to perform actual query work, and sends weight adjustment suggestions to the weight construction and normalization module when an exception occurs. The weight feedback and governance module evaluates the use of multi-dimensional weight vectors, outputs reports and suggestions to users, and provides popularity data and abnormal data to the weight construction and normalization module for historical performance analysis and rollback processing. The security and multi-tenancy control module provides weight templates and risk adjustment coefficients to the weight construction and normalization module, adjusting the three-layer weight model in the dimension of risk suppression.

2. The query optimization system based on weight calculation as described in claim 1, characterized in that, The weight-aware data statistics module is responsible for collecting data including field statistics, term statistics, segment health, latency distribution, and cache hit rate.

3. The query optimization system based on weight calculation as described in claim 1, characterized in that, The weight-aware data statistics module is responsible for collecting data, extracting statistical feature spectra and storing them in the weight warehouse. Specifically, it traverses the index segment and collects field sparsity, maximum number of documents, number of valid documents, total word frequency, and static statistics of document frequency. During query execution, dynamic metrics such as data segment health, thread queue length, cache hit rate, search result collection latency, and Explain call count are recorded; static statistics and dynamic metrics are combined to generate a statistical feature spectrum, providing input for weight construction. The feature spectrum offers configurable multiple time windows, taking into account both short-term fluctuations and long-term trends when analyzing historical performance.

4. The query optimization system based on weight calculation as described in claim 1, characterized in that, The three-layer weighting model includes a semantic weighting layer, which calculates the initial weights based on field type, query intent, and keyword matching degree; and a historical performance layer, which adjusts the weights using historical hit rate, click-through rate, and conversion rate. The resource status layer dynamically adjusts weights based on current CPU, memory, I / O utilization, and index load metrics.

5. The query optimization system based on weight calculation as described in claim 1, characterized in that, The weight construction and normalization module receives data collected by the weight awareness module and generates a three-layer weight model and a multi-dimensional weight vector. It also receives weight adjustments from other modules and processes them in real time. Specifically, for each query node, it generates a multi-dimensional vector consisting of <field relevance weight, historical performance weight, resource priority weight, operator cost weight, and risk suppression weight>. The first three are output by the three-layer weight model, the operator cost is determined by the query type, and the risk suppression is determined by the user type. Taking into account the required score, query type, and field sparsity, it selects the most suitable similarity algorithm for this query from BM25, language model, Hybrid, and non-scoring methods. It uses a custom normalization function to globally normalize the combined weight calculation results to obtain static weight values ​​and retain interpretable coefficients. During a normal query, the weights will change dynamically within a reasonable range. If the statistical feature spectrum exceeds the threshold or the query context is missing, the system will fall back to the initial safe weight template to ensure stability.

6. The query optimization system based on weight calculation as described in claim 5, characterized in that, Similarity algorithms include TF / IDF or BM25 algorithms.

7. The query optimization system based on weight calculation as described in claim 5, characterized in that, In the normalized calculation, the combined weight = semantic weight * k1 + historical weight * k2 + resource weight * k3, where k1~k3 are coefficients that are adjusted by operator cost and risk suppression. During operator adjustment, the must clause will increase the semantic coefficient k1, while the filter clause will decrease k1. In risk adjustment, the semantic coefficient k1 will decrease when a risky user queries sensitive words, and the resource coefficient k2 will increase when a high-priority user queries.

8. The query optimization system based on weight calculation as described in claim 1, characterized in that, The parallel query scheduling module uses a multi-dimensional weight vector to schedule multiple threads for actual query work. When an exception occurs, it sends weight adjustment suggestions to the weight construction and normalization module. Specifically, it uses a thread pool for parallel thread scheduling, and determines the number of execution threads, the number of document indexes that each thread needs to query, and the data distribution based on the segment data distribution, file size, IO bandwidth, and resource priority in the weight vector to achieve load balancing. It encapsulates the weight vector, statistical features, and thread budget information, and binds a unified weight context to each data slice to ensure the consistency of scoring among parallel threads when the weight is dynamically adjusted. Monitor the execution status of parallel thread slices. If a timeout / abnormality is found when collecting query results, decide whether to roll back or terminate based on the risk suppression dimension of the slice weight. For queries with high weight but high resource consumption, if a slice times out when collecting results, the thread slice triggers a partial rollback, while other slices adopt progressive rate limiting to ensure the fairness of query scheduling and ensure that the overall query can still return some results.

9. The query optimization system based on weight calculation as described in claim 1, characterized in that, The weight feedback and governance module evaluates the usage of multi-dimensional weight vectors, outputs reports and suggestions to users, and provides popularity data and anomaly data to the weight construction and normalization module for historical performance analysis and rollback processing. Specifically, it generates an initial Explain report containing weight contribution, field ranking, operator hit rate, cache status, and optimization suggestions; utilizes the query caching mechanism, when the access frequency of a query exceeds a certain threshold, the query result is cached, while recording weight usage and frequency, and calculating popularity by combining cached content and user historical performance; collects weight usage data, ensuring that the values ​​of each dimension are approximately normally distributed, performs outlier analysis on the weight vector based on popularity, and automatically triggers a strategy to roll back to the initial safe weight template if an abnormal change is found; and automatically generates executable items such as "weight retraining suggestions," "field index optimization suggestions," and "cache expansion suggestions" in the Explain report for timeout anomalies.

10. The query optimization system based on weight calculation as described in claim 1, characterized in that, The security and multi-tenancy control module provides weight templates and risk adjustment coefficients to the weight construction and normalization module. Specifically, the three-layer weight model is adjusted in the risk suppression dimension: each tenant has an independent weight template and an adjustable range of risk suppression weight parameters to prevent cross-tenant interference. Within the risk suppression weight, penalty factors are set for fields containing sensitive attributes to prevent the restriction of sensitive data from being exceeded when the weight is too high; all user weight modifications, parameter tuning, and Explain requests are recorded for easy auditing. The resource dimension in the weight is bound to the tenant quota. The scheduler limits and queues different tenants based on global resources, query weight, and resource usage prediction to prevent abuse of cluster resources.

Images