Real-time index construction and intelligent optimization method and system based on large model drive

Abstract

The present invention discloses a real-time index construction and intelligent optimization method and system based on large model driving, which belongs to the field of database index technology, and specifically includes: collecting and analyzing sample data characteristics and access patterns through data stream processing and dynamic preprocessing technology; integrating pre-trained language model and knowledge graph method, performing deep semantic analysis on sample data, and adopting adaptive hybrid index strategy to generate index scheme; implementing index construction, and applying multi-objective optimization algorithm to optimize index performance, and through real-time monitoring and machine learning prediction model, being able to dynamically adjust index configuration, improve database query efficiency with optimal index combination, and realize intelligent and efficient index management strategy.

Description

Technical Field

[0001] The present invention belongs to the technical field of database indexing, and more specifically to a method and system for real-time indexing construction and intelligent optimization driven by a large model. Background Art

[0002] An index is a data structure used to increase the speed of database queries. In a relational database, an index is designed to sort the values ​​of one or more columns in a database table and save these values ​​and a list of logical pointers to the data pages in the table that physically store these values. The role of an index is similar to that of a book catalog. An index can be used to quickly locate specific information in a database table without scanning the entire table. Index construction and optimization play a vital role in modern database management and information retrieval systems. With the explosive growth of data volume and the increasing complexity of query requirements, how to efficiently construct indexes and intelligently optimize them to improve the speed and accuracy of data retrieval has become an important research topic in the field of database and information technology.

[0003] Early index construction and optimization methods mainly relied on the index creation commands provided by the database management system. They used statistical language models to predict future words by calculating the generation probability of word sequences. However, due to the need to estimate exponential transition probabilities, the model faces the problem of dimensionality disaster when processing large-scale data, and the index update method usually lags behind the data update, resulting in reduced query performance and difficulty in meeting the diversity of query requirements.

[0004] For example, the patent with publication number CN116955348A discloses a method and device for building a database index, including: when receiving a data insertion request, inserting the data to be inserted into the position to be inserted, if the detection result of the first detection is that the node of the global index is increased, allocating the required address space for the newly added node based on the address space of the dynamic random access memory, if the detection result of the second detection is that the parent node of the newly added node is located in the non-volatile memory, inserting the newly added node into the quick index; if the used space of the dynamic random access memory accounts for a preset proportion, migrating the newly added node from the address space of the dynamic random access memory to the address space of the non-volatile memory; wherein, both the global index and the quick index adopt a skip table data structure. This technical solution adopts a skip table data structure, optimizes the index structure of the memory database, and reduces the space consumption of the index on DRAM while ensuring a high data response speed.

[0005] For example, the patent with publication number CN117971837A discloses a method for constructing a spatial index of a real-life 3D model, including: parsing the tile data of the real-life 3D model and parsing the LOD hierarchy; establishing a unified LOD hierarchy, creating a unified root tile and LOD parameters, and the number of layers is a multiple of 3, completing the reorganization of the LOD hierarchy; calculating the spatial bounding box of the real-life 3D model, re-dividing the entire space from top to bottom into a uniform quadtree structure, and numbering the grids, and then filling the tile data of each layer into the new grid to generate a new tile file. This technical solution optimizes the index structure, splits the index tree of the large model into small local indexes of basically the same size, facilitates fast loading, is conducive to the optimization of memory allocation, can reduce the pressure on the network when transmitting index files, and does not need to query from the root node of the entire model, thereby greatly improving the random reading performance of the model.

[0006] The above existing technologies all have the following problems: 1) insufficient intelligence and adaptability; 2) low complexity and flexibility in index construction; 3) lack of long-term learning and optimization capabilities. Summary of the invention

[0007] In view of the shortcomings of the existing technology, the present invention proposes a real-time index construction and intelligent optimization method and system driven by a large model. The method collects and analyzes the characteristics and access patterns of sample data through data stream processing and dynamic preprocessing technology; integrates the pre-trained language model and the knowledge graph method to perform deep semantic analysis on the sample data, and adopts an adaptive hybrid indexing strategy to generate an indexing scheme; implements index construction, and applies a multi-objective optimization algorithm to optimize the index performance. Through real-time monitoring and machine learning prediction models, the index configuration can be dynamically adjusted to improve the database query efficiency with the optimal index combination, thereby realizing an intelligent and efficient index management strategy.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] Real-time index construction and intelligent optimization methods driven by large models include:

[0010] Step S1: collect sample data using data stream processing and perform dynamic adaptive preprocessing. At the same time, analyze the characteristics of the sample data and evaluate the access pattern of the sample data.

[0011] Step S2: Use statistical methods to predict hot data based on historical access records and current data characteristics, and automatically trigger the loading mechanism according to the hot data prediction results to load it into the memory in advance;

[0012] Step S3: Load the pre-trained language model, combine the knowledge graph method, perform deep semantic analysis on the sample data, adopt an adaptive hybrid indexing strategy based on the analysis results, sample data characteristics, access patterns and hot data prediction results, combine the entity relationship information in the knowledge graph, generate a hybrid indexing scheme, and store it in the index database;

[0013] Step S4: According to the hybrid indexing scheme, index construction is implemented in the database system, and the hybrid indexing scheme is evaluated in multiple dimensions using a pre-trained language model. According to the evaluation results, a multi-objective optimization algorithm is used to automatically generate an index optimization scheme, and the optimized indexing scheme is implemented;

[0014] Step S5: Monitor the performance indicators of index construction and query processing in real time, use machine learning algorithms to combine historical query data and index performance data, predict the query optimization effects under different index combinations, and obtain the optimal index combination.

[0015] Specifically, the specific steps of step S3 include:

[0016] S3.1: Load the pre-trained language model and its matching word segmenter, and use the word segmenter of the pre-trained language model to segment and encode the input sample data;

[0017] S3.2: Input the encoded data into the pre-trained language model, run the forward propagation of the pre-trained language model, and extract the last hidden state of the model as the deep semantic features of the sample data;

[0018] S3.3: Use the DistMult method to obtain the embedded representation of relevant entities and relations in the knowledge graph, and use the concatenation and dimension transformation methods to combine the entity and relationship embeddings obtained from the knowledge graph with the deep semantic features extracted by the pre-trained language model to form a feature vector.

[0019] Specifically, the specific steps of step S3 also include:

[0020] S3.4: Collect database access logs, perform statistical analysis on the collected access logs, and obtain user access patterns and query requirements;

[0021] S3.5: Extract features from feature vectors based on user access patterns and query requirements to obtain index candidate features. Use simulation tests to evaluate the impact of different index candidate features on query performance. ,in, represents the influence factor of the nth index candidate feature, where n represents the number of index candidate features;

[0022] S3.6: Based on the evaluation results, the impact factors Sort in descending order and set the minimum impact factor threshold to ;

[0023] like , then the index feature is obtained ,Based on the index characteristics and query requirements, an adaptive hybrid index strategy is used to generate a hybrid index scheme, where, represents the mth index feature, and m represents the number of index features.

[0024] Specifically, the specific steps of step S3 also include:

[0025] S3.7: Introduce a dynamic index adjustment mechanism, based on a hybrid indexing solution, monitor the index query efficiency in real time, and dynamically adjust the index structure based on the monitoring results;

[0026] S3.8: Use the trained machine learning model to predict the trend of index queries, interpret and analyze the predicted trend results of index queries, identify query hot spots and pattern changes, and evaluate whether the index structure or parameters need to be adjusted to optimize the adaptive hybrid index strategy based on the predicted trend results of index queries and the current index;

[0027] S3.9: Write an adaptive hybrid index strategy construction script and execute it on a large-scale data set, monitoring resource consumption and performance indicators during the construction process;

[0028] S3.10: After the new indexing strategy is executed, the constructed hybrid index is stored in the index database.

[0029] Specifically, the specific steps of the multi-objective optimization algorithm in step S4 include:

[0030] S4.1: Set the objective function and constraints of index optimization, and generate a set of initial index configuration solutions as candidate solutions through heuristic methods. The formula is:

[0031] ;

[0032] in, Represents a given i-th index configuration scheme, It represents the average or total response time required to perform a query operation under the i-th index configuration scheme. Indicates the storage space occupied by the index structure under the i-th index configuration scheme. Indicates the time required to build the index under the i-th index configuration scheme. , , represents the weight coefficient;

[0033] S4.2: Use the pre-trained language model to perform multi-dimensional performance evaluation on each index configuration scheme in the candidate solution set;

[0034] S4.3: Based on the results of multi-dimensional performance evaluation, a genetic algorithm is used to select some solutions from the candidate solution set as parents, generate new solutions as offspring, and at the same time, merge the parents and offspring to form a new solution set, which is then evaluated.

[0035] Specifically, the specific steps of the multi-objective optimization algorithm in step S4 also include:

[0036] S4.4: In the new solution set, the solutions are layered according to their dominance relations, so that each layer contains a set of mutually non-dominated solutions, and different non-dominated layers are obtained. The crowding degree of all individuals in the same non-dominated layer is initialized to 0.

[0037] S4.5: For each individual in the non-dominated layer, sort them on each objective function respectively. For each objective function, calculate the objective function difference between the individual and its adjacent individuals and perform normalization.

[0038] S4.6: Add the normalized differences of all objective functions to obtain the crowding degree of the individual. At the same time, select the individuals of the next generation population based on the level and crowding degree of the non-dominated sorting and perform iterative operations;

[0039] S4.7: Set the maximum number of iterations. If the maximum number of iterations is met, stop the iteration and obtain the Pareto optimal solution. Otherwise, return to step S4.3 to continue the iteration.

[0040] S3.8: Generate an index optimization plan based on the Pareto optimal solution.

[0041] Specifically, the hybrid indexing scheme generated in S3.6 adopts hash table and bitmap indexing methods, combined with entity relationship information in the knowledge graph, to generate a hybrid indexing scheme.

[0042] A real-time index building and intelligent optimization system driven by a large model, including: data processing module, index generation module, index optimization module, and monitoring module;

[0043] The data processing module is used to collect sample data and perform dynamic adaptive preprocessing;

[0044] The index generation module is used to create an index structure based on the preprocessed sample data and generate a hybrid index scheme;

[0045] The index optimization module is used to implement index construction and adjust and optimize the index structure as needed;

[0046] The monitoring module is used to monitor the performance indicators of index construction and query processing in real time.

[0047] Specifically, the index generation module includes: a model loading unit, a knowledge graph fusion unit, a strategy formulation unit, and a storage unit;

[0048] The model loading unit is used to load the pre-trained language model and perform deep semantic representation;

[0049] The knowledge graph fusion unit is used to integrate the external knowledge graph or external knowledge into the index construction process, enhance the semantic understanding ability of the index, and improve the accuracy of the query;

[0050] The strategy formulation unit is used to formulate an adaptive hybrid index strategy based on the analysis results, data characteristics and access patterns of the sample data;

[0051] The storage unit is used to generate an indexing scheme according to the adaptive hybrid indexing strategy and store it in an index database to implement physical storage of the index and manage index files.

[0052] Specifically, a computer-readable storage medium stores computer instructions, which, when executed, execute the steps of a real-time index building and intelligent optimization method driven by a large model.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention proposes a real-time index construction and intelligent optimization system driven by a large model, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production work costs.

[0055] 2. The present invention proposes a real-time index construction and intelligent optimization method driven by a large model, sets an adaptive indexing method, and improves the intelligence and efficiency of index construction through deep semantic analysis, adaptive hybrid indexing strategy, multi-dimensional performance evaluation and dynamic adjustment, improves the adaptability of the index, optimizes the query performance, and improves the real-time nature of data processing and the operating efficiency of the index construction and intelligent optimization system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a real-time index construction and intelligent optimization method driven by a large model according to the present invention;

[0057] Figure 2 This is a principle flow chart of the real-time index construction and intelligent optimization method driven by a large model according to the present invention;

[0058] Figure 3This is a flowchart of the indexing scheme implementation of the real-time index construction and intelligent optimization method driven by a large model in the present invention;

[0059] Figure 4 This is the architecture diagram of the real-time index construction and intelligent optimization system driven by a large model in the present invention. DETAILED DESCRIPTION

[0060] Example 1

[0061] See also Figure 1-Figure 3 , an embodiment provided by the present invention: a real-time index construction and intelligent optimization method driven by a large model, comprising the following steps:

[0062] Step S1: collect sample data using data stream processing and perform dynamic adaptive preprocessing. At the same time, analyze the characteristics of the sample data and evaluate the access pattern of the sample data.

[0063] The sample data includes: data distribution, access frequency, and update frequency information.

[0064] Furthermore, the specific steps of step S1 include:

[0065] A1: Set up data stream access points, configure preprocessing strategy libraries, and initialize data analysis tools and index building systems;

[0066] A2: Receive new data items from the data source in real time, perform dynamic adaptive preprocessing on the new data items, store the preprocessed sample data in a temporary storage area or directly perform feature analysis;

[0067] A3: Regularly perform statistical analysis, pattern recognition, and correlation analysis on sample data in the storage area. At the same time, analyze access logs, mine access patterns and user behaviors, and store the analysis results in the analysis result database for subsequent index construction and optimization.

[0068] Step S2: Use statistical methods to predict hot data based on historical access records and current data characteristics, and automatically trigger the loading mechanism according to the hot data prediction results to load it into the memory in advance;

[0069] Step S3: Load the pre-trained language model, combine the knowledge graph method, perform deep semantic analysis on the sample data, adopt an adaptive hybrid indexing strategy based on the analysis results, sample data characteristics, access patterns and hot data prediction results, combine the entity relationship information in the knowledge graph, generate a hybrid indexing scheme, and store it in the index database;

[0070] Step S4: According to the hybrid indexing scheme, index construction is implemented in the database system, and the hybrid indexing scheme is evaluated in multiple dimensions using a pre-trained language model. According to the evaluation results, a multi-objective optimization algorithm is used to automatically generate an index optimization scheme, and the optimized indexing scheme is implemented;

[0071] Step S5: Monitor the performance indicators of index construction and query processing in real time, use machine learning algorithms to combine historical query data and index performance data, predict query optimization effects under different index combinations, and obtain the optimal index combination;

[0072] Step S6: Feedback the new query data and index performance data to the machine learning model, and use incremental learning to continuously update and optimize the machine learning model to improve prediction accuracy and adaptability. The incremental learning method is the prior art content in this field and is not the creative solution of this application, so it will not be elaborated here.

[0073] The specific steps of step S3 include:

[0074] S3.1: Load the pre-trained language model and its matching word segmenter, and use the word segmenter of the pre-trained language model to segment and encode the input sample data;

[0075] In the present invention, the pre-trained language model adopts GPT-3.

[0076] Furthermore, the step of S3.1 includes:

[0077] (1) Make sure that PyTorch and transformers libraries are installed in the environment, because the transformers library provides pre-trained language models and their word segmentators;

[0078] (2) Use AutoModel in the transformers library to load the pre-trained language model, and use AutoTokenizer to load the corresponding tokenizer;

[0079] (3) Use the loaded tokenizer to tokenize the input sample data and convert the tokenization results into an ID sequence that the model can understand, i.e., encoding. This usually includes splitting the text into subwords, adding special tokens such as [CLS] and [SEP], and mapping them to indexes in the vocabulary.

[0080] S3.2: Input the encoded data into the pre-trained language model, run the forward propagation of the pre-trained language model, and extract the last hidden state of the model as the deep semantic features of the sample data;

[0081] Furthermore, the specific steps of S3.2 include:

[0082] (1) Encode the text data into a format acceptable to the pre-trained language model through the tokenizer, and convert the encoded data into a tensor format suitable for the pre-trained language model input;

[0083] (2) Pass the encoded data as input to the pre-trained language model, which will perform forward propagation and output multiple results, including hidden states and attention weights;

[0084] (3) Extract the hidden state of the last layer from the output of the pre-trained language model by accessing outputs.last_hidden_state, where outputs.last_hidden_state refers to the hidden state of the last Transformer layer generated after the model processes the input data. It is usually composed of multiple stacked Transformer layers. Each layer performs a series of operations on the input data or the output of its previous layer, and finally outputs the hidden state of the layer. The series of operations include: self-attention mechanism, feedforward network;

[0085] (4) If necessary, the extracted hidden state is further processed, such as average pooling and taking vectors at specific positions, to obtain the deep semantic features of the sample data.

[0086] S3.3: Use DistMult to obtain the embedding representation of relevant entities and relations in the knowledge graph, and use concatenation and dimension transformation methods to combine the entity and relationship embeddings obtained from the knowledge graph with the deep semantic features extracted by the pre-trained language model to form a feature vector;

[0087] Furthermore, the specific steps of S3.3 include:

[0088] (1) Prepare triple data and text data of the knowledge graph;

[0089] (2) Load the DistMult model and train it to obtain the embedding representation of entities and relations;

[0090] (3) Load the pre-trained language model and extract deep semantic features of text data;

[0091] (4) Concatenate the knowledge graph embedding with the deep semantic features of the text or concatenate them after dimension transformation;

[0092] (5) Form the final feature vector.

[0093] S3.4: Collect database access logs, perform statistical analysis on the collected access logs, and obtain user access patterns and query requirements;

[0094] S3.5: Extract features from feature vectors based on user access patterns and query requirements to obtain index candidate features. Use simulation tests to evaluate the impact of different index candidate features on query performance. ,in, represents the influence factor of the nth index candidate feature, where n represents the number of index candidate features;

[0095] Among them, the influencing factors include the complexity of retrieval conditions, retrieval time, hit data blocks, number of hits, sorting method, and user satisfaction.

[0096] S3.6: Based on the evaluation results, the impact factors Sort in descending order and set the minimum impact factor threshold to ;

[0097] like , then the index feature is obtained ,Based on the index characteristics and query requirements, an adaptive hybrid index strategy is used to generate a hybrid index scheme, where, represents the mth index feature, m represents the number of index features, The impact factor is greater than The set of corresponding index candidate features forms an index feature set;

[0098] S3.7: Introduce a dynamic index adjustment mechanism, based on a hybrid indexing solution, monitor the index query efficiency in real time, and dynamically adjust the index structure based on the monitoring results;

[0099] S3.8: Use the trained machine learning model to predict the trend of index queries, interpret and analyze the predicted trend results of index queries, identify query hot spots and pattern changes, and evaluate whether the index structure or parameters need to be adjusted to optimize the adaptive hybrid index strategy based on the predicted trend results of index queries and the current index;

[0100] S3.9: Write an adaptive hybrid index strategy construction script and execute it on a large-scale data set, monitoring resource consumption and performance indicators during the construction process;

[0101] S3.10: After the new indexing strategy is executed, the constructed hybrid index is stored in the index database.

[0102] The specific steps of the multi-objective optimization algorithm in step S4 include:

[0103] S4.1: Set the objective function and constraints of index optimization, and generate a set of initial index configuration solutions as candidate solutions through heuristic methods. The formula is:

[0104] ;

[0105] in, Represents a given i-th index configuration scheme, It represents the average or total response time required to perform a query operation under the i-th index configuration scheme. Indicates the storage space occupied by the index structure under the i-th index configuration scheme. Indicates the time required to build the index under the i-th index configuration scheme. , , represents the weight coefficient;

[0106] S4.2: Use the pre-trained language model to perform multi-dimensional performance evaluation on each index configuration scheme in the candidate solution set;

[0107] Furthermore, the specific steps of S4.2 include:

[0108] (1) Collect each index configuration scheme in the candidate solution set and ensure that each scheme has clear definitions and parameters. At the same time, prepare a data set for evaluating the performance of the index configuration scheme. The data set should cover multiple dimensions to comprehensively evaluate the effect of the index configuration.

[0109] (2) Preprocess the text in the evaluation dataset, including cleaning, word segmentation, and removal of stop words, to ensure the quality of the text input to the pre-trained language model;

[0110] (3) Use the pre-trained language model to extract features from the pre-processed text. The pre-trained language model outputs deep semantic feature representations for each text sequence, which can be used for subsequent evaluation tasks.

[0111] (4) Determine performance evaluation indicators based on evaluation requirements, such as query efficiency, response time, accuracy, and recall rate. These indicators will be used to quantify the performance of the index configuration solution.

[0112] (5) Associating the features extracted by the pre-trained language model with the index configuration scheme, and evaluating the performance of different index configuration schemes on a specific dataset through simulated or actual queries;

[0113] (6) Score or rank each index configuration scheme using the determined performance evaluation index;

[0114] (7) Analyze the evaluation results and identify the index configuration scheme with excellent performance and its characteristics.

[0115] S4.3: Based on the multi-dimensional performance evaluation results, a genetic algorithm is used to select some solutions from the candidate solution set as parents, generate new solutions as offspring, and merge the parents and offspring to form a new solution set, which is then evaluated. The genetic algorithm is a prior art in this field and is not an inventive solution of the present application, so it will not be described in detail here.

[0116] S4.4: In the new solution set, the solutions are layered according to their dominance relations, so that each layer contains a set of mutually non-dominated solutions, and different non-dominated layers are obtained. The crowding degree of all individuals in the same non-dominated layer is initialized to 0.

[0117] S4.5: For each individual in the non-dominated layer, sort them on each objective function respectively. For each objective function, calculate the objective function difference between the individual and its adjacent individuals and perform normalization.

[0118] S4.6: Add the normalized differences of all objective functions to obtain the crowding degree of the individual. At the same time, select the individuals of the next generation population based on the level and crowding degree of the non-dominated sorting and perform iterative operations;

[0119] S4.7: Set the maximum number of iterations. If the maximum number of iterations is met, stop the iteration and obtain the Pareto optimal solution. Otherwise, return to step S4.3 to continue the iteration.

[0120] S4.8: Generate an index optimization plan based on the Pareto optimal solution.

[0121] The hybrid indexing scheme generated in S3.6 adopts hash table and bitmap indexing methods, combined with entity relationship information in the knowledge graph, to generate a hybrid indexing scheme.

[0122] Example 2

[0123] See also Figure 4 ,Another embodiment provided by the present invention: A real-time index construction and intelligent optimization system driven by a large model, comprising: a data processing module, an index generation module, an index optimization module, and a monitoring module;

[0124] Data processing module, used to collect sample data and perform dynamic adaptive preprocessing to ensure data quality and consistency;

[0125] The index generation module is used to create an index structure based on the preprocessed sample data, generate a hybrid index scheme, speed up the data query process, and reduce search time and resource consumption;

[0126] Index optimization module, which is used to implement index construction and adjust and optimize the index structure as needed to adapt to data changes and the evolution of query patterns;

[0127] The monitoring module is used to monitor the performance indicators of index construction and query processing in real time, which can promptly discover and solve problems and ensure the stability and reliability of the index construction and intelligent optimization system.

[0128] The data processing module includes: a data stream processing unit, a dynamic preprocessing unit, a characteristic analysis unit, and an access mode evaluation unit;

[0129] A data stream processing unit, which is used to receive and process sample data in the data stream in real time and can quickly identify and process new data input;

[0130] Dynamic preprocessing unit, used to preprocess sample data, such as data cleaning, missing value processing, data conversion, and dynamically adjust the preprocessing process according to sample data characteristics and access patterns to ensure data quality and improve data consistency and accuracy;

[0131] The feature analysis unit is used to analyze the statistical and distribution characteristics of sample data, perform feature selection and engineering, reduce redundant information, and enhance index features;

[0132] The access pattern evaluation unit is used to analyze the access frequency and access pattern of users or systems, predict query requirements, provide guidance for index construction and optimization, and ensure that the index structure meets query requirements.

[0133] The index generation module includes: model loading unit, knowledge graph fusion unit, strategy formulation unit, and storage unit;

[0134] Model loading unit, used to load pre-trained language model and perform deep semantic representation;

[0135] The knowledge graph fusion unit is used to integrate external knowledge graphs or external knowledge into the index construction process, enhance the semantic understanding ability of the index, and improve the accuracy of queries;

[0136] A strategy formulation unit, used to formulate an adaptive hybrid index strategy based on the analysis results, data characteristics and access patterns of sample data;

[0137] The storage unit is used to generate an index scheme according to the adaptive hybrid index strategy and store it in the index database to realize the physical storage of the index and manage the index file.

[0138] The index optimization module includes: index construction unit, performance evaluation unit, optimization algorithm unit, and optimization implementation unit;

[0139] An index building unit, used for building an index in a database system according to a hybrid indexing scheme;

[0140] A performance evaluation unit, which is used to test and evaluate the hybrid indexing scheme using a pre-trained language model, and collect query execution time and resource usage metrics such as query speed and storage space;

[0141] The optimization algorithm unit is used to use a multi-objective optimization algorithm to reconstruct, merge or delete unnecessary index items, improve index performance and reduce maintenance costs;

[0142] The optimization implementation unit is used to implement the optimized indexing scheme and update the index structure.

[0143] The monitoring module includes: performance monitoring unit, prediction model unit, and dynamic adjustment unit;

[0144] Performance monitoring unit, used to monitor the performance indicators of index building and query processing in real time, such as response time, CPU usage, and throughput;

[0145] The prediction model unit is used to use machine learning algorithms to predict the query optimization effects under different index combinations, plan resource allocation in advance, and prevent performance degradation;

[0146] The dynamic adjustment unit is used to dynamically adjust the configuration of the index construction and intelligent optimization system, such as index strategy and query optimization parameters, according to the monitoring results and prediction results, to ensure the stability and efficiency of the index construction and intelligent optimization system.

[0147] Example 3

[0148] A computer-readable storage medium stores computer instructions, which, when executed, execute the steps of a large-model-driven real-time index building and intelligent optimization method, wherein the storage medium may be a volatile or non-volatile computer-readable storage medium.

[0149] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention, and all of these are within the protection of the present invention.

Claims

1. A real-time index construction and intelligent optimization method driven by a large model, characterized in that: include: Step S1: collect sample data using data stream processing and perform dynamic adaptive preprocessing. At the same time, analyze the characteristics of the sample data and evaluate the access pattern of the sample data. Step S2: Use statistical methods to predict hot data based on historical access records and current data characteristics, and automatically trigger the loading mechanism according to the hot data prediction results to load it into the memory in advance; Step S3: Load the pre-trained language model, combine the knowledge graph method, perform deep semantic analysis on the sample data, adopt an adaptive hybrid indexing strategy based on the analysis results, sample data characteristics, access patterns and hot data prediction results, combine the entity relationship information in the knowledge graph, generate a hybrid indexing scheme, and store it in the index database; Step S4: According to the hybrid indexing scheme, index construction is implemented in the database system, and the hybrid indexing scheme is evaluated in multiple dimensions using a pre-trained language model. According to the evaluation results, a multi-objective optimization algorithm is used to automatically generate an index optimization scheme, and the optimized indexing scheme is implemented; Step S5: Monitor the performance indicators of index construction and query processing in real time, use machine learning algorithms to combine historical query data and index performance data, predict the query optimization effects under different index combinations, and obtain the optimal index combination.

2. The method for real-time index construction and intelligent optimization based on large model drive according to claim 1, characterized in that: The specific steps of step S3 include: S3.1: Load the pre-trained language model and its matching word segmenter, and use the word segmenter of the pre-trained language model to segment and encode the input sample data; S3.2: Input the encoded data into the pre-trained language model, run the forward propagation of the pre-trained language model, and extract the last hidden state of the model as the deep semantic features of the sample data; S3.3: Use the DistMult method to obtain the embedded representation of relevant entities and relations in the knowledge graph, and use the concatenation and dimension transformation methods to combine the entity and relationship embeddings obtained from the knowledge graph with the deep semantic features extracted by the pre-trained language model to form a feature vector.

3. The method for real-time index construction and intelligent optimization based on large model drive according to claim 2, characterized in that: The specific steps of step S3 also include: S3.4: Collect database access logs, perform statistical analysis on the collected access logs, and obtain user access patterns and query requirements; S3.5: Extract features from feature vectors based on user access patterns and query requirements to obtain index candidate features. Use simulation tests to evaluate the impact of different index candidate features on query performance. ,in, represents the influence factor of the nth index candidate feature, where n represents the number of index candidate features; S3.6: Based on the evaluation results, the impact factors Sort in descending order and set the minimum impact factor threshold to ; like , then the index feature is obtained ,Based on the index characteristics and query requirements, an adaptive hybrid index strategy is used to generate a hybrid index scheme, where, represents the mth index feature, and m represents the number of index features.

4. The method for real-time index construction and intelligent optimization based on large model drive according to claim 3, characterized in that: The specific steps of step S3 also include: S3.7: Introduce a dynamic index adjustment mechanism, based on a hybrid indexing solution, monitor the index query efficiency in real time, and dynamically adjust the index structure based on the monitoring results; S3.8: Use the trained machine learning model to predict the trend of index queries, interpret and analyze the predicted trend results of index queries, identify query hot spots and pattern changes, and evaluate whether the index structure or parameters need to be adjusted to optimize the adaptive hybrid index strategy based on the predicted trend results of index queries and the current index; S3.9: Write an adaptive hybrid index strategy construction script and execute it on a large-scale data set, monitoring resource consumption and performance indicators during the construction process; S3.10: After the new indexing strategy is executed, the constructed hybrid index is stored in the index database.

5. The method for real-time index construction and intelligent optimization based on large model drive according to claim 4, characterized in that: The specific steps of the multi-objective optimization algorithm in step S4 include: S4.1: Set the objective function and constraints of index optimization, and generate a set of initial index configuration solutions as candidate solutions through heuristic methods. The formula is: ； in, Represents a given i-th index configuration scheme, It represents the average or total response time required to perform a query operation under the i-th index configuration scheme. Indicates the storage space occupied by the index structure under the i-th index configuration scheme. Indicates the time required to build the index under the i-th index configuration scheme. , , represents the weight coefficient; S4.2: Use the pre-trained language model to perform multi-dimensional performance evaluation on each index configuration scheme in the candidate solution set; S4.3: Based on the results of multi-dimensional performance evaluation, a genetic algorithm is used to select some solutions from the candidate solution set as parents, generate new solutions as offspring, and at the same time, merge the parents and offspring to form a new solution set, which is then evaluated.

6. The method for real-time index construction and intelligent optimization based on large model drive according to claim 5, characterized in that: The specific steps of the multi-objective optimization algorithm in step S4 also include: S4.4: In the new solution set, the solutions are layered according to their dominance relations, so that each layer contains a set of mutually non-dominated solutions, and different non-dominated layers are obtained. The crowding degree of all individuals in the same non-dominated layer is initialized to 0. S4.5: For each individual in the non-dominated layer, sort them on each objective function respectively. For each objective function, calculate the objective function difference between the individual and its adjacent individuals and perform normalization. S4.6: Add the normalized differences of all objective functions to obtain the crowding degree of the individual. At the same time, select the individuals of the next generation population based on the level and crowding degree of the non-dominated sorting and perform iterative operations; S4.7: Set the maximum number of iterations. If the maximum number of iterations is met, stop the iteration and obtain the Pareto optimal solution. Otherwise, return to step S4.3 to continue the iteration. S4.8: Generate an index optimization plan based on the Pareto optimal solution.

7. The method for real-time index construction and intelligent optimization based on large model drive according to claim 6, characterized in that: The hybrid indexing scheme generated in S3.6 adopts hash table and bitmap indexing methods, combined with entity relationship information in the knowledge graph, to generate a hybrid indexing scheme.

8. A real-time index building and intelligent optimization system driven by a large model, which is used to implement the real-time index building and intelligent optimization method driven by a large model as described in any one of claims 1 to 7, characterized in that: include: Data processing module, index generation module, index optimization module, monitoring module; The data processing module is used to collect sample data and perform dynamic adaptive preprocessing; The index generation module is used to create an index structure based on the preprocessed sample data and generate a hybrid index scheme; The index optimization module is used to implement index construction and adjust and optimize the index structure as needed; The monitoring module is used to monitor the performance indicators of index construction and query processing in real time.

9. The large model-driven real-time index building and intelligent optimization system according to claim 8, characterized in that: The index generation module includes: a model loading unit, a knowledge graph fusion unit, a strategy formulation unit, and a storage unit; The model loading unit is used to load the pre-trained language model and perform deep semantic representation; The knowledge graph fusion unit is used to integrate the external knowledge graph or external knowledge into the index construction process, enhance the semantic understanding ability of the index, and improve the accuracy of the query; The strategy formulation unit is used to formulate an adaptive hybrid index strategy based on the analysis results, data characteristics and access patterns of the sample data; The storage unit is used to generate an indexing scheme according to the adaptive hybrid indexing strategy and store it in an index database to implement physical storage of the index and manage index files.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the steps of the large model-driven real-time index building and intelligent optimization method described in any one of claims 1 to 7 are executed.

Images