Dynamic index generation method and system based on file metadata
By using a dynamic index generation method based on archival metadata, the archival index is managed automatically and intelligently, solving the problem of cumbersome index maintenance in traditional archival information management systems and achieving high-efficiency query performance and flexible index configuration.
Patent Information
- Application Number
- CN202511325575.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-17
- Publication Date
- 2026-01-02
AI Technical Summary
In traditional archival information management systems, the maintenance of archival database indexes is cumbersome and error-prone, resulting in unstable query performance and difficulty in adapting to the needs of adjusting and adding archival categories.
By using a dynamic index generation method based on archive metadata, the metadata model is defined using a user interface. Machine learning algorithms are combined to evaluate query conditions and historical logs, automatically select and optimize index types, monitor and adjust the index structure in real time, and support dynamic changes in archive fields.
It significantly reduces the workload of index maintenance, improves query efficiency, adapts to changes in different file categories and query needs, and reduces operation and maintenance costs.
Smart Images
Figure CN121255804A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of archive information management, and in particular to a dynamic index generation method and system based on archive metadata. BACKGROUND
[0002] In traditional archive information management systems, the table structure of the archive library is often customized according to the archive category, and the archive fields of different categories differ significantly. With the continuous increase of archive data and the diversification of archive categories, manually maintaining the index of the archive library becomes a tedious and error-prone task. Each adjustment or addition of archive fields requires re-evaluation and adjustment of the index structure to ensure query efficiency. This process not only consumes time and effort, but also easily leads to improper index settings due to human negligence, thereby affecting query performance. SUMMARY
[0003] The present application provides a dynamic index generation method and system based on archive metadata to address the needs and deficiencies of current technology development, aiming to dynamically create, adjust and optimize archive data index through intelligent means.
[0004] In a first aspect, the present application provides a dynamic index generation method based on archive metadata to solve the above technical problems, and the technical solution is as follows:
[0005] A dynamic index generation method based on archive metadata, comprising the following steps:
[0006] S1, providing a user interface supporting field addition, modification and deletion, allowing users to define detailed metadata models according to archive categories and business requirements;
[0007] S2, defining query conditions related to archive fields; evaluating field query importance to set index priority according to query condition usage frequency and result impact degree, analyzing index performance standards combining query type and expected response time, and predicting future query patterns using historical query logs and machine learning algorithms to provide basis for dynamic index generation;
[0008] S3, automatically selecting or recommending suitable index types according to the data type of archive fields, query conditions and performance requirements, combining the scene adaptability of index structure and index generation algorithm, and dynamically optimizing index configuration;
[0009] S4, executing index creation statements in the database management system according to the determined index structure to generate physical indexes; testing the performance of the generated physical indexes to verify whether they meet the preset query performance requirements; deploying the verified indexes to the production environment and continuously monitoring the usage and query performance of the indexes;
[0010] S5, real-time collection of query performance data, visualization and analysis of performance data through data analysis tools, automatic generation of index optimization suggestions according to preset strategies, and dynamic adjustment of index structure according to optimization suggestions;
[0011] S6, automatically identify changes in archive fields, trigger corresponding index update processes, and continuously optimize index configuration based on machine learning algorithms and real-time performance monitoring data.
[0012] Optionally, step S1 is performed, and the defined metadata model includes four types of attributes: field name, data type, indexable or not, and index priority;
[0013] The metadata model is maintained, the field change history is recorded, and the metadata export and import functions are provided to support data exchange and change tracking.
[0014] Optionally, step S2 specifically includes:
[0015] S2.1, define query conditions related to archive fields, including exact match query, range query, fuzzy match query, multi-field combination query, field non-empty / empty value query, and field enumeration value filtering query;
[0016] S2.2, determine the usage frequency by counting the number of calls of each query condition within a specified period, determine the result influence degree by counting the proportion of query result covered data amount in total data amount or the influence of query result on core business, then evaluate the field query importance using weighted scoring method, and set the index priority according to the score from high to low;
[0017] S2.3, first determine the index adaptation requirements corresponding to the query type, then set the expected response time based on the business scenario, and then analyze the index performance standards;
[0018] S2.4, first extract key features from historical query logs, pre-process the features; then select a machine learning algorithm, train the model based on the pre-processed features; finally, output the query mode prediction results in the future specified period through the trained model, and provide basis for dynamic index generation.
[0019] Optionally, step S3 is performed, and the index structure includes four types: single field index, composite index, partial index, and adaptive index; wherein, the composite index is composed of at least two fields, and the field order is optimized according to the field query frequency and the influence of the field on the query result; the partial index only generates index for row data corresponding to field values that meet the preset range or condition; the adaptive index automatically adjusts the index structure based on the dynamic changes of query load.
[0020] Optionally, step S3 is performed, and the index generation algorithm comprises a rule-based index generation algorithm and a machine learning-based index recommendation algorithm; wherein the rule-based index generation algorithm automatically triggers the index generation process when the field meets the preset rule; and the machine learning-based index recommendation algorithm uses historical query data and archive features to predict the optimal index configuration scheme through a machine learning model.
[0021] In a second aspect, the present application provides a dynamic index generation system based on archive metadata, and the technical solutions adopted to solve the above technical problems are as follows:
[0022] A dynamic index generation system based on archive metadata, the structure of which comprises:
[0023] A metadata model definition module for providing a user interface supporting field addition, modification and deletion, and allowing a user to define a detailed metadata model according to archive categories and business requirements;
[0024] A query condition definition module for defining query conditions related to archive fields;
[0025] An index priority and dynamic prediction module for evaluating field query importance to set index priority according to the frequency of use of query conditions and the degree of influence of results, analyzing index performance standards in combination with query types and expected response times, and predicting future query modes using historical query logs and machine learning algorithms to provide a basis for dynamic index generation;
[0026] An index type adaptation and dynamic optimization module for automatically selecting or recommending an adapted index type according to the data type of the archive field, the query condition and the performance requirement, in combination with the scene adaptability of the index structure and the index generation algorithm, and dynamically optimizing index configuration;
[0027] An index implementation and test deployment module for automatically executing an index creation statement in a database management system according to the determined index structure, generating a physical index, subsequently performing performance testing on the generated physical index to verify whether it meets the preset query performance requirement, finally deploying the verified index to a production environment, and continuously monitoring the use of the index and the query performance;
[0028] An index optimization and adjustment module for collecting query performance data in real time, visualizing and analyzing the performance data through a data analysis tool, automatically generating index optimization suggestions according to a preset strategy, and dynamically adjusting the index structure according to the optimization suggestions;
[0029] An intelligent index management module for automatically identifying changes in archive fields, triggering corresponding index update processes, and continuously optimizing index configuration in combination with machine learning algorithms and real-time performance monitoring data.
[0030] Optionally, the user defines the metadata model through a user interface provided by the metadata model definition module, and the defined metadata model includes four types of attributes: field name, data type, indexability, and index priority;
[0031] The user maintains the metadata model through the metadata model definition module, records field change history, and provides metadata export and import functions to support data exchange and change tracking.
[0032] Optionally, the involved query condition definition module defines query conditions related to the archive field, and the involved query conditions include exact match query, range query, fuzzy match query, multi-field combination query, field non-empty / empty value query, and field enumeration value screening query.
[0033] Optionally, the involved index priority and dynamic prediction module specifically includes:
[0034] The index priority setting unit is used to determine the usage frequency by counting the number of calls of each query condition within a specified period, determine the result influence degree by counting the proportion of query result coverage data in total data or the impact of query result on core business, then evaluate the field query importance using weighted scoring method, and set the index priority according to the score from high to low;
[0035] The index performance standard analysis unit is used to first determine the index adaptation requirement corresponding to the query type, then set the expected response time based on the business scenario, and then analyze the index performance standard;
[0036] The index generation dynamic prediction unit is used to first extract key features from historical query logs, pre-process the features, then select a machine learning algorithm, train the model based on the pre-processed features, and finally output the query pattern prediction result in the future specified period through the trained model, to provide basis for dynamic index generation.
[0037] Optionally, the involved index structure includes four types: single field index, composite index, partial index, and adaptive index; wherein, the composite index is composed of at least two fields, and the field order is optimized according to the field query frequency and the influence degree of the field on the query result; the partial index only generates index for row data corresponding to field values that meet the pre-set range or condition; the adaptive index automatically adjusts the index structure based on the dynamic changes of query load;
[0038] The involved index generation algorithm includes rule-based index generation algorithm and machine learning-based index recommendation algorithm; wherein, the rule-based index generation algorithm automatically triggers the index generation process when the field meets the pre-set rule; the machine learning-based index recommendation algorithm uses historical query data and archive features to predict the optimal index configuration scheme through a machine learning model.
[0039] The method and system for generating a dynamic index based on archive metadata have the beneficial effects compared with the prior art that:
[0040] The present application significantly reduces the investment of archive management personnel in index maintenance and reduces operation and maintenance costs through automated and intelligent index generation and management. The dynamically optimized index structure can respond to query requests more quickly and improve the query efficiency of archive data. The present application supports dynamic addition, modification and deletion of archive fields and flexible configuration of index strategies, and adapts to changes in different archive categories and query requirements. BRIEF DESCRIPTION OF DRAWINGS
[0041] Figure 1 is a method flowchart of an embodiment of the present application; Figure 1
[0042] Figure 2 is a dynamic index generation strategy flowchart of an embodiment of the present application; Figure 2
[0043] Figure 3 is an index optimization and adjustment flowchart of an embodiment of the present application; Figure 3
[0044] Figure 4 is a module connection block diagram of an embodiment of the present application. Figure 4 DETAILED DESCRIPTION
[0045] In order to make the technical solutions, technical problems solved and technical effects of the present application clearer and more apparent, the technical solutions of the present application are described in detail below in combination with specific embodiments.
[0046] Embodiment 1
[0047] In combination with the accompanying drawings, Figure 1 the present embodiment proposes a dynamic index generation method based on archive metadata, which includes the following steps:
[0048] S1, archive metadata definition and maintenance.
[0049] A user interface supporting field addition, modification and deletion is provided to allow users to define detailed metadata models according to archive categories and business requirements. The metadata model specifically includes four types of attributes, i.e., field name, data type, whether indexable and index priority. For example, for personnel archives, fields such as "name", "gender", "age", "date of employment" can be defined, and the "name" field is set as indexable with high index priority.
[0050] At the same time, the metadata model is maintained, the field change history is recorded, and the metadata export and import functions are provided to support data exchange and change tracking.
[0051] S2, query condition and index demand analysis.
[0052] Defining query conditions related to the archive field; evaluating the importance of field queries according to the frequency of use and the impact of the results to set index priority, analyzing index performance standards in combination with query type and expected response time, and using historical query logs and machine learning algorithms to predict future query patterns to provide the basis for dynamic index generation. This process specifically includes:
[0053] S2.1, define query conditions related to the archive field, which includes exact match queries (e.g., for the "user ID" field, an exact match query condition can be defined, such as "user ID is 123"), range queries (e.g., for the "date of employment" field, a range query condition can be defined, such as "date of employment between January 1, 2020 and December 31, 2023"), fuzzy match queries (e.g., "username LIKE '% Zhang %'", "product name contains 'wireless earphones')", multi-field combination queries (e.g., for the combined field of "user city + order amount", a multi-field combination query condition can be defined, such as "user city = Beijing AND order amount > 500 yuan"), field non-empty (e.g., "shipping address IS NOT NULL") / null value query, and field enumeration value filtering query (e.g., "payment method IN ['WeChat payment', 'Alipay']");
[0054] S2.2, determine the frequency of use by counting the number of calls for each query condition within a specified period (e.g., the last 30 days) (e.g., "user ID query" has an average daily call of 1000 times, "product category query" has an average daily call of 200 times), determine the result impact degree by counting the proportion of query result coverage data to total data (e.g., a query result coverage data proportion ≤5% is high impact, a proportion ≥50% is low impact) or the impact of the query result on core business (e.g., order payment status query affects transaction closure, which is high impact), and then use a weighted scoring method (e.g., "importance score = frequency weight × call number + impact weight × impact coefficient", the weight can be set according to business needs, such as higher impact weight than frequency weight in core business scenarios) to evaluate the importance of field queries, and set index priority according to the score from high to low (e.g., the top 20% of fields are set to high priority, the 20%-50% of fields are set to medium priority, and the last 50% of fields are set to low priority);
[0055] S2.3, first determine the index adaptation requirements corresponding to the query type (e.g., real-time query, batch query), then set the expected response time based on the business scenario, and then analyze the index performance standards (e.g., for real-time queries, the index is required to have extremely high query speed, with a response time of no more than 100 milliseconds);
[0056] S2.4, first extract key features (such as query field combination, query condition type, query trigger time, query result data volume, user / business module identification) from historical query logs, preprocess the features (such as encoding discrete features "query type", normalizing continuous features "query time consumption"), then select machine learning algorithms (such as time series model ARIMA to predict the time trend of query frequency, classification algorithm random forest to predict the popularity of query field combination, clustering algorithm K-Means to divide query mode categories), train the model based on the preprocessed features; finally, output the query mode prediction results in the future specified period (such as the next 7 days) (such as "the combination query frequency of 'order status + order time' will increase by 30% from 9:00 to 12:00 on weekdays", "the joint query of 'product classification + user level' will become a new popular mode") through the trained model, to provide basis for dynamic index generation.
[0057] S3, dynamic index generation strategy.
[0058] Reference to the attached drawings Figure 2 According to the data type of the file field, the query condition and the performance requirement, combined with the scene adaptability of the index structure and the index generation algorithm, the adaptive index type (such as B-tree index, hash index, full-text index) is automatically selected or recommended, and the index configuration is dynamically optimized. For example, for the "name" field, B-tree index is selected to support exact match and range query; for the range query of the "date of employment" field, B-tree index is also selected; for the fuzzy query of the text content, such as "job description", full-text index is selected.
[0059] The index structure includes four types of single field index (for high importance and strict query performance requirement fields, such as "name", single field index is automatically generated), composite index (for example, it is found that "name" and "date of employment" fields often appear together in query conditions, so "name + date of employment" composite index is designed to optimize multi-field query performance), partial index (for example, only the "date of employment" field in the archive data of the last three years is indexed to reduce the index size and improve the query efficiency) and adaptive index (for example, when it is found that the query frequency of the "gender" field suddenly increases, the index for the field is automatically generated or the existing index structure is adjusted to include the field); among them, the composite index is composed of at least two fields, and the field order is optimized according to the field query frequency and the influence degree of the field on the query result; the partial index only generates index for the row data corresponding to the field value meeting the preset range or condition (such as only establishing index for the row of "order date > 2024-01-01"); the adaptive index automatically adjusts the index structure (such as adding / deleting index entries, optimizing index field order) based on the dynamic changes of query load (such as high-frequency query field change, query statement type adjustment).
[0060] The index generation algorithm includes a rule-based index generation algorithm and a machine learning-based index recommendation algorithm; wherein the rule-based index generation algorithm automatically triggers the index generation process when the field meets the preset rules (such as the query frequency of a certain field exceeds the preset threshold); the machine learning-based index recommendation algorithm uses historical query data and archive features to predict the optimal index configuration scheme through a machine learning model (such as decision tree, random forest, etc.). For example, a model is trained to predict which field combination is most suitable for generating a composite index to improve query efficiency.
[0061] S4, index implementation and deployment.
[0062] According to the determined index structure, the index creation statement is automatically executed in the database management system to generate a physical index; the generated physical index is tested for performance to verify whether it meets the preset query performance requirements (for example, a series of queries are executed using a test data set, and the query response time in the indexed and non-indexed cases is compared); the verified index is deployed to the production environment, and the use of the index and the query performance are continuously monitored;
[0063] S5, index optimization and adjustment.
[0064] Referring to the accompanying drawings Figure 3 , real-time collection of query performance data, visualization and analysis of performance data through data analysis tools (such as ELK Stack, Prometheus, etc.), automatic generation of index optimization suggestions according to preset strategies (for example, when it is found that the hit rate of a composite index is low, it is recommended to delete the index or adjust its field combination; when it is found that the query frequency of a certain field increases, it is recommended to generate an index for it or adjust the existing index structure), and dynamic adjustment of the index structure according to the optimization suggestions (for example, using the online index reconstruction function of the database to adjust the index structure without downtime maintenance).
[0065] S6, intelligent index management.
[0066] Automatically identify changes in archive fields (such as addition, modification, deletion), trigger corresponding index update processes (for example, when a user deletes a field, automatically delete all indexes related to the field; when a user adds a field, predict whether an index needs to be generated for it according to preset rules or machine learning models), combine machine learning algorithms and real-time performance monitoring data, and return to step S5 to realize continuous optimization of index configuration.
[0067] Embodiment two:
[0068] Combined with the accompanying drawings Figure 4 , the embodiment proposes a dynamic index generation system based on archive metadata, the structure of which includes:
[0069] A metadata model definition module for providing a user interface supporting field addition, modification, and deletion, allowing users to define detailed metadata models according to archive categories and business requirements;
[0070] A query condition definition module for defining query conditions related to archive fields;
[0071] An index priority and dynamic prediction module for evaluating field query importance to set index priority according to query condition usage frequency and result impact degree, analyzing index performance standards combining query type and expected response time, and predicting future query patterns using historical query logs and machine learning algorithms to provide basis for dynamic index generation;
[0072] An index type adaptation and dynamic optimization module for automatically selecting or recommending adapted index types according to archive field data types, query conditions, and performance requirements, combining index structure scenario adaptability and index generation algorithms, and dynamically optimizing index configuration;
[0073] An index implementation and test deployment module for automatically executing index creation statements in a database management system according to determined index structures, generating physical indexes, then performing performance tests on generated physical indexes to verify whether they meet preset query performance requirements (e.g., executing a series of queries using test data sets, comparing query response times with and without indexes), and finally deploying verified indexes to production environments and continuously monitoring index usage and query performance;
[0074] An index optimization and adjustment module for collecting query performance data in real time, visualizing and analyzing performance data through data analysis tools (e.g., ELKStack, Prometheus, etc.), automatically generating index optimization suggestions according to preset strategies, and dynamically adjusting index structures according to optimization suggestions (e.g., using database online index reconstruction functions to adjust index structures without downtime maintenance);
[0075] An intelligent index management module for automatically identifying changes in archive fields (e.g., addition, modification, deletion), triggering corresponding index update processes (e.g., when a user deletes a field, the system automatically deletes all indexes related to the field; when a user adds a field, the system predicts whether an index needs to be generated for it according to preset rules or machine learning models), and continuously optimizing index configuration combining machine learning algorithms and real-time performance monitoring data.
[0076] In this embodiment, the user defines the metadata model through the user interface provided by the metadata model definition module, and the defined metadata model includes four types of attributes: field name, data type, indexability, and index priority. For example, for personnel files, fields such as "name", "gender", "age", and "date of employment" can be defined, and the "name" field can be set as indexable with high index priority.
[0077] At the same time, the user maintains the metadata model through the metadata model definition module, records the field change history, and provides metadata export and import functions to support data exchange and change tracking.
[0078] In this embodiment, the query condition definition module defines query conditions related to the file fields. The query conditions include exact match queries (for example, for the "user ID" field, an exact match query condition such as "user ID is 123" can be defined), range queries (for example, for the "date of employment" field, a range query condition such as "date of employment is between January 1, 2020 and December 31, 2023" can be defined), fuzzy match queries (for example, "username LIKE '% Zhang %'", "product name contains 'wireless earphone'"), multi-field combination queries (for example, for the combined field "user city + order amount", a multi-field combination query condition such as "user city = Beijing AND order amount > 500 yuan" can be defined), field non-empty (such as "shipping address IS NOT NULL") / null value queries, and field enumeration value filtering queries (such as "payment method IN ['WeChat payment', 'Alipay']").
[0079] In this embodiment, the index priority and dynamic prediction module specifically includes:
[0080] The index priority setting unit is used to determine the usage frequency (such as "user ID query" with an average daily call of 1000 times, and "product category query" with an average daily call of 200 times) by counting the number of calls of each query condition in a specified period (such as the last 30 days), determine the result impact degree (such as order payment status query affecting transaction closure, belonging to high impact degree) by counting the proportion of query result coverage data in total data (such as query result coverage data accounting for ≤5% of total data as high impact degree, and ≥50% as low impact degree), and then using a weighted scoring method (such as "importance score = usage frequency weight x call number + impact degree weight x impact coefficient", and the weight can be set according to business needs, such as higher impact degree weight than frequency weight in core business scenarios) to evaluate the importance of field query, and set the index priority according to the score from high to low (such as setting the top 20% of fields with high priority, the 20%-50% of fields with medium priority, and the last 50% of fields with low priority);
[0081] The index performance standard analysis unit is used to first determine the index adaptation requirements corresponding to the query type (such as real-time query, batch query), and then set the expected response time based on the business scenario, and then analyze the index performance standard (for example, for real-time query, the index is required to have extremely high query speed, and the response time is not more than 100 milliseconds);
[0082] The index generation dynamic prediction unit is used to first extract key features (such as query field combination, query condition type, query trigger time, query result data volume, user / business module identification) from historical query logs, preprocess the features (such as encoding the discrete feature "query type" and normalizing the continuous feature "query time consumption"), then select a machine learning algorithm (such as the time series model ARIMA to predict the time trend of query frequency, the classification algorithm random forest to predict the popularity of query field combination, and the clustering algorithm K-Means to divide query mode categories), train the model based on the preprocessed features, and finally output the query mode prediction results (such as "the combination query frequency of 'order status + order time' will increase by 30% from 9:00 to 12:00 on weekdays", and "the joint query of 'product classification + user level' will become a new popular mode") in the future specified period (such as the next 7 days) through the trained model, providing basis for dynamic index generation.
[0083] In this embodiment, the index structure involved includes four types of single field index (for fields with high importance and strict query performance requirements, such as "name", automatically generating single field index), composite index (for example, it is found that "name" and "date of employment" fields often appear together in query conditions, so a composite index of "name + date of employment" is designed to optimize multi-field query performance), partial index (for example, only generate index for "date of employment" field in the archive data of the last three years, reduce index size, and improve query efficiency), and adaptive index (for example, when it is found that the query frequency of the "gender" field suddenly increases, automatically generate an index for this field or adjust the existing index structure to include this field); wherein the composite index is composed of at least two fields, and the field order is optimized according to the field query frequency and the influence degree of the field on the query result; the partial index only generates index for the row data corresponding to the field value that meets the preset range or condition (such as only establishing index for the row where "order date > 2024-01-01"); the adaptive index automatically adjusts the index structure (such as adding / deleting index entries, optimizing index field order) based on the dynamic changes of query load (such as high-frequency query field change, query statement type adjustment).
[0084] The index generation algorithm involved includes a rule-based index generation algorithm and a machine learning-based index recommendation algorithm; wherein the rule-based index generation algorithm automatically triggers the index generation process when the field meets the preset rule (such as the query frequency of a certain field exceeds the preset threshold); the machine learning-based index recommendation algorithm uses historical query data and archive features to predict the optimal index configuration scheme through a machine learning model (such as decision tree, random forest, etc.). For example, a model is trained to predict which field combination is most suitable for generating a composite index to improve query efficiency.
[0085] In summary, the dynamic index generation method and system based on archive metadata can automatically maintain and optimize the query conditions and corresponding index structure of the archives to adapt to the dynamic changes of different archive category fields, thereby reducing the workload of archive operation and maintenance and improving the archive data query efficiency.
[0086] The above application specific examples have described the principles and implementation modes of the present application in detail, and these examples are only used to help understand the core technical content of the present application. Based on the above specific embodiments of the present application, any improvement and modification of the present application made by the person skilled in the art without departing from the principles of the present application shall fall within the patent protection scope of the present application.
Claims
1. A method for generating a dynamic index based on archival metadata, characterized by, The method comprises the following steps: S1, providing a user interface supporting field addition, modification and deletion, allowing users to define detailed metadata models according to archive categories and business requirements; S2, defining query conditions related to archive fields; According to the frequency of use of the query conditions and the result influence degree, the importance of field query is evaluated to set the index priority, the index performance standard is analyzed in combination with the query type and the expected response time, and the future query mode is predicted by using the historical query log and the machine learning algorithm to provide the basis for dynamic index generation; S3, according to the data type of the archive field, the query condition and the performance requirement, in combination with the scene adaptability of the index structure and the index generation algorithm, the adaptive index type is automatically selected or recommended, and the index configuration is dynamically optimized; S4, according to the determined index structure, automatically executing index creation statements in the database management system to generate physical indexes; performance test is performed on the generated physical indexes to verify whether they meet the preset query performance requirements; the indexes that pass the verification are deployed to the production environment, and the use of the indexes and the query performance are continuously monitored; S5, real-time collection of query performance data, visual display and analysis of the performance data by using a data analysis tool, automatic generation of index optimization suggestions according to a preset strategy, and dynamic adjustment of the index structure according to the optimization suggestions; S6, automatically identifying changes of the archive fields, triggering corresponding index update processes, continuously optimizing the index configuration in combination with the machine learning algorithm and real-time performance monitoring data.
2. The method of claim 1, wherein, Step S1 is performed, and the defined metadata model includes four types of attributes: field name, data type, whether indexable, and index priority; The metadata model is maintained, the field change history is recorded, and metadata export and import functions are provided to support data exchange and change tracking.
3. The method of claim 1, wherein the method further comprises: The step S2 specifically comprises: S2.1, defining query conditions related to archive fields, the query conditions including exact match query, range query, fuzzy match query, multi-field combination query, field non-empty / empty value query and field enumeration value filtering query; S2.2, determine the frequency of use by counting the number of calls of each query condition in a specified period, determine the result influence degree by counting the proportion of the data covered by the query result in the total data or the influence of the query result on the core business, then evaluate the importance of field query by using the weighted scoring method, and set the index priority according to the score from high to low; S2.3, first, determine the index adaptation requirements corresponding to the query type, then set the expected response time based on the business scenario, and then analyze the index performance standard; S2.4, first, extract key features from the historical query log, and preprocess the features; then select a machine learning algorithm, train a model based on the preprocessed features; finally, output the query mode prediction results in the future specified period by the trained model to provide the basis for dynamic index generation.
4. The method of claim 1, wherein the method further comprises: The step S3 is performed, and the index structure includes four types of single-field index, composite index, partial index and adaptive index; wherein the composite index is composed of at least two fields, and the field order is optimized according to the field query frequency and the influence degree of the field on the query result; the partial index only generates index for the row data corresponding to the field value meeting the preset range or condition; the adaptive index automatically adjusts the index structure based on the dynamic change of query load.
5. The method of claim 1, wherein the method further comprises: The step S3 is performed, and the index generation algorithm includes a rule-based index generation algorithm and a machine learning-based index recommendation algorithm; wherein the rule-based index generation algorithm automatically triggers the index generation process when the field meets the preset rule; the machine learning-based index recommendation algorithm uses historical query data and archive features to predict the optimal index configuration scheme through a machine learning model.
6. An archival metadata based dynamic index generation system, characterized by, The structure comprises: A metadata model definition module for providing a user interface supporting field addition, modification and deletion, allowing users to define detailed metadata models according to archive categories and business requirements; A query condition definition module for defining query conditions related to archive fields; An index priority and dynamic prediction module for evaluating field query importance to set index priority according to query condition usage frequency and result influence degree, analyzing index performance standards in combination with query type and expected response time, and predicting future query patterns using historical query logs and machine learning algorithms to provide basis for dynamic index generation; An index type adaptation and dynamic optimization module for automatically selecting or recommending adapted index types in combination with index structure scenario adaptability and index generation algorithm according to archive field data types, query conditions and performance requirements, and dynamically optimizing index configuration; An index implementation and test deployment module for automatically executing index creation statements in a database management system according to the determined index structure, generating physical indexes, then testing the generated physical indexes for performance, verifying whether they meet the preset query performance requirements, finally deploying the verified indexes to the production environment, and continuously monitoring the usage and query performance of the indexes; An index optimization and adjustment module for collecting query performance data in real time, visualizing and analyzing the performance data through data analysis tools, automatically generating index optimization suggestions according to preset strategies, and dynamically adjusting the index structure according to the optimization suggestions; An intelligent index management module for automatically identifying changes in archive fields, triggering corresponding index update processes, and continuously optimizing index configuration in combination with machine learning algorithms and real-time performance monitoring data.
7. The system for generating dynamic index based on archival metadata as claimed in claim 6 wherein, The user defines the metadata model through the user interface provided by the metadata model definition module, and the metadata model includes four attributes of field name, data type, indexability and index priority; The user maintains the metadata model through the metadata model definition module, records field change history, and provides metadata export and import functions to support data exchange and change tracking.
8. The system for generating dynamic index based on archival metadata as claimed in claim 6 wherein, The query condition definition module defines a query condition related to an archive field, and the query condition includes an exact match query, a range query, a fuzzy match query, a multi-field combination query, a field non-empty / empty value query, and a field enumeration value screening query.
9. The system for generating dynamic index based on archival metadata as claimed in claim 8 wherein, The index priority and dynamic prediction module specifically includes: An index priority setting unit is configured to determine a use frequency by counting the number of calls of each query condition in a specified period, determine a result influence degree by counting the proportion of the amount of data covered by the query result in the total amount of data or the influence of the query result on core business, and then evaluate the importance of field query by using a weighted scoring method, and set the index priority according to the scores from high to low; An index performance standard analysis unit is configured to first determine the index adaptation requirement corresponding to the query type, then set an expected response time based on a business scenario, and then analyze the index performance standard; An index generation dynamic prediction unit is configured to first extract key features from historical query logs, pre-process the features, then select a machine learning algorithm, train a model based on the pre-processed features, and finally output the query mode prediction result in a future specified period by the trained model, to provide a basis for dynamic index generation.
10. The system for generating dynamic index based on archival metadata as claimed in claim 6 wherein, The index structure includes four types of single-field index, composite index, partial index, and adaptive index; wherein the composite index is composed of at least two fields, and the field order is optimized according to the field query frequency and the influence degree of the field on the query result; the partial index only generates an index for row data corresponding to field values that meet a preset range or condition; the adaptive index automatically adjusts the index structure based on the dynamic changes of query load; The index generation algorithm includes a rule-based index generation algorithm and a machine learning-based index recommendation algorithm; wherein the rule-based index generation algorithm automatically triggers the index generation process when the field meets the preset rule; the machine learning-based index recommendation algorithm uses historical query data and archive features to predict the optimal index configuration scheme through a machine learning model.