Data table processing method and device, storage medium and electronic equipment

Abstract

The invention relates to a data table processing method and device, a storage medium and electronic equipment. The method comprises the steps of obtaining real-time access operation information, historical access operation information and a data table initial index of a target data table; performing optimization analysis on the initial index of the data table according to the real-time access operation information by using an index adjustment algorithm corresponding to the target data table to obtain index optimization data; using an index prediction algorithm corresponding to the target data table to predict and analyze the initial index of the data table according to the historical access operation information to obtain index prediction data; searching a Pareto leading edge of the index optimization data and the index prediction data to obtain index optimal adjustment data; and optimizing the target data table according to the index optimal adjustment data to obtain an optimized data table. By adopting the method, the application efficiency of the index of the data table in complex, dynamic and large-scale data processing can be effectively improved, and a more efficient data table management and maintenance scheme is further provided.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data table processing method, device, storage medium and electronic device. Background Art

[0002] Data table processing methods are widely used in the fields of data analysis, storage and computing. This method mainly includes data cleaning, data conversion, data analysis and data visualization. Through data cleaning, invalid or erroneous data is eliminated to improve data quality; data conversion standardizes and formats multi-source heterogeneous data for subsequent operations; data analysis uses tools such as statistics and machine learning to extract valuable information; data visualization uses charts or dashboards to intuitively display results. In addition, modern data table processing often uses professional software (such as Excel, Google Sheets) or programming tools (such as Python's Pandas library) to achieve efficient and automated operations.

[0003] The index in the data table processing method in traditional technology has limited indexing efficiency for large-scale data sets. When the data volume is huge, the construction and query of the index may consume more resources. Furthermore, the index method of traditional technology has weak support for dynamic data. When the data is frequently updated or changed, the maintenance cost of the index is high. This leads to low application efficiency of data table index in complex, dynamic, and large-scale data processing. Summary of the invention

[0004] Based on this, it is necessary to provide a data table processing method, device, storage medium and electronic device that can effectively improve the application efficiency of data table indexes in complex, dynamic and large-scale data processing in response to the above technical problems.

[0005] In a first aspect, the present application provides a data table processing method, comprising:

[0006] Obtaining real-time access operation information, historical access operation information, and initial index of the target data table;

[0007] Using the index adjustment algorithm corresponding to the target data table, and according to the real-time access operation information, optimizing and analyzing the initial index of the data table to obtain index optimization data;

[0008] Using the index prediction algorithm corresponding to the target data table, and based on the historical access operation information, performing prediction analysis on the initial index of the data table to obtain index prediction data;

[0009] Finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data;

[0010] The target data table is optimized according to the index optimal adjustment data to obtain an optimized data table.

[0011] In a second aspect, the present application further provides a data table processing device, comprising:

[0012] A data acquisition module is used to acquire real-time access operation information, historical access operation information and initial index of the data table of the target data table;

[0013] An index analysis module, configured to use an index adjustment algorithm corresponding to the target data table to optimize and analyze the initial index of the data table according to the real-time access operation information to obtain index optimization data;

[0014] An index prediction module, configured to use an index prediction algorithm corresponding to the target data table to perform prediction analysis on the initial index of the data table according to the historical access operation information to obtain index prediction data;

[0015] An index optimization module, used to find the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data;

[0016] The table optimization module is used to optimize the target data table by optimally adjusting the data according to the index to obtain an optimized data table.

[0017] In a third aspect, the present application further provides a data table processing system, the system comprising an electronic device and a storage medium, the storage medium storing a computer program, and the computer program, when executed by the electronic device, implements the steps of a data table processing method.

[0018] The above-mentioned data table processing method, device, storage medium and electronic device can optimize and analyze the index through real-time access data, adjust the index strategy according to the current access mode, and improve the query efficiency; at the same time, the predictive analysis of historical access data can foresee the future access trend and further guide the index adjustment. Combined with the Pareto frontier analysis of the optimized data and the predicted data, the optimal balance point can be found between multiple optimization goals, thereby reducing the waste of storage space and the cost of index maintenance while ensuring the query response speed. Due to the combination of real-time access operation information, historical access operation information and the initial index of the data table, the targeted index adjustment algorithm and index prediction algorithm are adopted to realize the dynamic optimization and intelligent prediction of the data table index, which can significantly improve the query performance, storage efficiency and overall response speed of the data table, reduce the performance bottleneck caused by unreasonable index design, effectively improve the application efficiency of the data table index in complex, dynamic and large-scale data processing, and further provide a more efficient data table management and maintenance solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 An application environment diagram of a data table processing method in an embodiment;

[0021] Figure 2 is a flow chart of a data table processing method in one embodiment;

[0022] Figure 3 A schematic diagram of a flow chart of a first method for obtaining index optimization data in an embodiment;

[0023] Figure 4 A schematic diagram of a flow chart of a second method for obtaining index optimization data in an embodiment;

[0024] Figure 5 A flowchart of a method for obtaining a logical reconstruction index in one embodiment;

[0025] Figure 6 A schematic diagram of a flow chart of a first method for obtaining index prediction data in an embodiment;

[0026] Figure 7 A schematic diagram of a flow chart of a second method for obtaining index prediction data in an embodiment;

[0027] Figure 8 A schematic diagram of a flow chart of a method for obtaining optimal index adjustment data in one embodiment;

[0028] Fig. 9 A schematic flow chart of a method for optimizing a data table in one embodiment;

[0029] Fig.10 is a structural block diagram of a data table processing device in one embodiment;

[0030] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] A data table processing method provided in an embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the server 104 can be implemented with an independent server or a server cluster composed of multiple servers.

[0033] In an exemplary embodiment, Figure 2 As shown, a data table processing method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 210. Among them:

[0034] Step 202: Acquire real-time access operation information, historical access operation information, and an initial index of the target data table.

[0035] The target data table may be a specific table in the data table that needs to be optimized, and it contains a series of data and their associations.

[0036] The real-time access operation information may be a detailed record of various access operations performed by the user on the target data table at the current moment in the data table system.

[0037] The historical access operation information may be a record of accesses to the target data table by the data table system in the past period of time.

[0038] The data table initial index may be an original index structure created in the data table for the target data table, which is usually set when the data table is created or in an early optimization process. The initial index includes a single-column index, a composite index, and the like.

[0039] Specifically, the real-time access operation information of the target data table is extracted from the current interaction information between the computer and the operation object of the data table management system, including the current query request, operation type (such as insert, delete, update, query) and its frequency. At the same time, the historical access operation information of the target data table is extracted from the data table management system. The operation information covers the access records of the data table in the past period of time, such as query mode, operation mode and time series changes of data access. In addition, it is also necessary to obtain the initial index structure of the target data table, including basic information such as the existing index type, index field and index usage in the table.

[0040] Step 204 , using the index adjustment algorithm corresponding to the target data table, optimize and analyze the initial index of the data table according to the real-time access operation information to obtain index optimization data.

[0041] Among them, the index adjustment algorithm can be an algorithm that automatically adjusts the index structure of a data table according to real-time access operation information or specific optimization goals. Generally, a self-organizing algorithm is selected, such as one or more of self-organizing maps, ant colony algorithms, particle swarm algorithms, genetic algorithms, collective intelligence and swarm intelligence algorithms, and adaptive neural networks.

[0042] Among them, optimization analysis can evaluate the efficiency of existing indexes and query execution plans based on the current data table usage, query mode, access frequency, etc. Through this analysis, performance bottlenecks, redundant indexes or inefficient query paths can be identified, and corresponding optimization solutions can be proposed, such as creating new indexes, rebuilding existing indexes, etc.

[0043] The index optimization data may be a set of index optimization suggestions or results obtained through analysis of real-time access operation information. These data may include decisions on newly created indexes, modified indexes, or deletion of redundant indexes.

[0044] Specifically, the index adjustment algorithm corresponding to the target data table is used to analyze the real-time access operation information such as query frequency, query type, query field, and execution time of each query, so as to identify which fields are frequently queried, which indexes are redundant, or whether some queries do not use existing indexes. Based on the real-time query pattern, the index adjustment algorithm calculates the optimized data, such as creating a new composite index, reconstructing an existing index, or deleting an invalid index. The goal of the optimization analysis is to make the index structure more efficient, reduce unnecessary storage overhead, and improve the execution efficiency of queries. The final index optimization data includes new index structure data or adjustment data for existing indexes, such as creating composite indexes for frequently queried "user ID" and "order date" fields.

[0045] Step 206 , using the index prediction algorithm corresponding to the target data table, and based on the historical access operation information, perform prediction analysis on the initial index of the data table to obtain index prediction data.

[0046] Among them, the index prediction algorithm can be an algorithm that predicts the access pattern and query requirements of future data tables based on historical access operation information. Generally, one or more of a time series model, a regression model, and a deep learning model are selected.

[0047] Among them, predictive analysis can be achieved by in-depth analysis of historical access operation information to infer future data table access patterns or performance bottlenecks. The system can identify possible growing query demands or changing query patterns in advance, and provide forward-looking information support for the adjustment and optimization of index structures.

[0048] The index prediction data may be predictive data generated by an index prediction algorithm based on historical access operation information and used to guide future index optimization.

[0049] Specifically, because historical access operation information contains data query patterns within a certain period of time in the past, including the access frequency of query fields, the time distribution of operations, and query types, etc. By analyzing these historical data, the index prediction algorithm can identify access trends that may appear in the future, such as the query frequency of certain fields may increase, or the query patterns of certain table connections will become more common. Based on this trend analysis, the index prediction algorithm can predict which indexes will become more important in the future and which indexes may lose their effectiveness. At the same time, the index prediction algorithm will provide a series of index adjustment suggestions, such as adding indexes for specific fields, modifying the structure of existing indexes, or deleting indexes that are no longer frequently accessed. The final index prediction data includes the prediction results of future index adjustments, such as predicting that the query frequency of the "product classification" field will increase significantly in the future, so it is recommended to create a separate index for this field in advance.

[0050] Step 208, finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data.

[0051] The optimal index adjustment data may be an optimal index adjustment solution obtained after multi-objective optimization (such as query efficiency, storage cost, etc.) is performed on the index optimization data and the index prediction data.

[0052] Specifically, since index optimization data and index prediction data usually involve multiple optimization objectives, such as query response time, index storage space, index update overhead, query hit rate, etc., there may be a certain trade-off between these objectives. For example, adding more indexes can improve query efficiency, but it will increase storage overhead and index maintenance cost. Through Pareto front analysis, the optimal balance point can be found between multiple objectives, that is, the best trade-off between query performance, storage cost and maintenance overhead. The analysis results reveal the optimal index adjustment scheme under different objective conditions, which optimizes the index structure and reduces unnecessary costs and resource waste without sacrificing the overall performance of the data table. The final index optimal adjustment data is the best index adjustment strategy selected based on these multi-objective balances, such as deleting the currently uncommon "product description" field index and adding a composite index covering "user ID" and "order date" to achieve a better balance between query performance and storage efficiency.

[0053] Step 210, optimizing the target data table according to the optimal index adjustment data to obtain an optimized data table.

[0054] Among them, optimizing the data table can be the result of actually adjusting the target data table after obtaining the index optimal adjustment data, which includes operations such as modifying the index structure of the data table, deleting redundant indexes, and creating new efficient indexes.

[0055] Specifically, the target data table is optimized and adjusted according to the index optimal adjustment data, which includes creating, deleting or modifying indexes in the data table, adjusting the existing index structure to adapt to new query patterns and performance requirements, and optimizing interaction and adaptability. The optimized data table will have adjusted indexes and appropriate styles, which can significantly improve query efficiency and reduce maintenance costs, thereby generating an optimized data table.

[0056] In the above-mentioned data table processing method, by optimizing and analyzing the index through real-time access data, the index strategy can be adjusted according to the current access mode to improve the query efficiency; at the same time, the predictive analysis of historical access data can foresee the future access trend and further guide the index adjustment. Combined with the Pareto frontier analysis of the optimized data and the predicted data, the optimal balance point can be found between multiple optimization goals, thereby reducing the waste of storage space and the cost of index maintenance while ensuring the query response speed. Due to the combination of real-time access operation information, historical access operation information and the initial index of the data table, the targeted index adjustment algorithm and index prediction algorithm are adopted to realize the dynamic optimization and intelligent prediction of the data table index, which can significantly improve the query performance, storage efficiency and overall response speed of the data table, reduce the performance bottleneck caused by unreasonable index design, effectively improve the application efficiency of the data table index in complex, dynamic and large-scale data processing, and further provide a more efficient data table management and maintenance solution.

[0057] In an exemplary embodiment, Figure 3 As shown, using the index adjustment algorithm corresponding to the target data table, the initial index of the data table is optimized and analyzed according to the real-time access operation information to obtain index optimization data, including steps 302 to 306. Wherein:

[0058] Step 302: Identify the hot data blocks and high-frequency query types corresponding to the target data table from the real-time access operation information.

[0059] Specifically, the system analyzes the real-time access operation information in the current human-computer interaction data table to identify all operation records for the target data table, including the query fields, query frequency, data access path, etc. Based on the information obtained from the analysis, the system uses algorithms to identify which data blocks are frequently accessed (i.e., hot data blocks) and which query types appear more frequently (i.e., high-frequency query types). Among them, hot data blocks usually represent areas in the data table that users frequently access, while high-frequency query types indicate the field combinations or query patterns that users often use when querying.

[0060] Step 304: According to the hot data blocks and the high-frequency query types, the index structure of the initial index of the data table is adjusted to obtain an adjusted index of the data table.

[0061] Among them, the index structure can be a data storage structure created in a data table to speed up query. It usually consists of a set of data columns and related index fields, with the purpose of improving data retrieval efficiency.

[0062] The data table adjustment index may be an index obtained after modification or optimization such as reconstructing the existing data table index structure, deleting redundant indexes, adding new indexes, or changing the order of index fields.

[0063] Specifically, based on hot data blocks and high-frequency query types, the system optimizes and adjusts the index structure of the initial index of the target data table. For example, for frequently accessed hot data blocks, it may be necessary to create more targeted indexes to speed up queries on these specific data blocks; at the same time, for high-frequency query types, the system may recommend creating composite indexes for commonly used query fields, or adjusting existing indexes to support more efficient query paths. The adjusted index structure should be able to effectively improve query efficiency, reduce unnecessary calculations and data scanning, and obtain an adjusted index for the data table.

[0064] Step 306: Use the index adjustment algorithm to optimize the access mode of the adjusted index of the data table to obtain index optimization data.

[0065] Among them, the access mode can be the way and frequency in which users or applications access data in a data table, which includes aspects such as the query type of access, the frequency of field use, the time distribution of queries, and the path of data retrieval.

[0066] Specifically, after adjusting the index structure of the target data table, the index adjustment algorithm is used to further optimize the access mode of the adjusted index. In the process of optimizing the access mode, the performance of the new index after adjusting the index structure is evaluated mainly by simulating or analyzing the impact of the adjusted index on the actual query mode. Therefore, the index adjustment algorithm will further identify new query patterns, scanning paths and their efficiency, and adjust the adjusted index to improve its performance. The optimization process may include deleting redundant indexes, merging indexes, or adjusting the storage structure of the index. Through these optimizations, the final index optimization data will contain a better index structure configuration, effectively support efficient queries, reduce storage overhead, and ensure that index maintenance costs are minimized.

[0067] In this embodiment, by identifying hot data blocks and high-frequency query types from real-time access operation information, it can help to accurately locate the most frequently accessed and queried parts of the data table, thereby providing an optimization basis for the initial index structure of the data table. By adjusting the index structure according to the hot data blocks and high-frequency query types, the query efficiency can be improved and unnecessary data access can be reduced, making the query operation more efficient. Furthermore, using the index adjustment algorithm to optimize and analyze the adjusted index can further improve the access mode of the index, reduce disk I / O and memory usage, and ultimately significantly improve the overall performance of the data table, ensure the efficiency of data query and operation, and meet the real-time data access requirements. This optimization method can not only improve the response speed of the system, but also reduce resource consumption, providing users with a faster and more stable service experience.

[0068] In an exemplary embodiment, Figure 4 As shown, the index adjustment algorithm is used to optimize the access mode of the data table index adjustment to obtain index optimization data, including steps 402 to 404. Among them:

[0069] Step 402: reconstruct the index logic of the data table adjustment index according to the access frequency of the hot data block to obtain a logically reconstructed index.

[0070] Among them, index logic can be the design and organization of the index structure in the data table, which mainly involves the selection of index fields, the index hierarchy structure, and the storage and retrieval method of the data inside the index. It determines how the index is associated with the records in the data table and how to effectively locate and retrieve data through the index.

[0071] Among them, logical refactoring of indexes can be to optimize the data table indexes by adjusting the organization and structure of the indexes to adapt to new query patterns or access requirements. This usually includes changing the order of index fields, creating new composite indexes, adjusting the hierarchical structure of indexes, etc., in order to more effectively support frequent query operations or access patterns.

[0072] Specifically, the access frequency of hot data blocks is analyzed to identify the situations in which different hot data blocks are frequently accessed during the query process. These hot data blocks usually have bottlenecks or high-frequency access areas for data table queries. Based on the above analysis information, the index adjustment algorithm will rebuild the logical structure of the index, that is, adjust the selection of index fields, the organization of the index and its hierarchical structure, etc. By optimizing the index logic, frequently accessed data blocks can be made easier to access, thereby improving query efficiency. For example, a new composite index may be created for the hot data blocks, or the existing index may be optimized to ensure rapid positioning of these hot data blocks. The resulting logically reconstructed index will more efficiently support queries, especially high-frequency access to hot data blocks.

[0073] Step 404: According to the addition, deletion and modification operation information in the real-time access operation information, the refresh frequency and resource allocation information of the logical reconstruction index are adjusted to obtain index optimization data.

[0074] The add, delete and modify operation information may be detailed information of the add, delete and modify operations recorded in the data table, including inserting, deleting and updating data.

[0075] The refresh frequency may be the frequency of updating or rebuilding the index in the data table, which is usually closely related to the frequency of data changes.

[0076] The resource allocation information may be how the data table allocates and uses system resources (such as computing power, memory, disk space, etc.) when performing index update and query operations.

[0077] Specifically, the system further analyzes the add, delete, and modify operation information in real-time access operations, because these operations directly affect the update and maintenance of the index. Since the add, delete, and modify operations will cause changes in the records in the data table, which may affect the validity and accuracy of the index, the index adjustment algorithm will adjust the refresh frequency of the logical reconstruction index based on the information of the add, delete, and modify operations. For example, for frequently updated data, the refresh frequency of the index can be increased to ensure that the index is always kept up to date; for less updated data, the refresh frequency can be reduced to save computing and storage resources. In addition, the system will also synchronously optimize the computing resources and storage resources during the index update process based on the resource allocation situation to ensure efficient use of resources. The adjusted index optimization data includes a new index refresh strategy and resource allocation strategy to obtain index optimization data.

[0078] In this embodiment, by reconstructing the index logic of the data table to adjust the index according to the access frequency of the hot data blocks, it is possible to optimize the most frequently accessed data blocks so that these data blocks can be retrieved and operated more quickly, thereby improving query efficiency and response speed. The reconstructed index logic can adjust the organization of the index according to the actual access mode, making the access path of commonly used data more efficient. At the same time, combined with the addition, deletion and modification operations in the real-time access operation information, the refresh frequency and resource allocation of the adjustment logic reconstruction index ensure that the index can be updated in time and adapt to changes in data, avoiding performance degradation caused by outdated indexes or unreasonable resource allocation. This optimization not only improves the query efficiency of the data table, but also reduces the waste of system resources through refined resource management, further improving the overall performance and user experience.

[0079] In an exemplary embodiment, Figure 5 As shown, according to the access frequency of the hot data block, the index logic of the data table adjustment index is reconstructed to obtain the logical reconstruction index, including steps 502 to 506. Among them:

[0080] Step 502: rearrange the order of the index nodes in the data table adjustment index according to the access frequency of the hot data block to obtain a rearranged adjustment index.

[0081] Among them, the index node can be a basic unit in the data table index structure, which includes a pointer or a key-value pair pointing to a data record.

[0082] The rearranged and adjusted index may be a new index obtained by reorganizing or reordering the existing index structure according to changes in data access frequency or query mode.

[0083] Specifically, by analyzing the access frequency of hot data blocks, identifying the access situations of different hot data blocks in the query, and combining the access patterns of hot data blocks, the index adjustment algorithm will reorder the existing index nodes in the data table. Specifically, frequently accessed nodes will be prioritized at the front or higher levels of the index tree for quick positioning. The purpose of this is to optimize the query path so that hot data blocks that are frequently accessed by queries can be more efficiently retrieved in the index tree, thereby reducing the index search time and computing resource consumption. Through this rearrangement, the resulting "rearranged and adjusted index" can better respond to the high-frequency access needs of hot data blocks.

[0084] Step 504 , matching the access type of each index level according to the query response time and index hit rate corresponding to each index level in the rearranged adjustment index to obtain a type-adjusted index.

[0085] The index levels may be different levels in an index tree or index structure. Typically, the index levels are divided into root nodes, inner nodes, and leaf nodes. In a multi-level index structure, the root node points to the inner node, the inner node points to the leaf node, and the leaf node stores a specific data record or a pointer to the data.

[0086] The query response time is the time required for the data table system to process a query request and return the result. This time includes all the steps from the user initiating the query request to the final return of the result, including query parsing, optimization, execution, and data access.

[0087] The index hit rate may be the ratio of target data that can be directly located through the index in a data table query.

[0088] The access type can be different ways or modes used by a data table query request when accessing data. Common access types include exact match query (such as searching for records with a specific ID), range query (such as searching for records within a certain time period), sort query (such as the result of sorting by a certain field), etc.

[0089] The type-adjusted index may be an index obtained by adjusting the index structure to optimize the execution efficiency of a specific type of query based on an analysis of the query access type. For example, if the system finds that most queries are range queries, the index structure may be adjusted to use an index type that is more suitable for range queries (such as a B-tree index).

[0090] Specifically, the system further analyzes the query response time and index hit rate of each index level based on the rearranged index. Among them, the query response time refers to the time required for each query to access the index node, and the index hit rate indicates whether the query successfully uses the existing index to complete data retrieval. By analyzing these indicators, the system determines the access type of each index level. For example, some index levels may be more suitable for range queries (such as interval retrieval), while other levels are suitable for exact match queries (such as equality queries). Further, based on the analysis results of the access type of each index level, the access modes and access types of different levels are adjusted to ensure that the index of each level can best match its main query type, and finally form a "type-adjusted index" to optimize the overall query performance.

[0091] Step 506: Replace the adjustment frequency threshold and execution logic of each index level in the type adjustment index according to the scenario information of the real-time access operation information to obtain a logical reconstruction index.

[0092] The scenario information may be external or internal environmental data that affects the operation of the data table, and usually includes changes in query load, user access behavior patterns, and update frequency of data table content.

[0093] The adjustment frequency threshold may be a range condition for the system to update or adjust the index, which determines the frequency at which the index structure needs to be updated or reconstructed in a specific situation.

[0094] The execution logic may be the specific steps and strategies taken by the data table when executing a query or index update operation.

[0095] Specifically, the system adjusts the index refresh and update strategy according to the scenario information contained in the real-time access operation information. The scenario information can reveal different usage scenarios of data table operations. For example, some operations may be more frequent in a specific time period (such as a large number of queries during peak hours), while some query patterns may change suddenly. Based on this scenario information, the system will set a new adjustment frequency threshold for each index level in the "Type Adjustment Index", that is, determine the frequency range in which the index level needs to be updated, and adjust the execution logic of the index according to the real-time scenario, for example, increase the refresh frequency range of frequently accessed indexes, or reduce the maintenance frequency range of infrequently used indexes. Through this replacement and optimization, the final "logical reconstruction index" will be more adaptable to real-time query needs and scenario changes, thereby ensuring that the data table always maintains efficient query performance under changing loads.

[0096] In one embodiment, the data table system of an e-commerce platform finds that a certain "hot-selling product" category will have a large number of access requests during a big promotion. In order to adapt to this change, the system sets a higher update frequency threshold for the "product category" index level in the "type adjustment index" so that the index at this level can be refreshed more frequently to reflect real-time inventory changes and price adjustments. At the same time, based on real-time scenario analysis, the system adjusts the execution logic of the index to give priority to some high-frequency query requests (such as price queries and inventory queries) to ensure that queries can respond more quickly during the big promotion and avoid query performance degradation caused by lagging index updates.

[0097] In this embodiment, by rearranging the data table and adjusting the order of each index node in the index according to the access frequency of the hot data block, the data retrieval path can be optimized and the query efficiency can be improved. The rearranged index structure can effectively reduce unnecessary index scanning and data access, thereby speeding up the query response time. Furthermore, by analyzing the query response time and index hit rate of each level in the rearranged index and matching the access type, it can be ensured that the access method of each index level is more in line with the actual access requirements, thereby improving the index hit rate and query efficiency. Finally, the frequency threshold and execution logic of the index level are dynamically adjusted according to the scene information in the real-time access operation, so that the index can flexibly adapt to different operation scenarios, reduce frequent invalid index refreshes, and optimize the resource utilization of the system. Overall, the adjustment improves the flexibility and efficiency of the index structure, and ensures the efficiency and stability of the system in a variety of data access scenarios.

[0098] In an exemplary embodiment, Figure 6 As shown, using the index prediction algorithm corresponding to the target data table, the initial index of the data table is predicted and analyzed according to the historical access operation information to obtain the index prediction data, including steps 602 to 604. Among them:

[0099] Step 602 , extracting features from historical access operation information to obtain access frequency feature data, query complexity feature data, time series feature data, and change trend feature data.

[0100] The access frequency characteristic data may be characteristic data obtained by analyzing historical access operations and calculating the number of accesses or the frequency of accesses to each data block, field or index item within a certain time range.

[0101] The query complexity feature data may be an analysis of historical query operations to evaluate the complexity of the query, including feature data such as the number of tables and fields involved in the query, the complexity of JOIN operations, the frequency of aggregation operations, and the use of sorting and grouping.

[0102] The time series feature data may be feature data obtained by analyzing the distribution pattern of query requests on the time axis to identify changes in the number of visits in different time periods.

[0103] The change trend characteristic data may be characteristic data that analyzes the change trend of query patterns, data access patterns, or system load based on long-term historical data.

[0104] Specifically, the system uses an index prediction algorithm to analyze historical access operation information and extract multiple key features from it. This includes access frequency feature data, which is obtained by calculating the access frequency of each query field or data block within a certain time range to identify hot data; query complexity feature data, which is obtained by analyzing the type, field combination and execution plan of historical queries to understand the complexity of queries; time series feature data, which is obtained by analyzing the time distribution pattern of queries, identifying the change pattern of query volume over time, and distinguishing between peak and trough periods; change trend feature data, which is obtained by analyzing the long-term trend of access patterns and query characteristics to predict changes in access patterns in the future.

[0105] Step 604 , predicting the index structure change information and system resource allocation information of the initial index of the data table based on the access frequency feature data, the query complexity feature data, the time series feature data and the change trend feature data, to obtain index prediction data.

[0106] The index structure change information may be prediction and analysis data on possible changes in the existing index structure during the data table index optimization process.

[0107] The system resource allocation information may be prediction data on how to allocate hardware resources such as computing resources, storage resources, and network bandwidth when a data table executes query and index update operations.

[0108] Specifically, the system uses access frequency feature data, query complexity feature data, time series feature data, and change trend feature data to predict the future change trend of the initial index of the data table. It relies on the part of the index prediction algorithm about machine learning or statistical analysis models, and predicts the changes that may occur in the index structure under different conditions in the future by analyzing historical data and trends. For example, some fields may become new hot spots due to the increase in query frequency, resulting in the need to adjust the index structure; or as the query complexity changes, it may be necessary to add new composite indexes or adjust the existing index levels. The system will also predict the resource allocation needs of the system based on this data, for example, whether more computing resources are needed to maintain index updates during high-frequency query periods, or whether storage resources need to be optimized to reduce the space occupied by the index. Ultimately, these predictive analyses obtain "index prediction data", that is, the index structure and resource allocation strategy optimized for future access patterns.

[0109] In this embodiment, by extracting features from historical access operation information, the access pattern of the data table can be deeply analyzed, including access frequency, query complexity, time series changes, and trend characteristics. These feature data provide a basis for predicting and optimizing the index structure, so that the system can dynamically adjust and optimize the index structure of the data table to adapt to different query requirements and access patterns. By combining these feature data to predict the changing trend of the index structure and the allocation of system resources, potential performance bottlenecks can be identified in advance, waste of resources can be avoided, and the use of system resources can be optimized. Ultimately, the predictive optimization based on historical data can significantly improve the query efficiency of the data table, reduce the response time, and improve the overall performance and stability of the system.

[0110] In an exemplary embodiment, Figure 7 As shown, according to the access frequency feature data, query complexity feature data, time series feature data and change trend feature data, the index structure change information and system resource allocation information of the initial index of the data table are predicted to obtain index prediction data, including steps 702 to 706. Among them:

[0111] Step 702 , predicting specific index construction information of index structure change information and index granularity adjustment information based on access frequency feature data, query complexity feature data, time series feature data, and change trend feature data, to obtain index structure prediction information.

[0112] Among them, specific index construction can be a type of index specially designed and created to meet specific query requirements or access patterns during the data table optimization process. For example, if a field or data block appears frequently in the query, resulting in a long query response time, the system may decide to create a separate index for the field.

[0113] Among them, index granularity adjustment can be to adjust the fine-grained or coarse-grained structure of the index according to the changes in the query mode of the data table. For example, some query operations may frequently access certain specific ranges of data, and the system can optimize the efficiency of data access by adjusting the index granularity. An index with too fine granularity may result in an index that is too large and has too high storage overhead, while an index with too coarse granularity may not be able to effectively accelerate the query.

[0114] The index structure prediction information may be information for predicting and planning possible future index adjustments, including which fields or tables may require new indexes, which existing indexes may require adjustment or optimization, and changes in the granularity, order, and hierarchical structure of the index.

[0115] Specifically, the system analyzes access frequency feature data, query complexity feature data, time series feature data, and change trend feature data to predict future access patterns of data tables, where these data can help identify which fields, data blocks, or tables will become hot spots in the future, which queries will become more complex, or the number of requests for certain queries may increase significantly in a specific time period. For example, if the access frequency of a field rises sharply, the system may predict the need to build an index for the field separately instead of relying on existing composite indexes. At the same time, based on changes in query complexity, the system may predict the need to add or adjust certain composite indexes to handle more complex queries, or based on time series analysis, identify that certain query types will peak in a specific period, thereby dynamically adjusting the index granularity to adapt to the query load. Through these analyses, the system can predict specific index construction information for specific index structure change information and adjustment plans for index granularity adjustment information, such as adding new indexes, modifying the granularity or indexing method of existing indexes, and generating index structure prediction information.

[0116] Step 704, predicting cache adjustment information, storage space change information and interface adjustment information of system resource allocation information based on access frequency feature data, query complexity feature data, time series feature data and change trend feature data to obtain resource allocation prediction information.

[0117] The cache adjustment information may be how to adjust the allocation and use strategy of the memory cache when the data table executes queries and data operations.

[0118] The storage space change information may be how to adjust the allocation and use of storage space in the data table during resource management.

[0119] The interface adjustment information may be how to adjust the configuration or bandwidth of the input and output interfaces of the system when the data table responds to different access modes and resource requirements. For example, as the query load increases, especially queries involving large amounts of data transfer (such as large-scale report generation, data migration, etc.), the system may need to optimize the bandwidth or concurrent processing capabilities of the data interface.

[0120] Among them, resource allocation prediction information can be based on the analysis of access patterns, query loads, hardware resource usage and system performance, predicting future demand for system resources (such as computing, storage, network, etc.) and formulating corresponding resource allocation strategies. This prediction information includes cache adjustment, storage space management, computing resource allocation, network bandwidth configuration, etc. For example, if the system predicts that a query module will enter a high concurrency stage, more computing resources and cache space may be required; if the size of a data table increases, more storage space or reorganization of the index may be required.

[0121] Specifically, the system predicts the future resource requirements of the system through comprehensive analysis of access frequency feature data, query complexity feature data, time series feature data, and change trend feature data. Among them, the access frequency feature data can reveal which data will become more popular, thus requiring more cache space to speed up query response; the query complexity feature data may prompt the system to need more computing resources (such as CPU or memory) when executing complex queries. Time series feature data can help identify peak and trough periods, thereby predicting that in certain time periods, the system may need to increase bandwidth or computing power, or reduce resource consumption during periods with low query volume. Change trend feature data helps the system identify long-term change trends, such as the gradual increase in the number of visits to certain data sets, or certain query types gradually becoming mainstream, and then adjust the allocation of storage space and interface resources. Through these predictions, the system can plan and adjust cache adjustment information, storage space change information, and interface adjustment information in advance to avoid performance bottlenecks caused by insufficient resources, thereby generating resource allocation prediction information.

[0122] Step 706: fuse the index structure prediction information and the resource allocation prediction information to obtain index prediction data.

[0123] Specifically, the system combines the "index structure prediction information" and "resource allocation prediction information" obtained in the first two steps, taking into account changes in the index structure and resource allocation requirements. For example, the system may predict that the query frequency of a certain field will increase significantly, and a new index will need to be created for this field. It also predicts that this change will lead to an increase in demand for cache and storage resources. By combining these two types of information, the system can generate index prediction data that includes not only recommendations for index optimization and adjustment, but also corresponding resource allocation plans.

[0124] In this embodiment, by predicting changes in the index structure based on characteristic data such as access frequency, query complexity, time series and change trends, the index construction method and granularity can be dynamically adjusted for different query types and data access patterns, thereby optimizing data retrieval performance. At the same time, predicting the allocation of system resources, such as cache adjustment, storage space changes and interface adjustments, can ensure that system resources are reasonably configured, avoid over-allocation or under-allocation of resources, and improve the response speed and stability of the system. By integrating index structure prediction information and resource allocation prediction information, the system can achieve more sophisticated optimization, respond to potential performance bottlenecks in advance, and improve the query efficiency of the data table and the processing capacity of the overall system. The final prediction and optimization mechanism can significantly improve the flexibility, efficiency and scalability of the system, and provide users with faster and more stable services.

[0125] In an exemplary embodiment, Figure 8As shown, the Pareto frontier of index optimization data and index prediction data is found to obtain the index optimal adjustment data, including steps 802 to 806. Among them:

[0126] Step 802: Determine the index optimization target and the Pareto judgment condition according to the index optimization data and the index prediction data.

[0127] Among them, the index optimization goal can be multiple goals that need to be optimized and weighed during the data table index adjustment process, including query performance optimization (such as minimizing query response time), storage space optimization (such as minimizing the storage space occupied by the index), system resource consumption optimization (such as the use of CPU, memory and other resources), and the cost of index update and maintenance (such as reducing the frequency of index reconstruction).

[0128] The Pareto criterion may be a criterion for judging whether one solution is better than another solution in a multi-objective optimization problem.

[0129] Specifically, based on the index optimization data and index prediction data, the index optimization goals are clarified, where these goals usually include maximizing query performance (for example, minimizing query response time), minimizing storage overhead, and balancing the use of computing and storage resources. In addition, it is necessary to set Pareto judgment conditions, which are used to evaluate which solutions can achieve better performance on certain goals among different optimization solutions without causing significant losses on other goals. For example, the condition may be set that the optimization solution can significantly improve the query speed without significantly increasing the storage space. Through these goals and judgment conditions, the system can lay the foundation for subsequent Pareto frontier optimization.

[0130] Step 804: Based on the index optimization goal and the Pareto judgment condition, at least two Pareto optimization algorithms are used to optimize the index optimization data and the index prediction data to obtain the Pareto frontier solutions corresponding to the Pareto optimization algorithms.

[0131] Among them, the Pareto optimization algorithm can be an algorithm for multi-objective optimization problems, used to find the optimal solution set that satisfies the balance of multiple optimization objectives. These algorithms generate a set of solutions in an iterative manner, so that each solution is better than other solutions in at least one objective, and is not significantly inferior to any solution in other objectives. Among them, the Pareto optimization algorithm includes NSGA-II (non-dominated sorting genetic algorithm), SPEA2 (strength Pareto evolutionary algorithm), etc.

[0132] Among them, the Pareto front solution can be a set of optimal solutions in a multi-objective optimization problem, representing solutions that cannot be further improved in each objective dimension.

[0133] Specifically, at least two Pareto optimization algorithms (e.g., NSGA-II, SPEA2, etc.) are applied to perform multi-objective optimization solutions for index optimization data and index prediction data. These algorithms are based on pre-set index optimization goals and Pareto judgment conditions, and continuously adjust the index structure and resource allocation strategy to find the optimal compromise solution between different goals. Each algorithm generates a set of solutions, called "Pareto front solutions", which represent the best solutions that cannot be further improved under given conditions, that is, it is impossible to improve on one goal without affecting the performance of other goals. Different Pareto front solutions provide a variety of potential optimization solutions.

[0134] Step 806, performing super-front aggregation on each Pareto front solution to obtain index optimal adjustment data.

[0135] Among them, super-frontier aggregation can be the process of fusing multiple Pareto frontier solution sets from different Pareto optimization algorithms, aiming to generate a global optimal solution by removing redundant solutions and improving the quality of solutions.

[0136] Specifically, hyperfront aggregation aims to integrate Pareto front solutions from multiple Pareto optimization algorithms to obtain the optimal index adjustment plan. Therefore, in multi-objective optimization, each Pareto front solution represents an optimal solution in the objective space that cannot be improved by improving one objective without affecting other objectives. However, different optimization algorithms may generate different sets of front solutions, which may have repeated or redundant parts, or each have different advantages. Through hyperfront aggregation, the system compares and merges these solutions, removes duplicate and inefficient solutions, and selects solutions that can achieve the best balance between multiple optimization objectives (such as query response time, storage space consumption, system resource usage, etc.). In the hyper-aggregation process, the system will take into account the performance of each solution in multiple dimensions, and by weighing the importance of different objectives, select the index adjustment plan that best meets the actual needs. The aggregated index optimal adjustment data will provide a comprehensive and efficient adjustment plan for data table optimization, ensuring the best performance and resource utilization in actual applications.

[0137] In this embodiment, by determining the index optimization target and the Pareto judgment condition based on the index optimization data and the index prediction data, a clear direction and standard can be set for the optimization process to ensure that the optimal balance point is found between different optimization targets. At least two Pareto optimization algorithms are used to perform multi-dimensional optimization on the index optimization data and the index prediction data, and multiple optimization factors such as query speed, system resource consumption, and response time can be considered at the same time, thereby obtaining multiple Pareto front solutions, providing a series of possible optimization schemes. By performing super-front aggregation on these Pareto front solutions, the advantages of each scheme can be combined to finally obtain the optimal index adjustment data, achieve a balance between multiple targets, and improve the overall performance and efficiency of the system. This method based on Pareto optimization can accurately meet the balance between different queries and system resource requirements, and ensure that the data table can run efficiently under a variety of conditions.

[0138] In an exemplary embodiment, Fig. 9 As shown, according to the index optimal adjustment data, the target data table is optimized to obtain the optimized data table, including steps 902 to 910. Among them:

[0139] Step 902, determining a data table style according to the data structure information of the index optimally adjusted data.

[0140] Among them, data structure information can be a collection of information that describes the internal field organization, data type, constraints, index design, etc. of a data table. It includes the attributes of each column in the table (such as column name, data type, size, whether it is empty, etc.), table constraints (such as primary key, foreign key, unique constraint, check constraint, etc.), and relationships between tables (such as foreign key relationships).

[0141] Among them, the data table style can be the display form of the data table and its layout design, involving the arrangement order of fields in the table, the paging, partition or column strategies of the table, the visual presentation style, etc.

[0142] Specifically, data structure information is extracted from the index optimal adjustment data, and based on this information, it is decided how to organize and layout the fields and data of the target data table. These data structure information include column types, sizes, constraints, and data table partitioning methods. The system will select the appropriate data table style design based on different query modes, data access frequencies, and storage requirements. For example, vertical partitioning may be selected to store the most frequently queried fields, while infrequently queried fields may be partitioned for storage; or the data table may be divided into multiple small tables (such as partitioned tables) to improve query efficiency and maintainability. The determination of the data table style is not only about the physical storage form of the table, but also whether the design of the data table is suitable for efficient data retrieval, update, and insert operations. It is further combined with the goal of index optimization to ensure that the final table structure can support optimal query and storage performance.

[0143] Step 904, setting single information operation interface information and multi-information operation interface information according to the data table style and data structure information.

[0144] Among them, the single information operation interface information can be the data of the user interface design for displaying and operating a single data record. It is usually used to view, edit, delete or add a single data item, and only the detailed information of a single record is displayed on the interface.

[0145] Among them, the multi-information operation interface information can be the data of the user interface design for displaying and operating multiple data records. Different from the single-information operation interface, the multi-information operation interface usually needs to support batch operation, filtering, sorting, paging and other functions. It is often used in scenarios that process a large number of data records, such as data lists, report generation, etc.

[0146] Specifically, the system designs the user interaction interface based on the determined data table style and data structure information, including single-information operation interface information and multi-information operation interface information. The single-information operation interface is mainly used to display and edit a single data record. Only necessary fields are displayed on the interface, and users can efficiently view, modify or delete operations through a simple operation interface; while the multi-information operation interface is used to display and operate multiple records, and usually needs to provide more complex filtering, sorting, paging, batch operation and other functions. During the design, the system will take into account the layout of the interface, the way data is presented (such as tables, cards, etc.), the sorting of fields, the display logic, etc., to ensure that the interface can meet different operation requirements. At the same time, it is also necessary to ensure that users can easily operate data on the interface to avoid overly complex interaction designs that affect the user experience.

[0147] Step 906, performing data table responsive front-end optimization according to the data table style, single information operation interface information, and multi-information operation interface information to obtain data table hardware adaptation information.

[0148] The data table hardware adaptation information may be the adaptation requirements of different hardware devices (such as desktop computers, mobile devices, tablets, etc.) for table display and interaction taken into consideration when designing the data table.

[0149] Specifically, since the responsive front-end optimization of the data table is to enable the data table to provide the best display effect and interactive experience on different devices and screen sizes, the system will adaptively adjust the front-end interface according to the data table style, single information operation interface information and multi-information operation interface information corresponding to the operation interface design. Responsive design ensures that the data table can automatically adjust according to the screen size, resolution and browser characteristics of the device. For example, in a desktop display, the table may display complete fields and complex operation buttons, while on a mobile device, the table may save space by hiding secondary columns or using drop-down menus, and optimize the operation method (such as touch operation). In addition, the system also needs to optimize according to hardware characteristics (such as memory, processing power, network bandwidth, etc.) to ensure that the loading speed and interactive performance of the data table on different devices can reach the best state, and obtain the data table hardware adaptation information, where the data table hardware adaptation information includes the adjustment of interface elements under different devices, the table data loading method, the response time optimization and other contents.

[0150] Step 908, interactively optimize the target data table according to the data table style, the single information operation interface information, the multi-information operation interface information and the data table hardware adaptation information to obtain an interactively optimized data table.

[0151] Among them, interaction optimization can be to improve the interactive experience between users and data tables, with the aim of allowing users to operate and browse data more conveniently and quickly.

[0152] Among them, the interactively optimized data table can be a data table that has been interactively optimized, aiming to provide a more efficient, smooth and intuitive user operation experience. This type of data table usually supports functions such as fast data display, editing, deletion, filtering, sorting, and paging, and the interface design enables users to easily understand the structure and content of the data. For example, the interactively optimized data table can reduce loading time through dynamic loading technology, simplify the data processing process through drag and drop, and click operations, and enable users to quickly find the required data through intelligent filtering and sorting.

[0153] Specifically, based on the data table style, single-information operation interface information, multi-information operation interface information and hardware adaptation information, the interactive experience of the target data table is optimized; the interactive optimization mainly involves improving the efficiency and convenience of interaction between users and data tables. For example, the system may reduce the time and complexity of user operations by optimizing the interaction methods such as drag and drop sorting, paging switching, and batch data operations (such as batch deletion and batch modification) in the table; in addition, it is also possible to adjust the priority and loading order of interface elements based on user behavior analysis (such as the most frequently accessed fields and the most frequently performed operations) to improve user operation efficiency; at the same time, interactive optimization also includes optimizing the response speed of controls such as buttons, drop-down boxes, and check boxes to ensure a smooth use experience of the table on different devices, reduce the user experience differences caused by differences in device performance, and obtain an interactively optimized data table.

[0154] Step 910, performing multi-level view optimization on the interactive optimization data table to obtain an optimized data table.

[0155] Specifically, the system performs multi-level view optimization on the interactively optimized data table, with the goal of providing more flexible and diverse data display forms according to different user needs and usage scenarios. Multi-level view optimization refers to setting different view levels for the data table so that users can switch between different views as needed. For example, users can view the summary information and statistical data of the data table in the "Overview" view, and view the complete information of a certain record in the "Details" view. The system can also provide views such as chart display and report display of the data to help users better understand and analyze the data. To achieve this, the system will dynamically load data and use technologies such as data filtering, grouping and paging to ensure that the content of different views can respond quickly, avoiding slow data loading or interface freezes. In addition, the optimization of multi-level views also includes ensuring seamless switching between view levels and flexible user customization capabilities, ensuring that the display method of the data table can meet the needs of different users and application scenarios, and obtaining an optimized data table.

[0156] In this embodiment, by determining the data table style according to the data structure information of the index optimally adjusted data, it can be ensured that the data table is more in line with actual needs in display and operation, and the data content can be efficiently presented. According to the data table style and data structure information, the single information and multi-information operation interface are set, and flexible interaction methods can be provided for different operation scenarios, improving the convenience and accuracy of user operation. Further, through the responsive front-end optimization of data table style, operation interface information and hardware adaptation, the data table can be displayed and operated smoothly on different hardware devices (such as desktops, tablets and mobile phones), improving cross-platform compatibility and user experience. Through the design of interactive optimization data tables, the user operation process can be simplified, unnecessary clicks and redundant information can be reduced, and the interactive experience can be optimized. Finally, through multi-level view optimization, the visualization effect and hierarchy of the data table can be further improved, so that users can operate and view data more clearly and intuitively. A series of optimizations effectively improve the response speed of the system, user operation efficiency and visual experience, and provide users with a more efficient, convenient and pleasant data interaction environment.

[0157] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0158] Based on the same inventive concept, the embodiment of the present application also provides a data table processing device for implementing the data table processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data table processing device embodiments provided below can refer to the above limitations on a data table processing method, and will not be repeated here.

[0159] In an exemplary embodiment, Fig.10 As shown, a data table processing device is provided, including: a data acquisition module 1002, an index analysis module 1004, an index prediction module 1006, an index optimization module 1008 and a table optimization module 1010, wherein:

[0160] The data acquisition module 1002 is used to acquire the real-time access operation information, historical access operation information and the initial index of the data table of the target data table;

[0161] The index analysis module 1004 is used to use the index adjustment algorithm corresponding to the target data table to optimize and analyze the initial index of the data table according to the real-time access operation information to obtain index optimization data;

[0162] The index prediction module 1006 is used to use the index prediction algorithm corresponding to the target data table to perform prediction analysis on the initial index of the data table according to the historical access operation information to obtain index prediction data;

[0163] An index optimization module 1008 is used to find the Pareto front of index optimization data and index prediction data to obtain the index optimal adjustment data;

[0164] The table optimization module 1010 is used to optimize the target data table by adjusting the data according to the optimal index to obtain an optimized data table.

[0165] Each module in the above-mentioned data table processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0166] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a data table. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The data table of the computer device is used to store server data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data table processing method is implemented.

[0167] In one embodiment, a data table processing system is provided. The system includes an electronic device and a storage medium. The storage medium stores a computer program. When the computer program is executed by the electronic device, the steps of a data table processing method are implemented.

[0168] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0169] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. Among them, any reference to memory, data table or other medium used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. The data table involved in the embodiments provided in this application may include at least one of a relational data table and a non-relational data table. Non-relational data tables may include distributed data tables based on blockchains, etc., but are not limited to this. The processor involved in the embodiments provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A data table processing method, characterized in that: The method comprises: Obtaining real-time access operation information, historical access operation information, and initial index of the target data table; Using the index adjustment algorithm corresponding to the target data table, and according to the real-time access operation information, optimizing and analyzing the initial index of the data table to obtain index optimization data; Using the index prediction algorithm corresponding to the target data table, and based on the historical access operation information, performing prediction analysis on the initial index of the data table to obtain index prediction data; Finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data; The target data table is optimized according to the index optimal adjustment data to obtain an optimized data table.

2. The method according to claim 1, characterized in that The using the index adjustment algorithm corresponding to the target data table to optimize and analyze the initial index of the data table according to the real-time access operation information to obtain index optimization data includes: Identifying hot data blocks and high-frequency query types corresponding to the target data table from the real-time access operation information; According to the hot data block and the high-frequency query type, the index structure of the initial index of the data table is adjusted to obtain an adjusted index of the data table; The index adjustment algorithm is used to optimize the access mode of the adjusted index of the data table to obtain the index optimization data.

3. The method according to claim 2, characterized in that The using the index adjustment algorithm to optimize the access mode of the data table index adjustment to obtain the index optimization data includes: According to the access frequency of the hot data block, the index logic of the data table adjustment index is reconstructed to obtain a logically reconstructed index; According to the addition, deletion and modification operation information in the real-time access operation information, the refresh frequency and resource allocation information of the logical reconstruction index are adjusted to obtain the index optimization data.

4. The method according to claim 3, characterized in that The step of reconstructing the index logic of the data table adjustment index according to the access frequency of the hot data block to obtain the logically reconstructed index includes: According to the access frequency of the hot data block, the order of each index node in the data table adjustment index is rearranged to obtain a rearranged adjustment index; According to the query response time and index hit rate corresponding to each index level in the rearranged adjustment index, the access type of each index level is matched to obtain a type adjustment index; According to the scenario information of the real-time access operation information, the adjustment frequency threshold and the execution logic of each index level in the type adjustment index are replaced to obtain the logic reconstruction index.

5. The method according to claim 1, characterized in that: The using the index prediction algorithm corresponding to the target data table to perform prediction analysis on the initial index of the data table according to the historical access operation information to obtain index prediction data includes: Extracting features of the historical access operation information to obtain access frequency feature data, query complexity feature data, time series feature data, and change trend feature data; According to the access frequency characteristic data, the query complexity characteristic data, the time series characteristic data and the change trend characteristic data, the index structure change information and the system resource allocation information of the initial index of the data table are predicted to obtain the index prediction data.

6. The method according to claim 5, characterized in that The step of predicting the index structure change information and the system resource allocation information of the initial index of the data table according to the access frequency feature data, the query complexity feature data, the time series feature data, and the change trend feature data to obtain the index prediction data includes: According to the access frequency feature data, the query complexity feature data, the time series feature data and the change trend feature data, specific index construction information and index granularity adjustment information of the index structure change information are predicted to obtain index structure prediction information; According to the access frequency characteristic data, the query complexity characteristic data, the time series characteristic data and the change trend characteristic data, the cache adjustment information, the storage space change information and the interface adjustment information of the system resource allocation information are predicted to obtain resource allocation prediction information; The index structure prediction information and the resource allocation prediction information are integrated to obtain the index prediction data.

7. The method according to claim 1, characterized in that The step of finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data includes: Determining an index optimization target and a Pareto judgment condition according to the index optimization data and the index prediction data; Based on the index optimization target and the Pareto judgment condition, at least two Pareto optimization algorithms are used to optimize the index optimization data and the index prediction data to obtain Pareto frontier solutions corresponding to the Pareto optimization algorithms; Super-front aggregation is performed on each of the Pareto front solutions to obtain the index optimal adjustment data.

8. The method according to claim 1, characterized in that The step of optimizing the target data table according to the index optimal adjustment data to obtain an optimized data table includes: Determine a data table style according to the data structure information of the index optimally adjusting the data; According to the data table style and the data structure information, single information operation interface information and multi-information operation interface information are set; Perform data table responsive front-end optimization according to the data table style, the single information operation interface information, and the multi-information operation interface information to obtain data table hardware adaptation information; According to the data table style, the single information operation interface information, the multi-information operation interface information and the data table hardware adaptation information, the target data table is interactively optimized to obtain an interactively optimized data table; Multi-level view optimization is performed on the interactive optimization data table to obtain the optimized data table.

9. A data table processing device, characterized in that: The device comprises: A data acquisition module is used to acquire real-time access operation information, historical access operation information and initial index of the data table of the target data table; An index analysis module, configured to use an index adjustment algorithm corresponding to the target data table to optimize and analyze the initial index of the data table according to the real-time access operation information to obtain index optimization data; An index prediction module, configured to use an index prediction algorithm corresponding to the target data table to perform prediction analysis on the initial index of the data table according to the historical access operation information to obtain index prediction data; An index optimization module, used to find the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data; The table optimization module is used to optimize the target data table by optimally adjusting the data according to the index to obtain an optimized data table.

10. A data table processing system, characterized in that: The system comprises an electronic device and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the electronic device, the steps of the method according to any one of claims 1 to 8 are implemented.

Images