Data table processing method, device, storage medium and electronic equipment

Abstract

The present application relates to a data table processing method, device, storage medium and electronic device. The method includes: obtaining real-time access operation information, historical access operation information and initial index of the data table of the target data table; using the index adjustment algorithm corresponding to the target data table, according to the real-time access operation information, optimizing and analyzing the initial index of the data table to obtain index optimization data; using the index prediction algorithm corresponding to the target data table, according to the historical access operation information, predicting and analyzing the initial index of the data table to obtain index prediction data; finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data; optimizing the target data table according to the index optimal adjustment data to obtain the optimized data table. The use of this method can effectively improve the application efficiency of the index of the data table in complex, dynamic and large-scale data processing, and further provide a more efficient data table management and maintenance solution.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data table processing method, device, storage medium and electronic device. Background Art

[0002] Data table processing methods are widely used in data analysis, storage, and computing. This method primarily involves data cleaning, data conversion, data analysis, and data visualization. Data cleaning eliminates invalid or erroneous data and improves data quality; data conversion standardizes and formats multi-source, heterogeneous data for ease of subsequent manipulation; data analysis utilizes statistical and machine learning tools to extract valuable information; and data visualization presents results intuitively through charts or dashboards. Furthermore, modern data table processing often utilizes specialized software (such as Excel and Google Sheets) or programming tools (such as Python's Pandas library) for efficient and automated operations.

[0003] Traditional table indexing methods have limited efficiency for large-scale data sets. When the data volume is enormous, index construction and querying can consume significant resources. Furthermore, traditional indexing methods have limited support for dynamic data, resulting in high index maintenance costs when data is frequently updated or changed. This results in inefficient application of table indexes in complex, dynamic, and large-scale data processing. Summary of the Invention

[0004] Based on this, it is necessary to provide a data table processing method, device, storage medium and electronic device that can effectively improve the application efficiency of data table indexes in complex, dynamic and large-scale data processing to address the above technical problems.

[0005] In a first aspect, the present application provides a data table processing method, comprising:

[0006] Obtain the real-time access operation information, historical access operation information and initial index of the target data table;

[0007] Using the index adjustment algorithm corresponding to the target data table, and according to the real-time access operation information, optimizing and analyzing the initial index of the data table to obtain index optimization data;

[0008] Using the index prediction algorithm corresponding to the target data table, and based on the historical access operation information, performing a prediction analysis on the initial index of the data table to obtain index prediction data;

[0009] Finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data;

[0010] The target data table is optimized according to the index optimal adjustment data to obtain an optimized data table.

[0011] In a second aspect, the present application further provides a data table processing device, comprising:

[0012] The data acquisition module is used to obtain the real-time access operation information, historical access operation information and initial index of the target data table;

[0013] An index analysis module, configured to use an index adjustment algorithm corresponding to the target data table and, based on the real-time access operation information, perform optimization analysis on the initial index of the data table to obtain index optimization data;

[0014] An index prediction module, configured to use an index prediction algorithm corresponding to the target data table and, based on the historical access operation information, perform a prediction analysis on the initial index of the data table to obtain index prediction data;

[0015] An index optimization module, configured to find the Pareto front of the index optimization data and the index prediction data to obtain the optimal index adjustment data;

[0016] The table optimization module is used to optimize the target data table according to the optimal adjustment data of the index to obtain an optimized data table.

[0017] In a third aspect, the present application also provides a data table processing system, which includes an electronic device and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the electronic device, it implements the steps of a data table processing method.

[0018] The above-mentioned data table processing method, device, storage medium and electronic device optimize and analyze the index through real-time access data, and can adjust the index strategy according to the current access mode to improve query efficiency; at the same time, the predictive analysis of historical access data can foresee future access trends and further guide index adjustment. Combining Pareto frontier analysis of optimized data and predicted data, the optimal balance point can be found between multiple optimization goals, thereby reducing the waste of storage space and the cost of index maintenance while ensuring query response speed. Due to the combination of real-time access operation information, historical access operation information and the initial index of the data table, a targeted index adjustment algorithm and index prediction algorithm are adopted to achieve dynamic optimization and intelligent prediction of the data table index, which can significantly improve the query performance, storage efficiency and overall response speed of the data table, reduce performance bottlenecks caused by unreasonable index design, effectively improve the application efficiency of the data table index in complex, dynamic, large-scale data processing, and further provide a more efficient data table management and maintenance solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 An application environment diagram of a data table processing method in one embodiment;

[0021] Figure 2 1 is a flow chart of a data table processing method in one embodiment;

[0022] Figure 3 A schematic flow chart of a first method for obtaining index optimization data in one embodiment;

[0023] Figure 4 Schematic diagram of a flow chart of a second method for obtaining index optimization data in one embodiment;

[0024] Figure 5 A flowchart of a method for obtaining a logical reconstruction index in one embodiment is shown;

[0025] Figure 6 1 is a flow chart of a first method for obtaining index prediction data in one embodiment;

[0026] Figure 7 1 is a flow chart of a second method for obtaining index prediction data in one embodiment;

[0027] Figure 8 1 is a flow chart of a method for obtaining index optimal adjustment data in one embodiment;

[0028] Figure 9 A schematic flow chart of a method for optimizing a data table in one embodiment;

[0029] Figure 10 is a structural block diagram of a data table processing device in one embodiment;

[0030] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0032] The data table processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or placed on a cloud or other network server. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0033] In an exemplary embodiment, Figure 2 As shown, a data table processing method is provided, which is applied to Figure 1 The server in FIG. 1 is taken as an example to illustrate the method, including the following steps 202 to 210. Among them:

[0034] Step 202: Acquire real-time access operation information, historical access operation information, and an initial index of the target data table.

[0035] The target data table may be a specific table in the data table that needs to be optimized, and it contains a series of data and their associations.

[0036] The real-time access operation information may be a detailed record of various access operations performed by the user on the target data table at the current moment in the data table system.

[0037] The historical access operation information may be access records of the target data table by the data table system within a period of time in the past.

[0038] The initial index of the data table may be an original index structure created for the target data table in the data table, which is usually set when the data table is created or in the early optimization process. The initial index includes a single-column index, a composite index, and the like.

[0039] Specifically, the real-time access operation information of the target data table is extracted from the current interaction information between the computer and the operation object in the data table management system, including the current query request, operation type (such as insert, delete, update, query) and its frequency. At the same time, the historical access operation information of the target data table is extracted from the data table management system. This operation information covers the access records of the data table over a period of time, such as query mode, operation mode, and temporal changes in data access. In addition, the initial index structure of the target data table needs to be obtained, including basic information such as the existing index type, index fields, and index usage in the table.

[0040] Step 204 : Using the index adjustment algorithm corresponding to the target data table, the initial index of the data table is optimized and analyzed according to the real-time access operation information to obtain index optimization data.

[0041] Among them, the index adjustment algorithm can be an algorithm that automatically adjusts the index structure of a data table based on real-time access operation information or specific optimization goals. Generally, a self-organizing algorithm is selected, such as one or more of self-organizing algorithms, such as self-organizing maps, ant colony algorithms, particle swarm algorithms, genetic algorithms, collective intelligence and swarm intelligence algorithms, and adaptive neural networks.

[0042] Optimization analysis can evaluate the efficiency of existing indexes and query execution plans based on information such as current table usage, query patterns, and access frequency. This analysis can identify performance bottlenecks, redundant indexes, or inefficient query paths, and then propose corresponding optimization solutions, such as creating new indexes or rebuilding existing indexes.

[0043] The index optimization data may be a set of index optimization suggestions or results obtained through analysis of real-time access operation information. These data may include decisions on newly created indexes, modified indexes, or deletion of redundant indexes.

[0044] Specifically, the index adjustment algorithm corresponding to the target data table is used to analyze real-time access operation information such as query frequency, query type, query field, and execution time of each query, to identify which fields are frequently queried, which indexes are redundant, or whether some queries do not use existing indexes. Based on the real-time query pattern, the index adjustment algorithm calculates optimized data, such as creating a new composite index, reconstructing an existing index, or deleting an invalid index. The goal of the optimization analysis is to make the index structure more efficient, reduce unnecessary storage overhead, and improve the execution efficiency of queries. The final index optimization data includes new index structure data or adjustment data for existing indexes, such as creating a composite index for the frequently queried "User ID" and "Order Date" fields.

[0045] Step 206 : Using the index prediction algorithm corresponding to the target data table, perform a prediction analysis on the initial index of the data table according to the historical access operation information to obtain index prediction data.

[0046] Among them, the index prediction algorithm can be an algorithm that predicts future data table access patterns and query requirements based on historical access operation information. Generally, one or more of a time series model, a regression model, and a deep learning model are selected.

[0047] Among them, predictive analysis can be achieved by in-depth analysis of historical access operation information to infer future data table access patterns or performance bottlenecks. The system can identify possible growing query demands or changing query patterns in advance, providing forward-looking information support for the adjustment and optimization of index structures.

[0048] The index prediction data may be predictive data generated by an index prediction algorithm based on historical access operation information to guide future index optimization.

[0049] Specifically, because historical access operation information contains data query patterns within a certain period of time in the past, including the access frequency of query fields, the time distribution of operations, and query types, etc. By analyzing this historical data, the index prediction algorithm can identify access trends that may appear in the future, such as the query frequency of certain fields may increase, or the query patterns of certain table connections will become more common. Based on this trend analysis, the index prediction algorithm can predict which indexes will become more important in the future and which indexes may lose their effectiveness. At the same time, the index prediction algorithm will provide a series of index adjustment suggestions, such as adding indexes for specific fields, modifying the structure of existing indexes, or deleting indexes that are no longer frequently accessed. The final index prediction data includes the prediction results of future index adjustments. For example, it is predicted that the query frequency of the "product classification" field will increase significantly in the future, so it is recommended to create a separate index for this field in advance.

[0050] Step 208: Find the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data.

[0051] The optimal index adjustment data may be an optimal index adjustment solution obtained after performing multi-objective optimization (such as query efficiency, storage cost, etc.) on the index optimization data and the index prediction data.

[0052] Specifically, since index optimization and prediction data often involve multiple optimization objectives, such as query response time, index storage space, index update overhead, and query hit rate, trade-offs may exist between these objectives. For example, adding more indexes improves query efficiency but increases storage and maintenance costs. Pareto front analysis can help find the optimal balance between these multiple objectives—that is, the trade-off between query performance, storage costs, and maintenance overhead. The analysis reveals the optimal index adjustment scheme under different objective conditions. This optimizes the index structure and reduces unnecessary costs and resource waste without sacrificing overall table performance. The resulting optimal index adjustment data is the optimal index adjustment strategy selected based on these multi-objective balances. For example, removing the infrequently used "Product Description" field index and adding a composite index covering "User ID" and "Order Date" achieves a better balance between query performance and storage efficiency.

[0053] Step 210 : Optimize the target data table according to the index optimal adjustment data to obtain an optimized data table.

[0054] Optimizing the data table may be the result of actually adjusting the target data table after obtaining the optimal index adjustment data, which includes operations such as modifying the index structure of the data table, deleting redundant indexes, and creating new efficient indexes.

[0055] Specifically, we optimize the target data table based on the optimal index adjustment data. This includes creating, deleting, or modifying indexes within the table, adjusting the existing index structure to accommodate new query patterns and performance requirements, and optimizing interaction and adaptability. The optimized table will have adjusted indexes and appropriate styles, significantly improving query efficiency and reducing maintenance costs, resulting in an optimized data table.

[0056] In the above-mentioned data table processing method, by optimizing and analyzing the index through real-time access data, the index strategy can be adjusted according to the current access pattern to improve query efficiency; at the same time, the predictive analysis of historical access data can foresee future access trends and further guide index adjustment. Combining Pareto frontier analysis of optimized data and predicted data, the optimal balance point can be found between multiple optimization objectives, thereby reducing the waste of storage space and the cost of index maintenance while ensuring query response speed. Due to the combination of real-time access operation information, historical access operation information and the initial index of the data table, targeted index adjustment algorithms and index prediction algorithms are adopted to achieve dynamic optimization and intelligent prediction of the data table index, which can significantly improve the query performance, storage efficiency and overall response speed of the data table, reduce performance bottlenecks caused by unreasonable index design, effectively improve the application efficiency of the data table index in complex, dynamic, large-scale data processing, and further provide more efficient data table management and maintenance solutions.

[0057] In an exemplary embodiment, Figure 3 As shown, using the index adjustment algorithm corresponding to the target data table, the initial index of the data table is optimized and analyzed according to the real-time access operation information to obtain index optimization data, including steps 302 to 306.

[0058] Step 302: Identify the hot data blocks and high-frequency query types corresponding to the target data table from the real-time access operation information.

[0059] Specifically, the system analyzes the real-time access operation information in the current human-computer interaction data table to identify all operation records for the target data table. These records include the query fields, query frequency, data access path, etc. Based on the information obtained from the analysis, the system uses algorithms to identify which data blocks are frequently accessed (i.e., hot data blocks) and which query types appear frequently (i.e., high-frequency query types). Among them, hot data blocks generally represent areas in the data table that users frequently access, while high-frequency query types indicate the field combinations or query patterns that users often use when querying.

[0060] Step 304 : Adjust the index structure of the initial index of the data table according to the hot data blocks and the high-frequency query types to obtain an adjusted index of the data table.

[0061] Among them, the index structure can be a data storage structure created in a data table to speed up queries. It usually consists of a set of data columns and related index fields, with the purpose of improving data retrieval efficiency.

[0062] The adjusted index of the data table may be an index obtained by modifying or optimizing the existing data table index structure, deleting redundant indexes, adding new indexes, or changing the order of index fields.

[0063] Specifically, based on hot data blocks and high-frequency query types, the system optimizes and adjusts the index structure of the target data table's initial index. For example, for frequently accessed hot data blocks, it may be necessary to create more targeted indexes to speed up queries on these specific data blocks. At the same time, for high-frequency query types, the system may recommend creating composite indexes for commonly used query fields, or adjusting existing indexes to support more efficient query paths. The adjusted index structure should be able to effectively improve query efficiency, reduce unnecessary calculations and data scans, and obtain an adjusted index for the data table.

[0064] Step 306: Use an index adjustment algorithm to optimize the access mode of the adjusted index of the data table to obtain index optimization data.

[0065] Among them, the access pattern can be the way and frequency in which users or applications access data in a data table, which includes aspects such as the query type of access, frequency of field usage, time distribution of queries, and data retrieval path.

[0066] Specifically, after adjusting the index structure of the target data table, the index adjustment algorithm is used to further optimize the access pattern of the adjusted index. In the process of optimizing the access pattern, the performance of the new index after adjusting the index structure is evaluated mainly by simulating or analyzing the impact of the adjusted index on the actual query pattern. Therefore, the index adjustment algorithm will further identify the new query pattern, scanning path and its efficiency, and adjust the adjusted index to improve its performance. The optimization process may include deleting redundant indexes, merging indexes, or adjusting the storage structure of the index. Through these optimizations, the final index optimization data will contain a better index structure configuration, effectively support efficient queries, reduce storage overhead, and ensure that index maintenance costs are minimized.

[0067] In this embodiment, by identifying hot data blocks and high-frequency query types from real-time access operation information, it can help to accurately locate the most frequently accessed and queried parts of the data table, thereby providing an optimization basis for the initial index structure of the data table. By adjusting the index structure according to the hot data blocks and high-frequency query types, the query efficiency can be improved and unnecessary data access can be reduced, making the query operation more efficient. Furthermore, using the index adjustment algorithm to optimize and analyze the adjusted index can further improve the access pattern of the index, reduce disk I / O and memory usage, and ultimately significantly improve the overall performance of the data table, ensure the efficiency of data queries and operations, and meet real-time data access needs. This optimization method can not only improve the response speed of the system, but also reduce resource consumption, providing users with a faster and more stable service experience.

[0068] In an exemplary embodiment, Figure 4 As shown, the index adjustment algorithm is used to optimize the access mode of the data table index adjustment to obtain index optimization data, including steps 402 to 404. Among them:

[0069] Step 402 : Reconstruct the index logic of the data table adjustment index according to the access frequency of the hot data block to obtain a logically reconstructed index.

[0070] Index logic is the design and organization of the index structure within a data table. This primarily involves the selection of index fields, the index hierarchy, and how data within the index is stored and retrieved. It determines how indexes are associated with records in the data table and how data can be efficiently located and retrieved through the index.

[0071] Logical reindexing involves optimizing table indexes by adjusting the index's organization and structure to accommodate new query patterns or access requirements. This typically involves changing the order of index fields, creating new composite indexes, or adjusting the index's hierarchy to more effectively support frequent query operations or access patterns.

[0072] Specifically, the access frequency of hot data blocks is analyzed to identify situations in which different hot data blocks are frequently accessed during the query process. These hot data blocks usually have bottlenecks or high-frequency access areas for data table queries. Based on the above analysis information, the index adjustment algorithm will rebuild the logical structure of the index, that is, adjust the selection of index fields, the organization of the index and its hierarchical structure, etc. By optimizing the index logic, frequently accessed data blocks can be made easier to access, thereby improving query efficiency. For example, a new composite index may be created for the hot data blocks, or the existing index may be optimized to ensure rapid positioning of these hot data blocks. The resulting logically reconstructed index will more efficiently support queries, especially high-frequency access to hot data blocks.

[0073] Step 404 : According to the add, delete, and modify operation information in the real-time access operation information, the refresh frequency of the logical reconstruction index and the resource allocation information are adjusted to obtain index optimization data.

[0074] The add, delete, and modify operation information may be detailed information about the add, delete, and modify operations recorded in the data table, including inserting, deleting, and updating data.

[0075] The refresh frequency may be the frequency of updating or rebuilding an index in a data table, and is usually closely related to the frequency of data changes.

[0076] The resource allocation information may be how the data table allocates and uses system resources (such as computing power, memory, disk space, etc.) when performing index update and query operations.

[0077] Specifically, the system further analyzes the add, delete, and modify operation information in real-time access operations, because these operations directly affect the update and maintenance of the index. Since add, delete, and modify operations will cause changes in the records in the data table, which may affect the validity and accuracy of the index, the index adjustment algorithm will adjust the refresh frequency of the logical reconstruction index based on the information of the add, delete, and modify operations. For example, for frequently updated data, the refresh frequency of the index can be increased to ensure that the index is always kept up to date; for less frequently updated data, the refresh frequency can be reduced to save computing and storage resources. In addition, the system will also synchronously optimize the computing resources and storage resources during the index update process based on the resource allocation situation to ensure efficient use of resources. The adjusted index optimization data includes a new index refresh strategy and resource allocation strategy to obtain index optimization data.

[0078] In this embodiment, by reconstructing the index logic of the data table to adjust the index according to the access frequency of the hot data blocks, it is possible to optimize the most frequently accessed data blocks so that these data blocks can be retrieved and operated more quickly, thereby improving query efficiency and response speed. The reconstructed index logic can adjust the organization of the index according to the actual access pattern, making the access path of commonly used data more efficient. At the same time, combined with the add, delete, and modify operations in the real-time access operation information, the refresh frequency and resource allocation of the adjustment logic reconstructed index ensure that the index can be updated in time and adapt to changes in data, avoiding performance degradation caused by outdated indexes or unreasonable resource allocation. This optimization not only improves the query efficiency of the data table, but also reduces the waste of system resources through refined resource management, further improving the overall performance and user experience.

[0079] In an exemplary embodiment, Figure 5 As shown, based on the access frequency of the hot data block, the index logic of the data table adjustment index is rebuilt to obtain the logical reconstruction index, including steps 502 to 506.

[0080] Step 502 : rearrange the order of the index nodes in the data table adjustment index according to the access frequency of the hot data block to obtain a rearranged adjustment index.

[0081] The index node may be a basic unit in the index structure of a data table, and includes a pointer or a key-value pair pointing to a data record.

[0082] The rearranged and adjusted index may be a new index obtained by reorganizing or reordering the existing index structure according to changes in data access frequency or query patterns.

[0083] Specifically, by analyzing the access frequency of hot data blocks, identifying the access situations of different hot data blocks in the query, and combining the access patterns of hot data blocks, the index adjustment algorithm will reorder the existing index nodes in the data table. Specifically, frequently accessed nodes will be prioritized at the front or higher levels of the index tree for quick positioning. The purpose of this is to optimize the query path so that hot data blocks that are frequently accessed by queries can be retrieved more efficiently in the index tree, thereby reducing the index search time and computing resource consumption. Through this rearrangement, the resulting "rearranged and adjusted index" can better respond to the high-frequency access needs of hot data blocks.

[0084] Step 504 : Match the access type of each index level according to the query response time and index hit rate corresponding to each index level in the rearranged and adjusted index to obtain a type-adjusted index.

[0085] The index levels can be different levels in an index tree or index structure. Typically, the index levels are divided into root nodes, inner nodes, and leaf nodes. In a multi-level index structure, the root node points to the inner nodes, which point to the leaf nodes. The leaf nodes store specific data records or pointers to data.

[0086] Query response time is the time it takes for a database system to process a query request and return results. This time includes all steps from the time a user initiates a query request to the time the results are returned, including query parsing, optimization, execution, and data access.

[0087] The index hit rate can be the ratio of target data that can be directly located through the index in a data table query.

[0088] Access types are the different methods or modes used by table query requests to access data. Common access types include exact match queries (such as searching for records with a specific ID), range queries (such as searching for records within a certain time period), and sort queries (such as results sorted by a specific field).

[0089] A type-adjusted index can be an index that optimizes the execution efficiency of specific query types by adjusting the index structure based on an analysis of query access types. For example, if the system finds that most queries are range queries, the index structure can be adjusted to use an index type more suitable for range queries (such as a B-tree index).

[0090] Specifically, the system further analyzes the query response time and index hit rate of each index level based on the rearranged index. Among them, the query response time refers to the time required for each query to access the index node, and the index hit rate indicates whether the query successfully uses the existing index to complete data retrieval. By analyzing these indicators, the system will determine the access type of each index level. For example, some index levels may be more suitable for range queries (such as interval retrieval), while other levels are suitable for exact match queries (such as equality queries). Further, based on the analysis results of the access type of each index level, the access modes and access types of different levels are adjusted to ensure that the index of each level can best match its main query type, and finally form a "type-adjusted index" to optimize the overall query performance.

[0091] Step 506 : Replace the adjustment frequency threshold and execution logic of each index level in the type adjustment index according to the scenario information of the real-time access operation information to obtain a logical reconstruction index.

[0092] The scenario information may be external or internal environmental data that affects the operation of the data table, and generally includes changes in query load, user access behavior patterns, and the update frequency of the data table content.

[0093] The adjustment frequency threshold may be a range condition for the system to perform index update or adjustment operations, which determines the frequency at which the index structure needs to be updated or reconstructed under specific circumstances.

[0094] The execution logic may be the specific steps and strategies taken by a data table when executing a query or index update operation.

[0095] Specifically, the system adjusts the refresh and update strategy of the index based on the scenario information contained in the real-time access operation information. The scenario information can reveal different usage scenarios of data table operations. For example, some operations may be more frequent in a specific time period (such as a large number of queries during peak hours), while some query patterns may change suddenly. Based on this scenario information, the system will set a new adjustment frequency threshold for each index level in the "Type Adjustment Index", that is, determine the frequency range in which the index level needs to be updated, and adjust the execution logic of the index according to the real-time scenario, for example, increase the refresh frequency range of frequently accessed indexes, or reduce the maintenance frequency range of infrequently used indexes. Through this replacement and optimization, the final "logically reconstructed index" will be more adaptable to real-time query needs and scenario changes, thereby ensuring that the data table always maintains efficient query performance under constantly changing loads.

[0096] In one embodiment, the data table system of an e-commerce platform discovers that a certain "hot-selling product" category will receive a large number of access requests during a big promotion. To adapt to this change, the system sets a higher update frequency threshold for the "product category" index level in the "type adjustment index", so that the index at this level can be refreshed more frequently to reflect real-time inventory changes and price adjustments. At the same time, based on real-time scenario analysis, the system adjusts the execution logic of the index and prioritizes some high-frequency query requests (such as price queries and inventory queries) to ensure that queries can respond more quickly during the big promotion and avoid query performance degradation caused by lagging index updates.

[0097] In this embodiment, by rearranging the data table and adjusting the order of each index node in the index according to the access frequency of the hot data blocks, the data retrieval path can be optimized and the query efficiency can be improved. The rearranged index structure can effectively reduce unnecessary index scanning and data access, thereby speeding up the query response time. Furthermore, by analyzing the query response time and index hit rate of each level in the rearranged index and matching the access type, it can be ensured that the access method of each index level is more in line with the actual access needs, thereby improving the index hit rate and query efficiency. Finally, the frequency threshold and execution logic of the index level are dynamically adjusted according to the scenario information in the real-time access operation, so that the index can flexibly adapt to different operation scenarios, reduce frequent invalid index refreshes, and optimize the resource utilization of the system. Overall, the adjustment improves the flexibility and efficiency of the index structure, and ensures the efficiency and stability of the system in various data access scenarios.

[0098] In an exemplary embodiment, Figure 6 As shown, using the index prediction algorithm corresponding to the target data table, based on historical access operation information, the initial index of the data table is predicted and analyzed to obtain index prediction data, including steps 602 to 604.

[0099] Step 602 : extract features from historical access operation information to obtain access frequency feature data, query complexity feature data, time series feature data, and change trend feature data.

[0100] The access frequency characteristic data may be characteristic data obtained by analyzing historical access operations and calculating the number of accesses or the frequency of accesses to each data block, field, or index item within a certain time range.

[0101] The query complexity feature data may be an analysis of historical query operations to evaluate the complexity of the query, including feature data such as the number of tables and fields involved in the query, the complexity of JOIN operations, the frequency of aggregation operations, and the use of sorting and grouping.

[0102] The time series feature data may be feature data obtained by analyzing the distribution pattern of query requests on the time axis and identifying changes in the number of visits in different time periods.

[0103] The change trend characteristic data may be characteristic data that analyzes the change trend of query patterns, data access patterns, or system load based on long-term historical data.

[0104] Specifically, the system applies an index prediction algorithm to analyze historical access operation information and extract multiple key features from it. This includes access frequency feature data, which is obtained by calculating the access frequency of each query field or data block within a certain time range to identify hot data; query complexity feature data, which is obtained by analyzing the type, field combination, and execution plan of historical queries to understand the complexity of the query; time series feature data, which is obtained by analyzing the time distribution pattern of queries to identify the temporal changes in query volume and distinguish between peak and trough periods; and change trend feature data, which is obtained by analyzing the long-term trends of access patterns and query characteristics to predict changes in access patterns in the future.

[0105] Step 604 , predicting the index structure change information and system resource allocation information of the initial index of the data table based on the access frequency feature data, the query complexity feature data, the time series feature data, and the change trend feature data, to obtain index prediction data.

[0106] The index structure change information may be prediction and analysis data about possible changes in the existing index structure during the data table index optimization process.

[0107] The system resource allocation information may be prediction data on how to allocate hardware resources such as computing resources, storage resources, and network bandwidth when executing query and index update operations on a data table.

[0108] Specifically, the system uses access frequency feature data, query complexity feature data, time series feature data, and change trend feature data to predict future change trends of the initial index of the data table. It relies on the part of the index prediction algorithm related to machine learning or statistical analysis models, and by analyzing historical data and trends, predicts possible changes in the index structure under different conditions in the future. For example, certain fields may become new hot spots due to the increase in query frequency, resulting in the need to adjust the index structure; or as the query complexity changes, it may be necessary to add new composite indexes or adjust the existing index levels. The system will also predict the system's resource allocation needs based on this data. For example, during high-frequency query periods, whether more computing resources are needed to maintain index updates, or whether storage resources need to be optimized to reduce the space occupied by the index. Ultimately, through these predictive analyses, "index prediction data" is obtained, that is, the index structure and resource allocation strategy optimized for future access patterns.

[0109] In this embodiment, by extracting features from historical access operation information, it is possible to deeply analyze the access patterns of the data table, including access frequency, query complexity, time series changes, and trend characteristics. These feature data provide a basis for predicting and optimizing the index structure, allowing the system to dynamically adjust and optimize the index structure of the data table to adapt to different query requirements and access patterns. By combining these feature data to predict the changing trends of the index structure and the allocation of system resources, potential performance bottlenecks can be identified in advance, resource waste can be avoided, and the use of system resources can be optimized. Ultimately, predictive optimization based on historical data can significantly improve the query efficiency of the data table, reduce response time, and enhance the overall performance and stability of the system.

[0110] In an exemplary embodiment, Figure 7 As shown, based on the access frequency feature data, query complexity feature data, time series feature data and change trend feature data, the index structure change information and system resource allocation information of the initial index of the data table are predicted to obtain index prediction data, including steps 702 to 706.

[0111] Step 702 : predict specific index construction information of index structure change information and index granularity adjustment information based on access frequency feature data, query complexity feature data, time series feature data, and change trend feature data to obtain index structure prediction information.

[0112] Specifically, a specific index build can be created during table optimization to meet specific query requirements or access patterns. For example, if a field or data block appears frequently in queries, resulting in long query response times, the system might decide to create a separate index for that field.

[0113] Index granularity adjustment involves adjusting the fine-grained or coarse-grained structure of the index based on changes in table query patterns. For example, certain queries may frequently access a specific range of data. The system can optimize data access efficiency by adjusting the index granularity. Indexes that are too fine-grained may result in excessively large indexes and high storage overhead, while indexes that are too coarse-grained may not effectively accelerate queries.

[0114] Among them, index structure prediction information can be information for predicting and planning possible future index adjustments. It covers which fields or tables may need new indexes, which existing indexes may need to be adjusted or optimized, and changes in the index's granularity, order, and hierarchical structure.

[0115] Specifically, the system analyzes access frequency feature data, query complexity feature data, time series feature data, and change trend feature data to predict future access patterns of data tables. These data can help identify which fields, data blocks, or tables will become hot spots in the future, which queries will become more complex, or which queries may see a significant increase in request volume within a specific time period. For example, if the access frequency of a field rises sharply, the system may predict the need to build a separate index for that field instead of relying on existing composite indexes. At the same time, based on changes in query complexity, the system may predict the need to add or adjust certain composite indexes to handle more complex queries, or based on time series analysis, identify certain query types that will peak during specific periods, thereby dynamically adjusting the index granularity to adapt to the query load. Through these analyses, the system can predict specific index construction information for specific index structure change information and adjustment plans for index granularity adjustment information, such as adding new indexes, modifying the granularity or indexing method of existing indexes, and generating index structure prediction information.

[0116] Step 704 , predicting cache adjustment information, storage space change information, and interface adjustment information of system resource allocation information based on access frequency feature data, query complexity feature data, time series feature data, and change trend feature data, to obtain resource allocation prediction information.

[0117] The cache adjustment information may be how to adjust the allocation and usage strategy of the memory cache when the data table executes queries and data operations.

[0118] The storage space change information may be how the data table adjusts the allocation and use of storage space during resource management.

[0119] Interface adjustment information can include how the system's input and output interface configurations or bandwidth are adjusted to accommodate different access patterns and resource requirements. For example, as query loads increase, especially for queries involving large amounts of data transfer (such as large-scale report generation and data migration), the system may need to optimize the bandwidth or concurrent processing capabilities of the data interface.

[0120] Resource allocation prediction information can be based on an analysis of access patterns, query load, hardware resource usage, and system performance to predict future demands for system resources (such as computing, storage, and networking), and formulate corresponding resource allocation strategies. This prediction information includes cache adjustment, storage space management, computing resource allocation, network bandwidth configuration, and more. For example, if the system predicts that a query module will enter a high-concurrency phase, it may require more computing resources and cache space; if the size of a data table increases, it may require more storage space or index reorganization.

[0121] Specifically, the system predicts the system's future resource requirements through a comprehensive analysis of access frequency feature data, query complexity feature data, time series feature data, and change trend feature data. Access frequency feature data can reveal which data will become more popular, requiring more cache space to speed up query responses; query complexity feature data may indicate that the system will require more computing resources (such as CPU or memory) when executing complex queries. Time series feature data can help identify peak and off-peak periods, thereby predicting that the system may need to increase bandwidth or computing power during certain time periods, or reduce resource consumption during periods of low query volume. Change trend feature data helps the system identify long-term change trends, such as the gradual increase in access to certain data sets or the gradual rise in popularity of certain query types, and thereby adjust the allocation of storage space and interface resources. Through these predictions, the system can plan and adjust cache adjustment information, storage space change information, and interface adjustment information in advance to avoid performance bottlenecks caused by insufficient resources, thereby generating resource allocation prediction information.

[0122] Step 706: Fusing the index structure prediction information and the resource allocation prediction information to obtain index prediction data.

[0123] Specifically, the system combines the "index structure prediction information" and "resource allocation prediction information" obtained in the first two steps, comprehensively considering changes in the index structure and resource allocation needs. For example, the system may predict that the query frequency of a certain field will increase significantly, requiring the creation of a new index for this field. It also predicts that this change will lead to increased demand for cache and storage resources. By fusing these two types of information, the system can generate index prediction data that not only includes recommendations for index optimization and adjustment, but also includes corresponding resource allocation plans.

[0124] In this embodiment, by predicting changes in the index structure based on characteristic data such as access frequency, query complexity, time series and change trends, the index construction method and granularity can be dynamically adjusted for different query types and data access patterns, thereby optimizing data retrieval performance. At the same time, predicting the allocation of system resources, such as cache adjustment, storage space changes and interface adjustments, can ensure that system resources are reasonably configured, avoid over-allocation or under-allocation of resources, and improve the response speed and stability of the system. By integrating index structure prediction information and resource allocation prediction information, the system can achieve more refined optimization, respond to potential performance bottlenecks in advance, and improve the query efficiency of the data table and the processing capacity of the overall system. Ultimately, the prediction and optimization mechanism can significantly improve the flexibility, efficiency and scalability of the system, providing users with faster and more stable services.

[0125] In an exemplary embodiment, Figure 8As shown, searching for the Pareto front of index optimization data and index prediction data to obtain the index optimal adjustment data includes steps 802 to 806.

[0126] Step 802: Determine the index optimization target and the Pareto judgment condition based on the index optimization data and the index prediction data.

[0127] Among them, the index optimization goal can be multiple goals that need to be optimized and weighed during the data table index adjustment process, including query performance optimization (such as minimizing query response time), storage space optimization (such as minimizing the storage space occupied by the index), system resource consumption optimization (such as the use of CPU, memory and other resources), and the cost of index update and maintenance (such as reducing the frequency of index reconstruction).

[0128] The Pareto criterion may be a criterion for judging whether one solution is better than another solution in a multi-objective optimization problem.

[0129] Specifically, based on the index optimization data and index prediction data, the index optimization goals are clarified, where these goals usually include maximizing query performance (for example, minimizing query response time), minimizing storage overhead, and balancing the use of computing and storage resources. In addition, it is necessary to set Pareto judgment conditions, which are used to evaluate which optimization solutions can achieve better performance on certain goals without causing significant losses on other goals. For example, the condition may be set: the optimization solution can significantly improve the query speed without significantly increasing the storage space. Through these goals and judgment conditions, the system can lay the foundation for subsequent Pareto frontier optimization.

[0130] Step 804 : Based on the index optimization goal and the Pareto judgment condition, at least two Pareto optimization algorithms are used to optimize the index optimization data and the index prediction data to obtain the Pareto frontier solutions corresponding to the respective Pareto optimization algorithms.

[0131] Pareto optimization algorithms can be used for multi-objective optimization problems to find an optimal set of solutions that balance multiple optimization objectives. These algorithms iteratively generate a set of solutions such that each solution outperforms the others on at least one objective, while not significantly inferior to any other solution on other objectives. Examples of Pareto optimization algorithms include NSGA-II (Non-Dominated Sorting Genetic Algorithm) and SPEA2 (Strength Pareto Evolutionary Algorithm).

[0132] Among them, the Pareto front solution can be a set of optimal solutions in a multi-objective optimization problem, representing solutions that cannot be further improved in each objective dimension.

[0133] Specifically, at least two Pareto optimization algorithms (e.g., NSGA-II, SPEA2, etc.) are applied to perform multi-objective optimization on index optimization data and index prediction data. These algorithms, based on pre-set index optimization objectives and Pareto criteria, continuously adjust the index structure and resource allocation strategy to find the optimal compromise between different objectives. Each algorithm generates a set of solutions, called "Pareto frontier solutions," which represent the best solutions that cannot be further improved under given conditions. In other words, it is impossible to improve on one objective without affecting the performance of other objectives. Different Pareto frontier solutions provide multiple potential optimization solutions.

[0134] Step 806: Perform super-front aggregation on each Pareto front solution to obtain index optimal adjustment data.

[0135] Among them, super-front aggregation can be the process of fusing multiple Pareto front solution sets from different Pareto optimization algorithms, aiming to generate a global optimal solution by removing redundant solutions and improving the quality of solutions.

[0136] Specifically, hyperfront aggregation aims to integrate Pareto front solutions from multiple Pareto optimization algorithms to obtain the optimal index adjustment plan. Therefore, in multi-objective optimization, each Pareto front solution represents an optimal solution in the objective space where it is impossible to improve one objective without affecting other objectives. However, different optimization algorithms may generate different sets of front solutions, which may contain duplicate or redundant components and may each have different advantages. Through hyperfront aggregation, the system compares and integrates these solutions, removes duplicate and inefficient solutions, and selects the solution that achieves the best balance between multiple optimization objectives (such as query response time, storage space consumption, system resource utilization, etc.). During the hyper-aggregation process, the system considers the performance of each solution across multiple dimensions and selects the index adjustment plan that best meets actual needs by weighing the importance of different objectives. The aggregated optimal index adjustment data will provide a comprehensive and efficient adjustment plan for data table optimization, ensuring optimal performance and resource utilization in real-world applications.

[0137] In this embodiment, by determining the index optimization target and the Pareto judgment condition based on the index optimization data and the index prediction data, a clear direction and standard can be set for the optimization process to ensure that the optimal balance point is found between different optimization targets. At least two Pareto optimization algorithms are used to perform multi-dimensional optimization of the index optimization data and the index prediction data, which can simultaneously consider multiple optimization factors such as query speed, system resource consumption and response time, thereby obtaining multiple Pareto frontier solutions and providing a series of possible optimization schemes. By performing super-frontier aggregation on these Pareto frontier solutions, the advantages of each scheme can be combined to ultimately obtain the optimal index adjustment data, achieve a balance between multiple targets, and improve the overall performance and efficiency of the system. This method based on Pareto optimization can accurately meet the balance between different queries and system resource requirements, ensuring that the data table can run efficiently under various conditions.

[0138] In an exemplary embodiment, Figure 9 As shown, according to the index optimal adjustment data, the target data table is optimized to obtain the optimized data table, including steps 902 to 910.

[0139] Step 902: Determine a data table style based on the data structure information of the index optimally adjusted data.

[0140] Among them, data structure information can be a collection of information that describes the internal field organization, data type, constraints, index design, etc. of a data table. It includes the attributes of each column in the table (such as column name, data type, size, whether it is empty, etc.), table constraints (such as primary key, foreign key, unique constraint, check constraint, etc.), and relationships between tables (such as foreign key relationships).

[0141] The data table style may be the display form of the data table and its layout design, involving the arrangement order of fields in the table, the paging, partitioning or column separation strategy of the table, the visual presentation style, etc.

[0142] Specifically, data structure information is extracted from the index optimal adjustment data, and based on this information, it is decided how to organize and layout the fields and data of the target data table. These data structure information include column types, sizes, constraints, and data table partitioning methods. The system will select the appropriate data table style design based on different query patterns, data access frequencies, and storage requirements. For example, vertical partitioning may be chosen to store the most frequently queried fields, while infrequently queried fields may be partitioned for storage; or the data table may be divided into multiple small tables (such as partitioned tables) to improve query efficiency and maintainability. The determination of the data table style is not only about the physical storage form of the table, but also includes whether the design of the data table is suitable for efficient data retrieval, update, and insert operations. It is further combined with the goal of index optimization to ensure that the final table structure can support optimal query and storage performance.

[0143] Step 904 : setting single information operation interface information and multi-information operation interface information according to the data table style and data structure information.

[0144] Among them, the single information operation interface information can be the data of the user interface design for displaying and operating a single data record. It is usually used to view, edit, delete or add a single data item, and only the detailed information of a single record is displayed on the interface.

[0145] The multi-information operation interface information can be the user interface design data for displaying and manipulating multiple data records. Unlike single-information operation interfaces, multi-information operation interfaces usually need to support batch operations, filtering, sorting, paging, and other functions. It is often used in scenarios that process large amounts of data records, such as data lists and report generation.

[0146] Specifically, the system designs the user interaction interface based on the determined data table style and data structure information, including single-information operation interface information and multi-information operation interface information. The single-information operation interface is mainly used to display and edit a single data record. Only the necessary fields are displayed on the interface, and the user can efficiently view, modify or delete operations through a simple operation interface; while the multi-information operation interface is used to display and operate multiple records, and usually needs to provide more complex filtering, sorting, paging, batch operations and other functions. During the design, the system will take into account the layout of the interface, the way data is presented (such as tables, cards, etc.), the sorting of fields, the display logic, etc., to ensure that the interface can meet different operation requirements. At the same time, it is also necessary to ensure that users can operate data on the interface conveniently to avoid overly complex interaction design affecting the user experience.

[0147] Step 906 , performing data table responsive front-end optimization based on the data table style, single information operation interface information, and multi-information operation interface information to obtain data table hardware adaptation information.

[0148] The data table hardware adaptation information may be information that is obtained by considering the adaptation requirements of different hardware devices (such as desktop computers, mobile devices, tablets, etc.) for table display and interaction when designing the data table.

[0149] Specifically, since the responsive front-end optimization of the data table is to enable the data table to provide the best display effect and interactive experience on different devices and screen sizes, the system will adaptively adjust the front-end interface according to the data table style, single-information operation interface information, and the operation interface design corresponding to the multi-information operation interface information. Responsive design ensures that the data table can automatically adjust according to the screen size, resolution, and browser characteristics of the device. For example, in a desktop display, the table may display complete fields and complex operation buttons, while on a mobile device, the table may save space by hiding secondary columns or using drop-down menus, and optimize the operation method (such as touch operation). In addition, the system also needs to make optimizations based on hardware characteristics (such as memory, processing power, network bandwidth, etc.) to ensure that the loading speed and interactive performance of the data table on different devices can reach the best state, and obtain the data table hardware adaptation information, where the data table hardware adaptation information includes the adjustment of interface elements under different devices, the table data loading method, the response time optimization, etc.

[0150] Step 908 : interactively optimize the target data table according to the data table style, the single information operation interface information, the multi-information operation interface information, and the data table hardware adaptation information to obtain an interactively optimized data table.

[0151] Among them, interaction optimization can be to improve the interactive experience between users and data tables, with the aim of allowing users to operate and browse data more conveniently and quickly.

[0152] Interactively optimized data tables are interactively optimized data tables designed to provide a more efficient, smooth, and intuitive user experience. These tables typically support fast data display, editing, deleting, filtering, sorting, and paging, and their interface design makes it easy for users to understand the structure and content of the data. For example, interactively optimized data tables can reduce loading times through dynamic loading technology, simplify data processing through drag-and-drop and single-click operations, and enable users to quickly find the data they need through intelligent filtering and sorting.

[0153] Specifically, based on the data table style, single-information operation interface information, multi-information operation interface information and hardware adaptation information, the interactive experience of the target data table is optimized; the interactive optimization mainly involves improving the efficiency and convenience of the interaction between the user and the data table. For example, the system may reduce the time and complexity of user operations by optimizing the interaction methods such as drag and drop sorting, paging switching, and batch data operations (such as batch deletion and batch modification) in the table; in addition, it may also adjust the priority and loading order of interface elements based on user behavior analysis (such as the most frequently accessed fields and the most frequently performed operations) to improve user operation efficiency; at the same time, interactive optimization also includes optimizing the response speed of controls such as buttons, drop-down boxes, and check boxes to ensure a smooth use experience of the table on different devices, reduce the user experience differences caused by differences in device performance, and obtain an interactively optimized data table.

[0154] Step 910: Perform multi-level view optimization on the interactive optimization data table to obtain an optimized data table.

[0155] Specifically, the system performs multi-level view optimization on the interactively optimized data table, with the aim of providing more flexible and diverse data display formats based on different user needs and usage scenarios. Multi-level view optimization refers to setting different view levels for the data table so that users can switch between different views as needed. For example, users can view the summary information and statistical data of the data table in the "Overview" view, and view the complete information of a certain record in the "Detailed" view. The system can also provide views such as chart display and report display of the data to help users better understand and analyze the data. To achieve this, the system will dynamically load data and use technologies such as data filtering, grouping and paging to ensure that the content of different views can respond quickly, avoiding slow data loading or interface freezes. In addition, the optimization of multi-level views also includes ensuring seamless switching between view levels and flexible user customization capabilities to ensure that the display method of the data table can meet the needs of different users and application scenarios, and obtain an optimized data table.

[0156] In this embodiment, by determining the data table style based on the data structure information of the index optimally adjusted data, it is possible to ensure that the data table is more in line with actual needs in terms of display and operation, and can efficiently present the data content. According to the data table style and data structure information, single information and multi-information operation interfaces are set, which can provide flexible interaction methods for different operation scenarios and improve the convenience and accuracy of user operation. Further, by optimizing the responsive front-end of the data table style, operation interface information and hardware adaptation, the data table can be displayed and operated smoothly on different hardware devices (such as desktops, tablets and mobile phones), thereby improving cross-platform compatibility and user experience. By optimizing the design of the interactive data table, the user operation process can be simplified, unnecessary clicks and redundant information can be reduced, and the interactive experience can be optimized. Ultimately, through multi-level view optimization, the visualization effect and layering of the data table can be further improved, so that users can operate and view data more clearly and intuitively. A series of optimizations effectively improve the system's response speed, user operation efficiency and visual experience, providing users with a more efficient, convenient and pleasant data interaction environment.

[0157] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0158] Based on the same inventive concept, embodiments of the present application also provide a data table processing device for implementing the aforementioned data table processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in the one or more data table processing device embodiments provided below can be found in the above-mentioned limitations on the data table processing method and will not be further elaborated here.

[0159] In an exemplary embodiment, Figure 10 As shown, a data table processing device is provided, including: a data acquisition module 1002, an index analysis module 1004, an index prediction module 1006, an index optimization module 1008 and a table optimization module 1010, wherein:

[0160] The data acquisition module 1002 is used to acquire the real-time access operation information, historical access operation information and initial index of the target data table;

[0161] The index analysis module 1004 is used to optimize and analyze the initial index of the data table based on the real-time access operation information using the index adjustment algorithm corresponding to the target data table to obtain index optimization data;

[0162] The index prediction module 1006 is used to use the index prediction algorithm corresponding to the target data table to perform prediction analysis on the initial index of the data table based on historical access operation information to obtain index prediction data;

[0163] The index optimization module 1008 is used to find the Pareto front of index optimization data and index prediction data to obtain the optimal index adjustment data;

[0164] The table optimization module 1010 is used to optimize the target data table by adjusting the data according to the optimal index to obtain an optimized data table.

[0165] Each module in the aforementioned data table processing device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0166] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 11 As shown. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. Among them, the memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a data table. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The data table of the computer device is used to store server data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data table processing method is implemented.

[0167] In one embodiment, a data table processing system is provided. The system includes an electronic device and a storage medium. The storage medium stores a computer program. When the computer program is executed by the electronic device, the steps of a data table processing method are implemented.

[0168] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0169] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer programs. Among them, any reference to memory, data tables or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memories. The data tables involved in the embodiments provided in this application may include at least one of relational data tables and non-relational data tables. Non-relational data tables may include distributed data tables based on blockchain, etc., but are not limited to this. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, etc., but are not limited to this.

[0170] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A data table processing method, characterized in that: The method comprises: Obtain the real-time access operation information, historical access operation information and initial index of the target data table; Using the index adjustment algorithm corresponding to the target data table, and according to the real-time access operation information, optimizing and analyzing the initial index of the data table to obtain index optimization data; Extracting features from the historical access operation information to obtain access frequency feature data, query complexity feature data, time series feature data, and change trend feature data; Predicting specific index construction information and index granularity adjustment information of index structure change information of the initial index of the data table based on the access frequency feature data, the query complexity feature data, the time series feature data, and the change trend feature data to obtain index structure prediction information; Based on the access frequency characteristic data, the query complexity characteristic data, the time series characteristic data, and the change trend characteristic data, cache adjustment information, storage space change information, and interface adjustment information of the system resource allocation information of the initial index of the data table are predicted to obtain resource allocation prediction information; the interface adjustment information is the configuration or bandwidth of the input and output interfaces of the system adjusted when the data table responds to different access modes and resource requirements; fusing the index structure prediction information and the resource allocation prediction information to obtain index prediction data; Finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data; The target data table is optimized according to the index optimal adjustment data to obtain an optimized data table.

2. The method according to claim 1, characterized in that The using the index adjustment algorithm corresponding to the target data table and performing optimization analysis on the initial index of the data table according to the real-time access operation information to obtain index optimization data includes: Identifying hot data blocks and high-frequency query types corresponding to the target data table from the real-time access operation information; Adjusting the index structure of the initial index of the data table according to the hot data block and the high-frequency query type to obtain an adjusted index of the data table; The index adjustment algorithm is used to optimize the access mode of the adjusted index of the data table to obtain the index optimization data.

3. The method according to claim 2, characterized in that The step of using the index adjustment algorithm to optimize the access mode of the index adjusted on the data table to obtain the index optimization data includes: Reconstructing the index logic of the data table adjustment index according to the access frequency of the hot data block to obtain a logically reconstructed index; According to the addition, deletion, and modification operation information in the real-time access operation information, the refresh frequency and resource allocation information of the logical reconstruction index are adjusted to obtain the index optimization data.

4. The method according to claim 3, characterized in that The step of reconstructing the index logic of the data table adjustment index according to the access frequency of the hot data block to obtain a logically reconstructed index includes: Rearranging the order of the index nodes in the data table adjustment index according to the access frequency of the hot data block to obtain a rearranged adjustment index; Matching the access type of each index level according to the query response time and index hit rate corresponding to each index level in the rearranged and adjusted index to obtain a type-adjusted index; According to the scenario information of the real-time access operation information, the adjustment frequency threshold and the execution logic of each index level in the type adjustment index are replaced to obtain the logical reconstruction index.

5. The method according to claim 1, wherein The step of finding the Pareto front of the index optimization data and the index prediction data to obtain the index optimal adjustment data includes: Determining an index optimization target and a Pareto judgment condition based on the index optimization data and the index prediction data; Based on the index optimization goal and the Pareto judgment condition, using at least two Pareto optimization algorithms to optimize the index optimization data and the index prediction data, and obtain Pareto frontier solutions corresponding to the respective Pareto optimization algorithms; Performing super-front aggregation on each of the Pareto front solutions to obtain the index optimal adjustment data.

6. The method according to claim 1, characterized in that The step of optimizing the target data table according to the optimal index adjustment data to obtain an optimized data table includes: Determine a data table style according to the data structure information of the optimally adjusted data of the index; Setting single information operation interface information and multi-information operation interface information according to the data table style and the data structure information; Performing data table responsive front-end optimization according to the data table style, the single-information operation interface information, and the multi-information operation interface information to obtain data table hardware adaptation information; Interactively optimizing the target data table according to the data table style, the single-information operation interface information, the multi-information operation interface information, and the data table hardware adaptation information to obtain an interactively optimized data table; Multi-level view optimization is performed on the interactive optimization data table to obtain the optimized data table.

7. A data table processing device, characterized in that: The device comprises: The data acquisition module is used to obtain the real-time access operation information, historical access operation information and initial index of the target data table; An index analysis module, configured to use an index adjustment algorithm corresponding to the target data table and, based on the real-time access operation information, perform optimization analysis on the initial index of the data table to obtain index optimization data; An index prediction module is used to extract features from the historical access operation information to obtain access frequency feature data, query complexity feature data, time series feature data, and change trend feature data; Predicting specific index construction information and index granularity adjustment information of index structure change information of the initial index of the data table based on the access frequency feature data, the query complexity feature data, the time series feature data, and the change trend feature data to obtain index structure prediction information; Based on the access frequency characteristic data, the query complexity characteristic data, the time series characteristic data, and the change trend characteristic data, cache adjustment information, storage space change information, and interface adjustment information of the system resource allocation information of the initial index of the data table are predicted to obtain resource allocation prediction information; the interface adjustment information is the configuration or bandwidth of the input and output interfaces of the system adjusted when the data table responds to different access modes and resource requirements; fusing the index structure prediction information and the resource allocation prediction information to obtain index prediction data; An index optimization module, configured to find the Pareto front of the index optimization data and the index prediction data to obtain the optimal index adjustment data; The table optimization module is used to optimize the target data table according to the optimal adjustment data of the index to obtain an optimized data table.

8. A data table processing system, characterized in that: The system includes an electronic device and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the electronic device, the steps of the method according to any one of claims 1 to 6 are implemented.

Images