Data management method, system, device and medium for data storage warehouse

Abstract

The present invention discloses a data management method, system, device and medium for a data storage warehouse, which relates to the field of data processing and storage technology, and includes the steps of: obtaining the original data managed by an enterprise, and extracting the metadata of the original data; dynamically partitioning the metadata through a preset partitioning strategy and algorithm, and allocating corresponding storage resources and optimization strategies to each partition; establishing an adaptive index for each partition according to the result of the dynamic partitioning, and establishing a corresponding index structure for each partition according to the index result, and monitoring the update of the data; storing the metadata after dynamic partitioning and indexing processing in a data storage warehouse through a distributed storage technology, configuring storage nodes and a backup mechanism, and performing parallel processing and load balancing processing of data. The present invention can intelligently adapt to data changes and query requirements, automatically adjust storage and retrieval strategies, and achieve optimal performance and efficiency through dynamic partitioning and adaptive indexing technology.

Description

Technical Field

[0001] The present invention relates to the field of data processing and storage technology, and in particular to a data management method, system, device and medium for a data storage warehouse. Background Art

[0002] Driven by the wave of informatization and digitalization, the amount of global data has exploded. During the operation of the database, historical data is used too infrequently and accumulates in the business database, which will lead to a decline in query performance.

[0003] The data warehouse manages the data of various heterogeneous data source databases in a unified manner, and completes the elimination and format conversion of data with poor quality. Finally, it completes the transformation of the source data organization form according to a reasonable modeling method to better support front-end visual analysis.

[0004] When faced with massive, diverse, and rapidly changing data, traditional data storage warehouses only manage data according to a specific model and are unable to automatically adapt to data changes. They often exhibit problems such as low storage efficiency, slow retrieval speed, and complex management, and are unable to meet the needs of modern enterprises and organizations for efficient data processing and application. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and to provide a data management method, system, device and medium for a data storage warehouse, so as to solve the problems in the prior art that it cannot automatically adapt to changes in data, often exhibits problems such as low storage efficiency, slow retrieval speed, complex management, and cannot meet the needs of modern enterprises and organizations for efficient data processing and application.

[0006] The present invention specifically provides the following technical solution: a data management method for a data storage warehouse, comprising the following steps:

[0007] Acquire the original data managed by the enterprise and extract metadata of the original data; the metadata includes the type, size, access frequency and relevance information of the original data;

[0008] Divide the metadata into multiple logically independent and physically continuous dynamic partitions, allocate corresponding storage resources and optimization strategies to each partition, and obtain the result of dynamic partitioning; wherein, logically independent means that the logical structure of metadata in the partition has different logics, and physically continuous means that the metadata in the partition has continuous and complete metadata physical properties;

[0009] According to the result of the dynamic partitioning, an adaptive index is established for each partition, and a corresponding index structure is established for each partition according to the index result to monitor the update of the data;

[0010] The metadata after dynamic partitioning and indexing is stored in a data storage warehouse through distributed storage technology, and storage nodes and backup mechanisms are configured to perform parallel processing and load balancing of data.

[0011] Preferably, the step of extracting metadata of the original data further includes the following steps:

[0012] Check the quality of raw data and remove duplicate, invalid or erroneous data;

[0013] Perform data conversion and standardization according to the data type, format and structure;

[0014] Extract metadata from data after data transformation and standardization.

[0015] Preferably, dividing the metadata into a plurality of logically independent and physically continuous dynamic partitions comprises the following steps:

[0016] Analyze the metadata's relevance, access frequency, and size factors, determine the number and size of partitions, and select an appropriate partitioning algorithm based on the metadata's characteristics;

[0017] The metadata is divided into multiple logically independent and physically continuous partitions according to the partitioning algorithm.

[0018] Preferably, according to the result of the dynamic partitioning, an adaptive index is established for each partition, and according to the index result, a corresponding index structure is established for each partition, including the following steps:

[0019] Analyze the query mode, update frequency and data volume of the data, determine the index type and parameters, and select the appropriate index algorithm based on the characteristics of the data;

[0020] Establish a corresponding index structure for each partition based on the index results, monitor data updates in real time, and dynamically adjust the index.

[0021] Preferably, when establishing a corresponding index structure for each partition according to the index result, the following steps are also included:

[0022] By compressing and encoding the index data, the storage space occupied by the index is reduced;

[0023] By introducing approximate indexing and incremental indexing techniques, only the incremental part of the index is updated when data is updated.

[0024] Preferably, the metadata after dynamic partitioning and indexing is stored in a data storage warehouse through distributed storage technology, and storage nodes and backup mechanisms are configured, including the following steps:

[0025] Use distributed storage technology to disperse data to multiple storage nodes;

[0026] The data is compressed and encrypted for storage, and data backup and recovery functions are provided.

[0027] Preferably, after performing the parallel processing and load balancing of the data, the following steps are further included:

[0028] Centrally manage and maintain the data stored in the warehouse, and provide a data operation interface for data query, update and deletion, as well as multi-user concurrent access and transaction processing services;

[0029] Perform data version control and permission management functions, and conduct performance monitoring and optimization. Monitor the performance status of the storage warehouse in real time and make adjustments as needed.

[0030] The present invention also provides a data management system for a data storage warehouse, comprising:

[0031] A preprocessing module, used to obtain the original data managed by the enterprise and extract metadata of the original data; the metadata includes the type, size, access frequency and relevance information of the original data;

[0032] A dynamic partitioning module is used to divide the metadata into multiple logically independent and physically continuous dynamic partitions, allocate corresponding storage resources and optimization strategies to each partition, and obtain dynamic partitioning results; wherein, logically independent means that the metadata logical structure in the partition has different logics, and physically continuous means that the metadata in the partition has continuous and complete metadata physical properties;

[0033] An index module is used to establish an adaptive index for each partition according to the result of the dynamic partitioning, and to establish a corresponding index structure for each partition according to the index result, and monitor the update of the data;

[0034] The storage module is used to store the metadata after dynamic partitioning and indexing in a data storage warehouse through distributed storage technology, configure storage nodes and backup mechanisms, and perform parallel processing and load balancing of data.

[0035] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the data management method of the data storage warehouse.

[0036] The present invention also provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data management method for a data storage warehouse.

[0037] Compared with the prior art, the present invention has the following significant advantages:

[0038] The present invention divides metadata into multiple logically independent and physically continuous dynamic partitions through dynamic partitioning and adaptive indexing technology, can intelligently adapt to data changes and query requirements, establish adaptive indexes and index structures for each partition, automatically adjust storage and retrieval strategies to achieve optimal performance and efficiency, and store the processed metadata in a data storage warehouse through distributed storage technology, configure storage nodes and backup mechanisms, perform parallel processing and load balancing of data, and through the above-mentioned optimized partitioning and indexing strategies, can significantly improve data storage efficiency and retrieval speed, reduce storage costs, and enhance overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart provided by the present invention;

[0040] Figure 2 It is a data layer diagram in the system provided by the present invention; wherein (a)-(e) are respectively the data access layer, the data storage layer, the data processing layer, the index management layer and the user access layer. DETAILED DESCRIPTION

[0041] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] Deep reinforcement learning (DRL) is a new technology that combines deep learning and reinforcement learning. It learns the optimal strategy through the interaction between the agent and the environment. In intelligent scheduling, deep reinforcement learning is often used to solve problems such as dynamic scheduling and real-time decision-making. This method will open up new ideas, adapt deep reinforcement learning to the scheduling of emergency medical resources, and further combine a traditional evolutionary computing method (particle swarm optimization algorithm) to optimize its scheduling model and scheduling strategy, fully integrating the advantages of traditional methods and deep reinforcement learning, thereby improving the accuracy and efficiency of scheduling plan generation.

[0043] During the system initialization phase, a series of configuration and preparation work is required to ensure the normal operation of the system. First, according to business requirements and data characteristics, set the basic parameters and configuration of the system, such as partition strategy, index algorithm, storage strategy, etc. Secondly, establish the data cleaning rules, conversion standards and standardization methods required by the data preprocessing module to facilitate subsequent data preprocessing. Finally, build a distributed storage environment for the data storage layer, configure storage nodes and backup mechanisms to ensure data reliability and security.

[0044] Further implementation details of the data storage warehouse technology refinement enhancement system based on dynamic partitioning and adaptive indexing.

[0045] like Figure 1 As shown, an embodiment of the present invention provides a data management method for a data storage warehouse, comprising the following steps:

[0046] Step S1: Obtain the original data managed by the enterprise and extract metadata of the original data. The metadata includes the type, size, access frequency and relevance information of the original data.

[0047] In this step, the metadata of the original data is extracted, which also includes the following steps:

[0048] Step S11: Check the quality of the original data and remove duplicate, invalid or erroneous data.

[0049] Step S12: Perform data conversion and standardization according to the data type, format and structure to ensure data consistency and comparability.

[0050] Step S13: Extract metadata of the data after data conversion and standardization, including data type, size, access frequency and relevance information, to provide basic data for subsequent partitioning and indexing strategy formulation.

[0051] The above steps are completed through the data preprocessing module, which is the starting point of the entire system and is responsible for cleaning, converting and preliminary analysis of the raw data.

[0052] In the data preprocessing stage, in addition to basic cleaning, conversion and standardization, machine learning algorithms can be further introduced to optimize the preprocessing process. For example, unsupervised learning algorithms can be used to automatically discover outliers and noise in the data and process or exclude them appropriately. At the same time, supervised learning algorithms can be used to extract and select features from the data to extract features that are more critical for subsequent analysis and processing.

[0053] In addition, in order to further improve the efficiency and accuracy of data preprocessing, parallel preprocessing algorithms can be developed to accelerate the preprocessing process by using multi-core processors or distributed computing resources. Through parallel processing, the preprocessing time can be greatly reduced and the overall performance of the system can be improved.

[0054] Step S2: Divide the metadata into multiple logically independent and physically continuous dynamic partitions, allocate corresponding storage resources and optimization strategies to each partition, and obtain the result of dynamic partitioning.

[0055] This step specifically includes the following steps:

[0056] Step S21: Analyze the relevance, access frequency and size factors of the metadata, determine the number and size of partitions, and select a suitable partitioning algorithm according to the characteristics of the metadata, such as range-based partitioning, hash-based partitioning, clustering-based partitioning, etc.

[0057] Step S22: Divide the metadata into multiple logically independent and physically continuous partitions according to the partitioning algorithm, and allocate corresponding storage resources and optimization strategies to each partition.

[0058] The above steps are completed through the dynamic partitioning module, which dynamically partitions the data based on the information provided by the data preprocessing module and combined with the preset partitioning strategy and algorithm.

[0059] Specifically:

[0060] 1) Demand Analysis:

[0061] Analyze the storage requirements of the current system in detail, including the type, size, growth trend, and access pattern of data.

[0062] Evaluate existing partition usage and identify performance bottlenecks and potential optimization points.

[0063] 2) Choose a suitable partitioning algorithm:

[0064] Based on the results of the demand analysis, select the partitioning algorithm that best suits the current data characteristics and system requirements.

[0065] Consider factors such as how often the data is accessed, how often it is updated, and how large the data is distributed.

[0066] 3) Develop a partitioning strategy:

[0067] Design a reasonable partitioning strategy, including the size, number, and location of partitions.

[0068] Consider the locality and relevance of data, and try to store related data in similar partitions.

[0069] 4) Back up your data:

[0070] Before making any partition adjustments, be sure to back up important data to prevent data loss or corruption.

[0071] Ensure the integrity and availability of backups so that data can be quickly restored when needed.

[0072] 5) Adjust partitions:

[0073] Use professional disk partitioning tools or system management tools to adjust the size and location of the partitions.

[0074] Based on the established partitioning strategy, the existing partitions are gradually adjusted to adapt to the data growth and changes in access patterns.

[0075] 6) Monitoring and tuning:

[0076] After partition adjustments, continue to monitor system performance and partition usage.

[0077] Use performance monitoring tools to collect and analyze data and identify potential performance issues or bottlenecks.

[0078] Based on the monitoring results, the partitioning strategy is tuned to improve system performance and storage efficiency.

[0079] 7) Implement automated management:

[0080] Consider using automated tools and scripts to manage dynamic partitions to reduce the complexity and error of manual operations.

[0081] Use automated tools to regularly check and adjust partitions to adapt to changes in data and system needs.

[0082] 8) Regular evaluation and update:

[0083] Regularly evaluate the effectiveness of your partitioning strategy and update it based on system changes and new requirements.

[0084] As technology evolves and storage devices are updated, consider adopting new partitioning algorithms and management tools to further improve performance.

[0085] It should be noted that dynamic partition optimization is an ongoing process that requires regular evaluation and adjustment. At the same time, be careful when adjusting partitions to avoid data loss or system instability due to misoperation.

[0086] Finally, the importance of backing up data is emphasized. Before making any partition adjustments, be sure to have a complete backup strategy in place and verify the integrity and availability of the backup after the operation is complete. This helps to quickly restore data in the event of an unexpected situation, ensuring system stability and data integrity.

[0087] Step S3: According to the result of dynamic partitioning, an adaptive index is established for each partition, and a corresponding index structure is established for each partition according to the index result to monitor the update status of the data.

[0088] This step specifically includes the following steps:

[0089] Step S31: Analyze the query mode, update frequency and data size factors of the data, determine the type and parameters of the index, and select a suitable index algorithm according to the characteristics of the data, such as B-tree index, hash index, bitmap index, etc.

[0090] Step S32: Establish a corresponding index structure for each partition according to the index result, monitor the data update status in real time, and dynamically adjust and optimize the index.

[0091] In the adaptive indexing stage, index optimization techniques can be further introduced to improve the performance and efficiency of the index. First, an adaptive index adjustment algorithm based on query load can be designed. According to the frequency and pattern of query requests, the structure and parameters of the index are dynamically adjusted to better meet query requirements and reduce the overhead of index maintenance.

[0092] When establishing a corresponding index structure for each partition based on the index results, the following steps are also included:

[0093] Consider using compression technology to reduce the storage space occupied by the index. By compressing and encoding the index data, the storage space occupied by the index can be reduced, and the loading and query speed of the index can be improved. In order to further improve the query efficiency of the index, by introducing indexing technology of approximate index and incremental index, the approximate index can greatly improve the query speed at the expense of a certain degree of accuracy. The incremental index can only update the incremental part of the index when the data is updated.

[0094] The above steps are completed by the dynamic partitioning module, and the adaptive indexing module establishes an adaptive index for each partition according to the partitioning result divided by the dynamic partitioning module.

[0095] Step S4: The metadata after dynamic partitioning and indexing is stored in the data storage warehouse through distributed storage technology, and storage nodes and backup mechanisms are configured to perform parallel processing and load balancing of data.

[0096] In this step, the metadata after dynamic partitioning and indexing is stored in the data storage warehouse through distributed storage technology, and the storage nodes and backup mechanism are configured, including the following steps:

[0097] Step S41: Distributed storage technology is used to disperse data to multiple storage nodes to achieve parallel processing and load balancing of data.

[0098] Step S42: compress and encrypt the data to reduce storage space usage and improve data security, and provide data backup and recovery functions to ensure data reliability and availability.

[0099] The above steps are completed through the data storage module, which is responsible for storing the pre-processed, partitioned and indexed data in the data storage warehouse. Specifically:

[0100] In the data storage and management stage, the data storage strategy can be further optimized to improve storage efficiency and reliability. First, data deduplication technology can be introduced to eliminate redundant data in storage and reduce storage space usage. At the same time, data compression algorithms can be used to compress stored data to further reduce storage space consumption.

[0101] Secondly, in order to improve the data access speed, a cache-based data access mechanism can be designed to cache hot data and frequently used data in high-speed storage media to reduce the number of accesses to slow storage media and improve data reading speed.

[0102] In addition, in order to ensure the security of data, security mechanisms such as data encryption and access control can be introduced. Encrypted data storage and transmission can prevent data leakage and illegal access; at the same time, through identity authentication and permission verification, user access and operation rights to data can be controlled to ensure data security and integrity. Through data encryption, backup and recovery and other functions, data leakage and loss can be effectively prevented.

[0103] After parallel processing and load balancing of data, the following steps are also included:

[0104] The following steps are completed through the data management module. The data management module is the core management component of the system and is responsible for unified management and maintenance of data stored in the warehouse. The module provides a rich data operation interface for data query, update and deletion, and provides multi-user concurrent access and transaction processing services.

[0105] Perform data version control and permission management functions to ensure data integrity and security, and conduct performance monitoring and optimization, monitor the performance status of the storage warehouse in real time, and make optimization adjustments as needed.

[0106] In order to monitor the operating status and performance of the system in real time, a complete monitoring and alarm mechanism can be designed. By collecting the system's operating data, performance indicators and log information, the system can be fully monitored and analyzed. When the system is abnormal or performance degrades, the alarm mechanism is triggered in time to notify the administrator to troubleshoot and handle the problem. At the same time, visualization tools can be used to display monitoring data in the form of charts and reports, so that administrators can intuitively understand the system's operating status and performance status.

[0107] Based on the above method, the present invention provides a data management system for a data storage warehouse, including a preprocessing module, a dynamic partitioning module, an indexing module and a storage module.

[0108] Among them, the preprocessing module is used to obtain the original data managed by the enterprise and extract the metadata of the original data; the metadata includes the type, size, access frequency and correlation information of the original data; the dynamic partitioning module is used to divide the metadata into multiple logically independent and physically continuous dynamic partitions, allocate corresponding storage resources and optimization strategies to each partition, and obtain the results of dynamic partitioning; the indexing module is used to establish an adaptive index for each partition according to the results of dynamic partitioning, and establish a corresponding index structure for each partition according to the index results, and monitor the update of data; the storage module is used to store the metadata after dynamic partitioning and indexing processing in the data storage warehouse through distributed storage technology, configure storage nodes and backup mechanisms, and perform parallel processing and load balancing of data.

[0109] like Figure 2 As shown in (a), (b), (c), (d) and (e), in actual use, a data management system of a data storage warehouse of the present invention uses a data access layer as a preprocessing module, and the data access layer includes data collection tools and data cleaning and conversion; uses a data processing layer as a dynamic partitioning module, and the data processing layer includes data modeling and ETL, as well as task scheduling and data quality management; uses an index management layer as an index module, and the index management layer includes adaptive index creation and maintenance, as well as index optimization strategies; uses a data storage layer as a storage module, and the data storage layer includes dynamic partition management and a data warehouse; and sets a user interface layer, including a data query and analysis interface, as well as a data service and management interface, to facilitate users to perform data query processing.

[0110] The present invention also provides a computer device, including a memory and a processor. The memory stores a program. When the program is executed by the processor, the processor executes the steps of a data management method for a data storage warehouse.

[0111] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0112] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, a data management method for a data storage warehouse is implemented.

[0113] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0114] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A data management method for a data storage warehouse, characterized in that: The steps include: Acquire the original data managed by the enterprise and extract metadata of the original data; the metadata includes the type, size, access frequency and relevance information of the original data; Divide the metadata into multiple logically independent and physically continuous dynamic partitions, allocate corresponding storage resources and optimization strategies to each partition, and obtain the result of dynamic partitioning; wherein, logically independent means that the logical structure of metadata in the partition has different logics, and physically continuous means that the metadata in the partition has continuous and complete metadata physical properties; According to the result of the dynamic partitioning, an adaptive index is established for each partition, and a corresponding index structure is established for each partition according to the index result to monitor the update of metadata; The metadata after dynamic partitioning and indexing is stored in a data storage warehouse through distributed storage technology, and storage nodes and backup mechanisms are configured to perform parallel processing and load balancing of the metadata; Dividing the metadata into a plurality of logically independent and physically continuous dynamic partitions comprises the following steps: Analyze the relevance, access frequency and size factors of metadata, determine the number and size of partitions, and select a partitioning algorithm based on the characteristics of the metadata; the partitioning algorithm includes range-based partitioning, hash-based partitioning, and cluster-based partitioning; During demand analysis, analyze the storage requirements of the current system, including the type, size, growth trend, and access pattern of metadata, and evaluate the usage of existing partitions to identify performance bottlenecks and potential optimization points; According to the results of demand analysis, select the partitioning algorithm that best suits the current metadata characteristics and system requirements. The partitioning algorithm takes into account the access frequency, update frequency, and size distribution of metadata. Divide metadata into multiple logically independent and physically continuous partitions according to the partitioning algorithm; According to the result of the dynamic partitioning, an adaptive index is established for each partition, and a corresponding index structure is established for each partition according to the index result, including the following steps: Analyze the query mode, update frequency and data volume of metadata, determine the type and parameters of the index, and select an index algorithm according to the characteristics of the metadata; the index algorithm includes B-tree index, hash index and bitmap index; Establish a corresponding index structure for each partition based on the index results, monitor metadata updates in real time, and dynamically adjust the index.

2. A data management method for a data storage warehouse according to claim 1, characterized in that: The step of extracting metadata of the original data further includes the following steps: Check the quality of raw data and remove duplicate, invalid or erroneous data; Perform data conversion and standardization according to the data type, format and structure; Extract metadata from data after data transformation and standardization.

3. A data management method for a data storage warehouse according to claim 1, characterized in that: When the corresponding index structure is established for each partition according to the index result, the following steps are also included: By compressing and encoding the index data, the storage space occupied by the index is reduced; By introducing approximate indexing and incremental indexing techniques, only the incremental part of the index is updated when the metadata is updated.

4. A data management method for a data storage warehouse according to claim 1, characterized in that: The step of storing the metadata after dynamic partitioning and indexing in a data storage warehouse through distributed storage technology, configuring storage nodes and a backup mechanism, includes the following steps: Use distributed storage technology to spread metadata across multiple storage nodes; The metadata is compressed and encrypted for storage, and data backup and recovery functions are provided.

5. The data management method for a data storage warehouse according to claim 1, characterized in that: After the parallel processing and load balancing processing of metadata, the following steps are also included: Centrally manage and maintain the metadata stored in the warehouse, and provide a data operation interface for data query, update and deletion, as well as multi-user concurrent access and transaction processing services; Perform metadata version control and permission management, and perform performance monitoring and optimization. Monitor the performance status of the storage warehouse in real time and make adjustments as needed.

6. A data management system for a data storage warehouse, characterized in that: include: A preprocessing module, used to obtain the original data managed by the enterprise and extract metadata of the original data; the metadata includes the type, size, access frequency and relevance information of the original data; A dynamic partitioning module is used to divide the metadata into multiple logically independent and physically continuous dynamic partitions, allocate corresponding storage resources and optimization strategies to each partition, and obtain dynamic partitioning results; wherein, logically independent means that the metadata logical structure in the partition has different logics, and physically continuous means that the metadata in the partition has continuous and complete metadata physical properties; An index module, used to establish an adaptive index for each partition according to the result of the dynamic partitioning, and to establish a corresponding index structure for each partition according to the indexing result, and to monitor the update of metadata; A storage module is used to store the metadata after dynamic partitioning and indexing in a data storage warehouse through distributed storage technology, configure storage nodes and backup mechanisms, and perform parallel processing and load balancing of metadata; Dividing the metadata into a plurality of logically independent and physically continuous dynamic partitions comprises the following steps: Analyze the relevance, access frequency and size factors of metadata, determine the number and size of partitions, and select a partitioning algorithm based on the characteristics of the metadata; the partitioning algorithm includes range-based partitioning, hash-based partitioning, and cluster-based partitioning; During demand analysis, analyze the storage requirements of the current system, including the type, size, growth trend, and access pattern of metadata, and evaluate the usage of existing partitions to identify performance bottlenecks and potential optimization points; According to the results of demand analysis, select the partitioning algorithm that best suits the current metadata characteristics and system requirements. The partitioning algorithm takes into account the access frequency, update frequency, and size distribution of metadata. Divide metadata into multiple logically independent and physically continuous partitions according to the partitioning algorithm; According to the result of the dynamic partitioning, an adaptive index is established for each partition, and a corresponding index structure is established for each partition according to the index result, including the following steps: Analyze the query mode, update frequency and data volume of metadata, determine the type and parameters of the index, and select an index algorithm according to the characteristics of the metadata; the index algorithm includes B-tree index, hash index and bitmap index; Establish a corresponding index structure for each partition based on the index results, monitor metadata updates in real time, and dynamically adjust the index.

7. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor executes the steps of a data management method for a data storage warehouse as described in any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a data management method for a data storage warehouse according to any one of claims 1 to 5 are implemented.

Images