Data fragment storage index optimization method and system

Abstract

The invention discloses a data fragment storage index optimization method and system, and relates to the technical field of data fragment storage, and the method comprises the steps: determining an observation time period according to a data access mode and short-term fluctuation characteristics; monitoring data performance, and performing log data analysis; and performing data resource adjustment, and adjusting task allocation. According to the method, data loading is efficiently managed and adjusted, the user experience and reliability of a system are improved, the importance of a dynamic data management method based on the observation time period is highlighted, hot spot fragments can be recognized in time when the load is too high through setting of a dynamic threshold value, and therefore effective measures are taken, and the workload of a user is reduced. According to the method and the device, the waste of system resources is reduced, the use efficiency of the resources is improved, data operation can be completed under the condition that a main thread is not interfered, the data transmission efficiency and the storage utilization rate are improved, and the overall processing capacity and fluency of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data sharding storage, and particularly to a method and system for optimizing data sharding storage indexes. Background Art

[0002] In the context of the rapid development of information technology, data sharding storage technology has been widely applied in fields such as big data processing, cloud computing, and distributed systems. Traditional data storage methods are difficult to efficiently handle read and write requests for massive data. Therefore, data sharding came into being. Data sharding divides large-scale data into multiple segments. Through distributed storage and parallel processing technologies, it not only improves the data access speed but also enhances the scalability and fault tolerance of the system. With technological progress, most data management systems have begun to implement dynamic adjustment of sharding to adapt to different application scenarios and access patterns. However, with the diversification of data access patterns and the continuous changes in user requirements, how to optimize the index structure of data segments, improve performance, and reduce resource consumption has become an urgent problem to be solved.

[0003] Although existing sharding storage technologies have improved in data access efficiency, there are still many deficiencies, especially in the adaptability to data access patterns and the intelligence of resource management. Most existing methods rely on fixed time periods for data monitoring. This static time framework cannot respond in a timely manner to changes in data access patterns in the short term, resulting in the inability to effectively predict and adjust resources at critical moments. For example, during peak periods, some data segments may encounter a large number of read requests. However, due to lack of flexibility, the system fails to reallocate resources in a timely manner to accommodate this increased load, leading to an increase in response time and possibly serious consequences such as system downtime. Existing methods have deficiencies in performance monitoring and analysis. Many data systems use simple logging, which often fails to uncover deep-seated performance trends and potential bottlenecks. For example, in scenarios where data query and update operations are frequent, real-time performance metrics cannot be obtained in real time, resulting in a lag in decision-making. This lag not only affects the dynamic adjustment of data resources but also makes it difficult for technicians to grasp the system operation status, affecting the stability and reliability of the system. Most existing task allocation and scheduling algorithms are based on static rules and fixed logic and cannot adapt to sudden changes in data access volume in the short term. For example, when some data segments are at a high access frequency, existing scheduling strategies fail to dynamically allocate other idle computing resources to these segments. This unreasonable resource allocation leads to waste of memory and computing power and fails to efficiently utilize the performance of the entire system. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed.

[0005] Therefore, the technical problem to be solved by the present invention is that the existing sharded storage technology has insufficient monitoring of data access patterns, lacks effective log data analysis, is not suitable for data resource adjustment with short-term fluctuations, and how to achieve efficient utilization and performance optimization of data storage through dynamic adjustment and intelligent scheduling.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: A method for optimizing data sharded storage index, including determining an observation time period according to the characteristics of data access patterns and short-term fluctuations; monitoring data performance and performing log data analysis; performing data resource adjustment and adjusting task allocation.

[0007] As a preferred solution of the data sharded storage index optimization method of the present invention, wherein: the determining of the observation time period includes triggering a data migration operation when the load of a hot shard exceeds a set threshold.

[0008] When the load difference between the hot shard and the cold shard reaches a certain ratio, trigger a data migration operation.

[0009] Each shard node collects key performance indicator data, including the number of query requests, query response time, CPU usage rate, memory occupancy rate, number of disk I / O operations, and waiting time of disk I / O.

[0010] Removing outliers and error data includes temporary network failures, system failures, and sudden factors caused by system fluctuations, and removing them according to network log reports, design thresholds, and average value methods.

[0011] As a preferred solution of the data sharded storage index optimization method of the present invention, wherein: the determining of the observation time period further includes sorting and normalizing the values, determining the time t, for short-term fluctuations, the time period value is relatively short, and by analyzing the performance data of the current time period value, capturing sudden high-load situations.

[0012] Continuously collecting performance data includes indicators such as the number of query requests, response time, and resource utilization rate, analyzing the distribution of data in different time periods, determining whether there is periodicity or trend, when the data access pattern is unstable, determining whether there is large randomness or the periodic change is not obvious, and whether it is necessary to continue to maintain a relatively long observation time period.

[0013] When it is determined that data migration or resource adjustment operations need to be performed, avoid the peak data access period and select a relatively short time period for centralized migration and adjustment operations.

[0014] Based on the system recovery time, confirm that before the business becomes busy again, the system has enough time to complete data migration, index reconstruction, and resource rebalancing operations and restore to the normal performance level.

[0015] As a preferred solution of the data sharding storage index optimization method described in the present invention, among them: after determining a time period for the monitored data performance, calculate the weighted moving average E of the performance index data t , which is expressed as:

[0016] E t =α×x t +(1 - α)×E t-1

[0017] Among them, x t is the actual performance index value of time, α is the smoothing coefficient. By adjusting the value of α, control the influence degree of historical data and current data on the calculation of the base value. The larger α is, the more sensitive to short-term fluctuations. E t-1 represents the smoothed value of the previous moment and is used as the historical data for calculating the current smoothed value.

[0018] Calculate the data standard deviation SD, which is expressed as:

[0019]

[0020] Among them, n represents the total number of data, i = 1 represents the starting index of summation, x i represents the i-th data point, and E is the expected value of the sample.

[0021] According to business requirements and the requirements for system stability, set the multiple factor k, and calculate the dynamic threshold DT, which is expressed as:

[0022] DT = E + k×SD

[0023] When the actual performance index value exceeds the dynamic threshold, it means that the shard is in a hot state. When the actual performance index value is lower than the dynamic threshold, it means that the shard is in a cold state.

[0024] As a preferred solution of the data sharding storage index optimization method described in the present invention, among them: the log data analysis includes determining the access frequency of each data block. For hot shards, sort the data blocks from low to high according to the access frequency, and take the data blocks with higher access frequency as the objects to be migrated preferentially. Use metadata information to construct a data lineage graph, starting from the data generation source, trace the flow process of data in the entire system, including the links of data conversion, aggregation, and storage, determine the association relationship between each data block, and confirm the correct association during or after the migration according to the relevance.

[0025] As a preferred solution of the data sharding storage index optimization method described in the present invention, wherein: the data resource adjustment includes dividing the data blocks to be migrated into different task units, each task unit including a fixed size or a fixed number of data records, selecting a data compression algorithm according to the characteristics of the energy data, using a lossless compression algorithm for the text format energy data, and when the data distribution of the numerical data is regular, using a compression algorithm specifically for numerical data to obtain a compression ratio.

[0026] As a preferred solution of the data sharding storage index optimization method described in the present invention, wherein: the adjustment of task allocation includes, based on a dynamic task allocation strategy, adjusting the task allocation in real time according to the execution progress of the threads and the resource utilization situation, monitoring in real time the current load situation of the cold spot shards, including performance metrics such as CPU usage rate, memory occupancy rate, and disk I / O load, confirming that the target cold spot shard has resources to receive the migrated data, and during the data migration process, using an asynchronous I / O library or framework for asynchronous I / O operations.

[0027] Another object of the present invention is to provide a data sharding storage index optimization system, which can solve the problems of insufficient dynamic response ability and untimely recognition of system performance changes in the current data performance monitoring technology by monitoring data performance and analyzing log data.

[0028] As a preferred solution of the data sharding storage index optimization system described in the present invention, wherein:

[0029] It includes a time period determination module, a monitoring and analysis module, and an adjustment module.

[0030] The time period determination module is used to determine the observation time period according to the data access pattern and the characteristics of short-term fluctuations; the monitoring and analysis module is used to monitor data performance and analyze log data; the adjustment module is used to perform data resource adjustment and adjust task allocation.

[0031] A computer device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the data sharding storage index optimization method.

[0032] A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, it implements the steps of the data sharding storage index optimization method.

[0033] Advantages of the present invention: The data sharding storage index optimization method provided by the present invention determines the observation time period according to the data access pattern and the characteristics of short-term fluctuations. By efficiently managing and adjusting data loading, it improves the user experience and reliability of the system, highlighting the importance of the dynamic data management method based on the observation time period. It monitors data performance, analyzes log data, and through the setting of dynamic thresholds, can identify hot shards in a timely manner when the load is too high, thereby taking effective measures, which helps to reduce the waste of system resources, improve the utilization efficiency of resources, adjust data resources, and adjust task allocation. It can complete data operations without interfering with the main thread, not only improving the efficiency of data transmission and storage utilization, but also enhancing the overall processing capacity and fluency of the system. The present invention achieves better results in terms of the real-time performance monitoring of the system, the intelligence of data resource management, and the resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0035] Figure 1 It is the overall flowchart of a data sharding storage index optimization method provided by the first embodiment of the present invention.

[0036] Figure 2 It is the overall flowchart of a data sharding storage index optimization system provided by the third embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the following will make a detailed description of the specific embodiments of the present invention in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0038] Embodiment 1, referring to Figure 1 , which is an embodiment of the present invention, provides a data sharding storage index optimization method, including:

[0039] S1: Determine the observation time period according to the data access pattern and the characteristics of short-term fluctuations.

[0040] Furthermore, it is determined that the observation time period includes triggering a data migration operation when the load of the hot shard exceeds a set threshold (e.g., the query response time exceeds 1 second continuously for 5 minutes and the CPU usage rate remains higher than 80%).

[0041] When the load difference between the hot shard and the cold shard reaches a certain ratio (e.g., the number of query requests of the hot shard is more than 10 times that of the cold shard), a data migration operation is triggered.

[0042] Each shard node collects key performance indicator data, including the number of query requests, query response time, CPU usage rate, memory occupancy rate, number of disk I / O operations, and disk I / O waiting time.

[0043] Removing outliers and error data, including temporary network failures, system failures, and sudden factors caused by system fluctuations, is performed according to network log reports, design thresholds, and average value methods.

[0044] It should be noted that determining the observation time period also includes sorting and normalizing values, determining time t. For short-term fluctuations, the time period is relatively short. By analyzing the performance data of the current time period value, sudden high-load situations are captured.

[0045] Continuously collecting performance data, including indicators such as the number of query requests, response time, and resource utilization rate, analyzing the distribution of data in different time periods, determining whether there is periodicity or trend. When the data access pattern is unstable, determining whether there is large randomness or the periodic change is not obvious, and whether it is necessary to continue to maintain a relatively long observation time period.

[0046] When it is determined that data migration or resource adjustment operations are required, avoid the peak data access period and select a relatively short time period (such as from 2 am to 5 am) for centralized migration and adjustment operations.

[0047] Based on the system recovery time, confirm that the system has enough time to complete data migration, index reconstruction, and resource rebalancing operations and restore to the normal performance level before the business becomes busy again.

[0048] It should also be noted that by analyzing the data access pattern and short-term fluctuation characteristics to determine the observation time period, refined monitoring of the system load is achieved. Especially when the load of the hot shard exceeds the set threshold, a data migration operation is triggered, which can effectively handle sudden high-load situations. When the load difference between the hot shard and the cold shard reaches a certain ratio, a data migration operation is triggered based on this ratio, which plays an important role in solving the performance bottleneck problem caused by uneven load. A reasonable observation time period can not only capture short-term load fluctuations but also help accurately identify the system performance trend, so as to take timely measures to avoid waste or collapse of system resources; the process of sorting and normalizing numerical values and removing outliers ensures the accuracy of the monitored data, enabling accurate judgment of the system performance status and requirements during daily operation. The detailed analysis of system fluctuations helps identify potential risks and bottlenecks, thereby reducing service interruptions caused by improper data migration. The entire data flow process is controlled through a systematic method, thereby optimizing the system's response speed and effectiveness in a complex environment.

[0049] S2: Monitor data performance and perform log data analysis.

[0050] Furthermore, monitoring data performance includes calculating the weighted moving average E of performance metric data after determining the time period t , expressed as:

[0051] E t = α × x t +(1 - α) × E t-1

[0052] where x t is the actual performance metric value at a certain time, α is the smoothing coefficient. By adjusting the value of α, the influence degree of historical data and current data on the calculation of the base value is controlled. The larger α is, the more sensitive it is to short-term fluctuations. E t-1 represents the smoothed value at the previous moment and is used as the historical data for calculating the current smoothed value.

[0053] Calculate the data standard deviation SD, expressed as:

[0054]

[0055] where n represents the total number of data, i = 1 represents the starting index of summation, x i represents the i-th data point, and E is the expected value of the sample.

[0056] According to business requirements and the requirements for system stability, set the multiple factor k and calculate the dynamic threshold DT, expressed as:

[0057] DT = E + k × SD

[0058] When the actual performance metric value exceeds the dynamic threshold, it indicates that the shard is in a hot state. When the actual performance metric value is lower than the dynamic threshold, it indicates that the shard is in a cold state.

[0059] It should be noted that log data analysis includes determining the access frequency of each data block. For hot shards, the data blocks are sorted from low to high according to the access frequency, and the data blocks with higher access frequency are taken as the objects to be migrated preferentially. A data lineage graph is constructed using metadata information. Starting from the data generation source, the data flow process in the entire system is traced, including data transformation, aggregation, and storage links, to determine the association relationships between data blocks. Based on the associations, each data block is determined to ensure that the correct associations are maintained during or after the migration process.

[0060] It should also be noted that by monitoring data performance, especially through the calculation of weighted moving average and standard deviation, a comprehensive analysis of data access behavior can be achieved. When the system load approaches the critical value, the setting of the dynamic threshold can quickly respond to changes in hot shards, ensuring that the system can perform effective resource adjustment during high-load periods. By analyzing log data to determine the access frequency of each data block, it helps to formulate a more efficient data migration strategy. Constructing a data lineage graph using metadata information not only helps to understand the data flow process but also ensures that the association relationships between data blocks are not damaged during the data migration process, effectively improving the security and stability of data management. The ability of automated monitoring and analysis enhances the self-healing ability of the system, enabling it to quickly make resource adjustment decisions based on real-time data feedback. The system can not only operate stably during peak periods but also reasonably allocate resources under normal conditions, improving resource utilization efficiency and reducing operating costs.

[0061] S3: Perform data resource adjustment and adjust task allocation.

[0062] Furthermore, performing data resource adjustment includes dividing the data blocks to be migrated into different task units. Each task unit includes a fixed size or a fixed number of data records (such as 1000 energy consumption data records as a task unit). Select a data compression algorithm according to the characteristics of the energy data. For text-format energy data (such as energy reports, log files, etc.), use lossless compression algorithms such as variants of LZ77, LZ78, Deflate algorithms (such as gzip, zlib, etc.). For numerical data (such as energy measurement data), when the data distribution is regular, use compression algorithms specifically for numerical data, such as Delta coding, Run-Length Encoding, etc. to obtain the compression ratio.

[0063] It should be noted that adjusting task allocation includes real-time adjusting task allocation based on a dynamic task allocation strategy according to the execution progress and resource utilization of threads, and real-time monitoring the current load of cold spot shards, including performance metrics such as CPU usage, memory occupancy, and disk I / O load, to confirm that the target cold spot shard has resources to receive the migrated data. During the data migration process, an asynchronous I / O library or framework is used for asynchronous I / O operations.

[0064] It should also be noted that by dividing the data blocks to be migrated into different task units and selecting an appropriate compression algorithm according to the characteristics of energy data, the efficiency of the data migration process can be effectively improved. This mechanism ensures the best processing method for different types of data, so that when performing large-scale data migration, not only the transmission speed is increased, but also the occupation of network bandwidth is reduced, and the use of system resources is optimized; based on the dynamic task allocation strategy, real-time monitoring of the load of cold spot shards ensures that the target cold spot shard has sufficient resources to receive the migrated data. This dynamic adaptability effectively reduces the migration delay caused by insufficient resources. At the same time, combined with the implementation of asynchronous I / O operations, the flexibility and response speed of data migration are improved, ensuring that the overall performance of the system is not affected during the data migration process; this series of adjustment measures integrates the data mobility, optimized allocation of resources, and efficient operation of the system, laying a foundation for the stability and efficiency of the entire system, improving the system's ability to handle high-concurrency requests, ensuring stable operation of the service during peak periods, and enhancing the user experience and system reliability.

[0065] Embodiment 2 is an embodiment of the present invention, which provides a method for optimizing data sharding storage indexing. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0066] First, the experiment was conducted on a database system with 10 shard nodes, where each shard node had similar hardware resources (such as CPU, memory, and storage). During the observation period, the data performance metrics of the experimental group and the control group were collected in real-time through log data. The experimental steps were as follows: Determine the observation time period: Before the experiment, determine the metric thresholds. For example, when the query response time of any node exceeded 1 second for 5 consecutive minutes and the CPU utilization rate remained higher than 80%, the system would trigger a data migration operation; when the ratio of the number of query requests for hot shards to the number of cold shards reached 10 times, the data migration would also be triggered. The system continuously monitored and collected data, removing outliers caused by temporary network failures and system fluctuations to ensure the accuracy of the data. The observation time period was set from 2 am to 5 am to avoid data peaks and capture sudden load situations. During the observation time period, the shard nodes respectively collected the following key performance metric data: the number of query requests, query response time, CPU utilization rate, memory occupancy rate, number of disk I / O operations, and disk I / O waiting time. By analyzing the performance data of different time periods, the system would confirm whether there was periodicity or trend, and select to maintain or adjust the observation time period according to the stability of the data access pattern; Monitor data performance and log data analysis: During the observation period, the system calculated the weighted moving average in real-time and analyzed the performance metrics of each node. At the same time, by analyzing the log data, the access frequency of each data block was determined to identify hot shards. The formed data lineage graph could track the flow process of data in the entire system to ensure that the association relationship of each data block remained unchanged during the migration process. The setting of dynamic thresholds could ensure timely response and effective resource adjustment during peak loads, thus preventing system crashes caused by high loads; Perform data resource adjustment and task allocation: After monitoring and analysis, the system divided the data blocks to be migrated into task units of a fixed size (such as each task unit containing 1000 records), and selected an appropriate compression algorithm according to the data characteristics. Among them, the lossless compression algorithm (such as gzip) was applied to text-format data, and a special encoding algorithm (such as run-length encoding) was applied to numerical data. During the data migration process, the system based on the dynamic task allocation strategy, monitored the cold shards in real-time to ensure that the cold shards had sufficient resources to receive the migrated data. At the same time, asynchronous I / O operations were used to ensure the flexibility and efficiency of the data migration process and ensure that the overall performance was not affected. From the experimental results, it can be seen that the present invention is creative and novel in processing large-scale data, improves the stability and response speed of the system, effectively reduces the occupancy of network bandwidth, and improves the utilization efficiency of resources and the speed of data transmission.

[0067] Example 3, referring to Figure 2 , which is an embodiment of the present invention, provides a data sharding storage index optimization system, including a time period determination module, a monitoring and analysis module, and a system recovery module.

[0068] Among them, the time period determination module is used to determine the observation time period according to the data access pattern and the characteristics of short-term fluctuations; the monitoring and analysis module is used to monitor the data performance and perform log data analysis; the adjustment module is used to perform data resource adjustment and adjust task allocation.

[0069] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, etc., which can store program codes.

[0070] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0071] More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts (electronic devices) having one or more wirings, portable computer disk cartridges (magnetic devices), random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fiber devices, and portable compact disc read-only memories (CDROMs). Additionally, the computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then stored in a computer memory.

[0072] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc. It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

[0073] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A data shard storage index optimization method, characterized in that: include: Determine the observation period based on the data access pattern and the characteristics of short-term fluctuations; Monitor data performance and perform log data analysis; Make data resource adjustments and adjust task allocation.

2. The data shard storage index optimization method according to claim 1, characterized in that: Determining the observation time period includes triggering a data migration operation when the load of the hotspot shard exceeds a set threshold; When the load difference between the hot spot shard and the cold spot shard reaches a certain ratio, the data migration operation is triggered; Each shard node collects key performance indicator data, including the number of query requests, query response time, CPU usage, memory usage, number of disk I / O operations, and disk I / O waiting time; Removal of outliers and erroneous data includes temporary network failures, system failures, and emergencies caused by system fluctuations, which are removed based on network log reports, design thresholds, and average methods.

3. The data shard storage index optimization method according to claim 2, characterized in that: Determining the observation time period also includes sorting out the normalized values ​​and determining the time t. For short-term fluctuations, the time period is relatively short, and sudden high load conditions are captured by analyzing the performance data of the current time period. Continuously collect performance data including the number of query requests, response time, and resource utilization indicators, analyze the distribution of data in different time periods, determine whether there is periodicity or trend, and when the data access pattern is unstable, determine whether there is a large randomness or the periodic changes are not obvious, and whether it is necessary to continue to maintain a longer observation period; When it is determined that data migration or resource adjustment operations are required, avoid the peak period of data access and select a relatively short period of time for centralized migration and adjustment operations; Based on the system recovery time, confirm that the system has enough time to complete data migration, index reconstruction, and resource rebalancing operations and return to normal performance levels before business becomes busy again.

4. The data shard storage index optimization method according to claim 3, characterized in that: The monitoring data performance includes calculating the weighted moving average E of the performance index data after a certain period of time. t , expressed as: E t =α×x t +(1-a)×E t-1 Among them, x t is the actual performance index value of time, α is the smoothing coefficient. By adjusting the value of α, the influence of historical data and current data on the basic value calculation is controlled. The larger the α, the more sensitive it is to short-term fluctuations. t-1 Represents the smoothed value at the previous moment, which is used to calculate the historical data of the current smoothed value; Calculate the data standard deviation SD, expressed as: Where n represents the total number of data, i=1 represents the starting index of the summation, and x i represents the i-th data point, and E is the expected value of the sample; According to business needs and requirements for system stability, the multiplication factor k is set and the dynamic threshold DT is calculated, which is expressed as: DT=E+k×SD When the actual performance index value exceeds the dynamic threshold, it indicates that the shard is in a hot spot state. When the actual performance index value is lower than the dynamic threshold, it indicates that the shard is in a cold spot state.

5. The data shard storage index optimization method according to claim 4, characterized in that: The log data analysis includes determining the access frequency of each data block, sorting the data blocks from low to high according to the access frequency for hot shards, and making the data blocks with higher access frequency the objects of priority migration, and constructing a data lineage relationship diagram using metadata information, starting from the source of data generation, tracking the flow of data in the entire system, including data conversion, aggregation, and storage, determining the association between each data block, determining each data block according to the association, and confirming that the correct association is maintained during or after the migration.

6. The data shard storage index optimization method according to claim 5, characterized in that: The data resource adjustment includes dividing the data blocks to be migrated into different task units, each task unit includes a fixed size or a fixed number of data records, and selecting a data compression algorithm according to the characteristics of the energy data. For energy data in text format, a lossless compression algorithm is used. For numerical data, when the data distribution is regular, a compression algorithm specifically for numerical data is used to obtain a compression ratio.

7. The data shard storage index optimization method according to claim 6, characterized in that: The adjustment of task allocation includes adjusting task allocation in real time based on the dynamic task allocation strategy and the execution progress and resource utilization of the thread, monitoring the current load of the cold spot shard in real time, including the performance indicators of CPU usage, memory occupancy, and disk I / O load, confirming that the target cold spot shard has resources to receive the migrated data, and using an asynchronous I / O library or framework to perform asynchronous I / O operations during the data migration process.

8. A system using the data shard storage index optimization method according to any one of claims 1 to 7, characterized in that: It includes time period determination module, monitoring and analysis module, and adjustment module; The time period determination module is used to determine the observation time period according to the data access pattern and the characteristics of short-term fluctuations; The monitoring and analysis module is used to monitor data performance and perform log data analysis; The adjustment module is used to adjust data resources and adjust task allocation.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the data shard storage index optimization method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data shard storage index optimization method described in any one of claims 1 to 7 are implemented.

Images