Data storage processing method and device, equipment and medium

Abstract

The embodiment of the invention provides a data storage processing method and device, equipment and a medium. The method comprises the steps of obtaining a target access record of data; the target access record comprises access times corresponding to a plurality of preset time periods before the current time; acquiring an attenuation factor corresponding to the data; the attenuation factor is used for representing a weight cardinal number of the data; determining a target access weight according to an attenuation factor and the target access record; determining a target access frequency according to the target access record and the target access weight; and according to the target access frequency and a preset access frequency threshold, determining a storage processing strategy of the data, thereby ensuring that the hotspot data is subjected to high-performance storage support, and meanwhile, migrating the cold data to a low-frequency resource pool in time. The utilization efficiency of storage resources can be maximized while the data access efficiency is ensured, and the storage cost is remarkably reduced.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a data storage processing method, apparatus, device, and medium. Background Technology

[0002] In data storage systems, data is typically categorized into hot and cold data based on its access frequency. Hot data, due to its high access frequency, needs to be stored in a high-performance resource pool to ensure read / write efficiency, while cold data, due to its low access frequency, is more suitable for storage in a low-frequency resource pool to reduce costs. While this hot / cold tiered storage strategy effectively reduces data storage costs, it faces a critical problem in practical applications: the future access patterns of data are often unpredictable during initial upload, making it impossible to accurately define its hot / cold attribute. To ensure immediate data availability, it's common practice to default to storing all data in the high-performance resource pool. However, when data access frequency is clearly differentiated, this approach leads to a large amount of low-access-frequency data consuming high-performance resources, resulting in wasted storage performance and unnecessary cost increases. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data processing method, apparatus, device and medium that overcomes or at least partially solves the above problems.

[0004] To address the above problems, this invention discloses a data storage processing method, the method comprising:

[0005] The target access record for acquiring data includes the number of times the data was accessed within multiple preset time periods prior to the current time.

[0006] Obtain the attenuation factor corresponding to the data; the attenuation factor is used to represent the weight base of the data;

[0007] The target access weight is determined based on the attenuation factor and the target access record; the target access weight includes: access weights corresponding to multiple preset time periods prior to the current time;

[0008] The target access frequency is determined based on the target access record and the target access weight;

[0009] Based on the target access frequency and the preset access frequency threshold, determine the data storage and processing strategy;

[0010] The data is processed according to the storage processing strategy.

[0011] Optionally, determining the data storage and processing strategy based on the target access frequency and a preset access frequency threshold includes...

[0012] Obtain the storage pool type of the data; the storage pool type includes a first storage pool and a second storage pool.

[0013] Based on the target access frequency, the preset access frequency threshold, and the data storage pool type, a data storage and processing strategy is determined.

[0014] Optionally, determining the data storage and processing strategy based on the target access frequency, a preset access frequency threshold, and the data storage pool type includes:

[0015] If the data storage pool type is a first storage pool and the target access frequency is greater than or equal to the preset access frequency threshold, the data storage processing strategy is determined to be a first storage processing strategy; the first storage processing strategy is to migrate the data from the first storage pool to the second storage pool.

[0016] If the data storage pool type is a first storage pool and the target access frequency is less than the preset access frequency threshold, the data storage processing strategy is determined to be a third storage processing strategy; the third storage processing strategy is to maintain the data storage pool type unchanged.

[0017] Optionally, the method further includes:

[0018] If the data storage pool type is the second storage pool and the target access frequency is greater than or equal to the preset access frequency threshold, the data storage processing strategy is determined to be the third storage processing strategy.

[0019] If the data storage pool type is the second storage pool and the target access frequency is less than the preset access frequency threshold, the data storage processing strategy is determined to be the second storage processing strategy; the second storage processing strategy is to migrate the data from the second storage pool to the first storage pool.

[0020] Optionally, before acquiring the target access record for the data, the method further includes:

[0021] The processing time of the data is obtained; the processing time is used to represent the time when the data was last stored and processed.

[0022] The target access records for acquiring data include:

[0023] When the data processing time is greater than or equal to a preset time threshold, the target access record of the data is obtained.

[0024] Optionally, obtaining the attenuation factor corresponding to the data includes:

[0025] Obtain business information corresponding to the data; the business information includes the storage pool types of multiple data in the business corresponding to the data;

[0026] Based on the business information, obtain the attenuation factor corresponding to the business information;

[0027] The method further comprises:

[0028] Update the business information corresponding to the data according to the data storage and processing strategy.

[0029] Optionally, after processing the data according to the storage processing strategy, the method further includes: updating the processing time of the data according to the current time.

[0030] On the other hand, embodiments of the present invention also disclose a data storage processing apparatus, the apparatus comprising:

[0031] The data acquisition module is used to acquire target access records of data; the target access records include the number of times the data was accessed in multiple preset time periods before the current time;

[0032] A base acquisition module is used to acquire the attenuation factor corresponding to the data; the attenuation factor is used to represent the weight base of the data;

[0033] The weight determination module is used to determine the target access weight based on the attenuation factor and the target access record; the target access weight includes: access weights corresponding to multiple preset time periods before the current time;

[0034] The frequency acquisition module is used to determine the target access frequency based on the target access record and the target access weight;

[0035] The strategy acquisition module is used to determine the data storage and processing strategy based on the target access frequency and the preset access frequency threshold.

[0036] The data processing module is used to process the data according to the storage processing strategy.

[0037] Optionally, the strategy acquisition module includes

[0038] The storage pool type acquisition submodule is used to acquire the storage pool type of the data; the storage pool type includes a first storage pool and a second storage pool.

[0039] The storage strategy determination submodule is used to determine the data storage processing strategy based on the target access frequency, the preset access frequency threshold, and the data storage pool type.

[0040] Optionally, the storage strategy determination submodule includes:

[0041] The first processing subunit is configured to determine the data storage processing strategy as a first storage processing strategy when the data storage pool type is a first storage pool and the target access frequency is greater than or equal to the preset access frequency threshold; the first storage processing strategy is to migrate the data from the first storage pool to the second storage pool.

[0042] The second processing subunit is configured to determine the data storage processing strategy as a third storage processing strategy when the data storage pool type is a first storage pool and the target access frequency is less than the preset access frequency threshold; the third storage processing strategy is to maintain the data storage pool type unchanged.

[0043] Optionally, the device further includes:

[0044] The third processing submodule is used to determine the data storage processing strategy as the third storage processing strategy when the data storage pool type is the second storage pool and the target access frequency is greater than or equal to the preset access frequency threshold.

[0045] The fourth processing submodule is used to determine the data storage processing strategy as the second storage processing strategy when the data storage pool type is the second storage pool and the target access frequency is less than the preset access frequency threshold; the second storage processing strategy is to migrate the data from the second storage pool to the first storage pool.

[0046] Optionally, before acquiring the target access record for the data, the device further includes:

[0047] The processing time acquisition submodule is used to acquire the processing time of the data; the processing time is used to represent the time when the data was last stored and processed.

[0048] The target access records for acquiring data include:

[0049] When the data processing time is greater than or equal to a preset time threshold, the target access record of the data is obtained.

[0050] Optionally, the cardinality acquisition module includes:

[0051] The business information acquisition submodule is used to acquire business information corresponding to the data; the business information includes the storage pool types of multiple data in the business corresponding to the data;

[0052] The business base acquisition submodule is used to obtain the attenuation factor corresponding to the business information based on the business information.

[0053] The method apparatus further includes:

[0054] The business information update submodule is used to update the business information corresponding to the data according to the data storage and processing strategy.

[0055] Optionally, after processing the data according to the storage processing strategy, the apparatus further includes:

[0056] The processing time update submodule is used to update the processing time of the data based on the current time.

[0057] Accordingly, this invention discloses an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various steps of the above-described data storage processing method embodiments.

[0058] Accordingly, embodiments of the present invention disclose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the various steps of the above-described data storage and processing method embodiments.

[0059] The embodiments of this invention offer the following advantages: By dynamically adjusting the storage processing strategy based on the data's access frequency and corresponding attenuation factor, frequently accessed data can be stored in a high-performance storage pool, while infrequently accessed data can be migrated to a lower-performance or lower-cost storage pool. This dynamic adjustment method not only improves the utilization rate of storage resources but also reduces storage costs. By introducing an attenuation factor and a dynamic access weight evaluation mechanism, the problem of wasted storage resources caused by inaccurate initial data classification is effectively solved. The storage hierarchy of data can be dynamically adjusted based on the actual access records and attenuation factor, ensuring that hot data receives high-performance storage support while promptly migrating cold data to a low-frequency resource pool. This maximizes the utilization efficiency of storage resources while ensuring data access efficiency, significantly reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flowchart illustrating the steps of an embodiment of the data storage processing method of the present invention;

[0061] Figure 2 This is a flowchart illustrating the steps of another embodiment of the data storage processing method of the present invention;

[0062] Figure 3 This is a schematic diagram of an intelligent hierarchical system equipped with the data storage processing method of the present invention;

[0063] Figure 4This is a flowchart illustrating another embodiment of data storage processing;

[0064] Figure 5 This is a structural block diagram of an embodiment of a data storage processing device according to the present invention. DETAILED DESCRIPTION

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] One of the core concepts of this invention is to use an attenuation factor to reduce the impact of timing factors on data access records, thereby calculating the access frequency and adjusting the data storage strategy based on the access frequency.

[0067] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a data storage processing method according to the present invention, which may specifically include the following steps:

[0068] Step 101: Obtain the target access records of the data; the target access records include: the number of times the data was accessed in multiple preset time periods prior to the current time.

[0069] First, when performing data storage processing, it is necessary to obtain the accessed records within a preset time interval before the current time. By analyzing these records, strong support can be provided for subsequent data processing methods.

[0070] In the field of storage resource management, technicians need to rationally allocate and manage storage space with limited storage resources. By understanding the frequency of data access, it is possible to analyze which data is frequently accessed, and then migrate this data to high-speed storage media with better cost and performance, while migrating less frequently accessed data to lower-cost, lower-performance storage media.

[0071] In one example, a cache table can be created in the database to record access records for all objects. This cache table has fields: object name, cluster name, bucket name, current storage type, and an additional metadata field. The metadata field records the number of times an object is accessed each day for the past n days, in the form of an array, such as the access count for the past seven days [10, 2, 13, 44, 5, 26, 17]. From left to right, these represent the number of object accesses for the seven days preceding the current day. During queries, the object name, cluster name, bucket name, etc., can be used as the query criteria to retrieve the desired data access records, based on business needs.

[0072] Step 102: Obtain the attenuation factor corresponding to the data; the attenuation factor is used to represent the weight base of the data;

[0073] After obtaining the access records of the data, this embodiment introduces the concept of a decay factor to accurately quantify the access frequency and reflect the current popularity of the data. In practice, it can be found that access records closer to the query date better reflect the popularity of the data. However, existing technologies do not fully consider this time-series factor, leading to inaccurate calculation of access frequency and potentially incorrect classification of the data storage pool. The decay factor, as a weighting base for the data, is used to accurately guide the quantitative analysis of access records. By combining the decay factor, the impact of time series on data popularity assessment can be effectively reduced, thereby more accurately determining the popularity of the data.

[0074] In one example, the attenuation factor can be specified in advance based on experience and business needs, or it can be dynamically adjusted during business operations based on real-time conditions.

[0075] Step 103: Determine the target access weight based on the attenuation factor and the target access record; the target access weight includes: access weights corresponding to multiple preset time periods before the current time;

[0076] By combining the previously determined decay factor, the weight of the corresponding time interval in the access record can be determined, thereby using a weighted calculation method to perform more accurate quantitative analysis of the data. The method for determining the corresponding weight through the decay factor can be a mathematical method such as the exponential decay method, the half-life method, and the linear decay method, and this embodiment does not limit it to this method.

[0077] In one example, the weights could be determined by using the current time of the query as a baseline, and then setting the weight of the corresponding time period forward by a preset time period as the power of the corresponding time period for the decay factor.

[0078] For example: During a data access process, an array is obtained from the access records. This array is [N1, N2, N3, ..., N]. n Where N represents the number of accesses and its subscript represents the nth time interval from the current time, the meaning of this array is the number of accesses for the data in the first n corresponding time intervals starting from the current time of data acquisition; assuming the decay factor obtained at this time is α, then the weight corresponding to this array is [α 1 α 2 α 3 , ..., α n The method for determining the corresponding weight based on the attenuation factor can also be other mathematical methods, and the attenuation factor can also be represented in other forms. This embodiment does not limit this.

[0079] Step 104: Determine the target access frequency based on the target access record and the target access weight;

[0080] After obtaining the corresponding weight, the access frequency of the data can be determined by combining the corresponding calculation method.

[0081] In one example, the calculation can be performed using the following formula:

[0082]

[0083] Where, N avg This represents the target access frequency of the data, where n is the number of access records for the data corresponding to the time interval, [N1, N2, N3, ..., N]. n The number of times the data is accessed corresponds to the time interval, and the coefficient in front of it represents the corresponding weight determined based on the decay factor.

[0084] Step 105: Determine the data storage and processing strategy based on the target access frequency and the preset access frequency threshold;

[0085] Once the target access frequency for the data is determined, the storage and processing strategy for the data can be determined based on the calculated access frequency and the preset threshold determined by the business.

[0086] In one example, the data storage and processing strategy could be a change in the data storage type, a change in the data storage pool, or a change in the data storage rules.

[0087] Step 106: Process the data according to the storage processing strategy.

[0088] This invention dynamically adjusts storage processing strategies based on data access frequency and corresponding attenuation factors. This allows frequently accessed data to be stored in a high-performance storage pool, while infrequently accessed data is migrated to a lower-performance or lower-cost storage pool. This dynamic adjustment not only improves storage resource utilization but also reduces storage costs. By introducing an attenuation factor and dynamic access weight evaluation mechanism, it effectively solves the problem of wasted storage resources caused by inaccurate initial data classification. It can dynamically adjust the data storage hierarchy based on actual access records and attenuation factors, ensuring that hot data receives high-performance storage support while promptly migrating cold data to a low-frequency resource pool. This maximizes storage resource utilization efficiency while maintaining data access efficiency, significantly reducing storage costs.

[0089] Reference Figure 2 The flowchart illustrates another embodiment of the data storage processing method of the present invention, which may specifically include the following steps:

[0090] Step 201: Obtain the processing time of the data; the processing time is used to represent the time when the data was last stored and processed;

[0091] Each time data is processed, the record of the processing time in the data is updated to support the business rules for data processing.

[0092] In one example, recording and updating processing time can be achieved using the `CopySourceIfUnmodifiedSince` time attribute provided by the S3 protocol's general copy interface. Recording and updating data processing time can also be achieved using corresponding methods in other frameworks, adding timestamp fields, etc., and this invention does not limit this approach.

[0093] In one embodiment, step 201 includes the following sub-steps:

[0094] Sub-step S11: When the data processing time is greater than or equal to a preset time threshold, the target access record of the data is obtained.

[0095] When performing data storage processing, most data does not experience sudden surges or drops in access. By setting an adjustment threshold, data with a time greater than or equal to this threshold is processed, thus improving system performance during data storage processing.

[0096] In one example, to prevent large fluctuations in traffic of certain data within a short period of time, traffic monitoring can be performed on the data through methods such as log monitoring. When the traffic variable of the data exceeds a preset variable threshold within a certain time interval, an information summary will be output to the administrator so that the administrator can adjust the business rules for data storage and processing in a timely manner based on the information summary.

[0097] Step 202: When the data processing time is greater than or equal to a preset time threshold, obtain the target access record of the data;

[0098] First, when performing data storage processing, it is necessary to obtain the accessed records within a preset time interval before the current time. By analyzing these records, strong support can be provided for subsequent data processing methods.

[0099] In the field of storage resource management, technicians need to rationally allocate and manage storage space with limited storage resources. By understanding the frequency of data access, it is possible to analyze which data is frequently accessed, and then migrate this data to high-speed storage media with better cost and performance, while migrating less frequently accessed data to lower-cost, lower-performance storage media.

[0100] In one example, a cache table can be created in the database to record access records for all objects. This cache table has fields: object name, cluster name, bucket name, current storage type, and an additional metadata field. The metadata field records the number of times an object is accessed each day for the past n days, in the form of an array, such as the access count for the past seven days [10, 2, 13, 44, 5, 26, 17]. From left to right, these represent the number of object accesses for the seven days preceding the current day. During queries, the object name, cluster name, bucket name, etc., can be used as the query criteria to retrieve the desired data access records, based on business needs.

[0101] Step 203: Obtain the attenuation factor corresponding to the data; the attenuation factor is used to represent the weight base of the data;

[0102] After obtaining the access records of the data, this embodiment introduces the concept of a decay factor to accurately quantify the access frequency and reflect the current popularity of the data. In practice, it can be found that access records closer to the query date better reflect the popularity of the data. However, existing technologies do not fully consider this time-series factor, leading to inaccurate calculation of access frequency and potentially incorrect classification of the data storage pool. The decay factor, as a weighting base for the data, is used to accurately guide the quantitative analysis of access records. By combining the decay factor, the impact of time series on data popularity assessment can be effectively reduced, thereby more accurately determining the popularity of the data.

[0103] In one example, the attenuation factor can be specified in advance based on experience and business needs, or it can be dynamically adjusted during business operations based on real-time conditions.

[0104] In one embodiment, step 203 includes the following sub-steps:

[0105] Sub-step S21: Obtain business information corresponding to the data; the business information includes the storage pool type of multiple data in the business corresponding to the data;

[0106] During data storage, different services will have their own data, and the storage pool type for the corresponding data will also be different depending on the service. The data corresponding to hot services needs to withstand a large number of accesses, so the data of hot services needs to be stored in a higher performance storage pool in order to better support the service needs.

[0107] Generally, high-performance storage pools are primarily used to store data that requires rapid access and processing. This data typically belongs to high-performance, high-frequency business processes. High-performance storage pools can provide higher data read and write speeds, lower latency, and higher IOPS (Input / Output Operations Per Second) to meet the needs of high-performance businesses. Higher-performance storage pools typically employ advanced hardware architectures and optimized software algorithms, such as using solid-state drives as storage media and utilizing caching, parallel processing, and data compression technologies to improve performance. At the software level, high-performance storage pools may implement more complex I / O scheduling algorithms, data prefetching techniques, and write-back strategies to further optimize storage performance.

[0108] Low-performance storage pools are primarily used to store data with low performance requirements. This data typically falls under more cost-sensitive business operations, such as data archiving, long-term backups, and infrequently accessed historical data. Low-performance storage pools can reduce storage costs while ensuring data reliability and availability. They typically use lower-cost storage media, such as traditional hard disk drives (HDDs). While these drives have slower read / write speeds, their relatively low cost makes them suitable for storing infrequently accessed data. At the software level, low-performance storage pools may implement simpler I / O scheduling algorithms and reduce the use of data prefetching and write-back strategies to lower system complexity and cost.

[0109] Sub-step S22: Based on the service information, obtain the attenuation factor corresponding to the service information;

[0110] Different business types have different attenuation factors.

[0111] The method further comprises:

[0112] Update the business information corresponding to the data according to the data storage and processing strategy.

[0113] By tagging the storage type of data in the business, it is easier to analyze the business's popularity and improve work efficiency.

[0114] Step 204: Determine the target access weight based on the attenuation factor and the target access record; the target access weight includes: access weights corresponding to multiple preset time periods before the current time;

[0115] By combining the previously determined decay factor, the weight of the corresponding time interval in the access record can be determined, thereby using a weighted calculation method to perform more accurate quantitative analysis of the data. The method for determining the corresponding weight through the decay factor can be a mathematical method such as the exponential decay method, the half-life method, and the linear decay method, and this embodiment does not limit it to this method.

[0116] In one example, the weights could be determined by using the current time of the query as a baseline, and then setting the weight of the corresponding time period forward by a preset time period as the power of the corresponding time period for the decay factor.

[0117] For example: During a data access process, an array is obtained from the access records. This array is [N1, N2, N3, ..., N]. n Where N represents the number of accesses and its subscript represents the nth time interval from the current time, the meaning of this array is the number of accesses for the data in the first n corresponding time intervals starting from the current time of data acquisition; assuming the decay factor obtained at this time is α, then the weight corresponding to this array is [α 1 α 2 α 3 , ..., α n The method for determining the corresponding weight based on the attenuation factor can also be other mathematical methods, and the attenuation factor can also be represented in other forms. This embodiment does not limit this.

[0118] Step 205: Determine the target access frequency based on the target access record and the target access weight;

[0119] After obtaining the corresponding weight, the access frequency of the data can be determined by combining the corresponding calculation method.

[0120] In one example, the calculation can be performed using the following formula:

[0121]

[0122] Where, N avg This represents the target access frequency of the data, where n is the number of access records for the data corresponding to the time interval, [N1, N2, N3, ..., N]. n The number of times the data is accessed corresponds to the time interval, and the coefficient in front of it represents the corresponding weight determined based on the decay factor.

[0123] Step 206: Determine the data storage and processing strategy based on the target access frequency and the preset access frequency threshold;

[0124] Once the target access frequency for the data is determined, the storage and processing strategy for the data can be determined based on the calculated access frequency and the preset threshold determined by the business.

[0125] In one example, the data storage and processing strategy could be a change in the data storage type, a change in the data storage pool, or a change in the data storage rules.

[0126] In one embodiment, step 206 includes the following sub-steps:

[0127] Sub-step S31: Obtain the storage pool type of the data; the storage pool type includes a first storage pool and a second storage pool.

[0128] The first storage pool is a low-performance storage pool, and the second storage pool is a high-performance storage pool.

[0129] Sub-step S32: Determine the data storage and processing strategy based on the target access frequency, the preset access frequency threshold, and the data storage pool type.

[0130] By comparing the access frequency of the data with the preset access frequency, the real-time popularity of the data can be obtained, and the data storage strategy can be adjusted according to the real-time popularity to achieve intelligent data storage.

[0131] In one embodiment, sub-step S32 includes the following sub-steps:

[0132] Sub-step S321: If the data storage pool type is a first storage pool and the target access frequency is greater than or equal to the preset access frequency threshold, determine the data storage processing strategy as a first storage processing strategy; the first storage processing strategy is to migrate the data from the first storage pool to the second storage pool.

[0133] When data is in a low-performance storage pool but access frequency exceeds a preset frequency threshold, it means that the data has changed from cold data to hot data. Therefore, the data needs to be migrated from the low-performance storage pool to the high-performance storage pool.

[0134] In sub-step S322, if the data storage pool type is the first storage pool and the target access frequency is less than the preset access frequency threshold, the data storage processing strategy is determined to be the third storage processing strategy; the third storage processing strategy is to maintain the data storage pool type unchanged.

[0135] When data is in a low-performance storage pool and access frequency does not exceed a preset frequency threshold, the data's frequency remains unchanged, and no adjustment to the data's storage pool type is made. Storage strategies allow for more intelligent planning of data storage pools, reducing system load.

[0136] In one embodiment, the method further includes the following sub-steps:

[0137] Sub-step S41: If the data storage pool type is the second storage pool and the target access frequency is greater than or equal to the preset access frequency threshold, determine the data storage processing strategy as the third storage processing strategy.

[0138] If the data is in a high-performance storage pool and the access popularity still exceeds the preset popularity threshold, it means that the data popularity remains unchanged, and the data storage pool type will not be adjusted at this time.

[0139] In sub-step S42, if the data storage pool type is the second storage pool and the target access frequency is less than the preset access frequency threshold, the data storage processing strategy is determined to be the second storage processing strategy; the second storage processing strategy is to migrate the data from the second storage pool to the first storage pool.

[0140] When data is in a high-performance storage pool but access frequency does not exceed a preset frequency threshold, it means that the data has changed from hot data to cold data. At this time, the data needs to be migrated from the high-performance storage pool to the low-performance storage pool.

[0141] Step 207: Process the data according to the storage processing strategy.

[0142] In one embodiment, the method further includes the following sub-steps:

[0143] Sub-step S51: Update the processing time of the data according to the current time.

[0144] This invention dynamically adjusts storage processing strategies based on data access frequency and corresponding attenuation factors. This allows frequently accessed data to be stored in a high-performance storage pool, while infrequently accessed data is migrated to a lower-performance or lower-cost storage pool. This dynamic adjustment not only improves storage resource utilization but also reduces storage costs. By introducing an attenuation factor and dynamic access weight evaluation mechanism, it effectively solves the problem of wasted storage resources caused by inaccurate initial data classification. It can dynamically adjust the data storage hierarchy based on actual access records and attenuation factors, ensuring that hot data receives high-performance storage support while promptly migrating cold data to a low-frequency resource pool. This maximizes storage resource utilization efficiency while maintaining data access efficiency, significantly reducing storage costs.

[0145] Reference Figure 3 The diagram illustrates a schematic of an intelligent hierarchical system equipped with the data storage processing method according to an embodiment of the present invention.

[0146] The system consists of three main modules: a data scanning module, a data transfer module, and an analysis and calculation module.

[0147] When the system is working, the cross-cluster object storage gateway first manages the high-performance pool (high-performance storage pool) and the low-frequency pool (low-performance storage pool) in a unified manner; the cross-cluster object storage gateway manages the metadata of all objects in a unified manner through the database, and at the same time creates a cache table in the database to record the information of all data objects;

[0148] When scanning data, the data scanning module first performs rule prefix filtering; that is, it only processes data whose names have the same prefix as the set scanning prefix. This prefix can be set according to business rules, such as setting inclusion rules based on business naming. That is, when the scanning prefix is ​​set to "abc" and "abcd", the system can intelligently filter out the data with the prefix "abcd" and only scan the data with the prefix "abc".

[0149] After completing the rule prefix filtering, the data scanning module begins scanning for objects that meet the requirements and formulates a data object transformation strategy based on the business rules.

[0150] There are two conversion strategies: hot to cold and cold to hot, which correspond to two different queues.

[0151] The data conversion module reads data from the hot-to-cold queue and the cold-to-hot queue, and converts the data type by calling a preset interface. Specifically, it changes the storage type of the data, that is, from the high-performance pool to the low-frequency pool or from the low-frequency pool to the high-performance pool.

[0152] The cross-cluster object storage gateway accepts data transformation requests sent by the data transformation module via the copy interface and completes the transformation of the corresponding data.

[0153] During system operation, the analysis and computing module will periodically analyze the logs of the storage cluster and update the data of the corresponding cache table in the database based on the analysis results.

[0154] refer to Figure 4 The diagram illustrates a flowchart of another embodiment of the data storage processing of the present invention.

[0155] In this embodiment, the data type stored is an object, the storage container is a bucket, and the intelligent hierarchical rules of the bucket correspond to the data storage and processing rules. In object storage services, a bucket is a logical storage space used to store objects. It is equivalent to a folder or directory in a traditional file system, but provides more advanced functionality and more flexible access control.

[0156] First, obtain the intelligent hierarchical rules for the buckets to be processed. The intelligent hierarchical rules may include: configuration rules for the configuration prefix, values ​​for the attenuation factor, and thresholds for preset access frequencies, etc.

[0157] After obtaining the rules, it is determined whether the bucket is currently being processed. If it is, the currently being processed rule is obtained and compared with the rule being processed this time. If the rules do not match, the current processing scan task is terminated, and a new processing round is started using the rule being processed this time. If the rules match, the objects in the current processing queue are directly read and processed according to the processing rules of the corresponding objects. If the bucket is not currently being processed, a new processing round is started using the rule being processed this time.

[0158] Based on the configured prefix rules, the data objects that need to be processed are filtered out, and the access frequency of the data objects to be processed is calculated based on the attenuation factor. Based on the calculated results and the access frequency threshold pre-configured in the rules, the data objects are transformed and judged, and the data objects are written into the corresponding data transformation queue according to the judgment results.

[0159] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0160] Reference Figure 5 The diagram illustrates a structural block diagram of an embodiment of a data storage processing apparatus according to the present invention, which may specifically include the following modules:

[0161] The data acquisition module 301 is used to acquire the target access record of the data; the target access record includes: the number of times the data was accessed in multiple preset time periods before the current time;

[0162] First, when performing data storage processing, it is necessary to obtain the accessed records within a preset time interval before the current time. By analyzing these records, strong support can be provided for subsequent data processing methods.

[0163] In the field of storage resource management, technicians need to rationally allocate and manage storage space with limited storage resources. By understanding the frequency of data access, it is possible to analyze which data is frequently accessed, and then migrate this data to high-speed storage media with better cost and performance, while migrating less frequently accessed data to lower-cost, lower-performance storage media.

[0164] In one example, a cache table can be created in the database to record access records for all objects. This cache table has fields: object name, cluster name, bucket name, current storage type, and an additional metadata field. The metadata field records the number of times an object is accessed each day for the past n days, in the form of an array, such as the access count for the past seven days [10, 2, 13, 44, 5, 26, 17]. From left to right, these represent the number of object accesses for the seven days preceding the current day. During queries, the object name, cluster name, bucket name, etc., can be used as the query criteria to retrieve the desired data access records, based on business needs.

[0165] The base acquisition module 302 is used to acquire the attenuation factor corresponding to the data; the attenuation factor is used to represent the weight base of the data;

[0166] After obtaining the access records of the data, this embodiment introduces the concept of a decay factor to accurately quantify the access frequency and reflect the current popularity of the data. In practice, it can be found that access records closer to the query date better reflect the popularity of the data. However, existing technologies do not fully consider this time-series factor, leading to inaccurate calculation of access frequency and potentially incorrect classification of the data storage pool. The decay factor, as a weighting base for the data, is used to accurately guide the quantitative analysis of access records. By combining the decay factor, the impact of time series on data popularity assessment can be effectively reduced, thereby more accurately determining the popularity of the data.

[0167] In one example, the attenuation factor can be specified in advance based on experience and business needs, or it can be dynamically adjusted during business operations based on real-time conditions.

[0168] The weight determination module 303 is used to determine the target access weight based on the attenuation factor and the target access record; the target access weight includes: access weights corresponding to multiple preset time periods before the current time;

[0169] By combining the previously determined decay factor, the weight of the corresponding time interval in the access record can be determined, thereby using a weighted calculation method to perform more accurate quantitative analysis of the data. The method for determining the corresponding weight through the decay factor can be a mathematical method such as the exponential decay method, the half-life method, and the linear decay method, and this embodiment does not limit it to this method.

[0170] In one example, the weights could be determined by using the current time of the query as a baseline, and then setting the weight of the corresponding time period forward by a preset time period as the power of the corresponding time period for the decay factor.

[0171] For example: During a data access process, an array is obtained from the access records. This array is [N1, N2, N3, ..., N]. n Where N represents the number of accesses and its subscript represents the nth time interval from the current time, the meaning of this array is the number of accesses for the data in the first n corresponding time intervals starting from the current time of data acquisition; assuming the decay factor obtained at this time is α, then the weight corresponding to this array is [α 1 α 2 α 3 , ..., α n The method for determining the corresponding weight based on the attenuation factor can also be other mathematical methods, and the attenuation factor can also be represented in other forms. This embodiment does not limit this.

[0172] The frequency acquisition module 304 is used to determine the target access frequency based on the target access record and the target access weight;

[0173] After obtaining the corresponding weight, the access frequency of the data can be determined by combining the corresponding calculation method.

[0174] In one example, the calculation can be performed using the following formula:

[0175]

[0176] Where, N avg This represents the target access frequency of the data, where n is the number of access records for the data corresponding to the time interval, [N1, N2, N3, ..., N]. n The number of times the data is accessed corresponds to the time interval, and the coefficient in front of it represents the corresponding weight determined based on the decay factor.

[0177] The strategy acquisition module 305 is used to determine the data storage and processing strategy based on the target access frequency and the preset access frequency threshold.

[0178] Once the target access frequency for the data is determined, the storage and processing strategy for the data can be determined based on the calculated access frequency and the preset threshold determined by the business.

[0179] In one example, the data storage and processing strategy could be a change in the data storage type, a change in the data storage pool, or a change in the data storage rules.

[0180] The data processing module 306 is used to process the data according to the storage processing strategy.

[0181] In one embodiment, the policy acquisition module includes

[0182] The storage pool type acquisition submodule is used to acquire the storage pool type of the data; the storage pool type includes a first storage pool and a second storage pool.

[0183] The first storage pool is a low-performance storage pool, and the second storage pool is a high-performance storage pool.

[0184] The storage strategy determination submodule is used to determine the data storage processing strategy based on the target access frequency, the preset access frequency threshold, and the data storage pool type.

[0185] By comparing the access frequency of the data with the preset access frequency, the real-time popularity of the data can be obtained, and the data storage strategy can be adjusted according to the real-time popularity to achieve intelligent data storage.

[0186] In one embodiment, the storage policy determination submodule includes:

[0187] The first processing subunit is configured to determine the data storage processing strategy as a first storage processing strategy when the data storage pool type is a first storage pool and the target access frequency is greater than or equal to the preset access frequency threshold; the first storage processing strategy is to migrate the data from the first storage pool to the second storage pool.

[0188] When data is in a low-performance storage pool but access frequency exceeds a preset frequency threshold, it means that the data has changed from cold data to hot data. Therefore, the data needs to be migrated from the low-performance storage pool to the high-performance storage pool.

[0189] The second processing subunit is configured to determine the data storage processing strategy as a third storage processing strategy when the data storage pool type is a first storage pool and the target access frequency is less than the preset access frequency threshold; the third storage processing strategy is to maintain the data storage pool type unchanged.

[0190] When data is in a low-performance storage pool and access frequency does not exceed a preset frequency threshold, the data's frequency remains unchanged, and no adjustment to the data's storage pool type is made. Storage strategies allow for more intelligent planning of data storage pools, reducing system load.

[0191] In one embodiment, the device further includes:

[0192] The third processing submodule is used to determine the data storage processing strategy as the third storage processing strategy when the data storage pool type is the second storage pool and the target access frequency is greater than or equal to the preset access frequency threshold.

[0193] If the data is in a high-performance storage pool and the access popularity still exceeds the preset popularity threshold, it means that the data popularity remains unchanged, and the data storage pool type will not be adjusted at this time.

[0194] The fourth processing submodule is used to determine the data storage processing strategy as the second storage processing strategy when the data storage pool type is the second storage pool and the target access frequency is less than the preset access frequency threshold; the second storage processing strategy is to migrate the data from the second storage pool to the first storage pool.

[0195] When data is in a high-performance storage pool but access frequency does not exceed a preset frequency threshold, it means that the data has changed from hot data to cold data. At this time, the data needs to be migrated from the high-performance storage pool to the low-performance storage pool.

[0196] In one embodiment, prior to acquiring the target access record for the data, the apparatus further includes:

[0197] The processing time acquisition submodule is used to acquire the processing time of the data; the processing time is used to represent the time when the data was last stored and processed.

[0198] Each time data is processed, the record of the processing time in the data is updated to support the business rules for data processing.

[0199] In one example, recording and updating processing time can be achieved using the `CopySourceIfUnmodifiedSince` time attribute provided by the S3 protocol's general copy interface. Recording and updating data processing time can also be achieved using corresponding methods in other frameworks, adding timestamp fields, etc., and this invention does not limit this approach.

[0200] The target access records for acquiring data include:

[0201] When the data processing time is greater than or equal to a preset time threshold, the target access record of the data is obtained.

[0202] When performing data storage processing, most data does not experience sudden surges or drops in access. By setting an adjustment threshold, data with a time greater than or equal to this threshold is processed, thus improving system performance during data storage processing.

[0203] In one example, to prevent large fluctuations in traffic of certain data within a short period of time, traffic monitoring can be performed on the data through methods such as log monitoring. When the traffic variable of the data exceeds a preset variable threshold within a certain time interval, an information summary will be output to the administrator so that the administrator can adjust the business rules for data storage and processing in a timely manner based on the information summary.

[0204] In one embodiment, the cardinality acquisition module includes:

[0205] The business information acquisition submodule is used to acquire business information corresponding to the data; the business information includes the storage pool types of multiple data in the business corresponding to the data;

[0206] During data storage, different services will have their own data, and the storage pool type for the corresponding data will also be different depending on the service. The data corresponding to hot services needs to withstand a large number of accesses, so the data of hot services needs to be stored in a higher performance storage pool in order to better support the service needs.

[0207] Generally, high-performance storage pools are primarily used to store data that requires rapid access and processing. This data typically belongs to high-performance, high-frequency business processes. High-performance storage pools can provide higher data read and write speeds, lower latency, and higher IOPS (Input / Output Operations Per Second) to meet the needs of high-performance businesses. Higher-performance storage pools typically employ advanced hardware architectures and optimized software algorithms, such as using solid-state drives as storage media and utilizing caching, parallel processing, and data compression technologies to improve performance. At the software level, high-performance storage pools may implement more complex I / O scheduling algorithms, data prefetching techniques, and write-back strategies to further optimize storage performance.

[0208] Low-performance storage pools are primarily used to store data with low performance requirements. This data typically falls under more cost-sensitive business operations, such as data archiving, long-term backups, and infrequently accessed historical data. Low-performance storage pools can reduce storage costs while ensuring data reliability and availability. They typically use lower-cost storage media, such as traditional hard disk drives (HDDs). While these drives have slower read / write speeds, their relatively low cost makes them suitable for storing infrequently accessed data. At the software level, low-performance storage pools may implement simpler I / O scheduling algorithms and reduce the use of data prefetching and write-back strategies to lower system complexity and cost.

[0209] The business base acquisition submodule is used to obtain the attenuation factor corresponding to the business information based on the business information.

[0210] Different business types have different attenuation factors.

[0211] The method apparatus further includes:

[0212] The business information update submodule is used to update the business information corresponding to the data according to the data storage and processing strategy.

[0213] By tagging the storage type of data in the business, it is easier to analyze the business's popularity and improve work efficiency.

[0214] In one embodiment, after processing the data according to the storage processing strategy, the apparatus further includes:

[0215] The processing time update submodule is used to update the processing time of the data based on the current time.

[0216] This invention dynamically adjusts storage processing strategies based on data access frequency and corresponding attenuation factors. This allows frequently accessed data to be stored in a high-performance storage pool, while infrequently accessed data is migrated to a lower-performance or lower-cost storage pool. This dynamic adjustment not only improves storage resource utilization but also reduces storage costs. By introducing an attenuation factor and dynamic access weight evaluation mechanism, it effectively solves the problem of wasted storage resources caused by inaccurate initial data classification. It can dynamically adjust the data storage hierarchy based on actual access records and attenuation factors, ensuring that hot data receives high-performance storage support while promptly migrating cold data to a low-frequency resource pool. This maximizes storage resource utilization efficiency while maintaining data access efficiency, significantly reducing storage costs.

[0217] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0218] This invention also provides an electronic device, comprising:

[0219] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described data storage processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0220] This invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the various processes of the above-described data storage and processing method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0221] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0222] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0223] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0224] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0225] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0226] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0227] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0228] The present invention has provided a detailed description of a data storage processing method, apparatus, device, and medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data storage and processing method, characterized in that: The method comprises: Obtaining a target access record of data; the target access record includes: the number of accesses corresponding to multiple preset time periods before the current time; Acquire an attenuation factor corresponding to the data; the attenuation factor is used to represent the weight base of the data; Determining a target access weight according to the attenuation factor and the target access record; the target access weight includes: access weights corresponding to a plurality of preset time periods before the current time; Determining a target access frequency according to the target access record and the target access weight; Determining a data storage and processing strategy according to the target access frequency and a preset access frequency threshold; The data is processed according to the storage processing strategy.

2. The method according to claim 1, characterized in that Determining a data storage processing strategy according to the target access frequency and a preset access frequency threshold includes: Acquire a storage pool type of the data; the storage pool type includes a first storage pool and a second storage pool; A data storage processing strategy is determined according to the target access frequency, a preset access frequency threshold, and a storage pool type of the data.

3. The method according to claim 2, characterized in that The step of determining a data storage processing strategy according to the target access frequency, a preset access frequency threshold, and a storage pool type of the data includes: When the storage pool type of the data is a first storage pool and the target access frequency is greater than or equal to the preset access frequency threshold, determining that the storage processing strategy of the data is a first storage processing strategy; the first storage processing strategy is to migrate the data from the first storage pool to the second storage pool; When the storage pool type of the data is the first storage pool and the target access frequency is less than the preset access frequency threshold, the storage processing strategy of the data is determined to be the third storage processing strategy; the third storage processing strategy is to maintain the storage pool type of the data unchanged.

4. The method according to claim 3, characterized in that The method further comprises: When the storage pool type of the data is the second storage pool and the target access frequency is greater than or equal to the preset access frequency threshold, determining that the storage processing strategy of the data is the third storage processing strategy; When the storage pool type of the data is the second storage pool and the target access frequency is less than the preset access frequency threshold, the storage processing strategy of the data is determined to be the second storage processing strategy; the second storage processing strategy is to migrate the data from the second storage pool to the first storage pool.

5. The method according to any one of claims 1 to 4, characterized in that Before obtaining the target access record of the data, the method further includes: Acquire the processing time of the data; the processing time is used to indicate the time when the data was last stored and processed; The target access record for acquiring data includes: When the processing time of the data is greater than or equal to a preset time threshold, a target access record of the data is obtained.

6. The method according to claim 5, characterized in that The obtaining of the attenuation factor corresponding to the data comprises: Acquire business information corresponding to the data; the business information includes: storage pool types of multiple data in the business corresponding to the data; According to the service information, acquiring an attenuation factor corresponding to the service information; The method further comprises: According to the storage and processing strategy of the data, the business information corresponding to the data is updated.

7. The method according to claim 6, characterized in that After processing the data according to the storage processing strategy, the method further includes: updating the processing time of the data according to the current time.

8. A data storage and processing device, characterized in that: The device comprises: A data acquisition module, used to acquire target access records of data; the target access records include: the number of accesses corresponding to multiple preset time periods before the current time; A cardinality acquisition module, used to acquire an attenuation factor corresponding to the data; the attenuation factor is used to represent the weight cardinality of the data; A weight determination module, configured to determine a target access weight according to the attenuation factor and the target access record; the target access weight includes: access weights corresponding to a plurality of preset time periods before the current time; A frequency acquisition module, used to determine the target access frequency according to the target access record and the target access weight; A strategy acquisition module, used to determine a data storage processing strategy according to the target access frequency and a preset access frequency threshold; A data processing module is used to process the data according to the storage processing strategy.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of a data storage processing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a data storage and processing method as claimed in any one of claims 1 to 7 are implemented.

Images