Data storage method and device based on time partition

Abstract

The invention discloses a data storage method and device based on time partition, and the method comprises the steps: obtaining the access time, the receiving time and the current medium storage position of each stored data, and carrying out the threshold adjustment of a preset time partition threshold based on the access time and the receiving time; matching the receiving time with a plurality of medium storage positions corresponding to the current time partition threshold, and determining a target medium storage position of each stored data; determining to-be-migrated data of which the target medium storage position is inconsistent with the current medium storage position, wherein the medium read-write performance of the target medium storage position corresponding to the to-be-migrated data is higher than or lower than the medium read performance of the current medium storage position; and according to the target medium storage position corresponding to the to-be-migrated data, storage position migration is performed on the to-be-migrated data to complete data storage, so that the data storage efficiency and the utilization rate of the read-write performance of the storage medium can be improved, and the data access delay can be reduced.

Description

Technical Field

[0001] This application belongs to the field of data storage technology, specifically relating to a data storage method and apparatus based on time partitioning. Background Technology

[0002] With the rapid development of big data and real-time computing technologies, the data generated by various application systems is growing exponentially. In scenarios such as financial transactions, e-commerce platforms, and IoT monitoring, the data is not only massive in scale but also characterized by continuous inflow and high concurrency access. Data storage faces dual challenges in terms of cost and performance.

[0003] In existing technologies, data storage primarily involves manually setting fixed data storage cycle thresholds to classify data as hot or cold, storing hot data on high-performance media and cold data on low-performance media. However, this method relies on manual intervention and fixed storage cycle thresholds, resulting in low data storage efficiency, low utilization of storage media read / write performance, and consequently, poor data access performance. Summary of the Invention

[0004] This application provides a time-partitioned data storage method and apparatus, which solves the problems of low data storage efficiency and low utilization of storage medium read / write performance in the prior art, resulting in poor data access performance. By adjusting the preset time partition threshold using access time and reception time, the target storage location of each stored data and the data to be migrated are determined, and the data to be migrated is migrated to storage media with different read / write performance to complete data storage. This achieves the purpose of data partitioning and migration based on data access needs without relying on manual intervention, improving data storage efficiency and utilization of storage medium read / write performance, and helping to reduce data access latency.

[0005] In a first aspect, embodiments of this application provide a data storage method based on time partitioning, the method comprising: Obtain the access time, reception time, and current media storage location of each stored data. Adjust the preset time partition threshold based on the access time and reception time to obtain the current time partition threshold. The receiving time is matched with multiple media storage locations corresponding to the current time partition threshold, and the target media storage location of each stored data is determined based on the matching results. Identify data to be migrated from multiple stored data where the target medium storage location is inconsistent with the current medium storage location. The read / write performance of the first medium storage location corresponding to the target medium storage location of the data to be migrated is higher or lower than the read performance of the second medium storage location corresponding to the current medium storage location of the data to be migrated. The data to be migrated is moved to a new storage location based on the target media storage location corresponding to the data to be migrated, in order to complete the data storage.

[0006] Furthermore, the number of access times is multiple; The threshold for the preset time partition is adjusted based on the access time and the reception time, including: All stored data are grouped based on the reception time, and the access frequency of each data group in the data grouping result is determined according to multiple access times. Compare the frequency of accessing group data with the frequency of accessing preset hot data, and determine the adjustment direction of the preset time partition threshold based on the comparison results; Determine the time range corresponding to each data group, and adjust the preset time partition threshold according to the adjustment direction and the time range of the group.

[0007] Furthermore, the received time is matched with multiple media storage locations corresponding to the current time partition threshold, and the target media storage location for each stored data is determined based on the matching results, including: Based on the received time and multiple time partitions corresponding to the current time partition threshold, determine the target time partition for each stored data. Determine multiple time slices corresponding to each target time partition, as well as the time range and storage location of each time slice; The receiving time is matched with the fragment time range, and the target time fragment corresponding to each stored data is determined according to the matching result, as well as the target fragment medium storage location corresponding to the target time fragment, thus obtaining the target medium storage location of each stored data.

[0008] Furthermore, the target time partition includes hot data time partitions or cold data time partitions; Identify multiple time slices corresponding to each target time partition, including: Determine the partition separation time value between the preset hot data time partition and the preset cold data time partition based on the current time partitioning threshold, and correct the hot data time partitioning attribute in the preset hot data time partition or the cold data time partitioning attribute in the preset cold data time partition based on the partition separation time value; Determine the partition attribute information corresponding to the target time partition, and determine multiple time shards corresponding to each target time partition based on the partition attribute information and the correction results of hot data time shard attributes or cold data time shard attributes.

[0009] Furthermore, based on the target media storage location corresponding to the data to be migrated, the storage location of the data to be migrated is migrated, including: The data to be migrated is migrated to the corresponding target media storage location to obtain the initial data migration results; Determine the data storage cluster type corresponding to the data to be migrated, and the data storage tagging rules corresponding to the data storage cluster type; The initial data migration results are renamed and associated with the storage cluster according to the data storage tagging rules to obtain the final data migration results.

[0010] Furthermore, the data to be migrated will be migrated to the corresponding target media storage location, including: The data to be migrated is copied in its entirety, and the copy result is then pasted into the corresponding target media storage location to obtain the data to be migrated pasting result; Based on preset data integrity verification rules, the data to be migrated pasted in the target media storage location is verified for data integrity. If the data integrity verification of the data to be migrated pasted is passed, the initial migration of the data to be migrated is determined to be completed.

[0011] Furthermore, after data storage is completed, the method also includes: In response to a data query command, the system parses the data service identifier and multiple data association times in the data query command, and determines the target data storage cluster type corresponding to the data service identifier, as well as the time partition attribute corresponding to each data association time. Based on the data service identifier and the target data storage cluster type, multiple storage clusters in the data storage results are filtered to obtain the target storage cluster corresponding to the data query instruction. Based on the time partition attributes and the associated time of each data, determine multiple target data storage locations in the target storage cluster corresponding to the data query command, integrate the target data in multiple target data storage locations, and provide feedback on the target data integration result.

[0012] Secondly, embodiments of this application provide a time-partitioned data storage device, the device comprising: The partition threshold adjustment module is used to obtain the access time, reception time and current media storage location of each stored data, and adjust the preset time partition threshold based on the access time and reception time to obtain the current time partition threshold; The storage location determination module is used to match the received time with multiple media storage locations corresponding to the current time partition threshold, and determine the target media storage location for each stored data based on the matching results. The migration data determination module is used to determine the data to be migrated from multiple stored data whose target medium storage location is inconsistent with the current medium storage location. The read / write performance of the first medium storage location corresponding to the target medium storage location of the data to be migrated is higher or lower than the read performance of the second medium storage location corresponding to the current medium storage location of the data to be migrated. The data storage module is used to migrate the data to be migrated according to the target media storage location of the data to be migrated, so as to complete the data storage.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] Fifthly, embodiments of this application also provide a computer program product comprising a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the method described in the first aspect.

[0016] In this embodiment, the access time, reception time, and current media storage location of each stored data are obtained. A preset time partition threshold is adjusted based on the access time and reception time to obtain the current time partition threshold. The reception time is matched with multiple media storage locations corresponding to the current time partition threshold, and the target media storage location of each stored data is determined according to the matching result. Data to be migrated is identified among the multiple stored data whose target media storage location is inconsistent with the current media storage location. The first media read / write performance of the target media storage location corresponding to the data to be migrated is higher or lower than the second media read performance of the current media storage location corresponding to the data to be migrated. The data to be migrated is migrated according to the target media storage location corresponding to the data to be migrated to complete the data storage. The time-partition-based data storage method described above solves the problems of low data storage efficiency and low utilization of storage medium read / write performance in existing technologies, which in turn leads to poor data access performance. By adjusting the preset time partition threshold using access time and reception time, the target storage location of each stored data and the data to be migrated are determined. The data to be migrated is then migrated to storage media with different read / write performance to complete data storage. This method can achieve the goal of partitioning and migrating data based on data access needs without relying on manual intervention, thereby improving data storage efficiency and utilization of storage medium read / write performance, and helping to reduce data access latency. Attached Figure Description

[0017] Figure 1 This is a flowchart of a time-partition-based data storage method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the process of determining the target storage location of stored data provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the migration of data storage locations provided in an embodiment of this application; Figure 4 This is a structural block diagram of the single-cluster data storage module provided in the embodiments of this application; Figure 5 This is a structural block diagram of the dual-cluster data storage module provided in the embodiments of this application; Figure 6 This is a structural block diagram of a time-partition-based data storage device provided in an embodiment of this application; Figure 7 This is a structural block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application are described in detail below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] First, this solution can be used in scenarios involving data storage from various fields or systems, particularly for flexibly storing data generated in large-scale data domains to adapt to high-concurrency data read and write operations. By adjusting a preset time partition threshold using access and reception times, the target storage location of each stored data and the data to be migrated are determined. The data to be migrated is then moved to storage media with different read / write performance, completing data storage. This achieves the goal of partitioning and migrating data based on data access needs without relying on manual intervention, improving data storage efficiency and utilization of storage media read / write performance, and reducing data access latency. Based on the above application scenarios, it is understood that the execution entity of each step in this solution can be a computer device. This computer device refers to any electronic device with data computing, processing, and storage capabilities, such as mobile phones, PCs (Personal Computers), tablets, and other terminal devices, or servers, etc. This application embodiment does not limit this.

[0022] The following description, in conjunction with the accompanying drawings, details a time-partition-based data storage method and apparatus provided in this application through specific embodiments and application scenarios.

[0023] Figure 1 This is a flowchart illustrating a time-partition-based data storage method provided in an embodiment of this application. For example... Figure 1 As shown, the specific steps include the following: S101, obtain the access time, reception time and current media storage location of each stored data, and adjust the preset time partition threshold based on the access time and reception time to obtain the current time partition threshold.

[0024] The stored data can be data that a storage module such as a memory or database has received and initially partitioned into hot and cold data according to the reception time. The access time can be the time when an access instruction or request for each stored data item is received. The reception time can be the time when each stored data item arrives at the storage module's external interface. The current media storage location can be the current location of each stored data item within the storage medium. The preset time partitioning threshold can be a pre-set critical data storage duration for partitioning the data received by the storage module into hot and cold data. The preset time partitioning threshold can be an empirical value. The current time partitioning threshold can be a critical data storage duration for partitioning the stored data and future received data into hot and cold data, based on the access characteristics of the stored data in the storage module.

[0025] In one embodiment, the receiving record information, historical access record information, and storage record information of each stored data can be read from the storage module based on the identifier of each stored data to obtain the access time, receiving time, and current media storage location of each stored data. The access frequency of each stored data is determined according to the time interval between its corresponding access time and receiving time, and the variation pattern of its access frequency with storage duration is determined. Based on this variation pattern, multiple stored data with access frequencies lower than or close to zero are identified. The average storage duration of these multiple stored data is calculated based on the time when the receiving time and access frequency are close to zero. This average storage duration is used as the time partition threshold that matches the access characteristics of the stored data. The preset time partition threshold is then adjusted to this threshold to obtain the current time partition threshold. For example, if the preset time partition threshold is one year, but the actual storage duration of the stored data in the storage module reaches six months and is almost no longer accessed, the preset time partition threshold can be adjusted from one year to six months as the current time partition threshold.

[0026] In one embodiment, the number of access times is multiple; adjusting the preset time partition threshold based on the access time and the reception time includes: grouping all stored data based on the reception time, and determining the group data access frequency corresponding to each data group in the data grouping result according to the multiple access times; comparing the size relationship between the group data access frequency and the preset hot data access frequency, and determining the adjustment direction of the preset time partition threshold according to the size relationship comparison result; determining the group time range corresponding to each data group, and adjusting the preset time partition threshold according to the adjustment direction and the group time range.

[0027] The group data access frequency can be the average number of times all stored data in each group is accessed within a preset time period. The preset hot data access frequency can be the minimum frequency at which stored data is accessed within a preset time period if it is determined to be a hot data type. The adjustment direction can be to expand or shrink the preset time partition threshold. The group time range can be the time interval between the earliest and latest reception times of multiple stored data in each data group.

[0028] In one embodiment, the earliest and latest reception times of all stored data can be identified, the overall time interval of the stored data can be calculated, and the group time interval of each data group can be determined based on the overall time interval and the number of preset groups. All stored data can then be grouped according to the group time interval and the reception time of each stored data. The group data access frequency corresponding to each data group in the data grouping result is determined based on multiple access times and group time intervals of each stored data. The group data access frequency of each data group is compared with the preset hot data access frequency. Data groups with a group data access frequency lower than the preset hot data access frequency are designated as cold data groups. The shortest storage duration in the cold data group is determined based on the reception time of each stored data, and it is compared whether the shortest storage duration is less than a preset time partition threshold. If it is less, the adjustment direction of the preset time partition threshold is determined to be decreasing; otherwise, the adjustment direction is increasing. The earliest and latest reception times of multiple stored data in the cold data group corresponding to the shortest storage duration are identified, the group time range of that group is calculated, and the group time range is used as the adjustment step size. The preset time partition threshold is adjusted according to the adjustment direction and adjustment step size.

[0029] This solution improves the efficiency of adjusting the preset time partition threshold by grouping all stored data and determining the adjustment direction and range of the preset time partition threshold based on the relationship between the access frequency of the group data and the access frequency of the preset hot data.

[0030] S102, match the received time with multiple media storage locations corresponding to the current time partition threshold, and determine the target media storage location for each stored data based on the matching results.

[0031] The media storage location can be the location obtained by dividing the hardware media used to store data into regions. Media storage locations can include data storage locations obtained from high-performance storage media and data storage locations obtained from low-performance storage media. High-performance storage media can be media with high read / write performance or high speed, such as SSDs. Low-performance storage media can be media with low read / write performance or slow speed, such as OSS or tape libraries. High-performance and low-performance storage media have different costs. The target media storage location can be the location within the storage media corresponding to the access characteristics of each stored data. For example, if a stored data has a high access frequency and is a hot data type, then its corresponding target media storage location is a location within a high-performance storage medium.

[0032] In one embodiment, the current time partition threshold can correspond to the storage location of the high-performance storage medium and the storage location of the low-performance storage medium. The storage duration of each stored data can be determined based on the reception time. The storage duration of each stored data is compared with the current time partition threshold. The stored data with a storage duration less than the current time partition threshold is matched with the storage location of the high-performance storage medium, and the stored data with a storage duration greater than the current time partition threshold is matched with the storage location of the low-performance storage medium, thereby obtaining the target medium storage location of each stored data.

[0033] S103, identify data to be migrated from multiple stored data whose target medium storage location is inconsistent with the current medium storage location, wherein the first medium read / write performance of the target medium storage location corresponding to the data to be migrated is higher or lower than the second medium read performance of the current medium storage location corresponding to the data to be migrated.

[0034] The data to be migrated can be existing data whose storage location on the target medium differs from its current storage location. The first media read / write performance can be the read / write speed of the target medium. The second media read performance can be the read / write speed of the current medium.

[0035] In one embodiment, a consistency check can be performed between the target media storage location and the current media storage location in each stored data. Based on the check result, the stored data whose target media storage location and current media storage location are inconsistent among the multiple stored data can be determined as data to be migrated.

[0036] S104, migrate the storage location of the data to be migrated according to the target media storage location corresponding to the data to be migrated, so as to complete the data storage.

[0037] In one embodiment, the data to be migrated can be read from the current media storage location and sent to the target media storage location to realize the migration of the storage location of the data to be migrated, thereby completing data storage based on data access characteristics.

[0038] In one embodiment, after data storage is completed, the method further includes: responding to a data query instruction, parsing the data service identifier and multiple data association times in the data query instruction, and determining the target data storage cluster type corresponding to the data service identifier, as well as the time partition attribute corresponding to each data association time; performing cluster filtering on multiple storage clusters in the data storage result according to the data service identifier and the target data storage cluster type to obtain the target storage cluster corresponding to the data query instruction; determining multiple target data storage locations in the target storage cluster corresponding to the data query instruction based on the time partition attribute and each data association time, integrating the target data in the multiple target data storage locations, and feeding back the target data integration result.

[0039] The data query command can be an instruction used to read target business data within a target time range. Clients issue data query commands through a unified interface to retrieve data without needing to know the data storage location. The system can automatically route query requests, reading hot data directly from the corresponding high-performance media in local storage, and asynchronously loading and caching cold data from the corresponding low-performance media in the archive repository. The data business identifier can be a unique marker used to distinguish the business type to which the data belongs. The data association time can be a storage time range limited to the queried data. The target data storage cluster type can be a single data storage cluster or a dual data storage cluster. The target data storage cluster type corresponds to the data business type; that is, different business types of data are stored in different types of data storage clusters. Simple business types are stored in a single data storage cluster, while complex business types are stored in a dual data storage cluster. The complexity of the business type is related to the number of business processing steps; the more business processing steps, the more complex the corresponding business type. Cluster types include cold data clusters and hot data clusters. Time partitions can include hot data partitions and cold data partitions, and time partitions are obtained by dividing the current time partition threshold. Different time partitions can correspond to different storage media, meaning that different time partitions correspond to different media storage locations. Time partition attributes can be used to distinguish the hot and cold types of data corresponding to each time partition. Target data can be all data that matches the data business identifier in the data query command and the time associated with multiple data sets.

[0040] In one embodiment, after data storage is completed, a data query instruction can be received and responded to. Based on a preset business identifier format and a preset time format, the data business identifier and multiple data-related times in the data query instruction are identified. The business complexity is determined based on the data business identifier, and the target data storage cluster type corresponding to the data business identifier is determined based on the correspondence between business complexity and data storage cluster type. The time partition and its attributes corresponding to each data-related time are determined based on the storage time range corresponding to each time partition. Multiple data storage clusters of the same type as the target data storage cluster are determined based on the cluster type of each data storage cluster in the storage unit, and it is identified whether a data business identifier exists in each data storage cluster. The data storage cluster with the data business identifier is selected as the target storage cluster corresponding to the data query instruction. The target data attribute corresponding to the data query instruction is determined based on the time partition attribute. The target data attribute can be a cold data attribute or a hot data attribute. Since the target storage cluster includes both a hot data storage location list and a cold data storage location list, data storage locations in the target storage cluster that match the target data attribute can be used as multiple target data storage locations. The target data stored in each target data storage location is read, and the target data is merged and fed back.

[0041] This solution determines the target data storage cluster type and time partition attribute by parsing the data service identifier and multiple data association times in the data query command. Based on the target data storage cluster type and time partition attribute, it determines multiple target data storage locations corresponding to the data query command and integrates the target data in multiple target data storage locations for feedback. This achieves the goal of automatic data query routing and query feedback without prior acquisition of data storage locations, improving the feedback efficiency of data query and reducing data access latency.

[0042] The technical solution provided in this application involves obtaining the access time, reception time, and current media storage location of each stored data; adjusting a preset time partition threshold based on the access time and reception time to obtain the current time partition threshold; matching the reception time with multiple media storage locations corresponding to the current time partition threshold; determining the target media storage location for each stored data based on the matching result; identifying data to be migrated from among the multiple stored data whose target media storage location is inconsistent with the current media storage location; wherein the first media read / write performance of the target media storage location corresponding to the data to be migrated is higher or lower than the second media read performance of the current media storage location corresponding to the data to be migrated; and migrating the storage location of the data to be migrated according to the target media storage location corresponding to the data to be migrated to complete the data storage. The time-partition-based data storage method described above solves the problems of low data storage efficiency and low utilization of storage medium read / write performance in existing technologies, which in turn leads to poor data access performance. By adjusting the preset time partition threshold using access time and reception time, the target storage location of each stored data and the data to be migrated are determined. The data to be migrated is then migrated to storage media with different read / write performance to complete data storage. This method can achieve the goal of partitioning and migrating data based on data access needs without relying on manual intervention, thereby improving data storage efficiency and utilization of storage medium read / write performance, and helping to reduce data access latency.

[0043] Figure 2 This is a flowchart illustrating the process of determining the target medium storage location for stored data, as provided in an embodiment of this application. Figure 2 As shown, the specific steps include the following: S201, determine the target time partition for each stored data based on the received time and multiple time partitions corresponding to the current time partition threshold.

[0044] The time partition can be a range of storage durations obtained by dividing the data storage duration according to the current time partition threshold. Time partitions can include hot data time partitions or cold data time partitions. The target time partition can be a time partition whose storage duration matches the reception time of the stored data.

[0045] In one embodiment, the storage duration of each stored data can be determined based on the reception time, and the storage duration of each stored data can be compared with the storage duration range of each time partition corresponding to the current time partition threshold to determine the target time partition to which each stored data belongs.

[0046] S202, determine multiple time slices corresponding to each target time partition, as well as the slice time range and slice media storage location corresponding to each time slice.

[0047] Here, time sharding can be a storage duration range obtained by further dividing time partitions. The storage location of the sharding medium can be the location in the storage medium to which each time shard is bound, that is, the final storage location of the data corresponding to that time shard. The sharding time range can be the data storage duration range corresponding to the time shard.

[0048] In one embodiment, the number of shards corresponding to each time partition is determined based on the association between preset time partitions and time shards. Each time partition is then split into multiple non-overlapping time shards with equal coverage time lengths. These shards can be numbered according to the splitting order or the order of the corresponding time from oldest to newest. The time range corresponding to each time shard is determined based on the time length covered by each shard. Based on the correspondence between each time partition and the media storage location, and the shard number, a shard media storage location is allocated to each time shard. The shard number is then bound to the shard media storage location for subsequent storage of data corresponding to the storage duration of the stored or to-be-stored data in the corresponding shard media storage location.

[0049] In one embodiment, the target time partition includes a hot data time partition or a cold data time partition; determining multiple time slices corresponding to each target time partition includes: determining a partition separation time value between a preset hot data time partition and a preset cold data time partition based on a current time slice threshold; correcting the hot data time slice attribute in the preset hot data time partition or correcting the cold data time slice attribute in the preset cold data time partition based on the partition separation time value; determining partition attribute information corresponding to the target time partition; and determining multiple time slices corresponding to each target time partition based on the partition attribute information and the hot data time slice attribute correction result or the cold data time slice attribute correction result.

[0050] In this system, a hot data time partition can be a partition corresponding to the storage duration of hot data type data. A cold data time partition can be a partition corresponding to the storage duration of cold data type data. Hot data type data represents data that is accessed frequently and / or has been stored for a short time. Cold data type data represents data that is accessed infrequently and / or has been stored for a long time. The partition separator time value can be the boundary storage time value between hot data time partitions and cold data time partitions. The partition separator time value can be equal to the current time sharding threshold, or it can be increased forward or backward by a preset ratio based on the current time sharding threshold to flexibly expand or decrease the length of the hot data time partition. The hot data time sharding attribute can be that the data type corresponding to this time shard is hot data. The cold data time sharding attribute can be that the data type corresponding to this time shard is cold data. Partition attribute information can be the data type information corresponding to a specific time partition.

[0051] In one embodiment, the current time sharding threshold can be used as the partition separation time value between the preset hot data time partition and the preset cold data time partition. The temporal correlation between the sharding time range corresponding to each time shard and the partition separation time value is compared. Based on the temporal correlation, the target sharding attribute of each time shard is determined. The hot data time sharding attribute in the preset hot data time partition or the cold data time sharding attribute in the preset cold data time partition is corrected according to the target sharding attribute. The preset hot data time partition and the preset cold data time partition can be obtained by dividing the time partition based on the preset time partitioning threshold. The partitioning attribute information corresponding to the target time partition is determined, and the corrected hot data time sharding attribute or the corrected cold data time sharding attribute is matched with the partitioning attribute information corresponding to the target time partition to obtain multiple time shards corresponding to each target time partition.

[0052] This solution corrects the hot data time shard attributes in a preset hot data time partition or the cold data time shard attributes in a preset cold data time partition by partitioning and separating time values. Based on the partition attribute information and the time shard attribute correction results, multiple time shards corresponding to each target time partition are determined. This achieves the purpose of further splitting time partitions based on attribute information, improves the accuracy of time partition splitting, and is conducive to the orderly storage of data.

[0053] S203, match the received time with the fragment time range, and determine the target time fragment corresponding to each stored data in the multiple time fragments according to the matching result, as well as the target fragment medium storage location corresponding to the target time fragment, to obtain the target medium storage location of each stored data.

[0054] In one embodiment, the storage duration corresponding to the receiving time can be compared with the storage duration range corresponding to the fragmented time range to determine the target time fragment to which each stored data belongs among multiple time fragments. The target media storage location of each stored data is then determined based on the target fragment media storage location corresponding to the target time fragment.

[0055] The technical solution provided in this application, by determining the target time partition of each stored data, multiple time shards corresponding to each target time partition, and the shard time range and shard media storage location corresponding to each time shard, determines the target time shard corresponding to each stored data among multiple time shards, and the target shard media storage location corresponding to the target time shard, and obtains the target media storage location of each stored data, can achieve the purpose of data sharding and media-based storage, and improve the utilization rate of storage media read and write performance.

[0056] Figure 3 This is a flowchart illustrating the migration of data storage locations provided in an embodiment of this application. For example... Figure 3 As shown, the specific steps include the following: S301, migrate the data to be migrated to the corresponding target media storage location to obtain the initial data migration result.

[0057] The initial data migration result can be the data storage result obtained after copying the data to be migrated to its corresponding target media storage location.

[0058] In one embodiment, the data to be migrated can be copied to the target media storage location, and the data corresponding to the data to be migrated in the initial media storage location can be deleted to obtain the initial data migration result.

[0059] In one embodiment, migrating data to be migrated to the corresponding target media storage location includes: making a full copy of the data to be migrated and pasting the full copy result into the corresponding target media storage location to obtain a data pasting result; performing data integrity verification on the data pasting result in the target media storage location based on preset data integrity verification rules; and determining that the initial migration of the data to be migrated is completed if the data integrity verification of the data pasting result passes.

[0060] Among them, the preset data integrity verification rules can be regulations used to verify the integrity of the data to be migrated. For example, calculating the CRC checksum of the data before and after migration.

[0061] In one embodiment, the data to be migrated can be fully copied, and the copy result can be pasted into the corresponding target media storage location to obtain the data to be migrated pasted result. Based on a preset data integrity verification rule, the checksum of the pasted data in the target media storage location is calculated to see if it matches the checksum of the data to be migrated in the initial media storage location. If the checksums match, it can be determined that the data to be migrated has been completely pasted into its corresponding target media storage location. At this point, the corresponding data in the initial media storage location can be deleted, completing the initial migration of the data to be migrated.

[0062] This solution verifies the integrity of the pasted data to determine whether the initial migration of the data to be migrated has been completed, thus avoiding data migration omissions and ensuring the integrity and success rate of the data migration.

[0063] S302, determine the data storage cluster type corresponding to the data to be migrated, and the data storage tagging rules corresponding to the data storage cluster type.

[0064] The data storage tagging rules can be data naming conventions used to associate data, corresponding storage clusters, and media storage locations. Data storage cluster types can include single-cluster and dual-cluster types. A single cluster may contain only one complete cluster, where all data is managed uniformly within the cluster, sharing the cluster's computing, network, and storage resources. Physical isolation of data is achieved through time-partitioned table sharding. In a single cluster, hot and cold data belong to the same cluster list, and a set of hot and cold data is named by a single cluster list. A dual-cluster may contain two independent clusters: a master cluster and a slave cluster. Each independent cluster has its own independent computing, network, and storage resources, and physical or logical isolation of data can be achieved through the two independent clusters. In a dual-cluster, the master cluster is used only for storing hot data and is bound to high-performance storage media such as SSDs, while the slave cluster is used only for storing cold data and is bound to low-cost storage media such as OSS or tape. Hot and cold data belong to tables in different clusters, and a set of hot and cold data is named by combining the two cluster lists.

[0065] In one embodiment, the data storage cluster type corresponding to the data to be migrated can be determined based on the business type corresponding to the data to be migrated and the pre-defined correspondence between business types and data storage cluster types, and the data storage tagging rule corresponding to the data storage cluster type can be determined based on the pre-defined correspondence between data storage cluster types and data storage tagging rules.

[0066] S303, rename the initial data migration results according to the data storage tagging rules to obtain the final data migration results.

[0067] Storage cluster association renaming can be a naming operation that associates the initial data migration results with the storage cluster name. The storage cluster name can represent the cluster type corresponding to the storage cluster and the hot / cold attribute of the stored data.

[0068] In one embodiment, the storage cluster name can be determined based on the data storage cluster type corresponding to the initial data migration result. The initial data migration result is then renamed according to the data storage tagging rules and the storage cluster name to obtain the final data migration result.

[0069] Figure 4 This is a structural block diagram of a single-cluster data storage module provided in an embodiment of this application. For example... Figure 4 As shown, the single-cluster data storage module includes a service layer, a database layer, and a storage layer. The service layer includes metadata, configuration information, and a task management module. Metadata describes the status of hot and cold data, using binary strings to represent them. For example, 1 represents hot data, and 0 represents cold data. If multiple stored data items include five hot data items, the current database status can be represented as 1111100000... with all 1s at the beginning, and each bit representing a data shard. Metadata also records the time range and storage location of time-based shards, which is the core basis for the system to identify hot and cold data. Configuration information stores the storage rules for data hot / cold grading and media-based storage in this solution. It includes data table names, hot data storage cycles, cold data storage cycles, and hot / cold data migration execution times, serving as the basis for the system to perform data grading, migration, and routing. The task management module is used to configure bidirectional migration tasks for hot and cold storage, enabling bidirectional migration of stored data. This includes migrating all data in a hot data time shard to the corresponding low-performance medium for cold data, and migrating all data in a cold data time shard to the corresponding high-performance medium for hot data. After the data migration is complete, the migration information is synchronized to the metadata. The database layer includes a single-cluster data list, which contains hot data time partitions consisting of five time shards (P1-P5) and cold data time partitions consisting of multiple time shards (P6-Pn). The single-cluster list uses list names to isolate data, but they essentially belong to the same storage logic and the same storage system. The storage layer includes high-performance media for storing hot data and low-performance media for storing cold data. High-performance media corresponds to the hot data time partitions (P1-P5), meaning data with a storage duration belonging to the hot data time partitions is stored in high-performance media. Low-performance media corresponds to the cold data time partitions (P6-Pn), meaning data with a storage duration belonging to the cold data time partitions is stored in low-performance media.

[0070] Figure 5 This is a structural block diagram of the dual-cluster data storage module provided in an embodiment of this application. For example... Figure 5As shown, the dual-cluster data storage module includes a service layer, a database layer, and a storage layer. The service layer includes metadata, configuration information, a task management module, and a routing module. Metadata describes the status of hot and cold data, recording the time range and storage location of time shards, and is the core basis for the system to identify hot and cold data. Configuration information stores the storage rules for data hot / cold classification and media-based storage in this solution. The task management module configures bidirectional migration tasks for hot and cold storage, enabling bidirectional migration of stored data, including migrating the entire data in a hot data time shard to the low-performance medium corresponding to cold data, and migrating the entire data in a cold data time shard to the high-performance medium corresponding to hot data. After the data migration is complete, the migration information is synchronized to the metadata. The routing module configures query routing rules based on the hot / cold type of the target data in the query request, dynamically routing data to different databases for hot / cold data queries based on the data cluster and corresponding data storage location matched by the query conditions, and merging the results for query feedback. The database layer comprises two clusters: a master cluster and a slave cluster. The master cluster consists of five time shards (P1-P5) forming a hot data time partition, while the slave cluster consists of multiple time shards (P6-Pn) forming a cold data time partition. Each cluster has independent computing, network, and storage resources. The master cluster is coupled with high-performance storage media to ensure low-latency access to hot data, while the slave clusters are coupled with low-cost storage media to reduce the cost of cold data storage. The storage layer includes both high-performance and low-performance media.

[0071] The technical solution provided in this application provides an initial data migration result by migrating the data to be migrated to the corresponding target media storage location. The initial data migration result is then renamed according to the data storage tagging rules corresponding to the data to be migrated to obtain the final data migration result. This achieves the purpose of storing data in clusters and associating the stored data with the storage cluster, improving the flexibility of data storage and reducing subsequent data access latency.

[0072] Figure 6 This is a structural block diagram of a time-partitioned data storage device provided in an embodiment of this application. Figure 6 As shown, it specifically includes the following: The partition threshold adjustment module 601 is used to obtain the access time, reception time and current media storage location of each stored data, and adjust the preset time partition threshold based on the access time and reception time to obtain the current time partition threshold. The storage location determination module 602 is used to match the received time with multiple media storage locations corresponding to the current time partition threshold, and determine the target media storage location of each stored data based on the matching result. The migration data determination module 603 is used to determine the data to be migrated among multiple stored data whose target medium storage location is inconsistent with the current medium storage location. The first medium read / write performance of the target medium storage location corresponding to the data to be migrated is higher or lower than the second medium read performance of the current medium storage location corresponding to the data to be migrated. The data storage module 604 is used to migrate the storage location of the data to be migrated according to the target medium storage location corresponding to the data to be migrated, so as to complete the data storage.

[0073] Furthermore, the number of access times is multiple; The partition threshold adjustment module 601 is specifically used for: All stored data are grouped based on the reception time, and the access frequency of each data group in the data grouping result is determined according to multiple access times. Compare the frequency of accessing group data with the frequency of accessing preset hot data, and determine the adjustment direction of the preset time partition threshold based on the comparison results; Determine the time range corresponding to each data group, and adjust the preset time partition threshold according to the adjustment direction and the time range of the group.

[0074] Furthermore, the storage location determination module 602 is specifically used for: Based on the received time and multiple time partitions corresponding to the current time partition threshold, determine the target time partition for each stored data. Determine multiple time slices corresponding to each target time partition, as well as the time range and storage location of each time slice; The receiving time is matched with the fragment time range, and the target time fragment corresponding to each stored data is determined according to the matching result, as well as the target fragment medium storage location corresponding to the target time fragment, thus obtaining the target medium storage location of each stored data.

[0075] Furthermore, the target time partition includes hot data time partitions or cold data time partitions; Storage location determination module 602 is specifically used for: Determine the partition separation time value between the preset hot data time partition and the preset cold data time partition based on the current time partitioning threshold, and correct the hot data time partitioning attribute in the preset hot data time partition or the cold data time partitioning attribute in the preset cold data time partition based on the partition separation time value; Determine the partition attribute information corresponding to the target time partition, and determine multiple time shards corresponding to each target time partition based on the partition attribute information and the correction results of hot data time shard attributes or cold data time shard attributes.

[0076] Furthermore, the data storage module 604 is specifically used for: The data to be migrated is migrated to the corresponding target media storage location to obtain the initial data migration results; Determine the data storage cluster type corresponding to the data to be migrated, and the data storage tagging rules corresponding to the data storage cluster type; The initial data migration results are renamed and associated with the storage cluster according to the data storage tagging rules to obtain the final data migration results.

[0077] Furthermore, the data storage module 604 is specifically used for: The data to be migrated is copied in its entirety, and the copy result is then pasted into the corresponding target media storage location to obtain the data to be migrated pasting result; Based on preset data integrity verification rules, the data to be migrated pasted in the target media storage location is verified for data integrity. If the data integrity verification of the data to be migrated pasted is passed, the initial migration of the data to be migrated is determined to be completed.

[0078] Furthermore, the device also includes: The query parsing module is used to respond to data query commands, parse the data service identifier and multiple data association times in the data query command, and determine the target data storage cluster type corresponding to the data service identifier, as well as the time partition attribute corresponding to each data association time. The storage cluster determination module is used to filter multiple storage clusters in the data storage results based on the data service identifier and the target data storage cluster type to obtain the target storage cluster corresponding to the data query instruction. The query feedback module is used to determine multiple target data storage locations in the target storage cluster corresponding to the data query command based on time partition attributes and the associated time of each data, integrate the target data in multiple target data storage locations, and provide feedback on the integrated target data results.

[0079] The technical solution provided in this application includes a partition threshold adjustment module, which is used to obtain the access time, reception time, and current media storage location of each stored data, and adjust the preset time partition threshold based on the access time and reception time to obtain the current time partition threshold; a storage location determination module, which is used to match the reception time with multiple media storage locations corresponding to the current time partition threshold, and determine the target media storage location of each stored data according to the matching result; a migration data determination module, which is used to determine the data to be migrated among multiple stored data whose target media storage location is inconsistent with the current media storage location, wherein the first media read / write performance of the target media storage location corresponding to the data to be migrated is higher or lower than the second media read performance of the current media storage location corresponding to the data to be migrated; and a data storage module, which is used to migrate the storage location of the data to be migrated according to the target media storage location corresponding to the data to be migrated, so as to complete the data storage. The aforementioned time-partition-based data storage device solves the problems of low data storage efficiency and low utilization of storage medium read / write performance in existing technologies, which in turn lead to poor data access performance. By adjusting the preset time partition threshold using access time and reception time, the target storage location of each stored data and the data to be migrated are determined. The data to be migrated is then migrated to storage media with different read / write performance to complete data storage. This achieves the goal of partitioning and migrating data based on data access needs without relying on manual intervention, improving data storage efficiency and utilization of storage medium read / write performance, and helping to reduce data access latency.

[0080] The time-partition-based data storage device in this application embodiment can be configured in a device, or in a component, integrated circuit, or chip within a terminal. This device can be configured in a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0081] The time-partition-based data storage device in this application embodiment can be an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0082] The data storage device based on time partitioning provided in this application can implement the various processes implemented in the above-described method embodiments. To avoid repetition, it will not be described again here.

[0083] like Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701, a memory 702, and a program or instructions stored in the memory 702 and executable on the processor 701. When the program or instructions are executed by the processor 701, they implement the various processes of the above-described time-partition-based data storage method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0084] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0085] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described time-partition-based data storage method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0086] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0087] This application also provides a program product including program code. When the program product is run on a computer device, the program code causes the computer device to perform the steps of the methods described above according to various exemplary embodiments of this application. For example, the computer device can execute a time-partition-based data storage method described in an embodiment of this application. The program product can be implemented using any combination of one or more readable media.

[0088] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element. Furthermore, it should be noted that the scope of the methods and systems in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0090] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0091] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A data storage method based on time partitioning, characterized in that, The method includes: Obtain the access time, reception time, and current media storage location of each stored data; adjust the preset time partition threshold based on the access time and reception time to obtain the current time partition threshold. The receiving time is matched with multiple media storage locations corresponding to the current time partition threshold, and the target media storage location of each of the stored data is determined based on the matching results. Among the multiple stored data, data to be migrated whose target media storage location is inconsistent with the current media storage location is identified. The first media read / write performance of the target media storage location corresponding to the data to be migrated is higher or lower than the second media read performance of the current media storage location corresponding to the data to be migrated. The data to be migrated is migrated to a new storage location based on the target medium storage location corresponding to the data to be migrated, thereby completing the data storage.

2. The data storage method based on time partitioning according to claim 1, characterized in that, The number of access times is multiple; The step of adjusting the preset time partition threshold based on the access time and the reception time includes: Based on the receiving time, all the stored data are grouped into data groups, and the group data access frequency corresponding to each data group in the data grouping result is determined according to multiple access times; Compare the group data access frequency with the preset hot data access frequency, and determine the adjustment direction of the preset time partition threshold based on the comparison result; Determine the time range of each data group, and adjust the preset time partition threshold according to the adjustment direction and the time range of the data group.

3. The data storage method based on time partitioning according to claim 1, characterized in that, The step of matching the receiving time with multiple media storage locations corresponding to the current time partition threshold, and determining the target media storage location for each of the stored data based on the matching results, includes: Based on the received time and the multiple time partitions corresponding to the current time partition threshold, the target time partition for each of the stored data is determined; Determine multiple time slices corresponding to each of the target time partitions, as well as the slice time range and slice medium storage location corresponding to each of the time slices; The receiving time is matched with the segmented time range, and the target time segment corresponding to each of the stored data is determined according to the matching result, as well as the target segment medium storage location corresponding to the target time segment, to obtain the target medium storage location of each of the stored data.

4. The data storage method based on time partitioning according to claim 3, characterized in that, The target time partition includes a hot data time partition or a cold data time partition; The determination of multiple time slices corresponding to each of the target time partitions includes: The partition separation time value between the preset hot data time partition and the preset cold data time partition is determined based on the current time partitioning threshold. The hot data time partitioning attribute in the preset hot data time partition or the cold data time partitioning attribute in the preset cold data time partition is corrected based on the partition separation time value. Determine the partition attribute information corresponding to the target time partition, and determine multiple time slices corresponding to each target time partition based on the partition attribute information and the hot data time slice attribute correction result or the cold data time slice attribute correction result.

5. The data storage method based on time partitioning according to claim 1, characterized in that, The step of migrating the storage location of the data to be migrated according to the target media storage location corresponding to the data to be migrated includes: The data to be migrated is migrated to the corresponding target media storage location to obtain the initial data migration result; Determine the data storage cluster type corresponding to the data to be migrated, and the data storage tagging rules corresponding to the data storage cluster type; The initial data migration results are renamed and associated with the storage cluster according to the data storage tagging rules to obtain the final data migration results.

6. The data storage method based on time partitioning according to claim 5, characterized in that, The step of migrating the data to be migrated to the corresponding target media storage location includes: The data to be migrated is copied in its entirety, and the copy result is pasted into the corresponding target media storage location to obtain the data to be migrated pasting result; Based on preset data integrity verification rules, the data to be migrated pasted in the target media storage location is verified for data integrity. If the data integrity verification of the data to be migrated pasted is passed, the initial migration of the data to be migrated is determined to be completed.

7. The data storage method based on time partitioning according to claim 1, characterized in that, After data storage is completed, the method further includes: In response to a data query command, the data service identifier and multiple data association times in the data query command are parsed, and the target data storage cluster type corresponding to the data service identifier and the time partition attribute corresponding to each of the data association times are determined. Based on the data service identifier and the target data storage cluster type, multiple storage clusters in the data storage results are filtered to obtain the target storage cluster corresponding to the data query instruction; Based on the time partition attributes and the associated time of each data, determine multiple target data storage locations in the target storage cluster corresponding to the data query instruction, integrate the target data in the multiple target data storage locations, and provide feedback on the target data integration result.

8. A time-partitioned data storage device, characterized in that, The device includes: The partition threshold adjustment module is used to obtain the access time, reception time and current media storage location of each stored data, and adjust the preset time partition threshold based on the access time and reception time to obtain the current time partition threshold. The storage location determination module is used to match the receiving time with multiple media storage locations corresponding to the current time partition threshold, and determine the target media storage location for each of the stored data based on the matching results. The migration data determination module is used to determine the data to be migrated among the plurality of stored data whose target medium storage location is inconsistent with the current medium storage location, wherein the first medium read / write performance of the target medium storage location corresponding to the data to be migrated is higher or lower than the second medium read performance of the current medium storage location corresponding to the data to be migrated. The data storage module is used to migrate the data to be migrated according to the target medium storage location corresponding to the data to be migrated, so as to complete the data storage.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and running on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the time-partition-based data storage method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the time-partition-based data storage method as described in any one of claims 1-7.

Images