Data management method and device, computer program product and storage medium

Abstract

The invention discloses a data management method and device, a computer program product and a storage medium, belongs to the field of distributed storage systems, is used for fragmenting data from two dimensions of time and space and then storing the fragmented data into the distributed storage system, and solves the problems of high network overhead and high access delay when accessing the distributed storage system. Determining a plurality of target time segments and a plurality of target space regions according to the segment characteristics of the to-be-stored data in the time dimension and the region characteristics of the to-be-stored data in the space dimension; according to the method, data to be stored is subjected to data fragmentation according to a target time section and a target space area, then the data fragments are completely stored in corresponding nodes, and data with correlation between time and space dimensions are intensively stored in the nodes in a data fragmentation mode; therefore, when the data in the distributed storage system is accessed, the number of nodes needing to be accessed is theoretically reduced, so that the network overhead and the access delay are reduced.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage systems, and in particular, to a data management method, device, computer program product, and storage medium. Background Art

[0002] Since a distributed storage system includes many nodes for storing data, when storing data in the distributed storage system, if closely related data can be stored centrally in the same node, the network overhead and access latency will be reduced. However, in the related art, there is a lack of a mature data management method, resulting in data being possibly stored dispersedly in each node of the distributed storage system, thereby increasing the network overhead and access latency.

[0003] Therefore, how to provide a solution to the above technical problems is an issue that those skilled in the art need to solve currently. Summary of the Invention

[0004] The object of the present invention is to provide a data management method, device, computer program product, and storage medium. In the present invention, multiple target time intervals can be determined according to the interval characteristics of the data to be stored in the time dimension; multiple target spatial regions can be determined according to the regional characteristics of the data to be stored in the spatial dimension; then, the data to be stored is sliced according to the target time intervals and the target spatial regions respectively, so that the data in the data slices has relevance in both the time and spatial dimensions, and then the data slices are stored completely in the corresponding nodes. Since the data with "relevance in both the time and spatial dimensions" is stored centrally in the nodes in the form of data slices, when accessing the data in the distributed storage system, theoretically, the number of nodes to be accessed is reduced, thereby reducing the network overhead and access latency.

[0005] To solve the above technical problems, the present invention provides a data management method applied to a distributed storage system, including: Determining multiple target time intervals according to the interval characteristics of the data to be stored in the time dimension; Determining multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension; Slicing the data to be stored successively through the target time intervals and the target spatial regions to obtain a number of data slices, where a single data slice includes data of a single target time interval and a single target spatial region; For any data slice, storing the data slice in a node of the distributed storage system.

[0006] On the other hand, determining multiple target time intervals according to the interval characteristics of the data to be stored in the time dimension includes: Regarding each second preset time interval within a preset first preset time interval as a target time interval, where the union of the second preset time intervals is the first preset time interval, and there is no intersection among the second preset time intervals. On the other hand, determining multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension includes: Obtaining the spatial information of each sub-data in the data to be stored; According to a preset regional division strategy, dividing the first spatial region covered by the spatial information into several second spatial regions, and taking each second spatial region as a target spatial region.

[0007] On the other hand, according to a preset regional division strategy, dividing the first spatial region covered by the spatial information into several second spatial regions, and taking each second spatial region as a target spatial region includes: Taking the preset urban area granularity as a unit, dividing the first spatial region covered by the spatial information into several second spatial regions, and taking each second spatial region as a target spatial region; Among them, the preset urban area granularity includes districts or streets.

[0008] On the other hand, after obtaining the spatial information of each sub-data in the data to be stored, and before dividing the first spatial region covered by the spatial information into several second spatial regions according to a preset regional division strategy and taking each second spatial region as a target spatial region, the data management method further includes: Performing data cleaning on the spatial information to filter out abnormal spatial information.

[0009] On the other hand, after performing data sharding on the data to be stored successively through the target time segment and the target spatial region to obtain several data shards, before storing any data shard in a node of a distributed storage system, the data management method further includes: For any target time segment, determining the correlation between the data shards of each target spatial region under the target time segment; For the data shards of any pair of target spatial regions under the same target time segment, if the correlation meets the standard, then merging the data shards of the pair of target spatial regions.

[0010] On the other hand, after performing data sharding on the data to be stored successively through the target time segment and the target spatial region to obtain several data shards, before storing any data shard in a node of a distributed storage system, the data management method further includes: For any pair of data shards, determine the target ratio of the pair of data shards based on the access data of the corresponding historical data shard pair, where the target ratio is the ratio of the number of times the pair of data shards were accessed simultaneously in history to the total number of accesses; For any pair of data shards that meet the target ratio, store the data shards centrally; Storing a data shard in a node of a distributed storage system for any data shard includes: For any data shard that has not been stored, store the data shard in a node of the distributed storage system.

[0011] On the other hand, storing the data shards centrally for any pair of data shards that meet the target ratio includes: For any pair of data shards that meet the target ratio, if there is a node that meets the first preset constraint condition, use the node that meets the first preset constraint condition as the target storage node, where the first preset constraint condition is: the storage load is lower than the first preset threshold; If there is no node that meets the first preset constraint condition, use two nodes that meet the second preset constraint condition as the target storage nodes, where the second preset constraint condition is: the network distance is lower than the second preset threshold and the storage loads are both lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold; Store the pair of data shards that meet the target ratio in the target storage node.

[0012] On the other hand, the data management method further includes: Determine the load level of each node in the distributed storage system through preset node load monitoring metrics; For a node whose load level exceeds the standard, determine the data shards with an excessive access frequency of the node as the shards to be migrated; Migrate the shards to be migrated to a node whose load level meets the standard.

[0013] On the other hand, the preset node load monitoring metrics include the central processing unit usage rate, the memory usage rate, and the network bandwidth usage rate; Determining the load level of each node in the distributed storage system through preset node load monitoring metrics includes: For any node in the distributed storage system, perform proportional weighting on the preset node load monitoring metrics of the node as the load level of the node.

[0014] On the other hand, the data management method further includes: For any data storage object to be stored, the number of copies corresponding to the data type to which the data storage object belongs is used as the number of copies of the data storage object; Generate the corresponding number of copies of the data storage object and store them in the distributed storage system.

[0015] On the other hand, for any data storage object to be stored, using the number of copies corresponding to the data type to which the data storage object belongs as the number of copies of the data storage object includes: For any data storage object whose data type belongs to the preset key data type set, determine the number of copies of the data storage object according to the corresponding relationship between the preset data type and the number of copies; For any data storage object whose data type does not belong to the preset key data type set, determine the number of copies of the data storage object according to the access frequency of the data type to which the data storage object belongs and the corresponding relationship between the preset access frequency and the number of copies.

[0016] To solve the above technical problems, the present invention also provides a data management device, including: A memory for storing a computer program; A processor for implementing the steps of the data management method as described above when executing the computer program.

[0017] To solve the above technical problems, the present invention also provides a computer program product, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the data management method as described above are implemented.

[0018] To solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data management method as described above are implemented.

[0019] Beneficial effects: The present invention provides a data management method. Considering that data usually has distribution characteristics in both the time and space dimensions, and the probability of data within the same target time period or target space region being accessed simultaneously is relatively high. Therefore, in the present invention, multiple target time periods can be determined according to the section characteristics of the data to be stored in the time dimension; multiple target space regions can be determined according to the region characteristics of the data to be stored in the space dimension; then the data to be stored is fragmented according to the target time periods and target space regions respectively, so that the data in the data fragments has relevance in both the time and space dimensions, and then the data fragments are completely stored in the corresponding nodes. Since the data with "relevance in both the time and space dimensions" is centrally stored in the nodes in the form of data fragments, when accessing the data in the distributed storage system, theoretically, the number of nodes to be accessed is reduced, thereby reducing the network overhead and access latency.

[0020] The present invention also provides a data management device, a computer program product, and a storage medium, which have the same beneficial effects as the above data management method. Brief Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for use in the related art and embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 It is a flowchart of a data management method provided by the present invention; Figure 2 It is a structural diagram of a distributed storage system provided by the present invention; Figure 3 It is a flowchart of another data management method provided by the present invention; Figure 4 It is a structural diagram of a data management device provided by the present invention; Figure 5 It is a structural diagram of a computer-readable storage medium provided by the present invention. Detailed Embodiments

[0023] The core of the present invention is to provide a data management method, device, computer program product, and storage medium. In the present invention, multiple target time segments can be determined according to the segment characteristics of the data to be stored in the time dimension; multiple target spatial regions can be determined according to the regional characteristics of the data to be stored in the spatial dimension; then, the data to be stored is fragmented according to the target time segments and the target spatial regions respectively, so that the data in the data fragments has the correlation in the time and space dimensions, and then the data fragments are completely stored in the corresponding nodes. Since the data with "correlation in the time and space dimensions" is centrally stored in the nodes in the form of data fragments, when accessing the data in the distributed storage system, theoretically, the number of nodes to be accessed is reduced, thereby reducing the network overhead and access latency.

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0025] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a data management method provided by the present invention. This data management method is applied to a distributed storage system and includes: S101: Determine multiple target time segments according to the segment characteristics of the data to be stored in the time dimension; Specifically, considering the technical problems in the above background technology and also considering that data usually has distribution characteristics in both the time and space dimensions, data that is approximate in time or space may be accessed simultaneously in subsequent access processes. Therefore, it is possible to consider storing this data together from the time and space dimensions. Therefore, in the embodiments of the present invention, it is desired to divide approximate data into the same data fragment and store it from the time and space dimensions in order to improve the data access efficiency of the distributed storage system and reduce the network overhead and access latency.

[0026] Specifically, based on the above considerations, in this step, it is desired to first determine the respective target time segments of the data to be stored in the time dimension in order to perform data fragmentation based on the target time segments subsequently. Therefore, in this step, multiple target time segments can be determined according to the segment characteristics of the data to be stored in the time dimension and used as the data basis for the subsequent steps.

[0027] S102: Determine multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension; Specifically, in this step, the embodiment of the present invention can then start from the spatial dimension and determine multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension, so as to use them as the data basis to slice the data to be stored from the spatial dimension in the subsequent steps.

[0028] S103: Perform data slicing on the data to be stored successively through the target time period and the target spatial region to obtain a number of data slices, where a single data slice includes data of a single target time period and a single target spatial region; Specifically, after having the target time period and the target spatial region, data slicing can be performed on the data to be stored successively through the target time period and the target spatial region to obtain a number of data slices. For example, data slicing can be first performed on the data to be stored through each target time period, and then for the current slicing result, further data slicing can be performed through each target spatial region. The embodiment of the present invention does not make limitations here.

[0029] S104: For any data slice, store the data slice in a node of the distributed storage system.

[0030] Specifically, after data slicing and division are performed from the time dimension and the spatial dimension through the foregoing steps, the data to be stored in a single data slice has approximate rows in the time dimension and the spatial dimension, and the probability of being accessed simultaneously in the future is relatively high. Therefore, for any data slice, the data slice can be stored in a node of the distributed storage system to improve the efficiency when the distributed storage system is accessed and reduce the network overhead.

[0031] To better illustrate the embodiment of the present invention, please refer to Figure 2 , Figure 2 which is a schematic structural diagram of a distributed storage system provided by the present invention. The distributed storage system includes multiple nodes (such as Figure 2 the first node to the eighth node in

[0032] The present invention provides a data management method. Considering that data usually has distribution characteristics in both time and space dimensions, the probability of data in the same target time segment or target space area being accessed simultaneously is high. Therefore, in the present invention, multiple target time segments can be determined according to the segment characteristics of the data to be stored in the time dimension; multiple target space areas can be determined according to the area characteristics of the data to be stored in the space dimension; then, data to be stored is sliced according to the target time segments and the target space areas, so that the data in the data slices have correlation in the time and space dimensions, and then the data slices are completely stored in the corresponding nodes. Since the data "having correlation in the time and space dimensions" is centrally stored in the nodes in the form of data slices, when accessing the data in the distributed storage system, the number of nodes that need to be accessed is theoretically reduced, thereby reducing network overhead and access delay.

[0033] Based on the above embodiments: As an optional embodiment, according to the segment characteristics of the data to be stored in the time dimension, determining multiple target time segments includes: Each second preset time interval in the preset first preset time interval is used as the target time interval, wherein the collection of each second preset time interval is the first preset time interval, and each second preset time interval has no intersection.

[0034] Specifically, considering that a longer first preset time interval is divided into several shorter second preset time intervals, the segment characteristics of data with distribution characteristics in the time dimension can be accurately characterized. Therefore, in an embodiment of the present invention, each second preset time interval belonging to the first preset time interval can be preset for the data to be stored, and the second preset time interval can be used as the target time interval, which is efficient and accurate.

[0035] Among them, the first preset time interval and the second preset time interval can be set independently and flexibly. For example, the first preset time interval can be one day or one year, and the second preset time interval can be any complete hour from 0 to 24 hours in a day, or the night sleep period from 22:00 to 07:00 is used as a second preset time interval, and 07:00 to 22:00 is used as another second preset time interval, etc. The embodiment of the present invention is not limited here.

[0036] Of course, in addition to this division scheme, there may be many other ways to implement “determining multiple target time segments according to the segment characteristics of the data to be stored in the time dimension”, which is not limited in the embodiments of the present invention.

[0037] As an alternative embodiment, determining multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension includes: Obtain the spatial information of each sub-data in the data to be stored; According to a preset regional division strategy, divide the first spatial region covered by the spatial information into several second spatial regions, and use each second spatial region as a target spatial region.

[0038] Specifically, in the embodiments of the present invention, the data to be stored can be set with a timestamp and a spatial information stamp, which respectively represent the generation time of the data to be stored and the spatial information in the spatial dimension, so as to facilitate data sharding.

[0039] Among them, the spatial region can be divided according to the spatial information, that is, the spatial information of each sub-data in the data to be stored can be obtained, and according to a preset regional division strategy, the first spatial region covered by the spatial information is divided into several second spatial regions, and each second spatial region is used as a target spatial region. Through this method, several target spatial regions can be accurately divided from the first spatial region covered by the spatial information, which is convenient for improving the accuracy of subsequent data sharding based on the target spatial regions.

[0040] Specifically, the spatial information can be of various types. For example, it can include geographical location coordinates or spatial object identifiers (such as building identifiers or plot numbers), etc. The embodiments of the present invention do not limit this here.

[0041] Of course, in addition to this division scheme, there can be many other implementation methods for "determining multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension", and the embodiments of the present invention do not limit this here.

[0042] As an alternative embodiment, dividing the first spatial region covered by the spatial information into several second spatial regions according to a preset regional division strategy, and using each second spatial region as a target spatial region includes: Taking the preset urban area granularity as a unit, divide the first spatial region covered by the spatial information into several second spatial regions, and use each second spatial region as a target spatial region; Among them, the preset urban area granularity includes districts or streets.

[0043] Specifically, considering that for the data to be stored collected in the city, the probability of "being accessed simultaneously" for the data within the same urban area granularity is relatively high. Therefore, in the embodiments of the present invention, the first spatial region covered by the spatial information can be divided into several second spatial regions with the preset urban area granularity as a unit, and each second spatial region is used as a target spatial region.

[0044] Specifically, in addition to this specific method, the implementation method of "dividing the first spatial region covered by the spatial information into several second spatial regions according to a preset region division strategy and using each second spatial region as a target spatial region" can also be of other types, which are not limited in the embodiments of the present invention.

[0045] As an optional embodiment, after obtaining the spatial information of each sub-data in the data to be stored, before dividing the first spatial region covered by the spatial information into several second spatial regions according to a preset region division strategy and using each second spatial region as a target spatial region, the data management method further includes: Performing data cleaning on the spatial information to filter out abnormal spatial information.

[0046] Specifically, considering that there may be abnormal spatial information in the spatial information of the data to be stored, if the target spatial region is directly divided based on each spatial information containing abnormal spatial information, it may affect the division accuracy of the target spatial region, thereby reducing the data sharding accuracy and affecting the access efficiency and network overhead of the distributed storage system. Therefore, in the embodiments of the present invention, data cleaning can be performed on the spatial information before dividing the target spatial region to filter out abnormal spatial information.

[0047] Among them, the specific cleaning operations can be of various types. For example, it can be to check whether the geographical coordinate data is within a reasonable range of earth coordinates (longitude between -180° and 180°, latitude between -90° and 90°), filter out the coordinate data outside the range, or for the regional coding data, such as administrative division codes, check whether it conforms to the standard coding specification, and convert or correct the non-conforming codes; the embodiments of the present invention are not limited herein.

[0048] As an optional embodiment, after performing data sharding on the data to be stored by means of a target time period and a target spatial region in sequence to obtain several data shards, before storing any data shard in a node of the distributed storage system, the data management method further includes: For any target time period, determining the correlation between the data shards of each target spatial region under the target time period; For any pair of data shards of target spatial regions under the same target time period, if the correlation meets the standard, merging the data shards of the pair of target spatial regions.

[0049] Specifically, considering that there may be a strong correlation between "data shards of different target spatial regions" in the same target time period, for these data shards with strong correlation, if they can be merged into the same data shard, then the network overhead and latency of data access in the distributed storage system can be further reduced. Therefore, in the embodiments of the present invention, for any target time period, the correlation between the data shards of each target spatial region in the target time period can be determined, and then for any pair of data shards of the target spatial regions in the same target time period, if the correlation meets the standard, the data shards of this pair of target spatial regions will be merged.

[0050] Among them, there are various types of methods that can be used for correlation analysis. For example, it can be the Pearson correlation coefficient method, etc., and the embodiments of the present invention do not limit this here.

[0051] As an alternative embodiment, after data sharding the data to be stored through the target time period and the target spatial region in sequence to obtain a number of data shards, before storing any data shard in a node of the distributed storage system, the data management method further includes: For any pair of data shards, determine the target ratio of this pair of data shards through the access data of the historical data shard pair corresponding to this pair of data shards, where the target ratio is the ratio of the number of times this pair of data shards are accessed simultaneously in history to the total number of access times; For any pair of data shards with the target ratio meeting the standard, store the data shards centrally; For any data shard, storing the data shard in a node of the distributed storage system includes: For any data shard that has not been stored, store the data shard in a node of the distributed storage system.

[0052] Specifically, considering that even for the divided data shards, it is very likely that a certain pair of data shards will be accessed simultaneously, which can be found from the historical data. Therefore, before storing the data shards to be stored in the embodiments of the present invention, it is possible to determine in advance the data shard pairs that "are likely to be accessed simultaneously with high probability", and then store them centrally. Therefore, in the embodiments of the present invention, "for any pair of data shards, determine the target ratio of this pair of data shards through the access data of the historical data shard pair corresponding to this pair of data shards (the target ratio is the ratio of the number of times this pair of data shards are accessed simultaneously in history to the total number of access times)", and then for any pair of data shards with the target ratio meeting the standard, store the data shards centrally, so as to further improve the locality of data storage and reduce the network overhead and latency during data access in the distributed storage system.

[0053] As an alternative embodiment, for any pair of data shards that meet the target ratio standard, centralizing the storage of the data shards includes: For any pair of data shards that meet the target ratio standard, if there is a node that meets the first preset constraint condition, use the node that meets the first preset constraint condition as the target storage node, where the first preset constraint condition is: the storage load is lower than the first preset threshold; If there is no node that meets the first preset constraint condition, use two nodes that meet the second preset constraint condition as the target storage nodes, where the second preset constraint condition is: the network distance is lower than the second preset threshold and the storage loads are both lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold; Store the pair of data shards that meet the target ratio standard in the target storage node.

[0054] Specifically, considering that if the data shard pair can be stored in the same node, then the network overhead and latency of data access can be minimized. However, storing the data shard pair in the same node has relatively high requirements for storage load. Therefore, in the embodiments of the present invention, for any pair of data shards that meet the target ratio standard, if there is a node that meets the first preset constraint condition (the storage load is lower than the first preset threshold), use the node that meets the first preset constraint condition as the target storage node; if there is no node that meets the first preset constraint condition, use two nodes that meet the second preset constraint condition (the network distance is lower than the second preset threshold and the storage loads are both lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold) as the target storage nodes, so as to store the data shard pair in two nodes, with one data shard stored in one node, which can also further reduce the network overhead and latency of data access to a certain extent.

[0055] Among them, the first to third preset thresholds can be set independently, and the embodiments of the present invention do not limit this here.

[0056] As an alternative embodiment, the data management method further includes: Determine the load level of each node in the distributed storage system through preset node load monitoring metrics; For nodes with an excessive load level, determine the data shards with an excessive access frequency of the node as the shards to be migrated; Migrate the shards to be migrated to nodes with a qualified load level.

[0057] Specifically, considering the unreasonable distribution of data sharding storage, for example, the access rates of data shards in some nodes are relatively high, which seriously increases the load of these nodes and reduces the node performance. Therefore, in the embodiments of the present invention, the load levels of each node in the distributed storage system can be determined through preset node load monitoring metrics. Then, for the nodes with excessive load levels, the data shards with excessive access frequencies in the nodes are determined as the shards to be migrated, and finally, the shards to be migrated are migrated to the nodes with qualified load levels. In this way, the nodes with lighter load levels can share the load pressure of the nodes with heavier load levels, improving the overall performance of the distributed storage system.

[0058] To better illustrate the embodiments of the present invention, please refer to Figure 3 , Figure 3 FIG. is a schematic flowchart of another data management method provided by the present invention. On the one hand, for the data to be stored, after "sharding the data from the time dimension" and "sharding the data from the space dimension" respectively, the data shards can be "merged through correlation analysis", and finally the data shards are written (into the distributed storage system). On the other hand, the nodes in the distributed storage system can be monitored, the shards to be migrated are determined according to the node load levels, and the data shards are migrated.

[0059] As an optional embodiment, the preset node load monitoring metrics include the central processor usage rate, the memory usage rate, and the network bandwidth usage rate. Determining the load levels of each node in the distributed storage system through the preset node load monitoring metrics includes: For any node in the distributed storage system, the preset node load monitoring metrics of the node are weighted proportionally as the load level of the node.

[0060] Specifically, considering that the three metrics of the central processor usage rate, the memory usage rate, and the network bandwidth usage rate can comprehensively and comprehensively evaluate the load level of a node, the preset node load monitoring metrics in the embodiments of the present invention include the central processor usage rate, the memory usage rate, and the network bandwidth usage rate. Then, for any node in the distributed storage system, the preset node load monitoring metrics of the node are weighted proportionally as the load level of the node, so as to efficiently and accurately determine the load levels of each node.

[0061] Of course, in addition to this specific form, the implementation manner of "determining the load levels of each node in the distributed storage system through the preset node load monitoring metrics" can also be other types, which are not limited in the embodiments of the present invention.

[0062] As an optional embodiment, the data management method further includes: For any data storage object to be stored, use the number of copies corresponding to the data type to which the data storage object belongs as the number of copies of the data storage object; Generate the corresponding number of copies of the data storage object and store them in the distributed storage system.

[0063] Specifically, considering that in a distributed storage system, the number of copies of a data storage object to be stored affects data security and storage pressure, and the importance of a data storage object is often related to the data type. Therefore, in the embodiments of the present invention, for any data storage object to be stored, use the number of copies corresponding to the data type to which the data storage object belongs as the number of copies of the data storage object, and then generate the corresponding number of copies of the data storage object and store them in the distributed storage system, so as to balance data security and storage pressure.

[0064] Among them, when storing multiple copies of a data storage object in the distributed storage system, the storage can be carried out as scattered as possible. For example, multiple copies of the data storage object can be stored in nodes that are "in different failure domains" and have "low network latency" between each other.

[0065] As an optional embodiment, for any data storage object to be stored, using the number of copies corresponding to the data type to which the data storage object belongs as the number of copies of the data storage object includes: For any data storage object whose data type belongs to the preset key data type set, determine the number of copies of the data storage object according to the preset correspondence between the data type and the number of copies; For any data storage object whose data type does not belong to the preset key data type set, determine the number of copies of the data storage object according to the access frequency of the data type to which the data storage object belongs and the preset correspondence between the access frequency and the number of copies.

[0066] Specifically, considering that for some key data types, it may be necessary to specifically specify the number of copies, while for non-key data types, the number of copies can be determined according to their access frequencies. Therefore, in the embodiments of the present invention, for any data storage object whose data type belongs to the preset key data type set, determine the number of copies of the data storage object according to the preset correspondence between the data type and the number of copies; and for any data storage object whose data type does not belong to the preset key data type set, determine the number of copies of the data storage object according to the access frequency of the data type to which the data storage object belongs and the preset correspondence between the access frequency and the number of copies, so as to be able to more accurately determine the appropriate number of copies for each data storage object and better balance data security and storage pressure.

[0067] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of a data management device provided by the present invention. The data management device includes: A memory 41 for storing computer programs; A processor 42 for implementing the steps of the data management method in the foregoing embodiments when executing the computer program.

[0068] For the introduction of the data management device provided by the embodiments of the present invention, please refer to the embodiments of the foregoing data management method, and the embodiments of the present invention will not be described in detail herein.

[0069] The present invention also provides a computer program product, including computer programs / instructions, which implement the steps of the data management method in the foregoing embodiments when executed by a processor.

[0070] For the introduction of the computer program product provided by the embodiments of the present invention, please refer to the embodiments of the foregoing data management method, and the embodiments of the present invention will not be described in detail herein.

[0071] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of a computer-readable storage medium provided by the present invention. A computer program 52 is stored on the computer-readable storage medium 51, and when the computer program 51 is executed by a processor, it implements the steps of the data management method in the foregoing embodiments.

[0072] For the introduction of the computer-readable storage medium provided by the embodiments of the present invention, please refer to the embodiments of the foregoing data management method, and the embodiments of the present invention will not be described in detail herein.

[0073] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts among the embodiments can be referred to each other. It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0074] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data management method, characterized in that, Applied to distributed storage systems, including: According to the segment characteristics of the data to be stored in the time dimension, multiple target time segments are determined; Determine multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension; The data to be stored is sliced successively by the target time segment and the target space region to obtain a plurality of data slices, wherein a single data slice includes data of a single target time segment and a single target space region; For any data shard, the data shard is stored in a node of the distributed storage system.

2. The data management method according to claim 1, wherein According to the segment characteristics of the data to be stored in the time dimension, multiple target time segments are determined, including: Each second preset time interval in the preset first preset time interval is used as the target time interval, wherein the collection of each second preset time interval is the first preset time interval, and each second preset time interval has no intersection.

3. The data management method according to claim 1, wherein According to the regional characteristics of the data to be stored in the spatial dimension, multiple target spatial regions are determined, including: Obtaining spatial information of each sub-data in the data to be stored; According to a preset area division strategy, the first space area covered by the space information is divided into a plurality of second space areas, and each second space area is used as a target space area.

4. The data management method according to claim 3, characterized in that According to a preset area division strategy, the first space area covered by the space information is divided into a plurality of second space areas, and each second space area is used as a target space area including: Taking a preset urban area granularity as a unit, dividing the first spatial area covered by the spatial information into a plurality of second spatial areas, and taking each second spatial area as a target spatial area; The preset urban area granularity includes districts or streets.

5. The data management method according to claim 3, wherein After acquiring the spatial information of each sub-data in the data to be stored, dividing the first spatial area covered by the spatial information into a plurality of second spatial areas according to a preset area division strategy, and before taking each second spatial area as the target spatial area, the data management method further includes: Data cleaning is performed on the spatial information to filter out abnormal spatial information.

6. The data management method according to claim 1, wherein After the target time segment and the target spatial region are used to successively slice the data to be stored to obtain a plurality of data slices, for any data slice, before the data slice is stored in a node of the distributed storage system, the data management method further includes: For any target time segment, determining the correlation between the data slices of each target spatial area in the target time segment; For any pair of data shards of the target spatial region in the same target time segment, if the correlation meets the standard, the data shards of the pair of target spatial regions are merged.

7. The data management method according to claim 1, characterized in that After the target time segment and the target spatial region are used to successively slice the data to be stored to obtain a plurality of data slices, for any data slice, before the data slice is stored in a node of the distributed storage system, the data management method further includes: For any pair of data shards, determine the target ratio of the pair of data shards based on the access data of the corresponding historical data shard pair, where the target ratio is the ratio of the number of times the pair of data shards were accessed simultaneously in history to the total number of accesses; For any pair of data shards with the target ratio meeting the standard, store the data shards centrally; For any data shard, storing the data shard in a node of the distributed storage system includes: For any data shard that has not been stored, store the data shard in a node of the distributed storage system.

8. The data management method according to claim 7, wherein For any pair of data shards with the target ratio meeting the standard, centrally storing the data shards includes: For any pair of data shards with the target ratio meeting the standard, if there is a node that meets the first preset constraint condition, use the node that meets the first preset constraint condition as the target storage node, where the first preset constraint condition is: the storage load is lower than the first preset threshold; If there is no node that meets the first preset constraint condition, use two nodes that meet the second preset constraint condition as the target storage nodes, where the second preset constraint condition is: the network distance is lower than the second preset threshold and the storage loads are both lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold; Store the pair of data shards with the target ratio meeting the standard in the target storage node.

9. The data management method according to claim 1, wherein The data management method further includes: Determine the load level of each node in the distributed storage system through preset node load monitoring metrics; For a node with an excessive load level, determine the data shards with an excessive access frequency of the node as the shards to be migrated; Migrate the shards to be migrated to a node with a qualified load level.

10. The data management method according to claim 9, wherein The preset node load monitoring metrics include the central processing unit usage rate, the memory usage rate, and the network bandwidth usage rate; Determining the load level of each node in the distributed storage system through preset node load monitoring metrics includes: For any node in the distributed storage system, proportionally weight the preset node load monitoring metrics of the node as the load level of the node.

11. The data management method according to any one of claims 1 to 10, characterized in that, The data management method further includes: For any data storage object to be stored, use the number of replicas corresponding to the data type to which the data storage object belongs as the number of replicas of the data storage object; Generate the corresponding number of replicas for the data storage object and store them in the distributed storage system.

12. The data management method according to claim 11, wherein For any data storage object to be stored, using the number of replicas corresponding to the data type to which the data storage object belongs as the number of replicas of the data storage object includes: For any data storage object whose data type belongs to the preset key data type set, determine the number of replicas of the data storage object according to the preset correspondence between the data type and the number of replicas; For any data storage object whose data type does not belong to the preset key data type set, determine the number of replicas of the data storage object according to the access frequency of the data type to which the data storage object belongs and the preset correspondence between the access frequency and the number of replicas.

13. A data management device, characterized in that, Includes: A memory for storing a computer program; A processor, which is configured to implement the steps of the data management method according to any one of claims 1 to 12 when executing the computer program.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by a processor, the steps of the data management method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data management method according to any one of claims 1 to 12 are implemented.

Images