Data management method, device, computer program product and storage medium

Abstract

The present invention discloses a data management method, device, computer program product and storage medium, which belong to the field of distributed storage systems and are used to slice data from two dimensions, time and space, and then store it in a distributed storage system, solving the problems of high network overhead and high access delay when accessing a distributed storage system. According to the segment characteristics of the data to be stored in the time dimension and the regional characteristics in the space dimension, multiple target time segments and multiple target space regions are determined; then, the data to be stored is sliced ​​according to the target time segments and target space regions, and then the data slices are completely stored in the corresponding nodes. Since the data "with correlation in the time and space dimensions" is centrally stored in the nodes in the form of data slices, when accessing data in the distributed storage system, the number of nodes that need to be accessed is theoretically reduced, thereby reducing network overhead and access delay.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage systems, and in particular to a data management method, device, computer program product and storage medium. Background Art

[0002] Since a distributed storage system includes many nodes for storing data, when storing data in a distributed storage system, if closely related data can be stored in the same node, network overhead and access delay will be reduced. However, the relevant technology lacks a mature data management method, which may cause data to be stored in a dispersed manner in various nodes in the distributed storage system, thereby increasing network overhead and access delay.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the Invention

[0004] The purpose of the present invention is to provide a data management method, device, computer program product and storage medium. In the present invention, multiple target time segments can be determined according to the segment characteristics of the data to be stored in the time dimension; multiple target spatial areas can be determined according to the regional characteristics of the data to be stored in the spatial dimension; then the data to be stored is sliced ​​according to the target time segments and target spatial areas, so that the data in the data slices have correlation in the time and space dimensions, and then the data slices are completely stored in the corresponding nodes. Since the data "with correlation in the time and space dimensions" is centrally stored in the nodes in the form of data slices, when accessing data in a distributed storage system, the number of nodes that need to be accessed is theoretically reduced, thereby reducing network overhead and access delay.

[0005] To solve the above technical problems, the present invention provides a data management method applied to a distributed storage system, comprising:

[0006] Determine multiple target time segments based on the segment characteristics of the data to be stored in the time dimension;

[0007] Determine multiple target spatial regions based on the regional characteristics of the data to be stored in the spatial dimension;

[0008] Slicing the data to be stored sequentially by the target time segment and the target spatial region to obtain a plurality of data slices, wherein a single data slice includes data of a single target time segment and a single target spatial region;

[0009] For any data shard, the data shard is stored in a node of the distributed storage system.

[0010] On the other hand, according to the segment characteristics of the data to be stored in the time dimension, multiple target time segments are determined, including:

[0011] Each second preset time interval in the preset first preset time interval is used as a target time interval, wherein the collection of each second preset time interval is the first preset time interval, and each second preset time interval has no intersection.

[0012] On the other hand, based on the regional characteristics of the data to be stored in the spatial dimension, multiple target spatial regions are determined, including:

[0013] Obtaining spatial information of each sub-data in the data to be stored;

[0014] According to a preset region division strategy, the first spatial region covered by the spatial information is divided into a plurality of second spatial regions, and each second spatial region is used as a target spatial region.

[0015] On the other hand, according to a preset region division strategy, the first spatial region covered by the spatial information is divided into a plurality of second spatial regions, and each second spatial region is used as a target spatial region, including:

[0016] Divide the first spatial area covered by the spatial information into a plurality of second spatial areas based on a preset urban area granularity, and use each second spatial area as a target spatial area;

[0017] The preset urban area granularity includes districts or streets.

[0018] On the other hand, after obtaining the spatial information of each sub-data in the data to be stored, dividing the first spatial area covered by the spatial information into a plurality of second spatial areas according to a preset area division strategy, and before using each second spatial area as the target spatial area, the data management method further includes:

[0019] Data cleaning is performed on the spatial information to filter out abnormal spatial information.

[0020] On the other hand, after sharding the data to be stored by the target time segment and the target spatial area to obtain a plurality of data shards, for any data shard, before storing the data shard in a node of the distributed storage system, the data management method further includes:

[0021] For any target time segment, determining the correlation between the data slices of each target spatial area in the target time segment;

[0022] For any pair of data shards in the target spatial region within the same target time segment, if the correlation meets the criteria, the data shards in the pair of target spatial regions will be merged.

[0023] On the other hand, after sharding the data to be stored by the target time segment and the target spatial area to obtain a plurality of data shards, for any data shard, before storing the data shard in a node of the distributed storage system, the data management method further includes:

[0024] For any pair of data shards, determine the target ratio of the pair of data shards based on the access data of the historical data shard pairs corresponding to the pair of data shards, where the target ratio is the ratio of the number of times the pair of data shards have been accessed simultaneously in history to the total number of accesses;

[0025] For any pair of data shards whose target ratios meet the requirements, the data shards are centrally stored;

[0026] For any data shard, storing the data shard in a node of the distributed storage system includes:

[0027] For any data shard that is not stored, the data shard is stored in a node of the distributed storage system.

[0028] On the other hand, for any pair of data shards whose target ratios meet the requirements, centrally storing the data shards includes:

[0029] For any pair of data shards that meet the target ratio, if there is a node that meets the first preset constraint condition, the node that meets the first preset constraint condition is used as the target storage node, wherein the first preset constraint condition is: the storage load is lower than a first preset threshold;

[0030] If there is no node that meets the first preset constraint, two nodes that meet the second preset constraint are used as target storage nodes, where the second preset constraint is: the network distance is lower than the second preset threshold and the storage load is lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold;

[0031] The pair of data shards that meet the target ratio are stored in the target storage node.

[0032] On the other hand, the data management method further includes:

[0033] Determine the load level of each node in the distributed storage system through the preset node load monitoring indicators;

[0034] For the node with an excessive load, determining a data shard whose access frequency of the node exceeds the standard as a shard to be migrated;

[0035] Migrate the shards to be migrated to nodes with a load level that meets the requirements.

[0036] On the other hand, the preset node load monitoring indicators include CPU usage, memory usage, and network bandwidth usage;

[0037] The load level of each node in the distributed storage system is determined by using preset node load monitoring indicators, including:

[0038] For any node in the distributed storage system, the preset node load monitoring indicators of the node are proportionally weighted to serve as the load level of the node.

[0039] On the other hand, the data management method further includes:

[0040] For any data storage object to be stored, the number of copies corresponding to the data type of the data storage object is used as the number of copies of the data storage object;

[0041] Generate the corresponding number of copies for the data storage object and store them in the distributed storage system.

[0042] On the other hand, for any data storage object to be stored, taking the number of replicas corresponding to the data type of the data storage object as the number of replicas of the data storage object includes:

[0043] For any data storage object whose data type belongs to a preset key data type set, determining the number of copies of the data storage object according to a preset correspondence between the data type and the number of copies;

[0044] For any data storage object whose data type does not belong to the preset key data type set, the number of copies of the data storage object is determined based on the access frequency of the data type to which the data storage object belongs and the correspondence between the preset access frequency and the number of copies.

[0045] To solve the above technical problems, the present invention further provides a data management device, comprising:

[0046] memory for storing computer programs;

[0047] A processor is used to implement the steps of the data management method described above when executing the computer program.

[0048] To solve the above technical problems, the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above data management method when executed by a processor.

[0049] To solve the above technical problems, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above data management method are implemented.

[0050] Beneficial effects: The present invention provides a data management method. Considering that data usually has distribution characteristics in both time and space dimensions, the probability of data in the same target time segment or target space area being accessed simultaneously is high. Therefore, the present invention can determine multiple target time segments according to the segment characteristics of the data to be stored in the time dimension; determine multiple target space areas according to the area characteristics of the data to be stored in the space dimension; then, data to be stored is sliced ​​according to the target time segments and target space areas, so that the data in the data slices have correlation in time and space dimensions, and then the data slices are completely stored in the corresponding nodes. Since the data "with correlation in time and space dimensions" is centrally stored in the nodes in the form of data slices, when accessing data in a distributed storage system, the number of nodes that need to be accessed is theoretically reduced, thereby reducing network overhead and access delay.

[0051] The present invention also provides a data management device, a computer program product, and a storage medium, which have the same beneficial effects as the above data management method. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the relevant technologies and the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flow chart of a data management method provided by the present invention;

[0054] Figure 2 A schematic diagram of the structure of a distributed storage system provided by the present invention;

[0055] Figure 3 A flow chart of another data management method provided by the present invention;

[0056] Figure 4 A structural diagram of a data management device provided by the present invention;

[0057] Figure 5 A schematic structural diagram of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0058] The core of the present invention is to provide a data management method, device, computer program product and storage medium. In the present invention, multiple target time segments can be determined according to the segment characteristics of the data to be stored in the time dimension; multiple target spatial areas can be determined according to the regional characteristics of the data to be stored in the spatial dimension; then the data to be stored is sliced ​​according to the target time segments and target spatial areas, so that the data in the data slices have correlation in the time and space dimensions, and then the data slices are completely stored in the corresponding nodes. Since the data "with correlation in the time and space dimensions" is centrally stored in the nodes in the form of data slices, when accessing data in the distributed storage system, the number of nodes that need to be accessed is theoretically reduced, thereby reducing network overhead and access delay.

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0060] Please refer to Figure 1 , Figure 1 This is a flow chart of a data management method provided by the present invention, which is applied to a distributed storage system and includes:

[0061] S101: Determine multiple target time segments based on the segment characteristics of the data to be stored in the time dimension;

[0062] Specifically, taking into account the technical problems in the above background technology, and considering that data usually has distribution characteristics in both time and space dimensions, data that are similar in time or space may be accessed simultaneously in subsequent access processes. Therefore, it is possible to consider storing these data together from the time and space dimensions. Therefore, in an embodiment of the present invention, the similar data is divided into the same data shard from the time and space dimensions and stored, so as to improve the efficiency of data access in the distributed storage system and reduce network overhead and access delay.

[0063] Specifically, based on the above considerations, in this step, we first want to determine the target time segments of the data to be stored from the time dimension, so that we can subsequently perform data sharding based on the target time segments. Therefore, in this step, we can determine multiple target time segments based on the segment characteristics of the data to be stored in the time dimension, and use them as the data basis for subsequent steps.

[0064] S102: Determine multiple target spatial regions based on the regional characteristics of the data to be stored in the spatial dimension;

[0065] Specifically, in this step, the embodiment of the present invention can then start from the spatial dimension and determine multiple target spatial areas based on the regional characteristics of the data to be stored in the spatial dimension, so as to use them as the data basis to segment the data to be stored from the spatial dimension in subsequent steps.

[0066] S103: Slicing the data to be stored sequentially by target time segment and target spatial region to obtain a plurality of data slices, wherein a single data slice includes data of a single target time segment and a single target spatial region;

[0067] Specifically, after having the target time segment and the target space area, the data to be stored can be sharded successively by the target time segment and the target space area respectively to obtain several data shards. For example, the data to be stored can be sharded first by each target time segment, and then the current sharding result can be further sharded by each target space area. The embodiments of the present invention are not limited here.

[0068] S104: For any data shard, store the data shard in a node of the distributed storage system.

[0069] Specifically, after the data is sharded along the time and space dimensions through the aforementioned steps, the data to be stored in a single data shard will have approximate rows in the time and space dimensions, and the probability of being accessed simultaneously in the future is high. Therefore, for any data shard, the data shard can be stored in a node of the distributed storage system to improve the efficiency of the distributed storage system when it is accessed and reduce network overhead.

[0070] To better illustrate the embodiments of the present invention, please refer to Figure 2 , Figure 2 This is a structural diagram of a distributed storage system provided by the present invention, which includes multiple nodes (e.g. Figure 2 From the first node to the eighth node in the , each node can communicate directly with each other.

[0071] The present invention provides a data management method. Considering that data usually has distribution characteristics in both time and space dimensions, the probability of data in the same target time segment or target space area being accessed simultaneously is high. Therefore, the present invention can determine multiple target time segments according to the segment characteristics of the data to be stored in the time dimension; determine multiple target space areas according to the area characteristics of the data to be stored in the space dimension; then, data to be stored is sliced ​​according to the target time segments and target space areas, so that the data in the data slices have correlation in the time and space dimensions, and then the data slices are completely stored in the corresponding nodes. Since the data "with correlation in the time and space dimensions" is centrally stored in the nodes in the form of data slices, when accessing data in the distributed storage system, the number of nodes that need to be accessed is theoretically reduced, thereby reducing network overhead and access delay.

[0072] Based on the above embodiment:

[0073] As an optional embodiment, determining multiple target time segments based on the segment characteristics of the data to be stored in the time dimension includes:

[0074] Each second preset time interval in the preset first preset time interval is used as a target time interval, wherein the collection of each second preset time interval is the first preset time interval, and each second preset time interval has no intersection.

[0075] Specifically, considering that a longer first preset time interval is divided into several shorter second preset time intervals, the segment characteristics of data with distribution characteristics in the time dimension can be accurately characterized. Therefore, in an embodiment of the present invention, each second preset time interval belonging to the first preset time interval can be preset for the data to be stored, and the second preset time interval can be used as the target time interval, which is efficient and accurate.

[0076] Among them, the first preset time interval and the second preset time interval can be set independently and flexibly. For example, the first preset time interval can be one day or one year, and the second preset time interval can be any complete hour from 0 to 24 hours in a day, or the night sleep period from 22:00 to 07:00 can be used as a second preset time interval, and 07:00 to 22:00 can be used as another second preset time interval, etc. The embodiment of the present invention is not limited here.

[0077] Of course, in addition to this division scheme, there may be many other ways to implement "determining multiple target time segments according to the segment characteristics of the data to be stored in the time dimension", which is not limited in the embodiment of the present invention.

[0078] As an optional embodiment, determining multiple target spatial regions based on regional characteristics of the data to be stored in a spatial dimension includes:

[0079] Obtaining spatial information of each sub-data in the data to be stored;

[0080] According to a preset area division strategy, the first spatial area covered by the spatial information is divided into a plurality of second spatial areas, and each second spatial area is used as a target spatial area.

[0081] Specifically, the data to be stored in the embodiment of the present invention can be provided with a timestamp and a spatial information stamp, which respectively represent the generation time of the data to be stored and the spatial information in the spatial dimension, thereby facilitating data sharding.

[0082] Among them, the spatial area can be divided according to the spatial information, that is, the spatial information of each sub-data in the data to be stored can be obtained, and according to the preset area division strategy, the first spatial area covered by the spatial information can be divided into several second spatial areas, and each second spatial area can be used as the target spatial area. In this way, the first spatial area covered by the spatial information can be accurately divided into several target spatial areas, which is convenient for improving the accuracy of subsequent data segmentation based on the target spatial area.

[0083] Specifically, the spatial information may be of various types, for example, geographic location coordinates or spatial object identifiers (such as building identifiers or plot numbers), etc., which is not limited in the embodiment of the present invention.

[0084] Of course, in addition to this division scheme, there may be many other ways to implement "determining multiple target spatial regions according to the regional characteristics of the data to be stored in the spatial dimension", which is not limited in the embodiments of the present invention.

[0085] As an optional embodiment, according to a preset region division strategy, the first spatial region covered by the spatial information is divided into a plurality of second spatial regions, and each second spatial region is used as a target spatial region, including:

[0086] Using a preset urban area granularity as a unit, the first spatial area covered by the spatial information is divided into a plurality of second spatial areas, and each second spatial area is used as a target spatial area;

[0087] The preset urban area granularity includes districts or streets.

[0088] Specifically, considering that for the data to be stored collected in the city, the probability of data within the same urban area granularity being "accessed simultaneously" is relatively high, therefore, in an embodiment of the present invention, the first spatial area covered by the spatial information can be divided into several second spatial areas based on the preset urban area granularity, and each second spatial area can be used as the target spatial area.

[0089] Specifically, in addition to this specific method, the implementation method of "dividing the first spatial area covered by the spatial information into several second spatial areas according to a preset area division strategy, and using each second spatial area as the target spatial area" can also be other types, which are not limited in the embodiments of the present invention.

[0090] As an optional embodiment, after obtaining the spatial information of each sub-data in the data to be stored, dividing the first spatial area covered by the spatial information into a plurality of second spatial areas according to a preset area division strategy, and before using each second spatial area as the target spatial area, the data management method further includes:

[0091] Perform data cleaning on spatial information to filter out abnormal spatial information.

[0092] Specifically, considering that there may be abnormal spatial information in the spatial information of the data to be stored, if the target spatial area is divided directly according to the spatial information containing abnormal spatial information, it may affect the division accuracy of the target spatial area, thereby reducing the data sharding accuracy and affecting the access efficiency and network overhead of the distributed storage system. Therefore, in an embodiment of the present invention, before dividing the target spatial area, the spatial information can be cleaned to filter out abnormal spatial information.

[0093] Among them, the specific cleaning operations can be of various types, for example, it can be to check whether the geographic coordinate data is within a reasonable earth coordinate range (longitude between -180° and 180°, latitude between -90° and 90°), filter out the coordinate data that exceeds the range, or for regional coding data, such as administrative division codes, check whether it complies with standard coding specifications, and convert or correct the coding that does not comply with the specifications; the embodiments of the present invention are not limited here.

[0094] As an optional embodiment, after sharding the data to be stored by the target time segment and the target spatial area to obtain a plurality of data shards, before storing any data shard in a node of the distributed storage system, the data management method further includes:

[0095] For any target time segment, determine the correlation between the data slices of each target spatial area within the target time segment;

[0096] For any pair of data shards in the target spatial region within the same target time segment, if the correlation meets the criteria, the data shards in the pair of target spatial regions will be merged.

[0097] Specifically, considering that the "data shards of different target spatial areas" under the same target time segment may have a strong correlation, if these data shards with a strong correlation can be merged into the same data shard, then the network overhead and delay of data access in the distributed storage system can be further reduced. Therefore, in an embodiment of the present invention, for any target time segment, the correlation between the data shards of each target spatial area under the target time segment can be determined, and then for any pair of data shards of the target spatial area under the same target time segment, if the correlation meets the standard, the data shards of the pair of target spatial areas will be merged.

[0098] There are many types of methods that can be used for correlation analysis, such as the Pearson correlation coefficient method, etc., which is not limited in the embodiment of the present invention.

[0099] As an optional embodiment, after sharding the data to be stored by the target time segment and the target spatial area to obtain a plurality of data shards, before storing any data shard in a node of the distributed storage system, the data management method further includes:

[0100] For any pair of data shards, determine the target ratio of the pair of data shards based on the access data of the historical data shard pairs corresponding to the pair of data shards, where the target ratio is the ratio of the number of times the pair of data shards have been accessed simultaneously in history to the total number of accesses;

[0101] For any pair of data shards that meet the target ratio, the data shards are stored centrally;

[0102] For any data shard, a node storing the data shard in a distributed storage system includes:

[0103] For any data shard that is not stored, the data shard is stored in a node of the distributed storage system.

[0104] Specifically, considering that even for divided data shards, a pair of data shards is very likely to be accessed simultaneously, which can be found from historical data, the embodiment of the present invention can determine in advance the data shard pairs that are "most likely to be accessed simultaneously" before storing the data shards to be stored, and then store them centrally. Therefore, the embodiment of the present invention can "for any pair of data shards, determine the target ratio of the pair of data shards through the access data of the historical data shard pairs corresponding to the pair of data shards (the target ratio is the ratio of the number of times the pair of data shards have been accessed simultaneously in history to the total number of accesses)", and then for any pair of data shards that meet the target ratio, the data shards are stored centrally, which can further improve the locality of data storage and reduce the network overhead and delay of data access in the distributed storage system.

[0105] As an optional embodiment, for any pair of data shards that meet the target ratio, centrally storing the data shards includes:

[0106] For any pair of data shards that meet the target ratio, if there is a node that meets the first preset constraint condition, the node that meets the first preset constraint condition is used as the target storage node, wherein the first preset constraint condition is: the storage load is lower than the first preset threshold;

[0107] If there is no node that meets the first preset constraint, two nodes that meet the second preset constraint are used as target storage nodes, where the second preset constraint is: the network distance is lower than the second preset threshold and the storage load is lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold;

[0108] The pair of data shards that meet the target ratio are stored in the target storage node.

[0109] Specifically, considering that if the data shard pair can be stored in the same node, the network overhead and delay of data access can be minimized. However, storing the data shard pair in the same node has high requirements for the storage load. Therefore, in an embodiment of the present invention, for any pair of data shards that meet the target ratio, if there is a node that meets the first preset constraint (storage load is lower than the first preset threshold), the node that meets the first preset constraint will be used as the target storage node; and if there is no node that meets the first preset constraint, two nodes that meet the second preset constraint (network distance is lower than the second preset threshold and storage load is lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold) will be used as the target storage nodes, so that the data shard pair is stored in two nodes, and one data shard is stored in one node, which can also further reduce the network overhead and delay of data access to a certain extent.

[0110] The first to third preset thresholds can be set independently, and are not limited in this embodiment of the present invention.

[0111] As an optional embodiment, the data management method further includes:

[0112] Determine the load level of each node in the distributed storage system through the preset node load monitoring indicators;

[0113] For nodes with excessive load, the data shards with excessive access frequency are identified as the shards to be migrated.

[0114] Migrate the shards to be migrated to nodes with sufficient load.

[0115] Specifically, taking into account the irrationality of the data shard storage distribution, for example, the access rate of data shards in some nodes is relatively high, which makes the load level of the node serious and reduces the node performance. Therefore, in the embodiment of the present invention, the load level of each node in the distributed storage system can be determined through preset node load monitoring indicators, and then for the nodes with excessive load levels, the data shards with excessive access frequency of the nodes are determined as the shards to be migrated, and finally the shards to be migrated are migrated to nodes with qualified load levels; in this way, the nodes with lighter load levels can share the load pressure of the nodes with heavier load levels, thereby improving the overall performance of the distributed storage system.

[0116] To better illustrate the embodiments of the present invention, please refer to Figure 3 , Figure 3 A flow chart of another data management method provided by the present invention. On the one hand, for the data to be stored, after "sharding the data from the time dimension" and "sharding the data from the space dimension" respectively, the data shards can be "merged through correlation analysis" and finally the data shards can be written (to the distributed storage system). On the other hand, the nodes in the distributed storage system can be monitored, the shards to be migrated can be determined according to the node load level, and the data shards can be migrated.

[0117] As an optional embodiment, the preset node load monitoring indicators include CPU usage, memory usage, and network bandwidth usage;

[0118] The load level of each node in the distributed storage system is determined by using preset node load monitoring indicators, including:

[0119] For any node in the distributed storage system, the preset node load monitoring indicators of the node are proportionally weighted to serve as the load level of the node.

[0120] Specifically, considering that the three indicators of central processing unit utilization, memory utilization and network bandwidth utilization can comprehensively evaluate the load level of a node, the preset node load monitoring indicators in the embodiment of the present invention include central processing unit utilization, memory utilization and network bandwidth utilization. Then, for any node in the distributed storage system, the preset node load monitoring indicators of the node are proportionally weighted as the load level of the node, thereby efficiently and accurately determining the load level of each node.

[0121] Of course, in addition to this specific form, the implementation method of "determining the load level of each node in the distributed storage system through preset node load monitoring indicators" can also be other types, and the embodiments of the present invention are not limited here.

[0122] As an optional embodiment, the data management method further includes:

[0123] For any data storage object to be stored, the number of replicas corresponding to the data type of the data storage object is used as the number of replicas of the data storage object;

[0124] Generate the corresponding number of copies for the data storage object and store them in the distributed storage system.

[0125] Specifically, considering that in a distributed storage system, the number of copies of the data storage object to be stored affects data security and storage pressure, and the importance of the data storage object is often related to the data type, in an embodiment of the present invention, for any data storage object to be stored, the number of copies corresponding to the data type to which the data storage object belongs can be used as the number of copies of the data storage object, and then the corresponding number of copies are generated for the data storage object and stored in the distributed storage system, thereby balancing data security and storage pressure.

[0126] When storing multiple copies of a data storage object in a distributed storage system, the storage can be as dispersed as possible. For example, multiple copies of the data storage object can be stored in nodes that are "located in different fault domains" and have "low network latency between each other."

[0127] As an optional embodiment, for any data storage object to be stored, the number of replicas corresponding to the data type of the data storage object as the number of replicas of the data storage object includes:

[0128] For any data storage object whose data type belongs to a preset key data type set, determining the number of copies of the data storage object according to the preset correspondence between the data type and the number of copies;

[0129] For any data storage object whose data type does not belong to the preset key data type set, the number of copies of the data storage object is determined based on the access frequency of the data type to which the data storage object belongs and the correspondence between the preset access frequency and the number of copies.

[0130] Specifically, considering that for certain key data types, it may be necessary to specifically specify the number of copies, while for non-key data types, the number of copies can be determined based on their access frequency, therefore, in an embodiment of the present invention, for any data storage object whose data type belongs to a preset set of key data types, the number of copies of the data storage object can be determined based on the correspondence between the preset data type and the number of copies; and for any data storage object whose data type does not belong to the preset set of key data types, the number of copies of the data storage object can be determined based on the access frequency of the data type to which the data storage object belongs and the correspondence between the preset access frequency and the number of copies, thereby being able to more accurately determine the appropriate number of copies for each data storage object and better balance data security and storage pressure.

[0131] Please refer to Figure 4 , Figure 4 A schematic diagram of the structure of a data management device provided by the present invention, the data management device comprising:

[0132] Memory 41, for storing computer programs;

[0133] The processor 42 is configured to implement the steps of the data management method in the aforementioned embodiment when executing a computer program.

[0134] For an introduction to the data management device provided by the embodiment of the present invention, please refer to the aforementioned embodiment of the data management method, and the embodiment of the present invention will not be described in detail here.

[0135] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the data management method in the aforementioned embodiment when executed by a processor.

[0136] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the aforementioned embodiment of the data management method, and the embodiment of the present invention will not be described in detail here.

[0137] Please refer to Figure 5 , Figure 5 This is a structural diagram of a computer-readable storage medium provided by the present invention. A computer program 52 is stored on the computer-readable storage medium 51. When the computer program 52 is executed by a processor, the steps of the data management method in the aforementioned embodiment are implemented.

[0138] For an introduction to the computer-readable storage medium provided in an embodiment of the present invention, please refer to the aforementioned embodiment of the data management method, and the embodiment of the present invention will not be described in detail here.

[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0140] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data management method, characterized in that: Applied to distributed storage systems, including: Determine multiple target time segments based on the segment characteristics of the data to be stored in the time dimension; Determine multiple target spatial regions based on the regional characteristics of the data to be stored in the spatial dimension; Slicing the data to be stored sequentially by the target time segment and the target spatial region to obtain a plurality of data slices, wherein a single data slice includes data of a single target time segment and a single target spatial region; For any pair of data shards, determine the target ratio of the pair of data shards based on the access data of the historical data shard pairs corresponding to the pair of data shards, where the target ratio is the ratio of the number of times the pair of data shards have been accessed simultaneously in history to the total number of accesses; For any pair of data shards whose target ratios meet the requirements, the data shards are centrally stored; For any data shard that is not stored, the data shard is stored in a node of the distributed storage system.

2. The data management method according to claim 1, wherein: Based on the segment characteristics of the data to be stored in the time dimension, multiple target time segments are determined, including: Each second preset time interval in the preset first preset time interval is used as a target time interval, wherein the collection of each second preset time interval is the first preset time interval, and each second preset time interval has no intersection.

3. The data management method according to claim 1, wherein: Based on the regional characteristics of the data to be stored in the spatial dimension, multiple target spatial regions are determined, including: Obtaining spatial information of each sub-data in the data to be stored; According to a preset region division strategy, the first spatial region covered by the spatial information is divided into a plurality of second spatial regions, and each second spatial region is used as a target spatial region.

4. The data management method according to claim 3, wherein: According to a preset region division strategy, the first spatial region covered by the spatial information is divided into a plurality of second spatial regions, and each second spatial region is used as a target spatial region, including: Divide the first spatial area covered by the spatial information into a plurality of second spatial areas based on a preset urban area granularity, and use each second spatial area as a target spatial area; The preset urban area granularity includes districts or streets.

5. The data management method according to claim 3, characterized in that: After obtaining the spatial information of each sub-data in the data to be stored, dividing the first spatial area covered by the spatial information into a plurality of second spatial areas according to a preset area division strategy, and before using each second spatial area as the target spatial area, the data management method further includes: Data cleaning is performed on the spatial information to filter out abnormal spatial information.

6. The data management method according to claim 1, wherein: After successively sharding the data to be stored by the target time segment and the target spatial region to obtain a plurality of data shards, for any data shard, before storing the data shard in a node of the distributed storage system, the data management method further includes: For any target time segment, determining the correlation between the data slices of each target spatial area in the target time segment; For any pair of data shards in the target spatial region within the same target time segment, if the correlation meets the criteria, the data shards in the pair of target spatial regions will be merged.

7. The data management method according to claim 1, wherein: For any pair of data shards whose target ratios meet the requirements, centrally storing the data shards includes: For any pair of data shards that meet the target ratio, if there is a node that meets the first preset constraint condition, the node that meets the first preset constraint condition is used as the target storage node, wherein the first preset constraint condition is: the storage load is lower than a first preset threshold; If there is no node that meets the first preset constraint, two nodes that meet the second preset constraint are used as target storage nodes, where the second preset constraint is: the network distance is lower than the second preset threshold and the storage load is lower than the third preset threshold, and the third preset threshold is greater than the first preset threshold; The pair of data shards that meet the target ratio are stored in the target storage node.

8. The data management method according to claim 1, wherein: The data management method further includes: Determine the load level of each node in the distributed storage system through the preset node load monitoring indicators; For the node with an excessive load, determining a data shard whose access frequency of the node exceeds the standard as a shard to be migrated; Migrate the shards to be migrated to nodes with a load level that meets the requirements.

9. The data management method according to claim 8, characterized in that: The preset node load monitoring indicators include CPU usage, memory usage, and network bandwidth usage; The load level of each node in the distributed storage system is determined by using preset node load monitoring indicators, including: For any node in the distributed storage system, the preset node load monitoring indicators of the node are proportionally weighted to serve as the load level of the node.

10. The data management method according to any one of claims 1 to 9, characterized in that: The data management method further includes: For any data storage object to be stored, the number of copies corresponding to the data type of the data storage object is used as the number of copies of the data storage object; Generate the corresponding number of copies for the data storage object and store them in the distributed storage system.

11. The data management method according to claim 10, characterized in that: For any data storage object to be stored, the number of replicas corresponding to the data type of the data storage object as the number of replicas of the data storage object includes: For any data storage object whose data type belongs to a preset key data type set, determining the number of copies of the data storage object according to a preset correspondence between the data type and the number of copies; For any data storage object whose data type does not belong to the preset key data type set, the number of copies of the data storage object is determined based on the access frequency of the data type to which the data storage object belongs and the correspondence between the preset access frequency and the number of copies.

12. A data management device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data management method according to any one of claims 1 to 11 when executing the computer program.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data management method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data management method according to any one of claims 1 to 11.

Images