Data storage method and system

Abstract

The present invention discloses a data storage method and system, and relates to the field of computer technology. The implementation method of the method includes: grouping multiple servers to obtain multiple server groups, grouping data to be stored to obtain multiple data groups to be stored; determining the storage mapping relationship between the data groups to be stored and the server groups, and storing according to the storage mapping relationship; determining the first server group and the second server group according to the load value of each server group; determining the first target data stored on the first server group, and determining the second target data stored on the second server group; updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group. This implementation method can effectively alleviate storage pressure, save server resources, and can dynamically adjust the storage location of data, automatically disperse hot data and access pressure, and achieve load balancing.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method and system. Background Art

[0002] With the development of Internet technology, data has exploded, and data storage technology has become particularly important for dealing with massive data. Specifically, data storage has requirements such as large access volume, large data volume, and high concurrency. At present, the commonly used data storage system is a centralized data system, that is, data is stored on each server. This storage system consumes a lot of server memory resources, and storing the same data on each server is also a waste of storage resources. In short, this storage system can no longer meet the needs of large-scale data storage. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data storage method and system, which can solve the problem that the server memory cannot store all the data in the case of large data volumes, effectively alleviate the server memory pressure, save server resources, and ensure the reliability, security and access efficiency of data storage; at the same time, it can automatically adjust the data storage location, automatically disperse hot data and access pressure, and achieve load balancing.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a data storage method is provided, the method comprising a plurality of servers, and the method further comprising:

[0005] The multiple servers are grouped to obtain multiple server groups, and the data to be stored are grouped according to a preset rule to obtain multiple data groups to be stored;

[0006] Determining a storage mapping relationship between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relationship;

[0007] Determine a load value of each server group, and determine a first server group and a second server group from all server groups according to the load value;

[0008] Determine first target data stored on the first server group according to a preset first condition, and determine second target data stored on the second server group according to a preset second condition;

[0009] The storage mapping relationship is updated, the first target data is migrated to the second server group, and the second target storage data is migrated to the first server group.

[0010] Optionally, determining the load value of each server group includes: for each server group, determining the load index value of each server in the server group; determining the load value of each server based on the load index value and its corresponding weight; and determining the load value of the server group based on the load value of each server.

[0011] Optionally, the load indicator value includes one or more of the following: CPU usage, memory usage and interface response time.

[0012] Optionally, determining the first server group and the second server group from all server groups according to the load value includes: taking the server group whose load value in all server groups is greater than or equal to a preset threshold as the first server group; determining the load average value according to the load value of each server group; and taking the server group whose load value in all server groups is less than the load average value as the second server group.

[0013] Optionally, according to a preset first condition, determining the first target data stored on the first server group includes:

[0014] determining the amount of first target data;

[0015] When the number of the first server group is 1, first data is obtained from the first server group, and the number of the first data is equal to the number of the first target data;

[0016] When the number of first server groups is greater than 1, determining a difference ratio between each first server group according to a difference between a load value of each first server group and the load average value; obtaining first data from each first server group according to the difference ratio, and using the first data as first target data, wherein the number ratio between the first data is equal to the difference ratio, and the sum of the numbers of the first data is equal to the number of the first target data;

[0017] According to the preset second condition, determining the second target data stored on the second server group includes:

[0018] Determining the amount of second target data, wherein the amount of the second target data is equal to the amount of the first target data;

[0019] When the number of the second server group is 1, obtaining second data from the second server group, the number of the second data being equal to the number of the second target data;

[0020] When the number of second server groups is greater than 1, the difference ratio between each second server group is determined based on the difference between the load value of each second server group and the load average value; based on the difference ratio, second data is obtained from each second server group, and the second data is used as second target data, wherein the quantity ratio between the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

[0021] Optionally, the first data is determined according to the following process:

[0022] determining a number of accesses to data stored on the first server group;

[0023] sorting the data stored in the first server group according to the number of accesses, wherein the greater the number of accesses, the higher the ranking of the corresponding data;

[0024] Using at least one data with a top ranking among the data stored on the first server group as first data;

[0025] The second data is determined according to the following process:

[0026] determining a number of accesses to data stored on the second server group;

[0027] sorting the data stored on the second server group according to the number of accesses, wherein the smaller the number of accesses, the higher the ranking of the corresponding data;

[0028] At least one data with a top ranking among the data stored in the second server group is used as the second data.

[0029] Optionally, updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group includes:

[0030] The first target data with the highest number of accesses is migrated to the second server group with the lowest load value, and the second target data with the lowest number of accesses is migrated to the first server group with the highest load value.

[0031] To achieve the above object, according to another aspect of an embodiment of the present invention, a data storage system is provided, the system comprising a plurality of servers, and the system further comprising:

[0032] A grouping module, used to group the multiple servers to obtain multiple server groups, and group the data to be stored according to a preset rule to obtain multiple data groups to be stored;

[0033] A storage module, used to determine a storage mapping relationship between the data group to be stored and the server group, and store the data group to be stored on the corresponding server group according to the storage mapping relationship;

[0034] A server group determination module, used to determine a load value of each server group, and determine a first server group and a second server group from all server groups according to the load value;

[0035] a target data determination module, configured to determine first target data stored on the first server group according to a preset first condition, and to determine second target data stored on the second server group according to a preset second condition;

[0036] The update module is used to update the storage mapping relationship, migrate the first target data to the second server group, and migrate the second target storage data to the first server group.

[0037] Optionally, the server group determination module is also used to: for each server group, determine the load index value of each server in the server group; determine the load value of each server based on the load index value and its corresponding weight; determine the load value of the server group based on the load value of each server.

[0038] Optionally, the load indicator value includes one or more of the following: CPU usage, memory usage and interface response time.

[0039] Optionally, the server group determination module is also used to: take the server group whose load value is greater than or equal to a preset threshold among all server groups as the first server group; determine the load average value according to the load value of each server group; and take the server group whose load value is less than the load average value among all server groups as the second server group.

[0040] Optionally, the target data determination module is further used to:

[0041] determining the amount of first target data;

[0042] When the number of the first server group is 1, first data is obtained from the first server group, and the number of the first data is equal to the number of the first target data;

[0043] When the number of first server groups is greater than 1, determining a difference ratio between each first server group according to a difference between a load value of each first server group and the load average value; obtaining first data from each first server group according to the difference ratio, and using the first data as first target data, wherein the number ratio between the first data is equal to the difference ratio, and the sum of the numbers of the first data is equal to the number of the first target data;

[0044] Determining the amount of second target data, wherein the amount of the second target data is equal to the amount of the first target data;

[0045] When the number of the second server group is 1, obtaining second data from the second server group, the number of the second data being equal to the number of the second target data;

[0046] When the number of second server groups is greater than 1, the difference ratio between each second server group is determined based on the difference between the load value of each second server group and the load average value; based on the difference ratio, second data is obtained from each second server group, and the second data is used as second target data, wherein the quantity ratio between the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

[0047] Optionally, the target data determination module is further used to:

[0048] determining a number of accesses to data stored on the first server group;

[0049] sorting the data stored in the first server group according to the number of accesses, wherein the greater the number of accesses, the higher the ranking of the corresponding data;

[0050] Using at least one data with a top ranking among the data stored on the first server group as first data;

[0051] determining a number of accesses to data stored on the second server group;

[0052] sorting the data stored on the second server group according to the number of accesses, wherein the smaller the number of accesses, the higher the ranking of the corresponding data;

[0053] At least one data with a top ranking among the data stored in the second server group is used as the second data.

[0054] Optionally, the update module is further used to: migrate the first target data with the highest number of accesses to the second server group with the lowest load value, and migrate the second target data with the lowest number of accesses to the first server group with the highest load value.

[0055] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method of an embodiment of the present invention.

[0056] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data storage method of the embodiment of the present invention is implemented.

[0057] An embodiment of the above invention has the following advantages or beneficial effects: by grouping the multiple servers to obtain multiple server groups, the data to be stored are grouped according to preset rules to obtain multiple data groups to be stored; determining the storage mapping relationship between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relationship, it can solve the problem that the server memory cannot store all the data in the case of large data volume, effectively alleviate the server memory pressure, save server resources, and ensure the reliability, security and access efficiency of data storage; by determining the load value of each server group, and determining the first server group and the second server group from all server groups according to the load value; according to the preset first condition, determining the first target data stored on the first server group, and according to the preset second condition, determining the second target data stored on the second server group; updating the storage mapping relationship, migrating the first target data to the second server group and migrating the second target storage data to the first server group, it can automatically adjust the data storage location, automatically disperse hot data and access pressure, and achieve load balancing.

[0058] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0060] Figure 1 is a schematic diagram of the main process of the data storage method according to an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of a sub-process of a data storage method according to an embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of a sub-process of a data storage method according to an embodiment of the present invention;

[0063] Figure 4 is a schematic diagram of main modules of a data storage system according to an embodiment of the present invention;

[0064] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0065] Figure 6It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0067] Figure 1 FIG. 1 is a schematic diagram of the main process of a data storage method according to an embodiment of the present invention, wherein the data storage method includes multiple servers. Figure 1 As shown, the method includes:

[0068] Step S101: Group the multiple servers to obtain multiple server groups, and group the data to be stored according to a preset rule to obtain multiple data groups to be stored.

[0069] In this embodiment, the number of server groups and the number of data groups to be stored can be flexibly set, and the present invention does not limit this. It is worth noting that each server group includes at least two servers to store the same data to be stored on at least two servers, thereby ensuring that the data to be stored has a backup. When grouping the data to be stored, the grouping rules can be flexibly set according to the specific content of the data, and the present invention does not limit this. For example, if the data to be stored is data related to a commodity, it can be grouped according to the category to which the commodity belongs. In other optional embodiments, the data to be stored can also be named with a unique identifier (such as ID), and then grouped according to the unique identifier, or grouped according to a Hash algorithm. Among them, the Hash algorithm can convert a data into a flag, and this flag has a very close relationship with each byte of the source data.

[0070] As a specific example, assuming that there are 20 servers in total, they can be divided into 4 server groups: server group A, server group B, server group C and server group D. Each server group has 5 servers. Assuming that the data to be stored is commodity data of 160 categories, the data to be stored can be divided into 4 groups according to the categories: data group to be stored T1, data group to be stored T2, data group to be stored T3 and data group to be stored T4. Each data group to be stored contains commodity data of 40 categories.

[0071] Step S102: determining a storage mapping relationship between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relationship.

[0072] The storage mapping relationship is used to represent the storage relationship between the data group to be stored and the server group. When storing the data group to be stored, all the data to be stored in the data group to be stored can be stored on each server in the corresponding server group, that is, if there are 5 servers in the corresponding server group, the same data group to be stored is stored on each server, which can ensure that the data has a backup and can also disperse the access request to different servers when accessing the data later to reduce the access pressure.

[0073] Continuing with the above example, the storage mapping relationship may be as shown in Table 1 below:

[0074] Table 1:

[0075] Data group to be stored Storage Location Data group to be stored T1 Server Group A Data group to be stored T2 Server Group B Data group to be stored T3 Server Group C Data group to be stored T4 Server Group D

[0076] In this example, each server group has five servers, and one of the five servers can be set as a central server, which is responsible for the distribution, storage, and synchronization of the data group to be stored.

[0077] In a specific embodiment, after determining the storage mapping relationship, the storage mapping relationship can be recorded in the configuration file, that is, the IP of the data group to be stored and its corresponding server group is recorded in the configuration file, and the IP of the central server of each server group can also be recorded. More specifically, two tables can be set before storing data, one of which is used to record the state of data storage of each server (this table can be called Table A), and the other is used to record the storage state of each data group to be stored (this table can be called Table B). The role of Table B is to manually intervene in the loading state of a certain data to achieve flexible control of individual data. At the beginning of storing data, the content of the configuration file is first obtained, and the central server of each server group determines the data group that needs to be stored by this server group according to the content of the configuration file, and then obtains the data group that needs to be stored by this server group and whose status in Table B is "not stored", distributes it to each server in the server group (including the central server) through the data interface, and then updates the status of the data group in Table B to "storing". When storing data, the central server also needs to update the status of the corresponding server in Table A to "storing". When the server storage is completed, the status of the corresponding server in Table A is updated to "storage completed". If the status corresponding to all servers in the server group in Table A is "Storage Completed", the status of the data group in Table B is updated to "Completed".

[0078] After steps S101 and S102, the data group to be stored has been stored on the corresponding server group. Subsequently, the data will be randomly accessed, and the load of each server group will be different. It may happen that the access frequency of a certain data on a certain server group far exceeds that of other data, that is, hot data appears, resulting in excessive access pressure on the server group, while the access pressure of other server groups is relatively small. Therefore, in steps S103-S105, it is necessary to disperse the hot data and evenly load each server group. The specific process is as follows.

[0079] Step S103: determining the load value of each server group, and determining a first server group and a second server group from all server groups according to the load value.

[0080] Among them, Figure 2 As shown, the load value of each server group can be determined according to the following process:

[0081] Step S201: for each server group, determine the load index value of each server in the server group; wherein the load index value includes one or more of the following: cpu usage, memory usage and interface response time. The above load index values ​​can be obtained by existing methods, and the present invention will not be repeated here.

[0082] Step S202: Determine the load value of each server according to the load index value and its corresponding weight; specifically, each load index value and its corresponding weight may be multiplied, and then the products are added together to obtain the load value of the server.

[0083] Step S203: Determine the load value of the server group according to the load value of each server. Specifically, the load value of all servers in the server group may be added together to obtain the load value of the server group.

[0084] After the load values ​​of the server groups are obtained, the first server group and the second server group may be determined from all the server groups according to the load values. The specific process may include:

[0085] The server group whose load value is greater than or equal to the preset threshold value among all server groups is taken as the first server group; wherein the preset threshold value can be flexibly set according to the application scenario, and the present invention does not limit it here. In this embodiment, when the load value of a server group is greater than the preset threshold value, it means that the load capacity of the server group has reached a certain limit, and other server groups are required to share the load.

[0086] Determine the load average value based on the load value of each server group;

[0087] The server group whose load value is smaller than the load average value among all the server groups is taken as the second server group.

[0088] For example, the preset threshold is 5, the load value of server group A is 6, the load value of server group B is 4, the load value of server group C is 8, and the load value of server group D is 2, then the load average value is 5, the first server group is server group A and server group C, and the second server group is server group B and server group D.

[0089] In this embodiment, the number of the first server group and the second server group can be one or more. When the number of the first server group and the second server group is one respectively, the storage locations of the first target data and the second target data can be directly exchanged. When the number of the first server group is multiple and the number of the second server group is one, or when the number of the second server group is multiple and the number of the first server group is one, or when the number of the first server group and the second server group are both multiple, the first target data and the second target data need to be distributed among the multiple first server groups and the second server groups according to certain rules. Specifically, as shown in step S104.

[0090] Step S104: determining first target data stored on the first server group according to a preset first condition, and determining second target data stored on the second server group according to a preset second condition.

[0091] The process of determining the first target data stored on the first server group according to the preset first condition includes:

[0092] (1) Determine the quantity of the first target data. In this embodiment, the quantity of the first target data can be flexibly set according to the total amount of data stored on each server and the application scenario. For example, if the data stored on each server is related data of 40 categories of goods, the quantity of the first target data can be set to 40×10%=4, that is, the first target data of 4 categories needs to be determined.

[0093] (2) When the number of the first server group is 1, the first data is obtained from the first server group, and the number of the first data is equal to the number of the first target data. That is, when there is only one first server group, it is necessary to determine the first data of four categories on the server group as the first target data, wherein the process of determining the first data is described below.

[0094] (3) When the number of first server groups is greater than 1, determine the difference ratio between each first server group according to the difference between the load value of each first server group and the load average value; obtain first data from each first server group according to the difference ratio, and use the first data as the first target data, wherein the number ratio between the first data is equal to the difference ratio, and the sum of the numbers of the first data is equal to the number of the first target data.

[0095] Continuing with the above example, the first server group is server group A and server group C. The difference between the load value 6 of server group A and the load average value 5 is 1, and the difference between the load value 8 of server group C and the load average value 5 is 3. The difference ratio between the first server groups (that is, the difference ratio between server group A and server group C) is 1:3. Because the number of first target data is 4, it is necessary to obtain commodity data of 1 category from server group A as the first data, and obtain commodity data of 3 categories from server group C as the first data.

[0096] Similarly, according to the preset second condition, the process of determining the second target data stored on the second server group includes:

[0097] (1) Determine the number of second target data, wherein the number of the second target data is equal to the number of the first target data. For example, if the number of the first target data is commodity data of 4 categories, then the number of the second target data is also commodity data of 4 categories. In this embodiment, the number of the second target data is set to be equal to the number of the first target data so that the amount of data stored on each server group is equal after the storage mapping relationship is updated and the data storage location is reallocated.

[0098] (2) When the number of the second server group is 1, the second data is obtained from the second server group, and the number of the second data is equal to the number of the second target data. That is, when there is only one second server group, it is necessary to determine the second data of 4 categories on the server group as the second target data, wherein the process of determining the second data is described below.

[0099] (3) When the number of second server groups is greater than 1, determine the difference ratio between each second server group based on the difference between the load value of each second server group and the load average value; obtain second data from each second server group based on the difference ratio, and use the second data as second target data, wherein the number ratio between the second data is equal to the difference ratio, and the sum of the numbers of the second data is equal to the number of the second target data.

[0100] Continuing with the above example, the second server group is server group B and server group D. The difference between the load value 4 of server group B and the load average value 5 is 1, the difference between the load value 2 of server group D and the load average value 5 is 3, and the difference ratio between the second server groups (that is, the difference ratio between server group B and server group D) is 1:3, then it is necessary to obtain commodity data of 1 category from server group B as the second data, and obtain commodity data of 3 categories from server group D as the second data.

[0101] Further, such as Figure 3 As shown, the first data can be determined according to the following process:

[0102] Step S301: Determine the number of accesses to data stored on the first server group; for the same data, the data is stored on all servers in the corresponding server group, so the number of accesses to the data is equal to the sum of the number of accesses to the data on all servers in the server group.

[0103] Step S302: sorting the data stored in the first server group according to the number of accesses, wherein the greater the number of accesses, the higher the ranking of the corresponding data;

[0104] Step S303: Take at least one of the top-ranked data stored on the first server group as the first data. Specifically, the data can be selected according to the number of first data to be obtained by each first server group as determined above. Continuing with the above example, for server group A, it is necessary to obtain commodity data of 1 category from the server group as the first target data, and the data ranked first is taken as the first data; for server group C, it is necessary to obtain commodity data of 3 categories from the server group as the first target data, and the data ranked first, second and third are taken as the first data.

[0105] Furthermore, the second data may be determined according to the following process:

[0106] Determine the number of accesses to data stored on the second server group; for the same data, the data is stored on all servers in the corresponding server group, so the number of accesses to the data is equal to the sum of the number of accesses to the data on all servers in the server group.

[0107] sorting the data stored on the second server group according to the number of accesses, wherein the smaller the number of accesses, the higher the ranking of the corresponding data;

[0108] At least one of the top-ranked data in the data stored on the second server group is used as the second data. Specifically, the data can be selected according to the number of second data that each second server group needs to obtain as determined above. Continuing with the above example, for server group B, it is necessary to obtain product data of 1 category from the server group as the second target data, then the data ranked first is taken as the second data; for server group D, it is necessary to obtain product data of 3 categories from the server group as the second target data, then the data ranked first, second and third is taken as the second data.

[0109] Step S105: updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group.

[0110] In this embodiment, the first target data is the data with the highest number of accesses on the server group with the largest load, which can be called hot data. The second target data is the data with the lowest number of accesses on the server group with the smallest load. By exchanging the storage locations of the two, the load of each server group can be balanced, the hot data can be dispersed, and the stability and efficiency of data access can be improved.

[0111] Since there may be multiple first target data, second target data, first server group and second server group, it is necessary to follow certain rules when updating the storage mapping relationship. Specifically, the first target data with the highest number of accesses is migrated to the second server group with the lowest load value, and the second target data with the lowest number of accesses is migrated to the first server group with the highest load value. By migrating the first target data with the highest number of accesses to the second server group with the lowest load value, the load value of the second server group can be increased, and by migrating the second target data with the lowest number of accesses to the first server group with the highest load value, the load value of the first server group can be reduced, thereby dispersing hot data, balancing the load of each server group, and improving data access stability and efficiency.

[0112] The data storage method of the embodiment of the present invention obtains multiple server groups by grouping the multiple servers, and obtains multiple data groups to be stored by grouping the data to be stored according to preset rules; determines the storage mapping relationship between the data group to be stored and the server group, and stores the data group to be stored on the corresponding server group according to the storage mapping relationship, which can solve the problem that the server memory cannot store all the data in the case of large data volume, effectively alleviates the server memory pressure, saves server resources, and ensures the reliability, security and access efficiency of data storage; determines the load value of each server group, and determines the first server group and the second server group from all server groups according to the load value; determines the first target data stored on the first server group according to a preset first condition, and determines the second target data stored on the second server group according to a preset second condition; updates the storage mapping relationship to store the first target data on the second server group, and stores the second target storage data on the first server group, which can automatically adjust the data storage location, automatically disperse hot data and access pressure, and achieve load balancing.

[0113] In order to make the data storage method of the embodiment of the present invention clearer, the following example is used for description.

[0114] (1) Assume that there are 20 servers in total. The 20 servers are divided into 4 groups, each with 5 servers (hereinafter referred to as a shard). Then, a server is set in each shard as a central server (also called a leader host). The central server is responsible for data distribution, loading (i.e. storage) and synchronization of data storage task completion. There are 160 categories of product data that need to be stored. The data to be stored is grouped according to the category and divided into 4 data groups, each with 40 categories of product data.

[0115] (2) Set up the storage mapping relationship between the data group to be stored and each shard. Specifically, the commodity data of each category in the data group to be stored and the corresponding shard information can be written into the configuration file, that is, the category of the data group 1 to be stored, the server IP corresponding to shard 1 and the IP of the leader host are written into the configuration file. The database sets up two tables, one for recording the status of each server loading data in the future (Table A), with one server IP and one category ID per row. The other records the loading status of each category of data (Table B).

[0116] (4) The data storage task starts running, obtains the content of the configuration file, and the leader host of each shard starts to obtain the product data that needs to be loaded in this shard and whose status in Table B is "Not loaded", distributes it to each server in the shard (including the local machine) through the interface, and then updates the status of the corresponding category in Table B to "Loading". Each server stores the received category ID in the cache queue, and then starts multiple threads to consume the data in the queue.

[0117] (5) Before loading the product data, call the leader host's interface and notify it to update the status of the corresponding record in Table A to "loading". Then start loading the product data under each category ID. After loading is complete, notify the leader host to update the status of the corresponding record in Table A to "loading completed".

[0118] (6) The leader host periodically scans the loading status of each server in the shard in Table A according to the category ID. If all are "loaded", the status of the corresponding category ID in Table B is updated to "completed". The role of Table B is to manually intervene in the loading status of the product data of a certain category to flexibly control the loading of product data under individual categories.

[0119] (7) After steps (1) to (6) have completed loading the data to be stored, each shard will continuously monitor the access frequency of the data during the data access process, and calculate the number of times each category is accessed through memory. When the server load capacity of the shard reaches a certain limit (i.e., when the server load value is greater than or equal to the preset threshold), each server in the shard will count the T category IDs with the highest number of accesses (T = number of categories loaded on the server * 10%). Then the interface of the leader host is called and synchronized to the leader host.

[0120] As an example, the preset threshold is 7, and the load values ​​of shard 1, shard 2, shard 3 and shard 4 are 6, 4, 8, and 2 respectively. Then, each server of shard 3 will sort the number of accesses to the 40 categories of product data stored thereon in descending order, take the 4 category IDs with the highest number of accesses and send them to the leader host. The leader host will merge the number of accesses for the same category ID, and finally obtain the 4 category IDs with the highest number of accesses.

[0121] (8) When the load capacity reaches a certain limit, the shard calls the interface of each shard leader host to query the load status of each shard, find out the M shards with the lowest load (M = the number of shards below the cluster load average), calculate the difference between the load values ​​of the M shards and the cluster load average, and calculate the number of categories allocated to each shard based on the difference ratio between the M shards (giving priority to categories with high access times), and then allocate the categories to each shard by modifying the configuration file.

[0122] For example, the leader host of shard 3 queries the load of shards 1, 2, and 4 through the interface, and calculates the load average value to be (6+4+8+2) / 4=5. Then shards 2 and shard 4 are the shards with loads below the cluster average value, M=2. Calculate the difference ratio (5-4):(5-2)=1:3. Then the number of categories to be obtained by shard 2 is (1 / 4)*4=1, and the number of categories to be obtained by shard 4 is (3 / 4)*4=3. When allocating, the data with the highest number of accesses is preferentially allocated to the shard with the lowest load, that is, the data with the highest number of accesses is preferentially allocated to shard 4.

[0123] (9) The shards assigned to the new categories will perform steps (4) to (6) to load. At the same time, the leader host of each shard will distribute the X categories with the lowest number of accesses on their respective shards (the value of X is equal to the number of new categories assigned) to the shards with high load in step (7). The shards with high load will also perform steps (4) to (6) to load data.

[0124] For example, shard 2 is assigned to 1 category, so the leader host of shard 2 obtains a category with the lowest number of visits from the 5 servers in its shard, and assigns it to shard 3 through the leader host interface of shard 3. Shard 4 performs the same operation to find the 3 categories with the lowest number of visits, and assigns them to shard 3 through the leader host interface of shard 3.

[0125] (10) After the high-load shard completes loading new data or the low-load shard completes loading new data, the storage mapping relationship is updated, and the leader host interface of the high-load shard and the low-load shard is called to delete the data.

[0126] (11) At this point, the hotspot data transfer is completed, reducing the load of the shards. Steps (7)-(10) can be performed periodically during the server operation.

[0127] Figure 4 FIG. 4 is a schematic diagram of the main modules of a data storage system 400 according to an embodiment of the present invention, wherein the system 400 includes a plurality of servers. Figure 4 As shown, the data storage system 400 also includes:

[0128] The grouping module 401 is used to group the multiple servers to obtain multiple server groups, and group the data to be stored according to a preset rule to obtain multiple data groups to be stored;

[0129] The storage module 402 is used to determine the storage mapping relationship between the data group to be stored and the server group, and store the data group to be stored on the corresponding server group according to the storage mapping relationship;

[0130] A server group determination module 403, configured to determine a load value of each server group, and determine a first server group and a second server group from all server groups according to the load value;

[0131] The target data determination module 404 is used to determine the first target data stored on the first server group according to a preset first condition, and to determine the second target data stored on the second server group according to a preset second condition;

[0132] The updating module 405 is used to update the storage mapping relationship, migrate the first target data to the second server group and migrate the second target storage data to the first server group.

[0133] Optionally, the server group determination module 403 is also used to: for each server group, determine the load index value of each server in the server group; determine the load value of each server based on the load index value and its corresponding weight; determine the load value of the server group based on the load value of each server.

[0134] Optionally, the load indicator value includes one or more of the following: CPU usage, memory usage and interface response time.

[0135] Optionally, the server group determination module 403 is also used to: take the server group whose load value is greater than or equal to a preset threshold among all server groups as the first server group; determine the load average value according to the load value of each server group; and take the server group whose load value is less than the load average value among all server groups as the second server group.

[0136] Optionally, the target data determination module 404 is further configured to:

[0137] determining the amount of first target data;

[0138] When the number of the first server group is 1, first data is obtained from the first server group, and the number of the first data is equal to the number of the first target data;

[0139] When the number of first server groups is greater than 1, determining a difference ratio between each first server group according to a difference between a load value of each first server group and the load average value; obtaining first data from each first server group according to the difference ratio, and using the first data as first target data, wherein the number ratio between the first data is equal to the difference ratio, and the sum of the numbers of the first data is equal to the number of the first target data;

[0140] Determining the amount of second target data, wherein the amount of the second target data is equal to the amount of the first target data;

[0141] When the number of the second server group is 1, obtaining second data from the second server group, the number of the second data being equal to the number of the second target data;

[0142] When the number of second server groups is greater than 1, the difference ratio between each second server group is determined based on the difference between the load value of each second server group and the load average value; based on the difference ratio, second data is obtained from each second server group, and the second data is used as second target data, wherein the quantity ratio between the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

[0143] Optionally, the target data determination module 404 is further configured to:

[0144] determining a number of accesses to data stored on the first server group;

[0145] sorting the data stored in the first server group according to the number of accesses, wherein the greater the number of accesses, the higher the ranking of the corresponding data;

[0146] Using at least one data with a top ranking among the data stored on the first server group as first data;

[0147] determining a number of accesses to data stored on the second server group;

[0148] sorting the data stored on the second server group according to the number of accesses, wherein the smaller the number of accesses, the higher the ranking of the corresponding data;

[0149] At least one data with a top ranking among the data stored in the second server group is used as the second data.

[0150] Optionally, the updating module 405 is further configured to: migrate the first target data with the highest number of accesses to the second server group with the lowest load value, and migrate the second target data with the lowest number of accesses to the first server group with the highest load value.

[0151] The data storage system of the embodiment of the present invention obtains multiple server groups by grouping the multiple servers, and obtains multiple data groups to be stored by grouping the data to be stored according to preset rules; determines the storage mapping relationship between the data group to be stored and the server group, and stores the data group to be stored on the corresponding server group according to the storage mapping relationship, which can solve the problem that the server memory cannot store all the data in the case of large data volume, effectively alleviates the server memory pressure, saves server resources, and ensures the reliability, security and access efficiency of data storage; determines the load value of each server group, and determines the first server group and the second server group from all server groups according to the load value; determines the first target data stored on the first server group according to a preset first condition, and determines the second target data stored on the second server group according to a preset second condition; updates the storage mapping relationship, migrates the first target data to the second server group and migrates the second target storage data to the first server group, which can automatically adjust the data storage location, automatically disperse hot data and access pressure, and realize load balancing.

[0152] The above data storage system can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0153] Figure 5 An exemplary system architecture 500 is shown to which the data storage method or data storage system according to the embodiment of the present invention can be applied.

[0154] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0155] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0156] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0157] The server 505 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information and product information) to the terminal device.

[0158] It should be noted that the data storage method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data storage system is generally set in the server 505 .

[0159] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0160] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0161] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0162] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0163] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0164] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a sending module, an acquisition module, a determination module, and a first processing module. The names of these modules do not, in some cases, constitute a limitation on the unit itself. For example, the sending module may also be described as a "module for sending a picture acquisition request to the connected server."

[0167] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0168] Grouping multiple servers to obtain multiple server groups, and grouping the data to be stored according to a preset rule to obtain multiple data groups to be stored;

[0169] Determining a storage mapping relationship between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relationship;

[0170] Determine a load value of each server group, and determine a first server group and a second server group from all server groups according to the load value;

[0171] Determine first target data stored on the first server group according to a preset first condition, and determine second target data stored on the second server group according to a preset second condition;

[0172] The storage mapping relationship is updated, the first target data is migrated to the second server group, and the second target storage data is migrated to the first server group.

[0173] The technical solution of the embodiment of the present invention is to obtain multiple server groups by grouping the multiple servers, and to obtain multiple data groups to be stored by grouping the data to be stored according to preset rules; determine the storage mapping relationship between the data group to be stored and the server group, and store the data group to be stored on the corresponding server group according to the storage mapping relationship, which can solve the problem that the server memory cannot store all the data in the case of large data volume, effectively alleviate the server memory pressure, save server resources, and ensure the reliability, security and access efficiency of data storage; by determining the load value of each server group, and determining the first server group and the second server group from all server groups according to the load value; according to a preset first condition, determine the first target data stored on the first server group, and according to a preset second condition, determine the second target data stored on the second server group; update the storage mapping relationship, migrate the first target data to the second server group and migrate the second target storage data to the first server group, which can automatically adjust the data storage location, automatically disperse hot data and access pressure, and achieve load balancing.

[0174] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data storage method, characterized in that: The method includes a plurality of servers, the method further comprising: Grouping the multiple servers to obtain multiple server groups, identifying specific content of the data to be stored, and selecting a preset rule to group the data to be stored to obtain multiple data groups to be stored; Determining a storage mapping relationship between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relationship; determining a load value of each server group, and determining a first server group and a second server group from all server groups according to the load value; Determine, based on a preset first condition, first target data stored on the first server group, and determine, based on a preset second condition, second target data stored on the second server group; wherein, based on the preset first condition, determining the first target data stored on the first server group includes: determining the quantity of the first target data; when the quantity of the first server group is 1, obtaining first data from the first server group, the quantity of the first data being equal to the quantity of the first target data; when the quantity of the first server group is greater than 1, determining a difference ratio between each first server group based on a difference between a load value of each first server group and a load average value; based on the difference ratio, obtain first data from each first server group and use the first data as the first target data, wherein the quantity ratio between the first data is equal to the difference ratio, and the sum of the quantities of the first data is equal to the quantity of the first target data; Determining, according to a preset second condition, the second target data stored on the second server group includes: determining the amount of the second target data, wherein the amount of the second target data is equal to the amount of the first target data; when the number of the second server groups is 1, obtaining second data from the second server group, wherein the amount of the second data is equal to the amount of the second target data; when the number of the second server groups is greater than 1, determining a difference ratio between each second server group based on a difference between a load value of each second server group and the load average value; obtaining second data from each second server group based on the difference ratio, and using the second data as the second target data, wherein the amount ratio between the second data is equal to the difference ratio, and the sum of the amounts of the second data is equal to the amount of the second target data; The storage mapping relationship is updated, the first target data is migrated to the second server group, and the second target storage data is migrated to the first server group.

2. The method according to claim 1, characterized in that Determining the load value of each server group includes: For each server group, determining a load index value of each server in the server group; Determine the load value of each server according to the load index value and its corresponding weight; The load value of the server group is determined according to the load value of each server.

3. The method according to claim 2, characterized in that The load indicator value includes one or more of the following: CPU usage, memory usage and interface response time.

4. The method according to claim 2, characterized in that Determining the first server group and the second server group from all server groups according to the load value includes: The server group whose load value is greater than or equal to the preset threshold among all the server groups is taken as the first server group; Determine the load average based on the load value of each server group; The server group whose load value is smaller than the load average value among all the server groups is used as the second server group.

5. The method according to claim 1, wherein The first data is determined according to the following process: determining a number of accesses to data stored on the first server group; sorting the data stored on the first server group according to the number of accesses, wherein the data with a greater number of accesses is ranked higher; using at least one data ranked first among the data stored on the first server group as first data; The second data is determined according to the following process: determining a number of accesses to data stored on the second server group; sorting the data stored on the second server group according to the number of accesses, wherein the smaller the number of accesses, the higher the ranking of the corresponding data; At least one data with a higher ranking among the data stored in the second server group is used as the second data.

6. The method according to claim 5, characterized in that Updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group includes: The first target data with the highest number of accesses is migrated to the second server group with the lowest load value, and the second target data with the lowest number of accesses is migrated to the first server group with the highest load value.

7. A data storage system, characterized in that: The system includes a plurality of servers, and further includes: a grouping module, configured to group the plurality of servers to obtain a plurality of server groups, identify specific content of the data to be stored, and select a preset rule to group the data to be stored to obtain a plurality of data groups to be stored; a storage module, configured to determine a storage mapping relationship between the data group to be stored and the server group, and store the data group to be stored on the corresponding server group according to the storage mapping relationship; A server group determination module, configured to determine a load value of each server group, and determine a first server group and a second server group from all server groups according to the load value; A target data determination module, configured to determine, based on a preset first condition, first target data stored on a first server group, and to determine, based on a preset second condition, second target data stored on a second server group; wherein, based on the preset first condition, determining the first target data stored on the first server group comprises: determining the quantity of the first target data; when the quantity of the first server group is 1, obtaining first data from the first server group, the quantity of the first data being equal to the quantity of the first target data; when the quantity of the first server group is greater than 1, determining a difference ratio between each first server group based on a difference between a load value of each first server group and a load average value; based on the difference ratio, obtaining first data from each first server group, and using the first data as the first target data, wherein the quantity ratio between the first data is equal to the difference ratio, and the sum of the quantities of the first data is equal to the quantity of the first target data; Determining, according to a preset second condition, the second target data stored on the second server group includes: determining the amount of the second target data, wherein the amount of the second target data is equal to the amount of the first target data; when the number of the second server groups is 1, obtaining second data from the second server group, wherein the amount of the second data is equal to the amount of the second target data; when the number of the second server groups is greater than 1, determining a difference ratio between each second server group based on a difference between a load value of each second server group and the load average value; obtaining second data from each second server group based on the difference ratio, and using the second data as the second target data, wherein the amount ratio between the second data is equal to the difference ratio, and the sum of the amounts of the second data is equal to the amount of the second target data; An updating module is configured to update the storage mapping relationship, migrate the first target data to the second server group, and migrate the second target storage data to the first server group.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images