Data storage method and device

Abstract

The invention discloses a data storage method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the steps of obtaining to-be-stored data of a target first user, and determining a grouping serial number of the target first user based on a pre-configured grouping result; determining a date dimension according to the data volume acceleration of the to-be-stored data of the target first user; determining a routing strategy of the target first user based on the grouping serial number and the date dimension; and storing the to-be-stored data of the target first user in different libraries and different tables according to the routing policy. According to the embodiment, the storage requirement of a large data volume can be met, and the uniformity of data storage is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method and device for data storage. Background Art

[0002] With the rapid development of the Internet, the traditional single-database single-table storage method is difficult to meet the storage needs of related business data. In addition, the read and write performance of a single database and a single table is easily greatly affected when the data volume is large. In order to solve the problem of too much data in a single table, which leads to an excessively large database, the current common method is to split the data in a single table into multiple tables according to a certain field or several fields, and multiple tables are split into multiple databases.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] It cannot meet the storage needs of large data volumes. When the data volume is uneven, uneven storage is prone to occur, causing data skew. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data storage method that can meet the storage requirements of a large amount of data and improve the uniformity of data storage.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for data storage is provided, comprising:

[0007] Acquire the data to be stored of the target first user, and determine the grouping sequence number of the target first user based on the pre-configured grouping result;

[0008] Determining a date dimension according to a data volume growth rate of the target first user's data to be stored;

[0009] Determining a routing strategy for the target first user based on the group sequence number and date dimension;

[0010] The data to be stored of the target first user is stored in separate libraries and tables according to the routing strategy.

[0011] Furthermore, the pre-configured grouping result is obtained by the following steps:

[0012] Determining the number of groups of the first user set based on the acquired data to be stored of the first user set;

[0013] Determine a predicted value of the data volume of each group based on the data volume and the number of groups of the data to be stored of the first user set;

[0014] The amounts of data to be stored of the first users in the first user set are evenly grouped according to the predicted value of the amount of data of each group, to obtain a grouping result for the first users.

[0015] Furthermore, it is characterized in that before determining the number of groups of the first user set based on the acquired data to be stored of the first user set, the method further comprises:

[0016] Dividing the user into a first user and a second user based on a preset division strategy; wherein the amount of data to be stored of the first user is greater than the amount of data to be stored of the second user;

[0017] Setting distinguishing marks for the first user and the second user respectively;

[0018] In response to receiving the data query request, querying the first user based on the distinguishing identifier and generating a first user set;

[0019] The data to be stored of the first user set is obtained.

[0020] Furthermore, the method further comprises:

[0021] Querying a second user based on the distinguishing identifier;

[0022] Performing hash calculation and modulo calculation on the second user using a preset hash function to obtain a calculation modulo result;

[0023] Based on the calculation modulo result, a target database and a target table of the second user are determined, and the acquired data to be stored of the second user is stored in the target table of the target database.

[0024] Furthermore, the date dimension includes a day dimension, a week dimension and a month dimension.

[0025] Further, based on the group sequence number and date dimension, determining the routing strategy of the target first user includes:

[0026] If the date dimension is a day dimension, the group sequence number of the target first user and the transaction date of the data to be stored of the target first user are summed up on the nth day of the year to obtain a sum calculation result, where n is a positive integer;

[0027] The sum calculation result is modulo the total number of databases, and the routing strategy of the target first user is determined according to the modulo result.

[0028] According to a second aspect of an embodiment of the present invention, a data storage device is provided, including:

[0029] A group determination module, used to obtain the data to be stored of the target first user, and determine the group sequence number of the target first user based on the pre-configured grouping result;

[0030] A dimension determination module, used to determine a date dimension according to a data volume growth rate of the data to be stored of the target first user;

[0031] Strategy determination, for determining a routing strategy for the target first user based on the group sequence number and date dimensions;

[0032] The data storage module is used to store the to-be-stored data of the target first user in separate libraries and tables according to the routing strategy.

[0033] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0034] one or more processors;

[0035] a storage device for storing one or more programs,

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0037] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0038] According to a fifth aspect of an embodiment of the present invention, there is provided a computer program product, including a computer program, which implements the method of any of the above embodiments when the computer program is executed by a processor.

[0039] An embodiment of the above invention has the following advantages or beneficial effects: by obtaining the data to be stored of the target first user, determining the grouping number of the target first user based on the pre-configured grouping result; determining the date dimension according to the data volume growth rate of the data to be stored of the target first user; determining the routing strategy of the target first user based on the grouping number and the date dimension; storing the data to be stored of the target first user in separate libraries and tables according to the routing strategy; being able to meet the storage requirements of a large amount of data and improve the uniformity of data storage.

[0040] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0042] Figure 1 is a schematic diagram of the main process of the data storage method according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of the main process of a data storage method according to a reference embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of the main process of a data storage method according to another reference embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of the main process of a data storage method according to another reference embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of main modules of a data storage device according to an embodiment of the present invention;

[0047] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0048] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0050] It should be noted that the acquisition, storage and application of personal information etc. involved in the embodiments of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0051] With the rapid development of the Internet, the traditional single-database single-table storage method is difficult to meet the storage needs of related business data. In addition, the read and write performance of a single database and a single table is easily affected when the data volume is large. In order to solve the problem of excessive data volume in a single table, which leads to an excessively large database, the current commonly used method is to split the data in a single table into multiple tables according to a certain field or several fields, and multiple tables are split into multiple databases. However, when the data volume is uneven, storage unevenness is prone to occur, resulting in data skew, which in turn leads to storage disk skew, reduced read and write performance, and increased operation and maintenance costs.

[0052] In view of this, according to one aspect of an embodiment of the present invention, a method for data storage is provided.

[0053] Figure 1 FIG. 1 is a schematic diagram of the main process of the method for storing data according to an embodiment of the present invention. Figure 1 As shown, the data storage method according to an embodiment of the present invention includes:

[0054] Step S101: acquiring data to be stored of a target first user, and determining a grouping sequence number of the target first user based on a pre-configured grouping result.

[0055] The target first user is a user with a large amount of data, such as a large enterprise, a large merchant, or other users with high transaction volume, complex business storage requirements, or high requirements for storage performance and stability. The data to be stored by the target first user includes the target first user's transaction records, personal information, transaction frequency, data growth trend and other information.

[0056] The pre-configured grouping results are analyzed to obtain the characteristics and data volume of each group, and the group number to which the target first user should belong is determined based on the data volume and other characteristics of the target first user's data to be stored. For example, the judgment can be made based on the predicted value of the group data volume and the data volume of the target first user.

[0057] Step S102: determining a date dimension according to a data volume growth rate of the target first user's data to be stored.

[0058] The data volume growth rate of the target first user's data to be stored is analyzed to obtain the growth trend and speed of the data to be stored, and the selection of the date dimension is determined according to the data growth rate. Optionally, the date dimension includes a day dimension, a week dimension, and a month dimension, etc., that is, it can be in different forms such as by day, by week, and by month.

[0059] Step S103: determining a routing strategy for the target first user based on the group sequence number and date dimension.

[0060] The routing strategy of the target first user is a data storage and processing strategy formulated for the target first user with large data volume, high transaction frequency or other important characteristics; for example, data distribution strategy: determine in which databases or data storage units the data of the target first user should be stored; data storage optimization strategy: determine the data storage method and technology of the target first user, including data storage format, storage medium, storage structure, etc.

[0061] Combine the group number of the first target user and the determined date dimension, perform data analysis and calculation, and obtain the data storage situation and analysis results of the first target user; determine the routing strategy of the first target user based on the analysis results. The routing strategy includes the location of data storage (such as database number), the date dimension of storage, and the storage method (such as sub-table naming strategy, etc.). Among them, the sub-table naming strategy is the concatenation of group number, transaction date, and database number.

[0062] Specifically, according to the grouping result, a hash calculation can be performed on the target first user, and the hash value can be modulo the number of databases to determine the database number where the target first user's data to be stored is stored; the target first user's data to be stored can also be allocated to different databases according to the grouping result and the transaction volume of the target first user; the data to be stored can also be routed to the corresponding database according to the grouping result and the data growth trend of the target first user; the data to be stored can also be allocated to different databases according to the grouping result and the transaction frequency of the target first user; it should be noted that a suitable routing strategy can be selected according to the actual situation of the target first user.

[0063] Step S104: storing the target first user's data to be stored in separate databases and tables according to the routing strategy

[0064] Specifically, according to the selected routing strategy, determine in which databases the target first user should be stored. In the determined database, the target first user's data to be stored is divided into different tables for storage according to the table sub-naming strategy. The sub-table naming can be determined according to business needs and routing strategies, for example, using the grouping sequence number, transaction date, database number, etc. as the table name. The target first user's data to be stored after the database and table sub-processing is stored in the corresponding database and data table.

[0065] In addition, you can also set up appropriate data access control policies so that only authorized users can access and operate the data of the target first user, such as permission control, identity authentication, encryption and other policies. Establish a monitoring system to monitor the data stored in the sub-databases and sub-tables, promptly discover and solve problems that arise during the storage process, and at the same time, optimize data storage and access to improve system performance and reliability. Set up appropriate data backup and disaster recovery strategies to improve the security and recoverability of data in unexpected situations, such as regular backup, off-site backup, disaster recovery solutions, etc. As the business develops and the amount of data grows, continue to optimize the sub-database and sub-table storage solutions, adjust the routing strategy and storage architecture according to actual conditions, and ensure the stability and scalability of the system.

[0066] The embodiment of the present invention obtains the data to be stored of the target first user, determines the grouping sequence number of the target first user based on the pre-configured grouping result; determines the date dimension according to the data volume growth rate of the data to be stored of the target first user; determines the routing strategy of the target first user based on the grouping sequence number and the date dimension; stores the data to be stored of the target first user in separate libraries and tables according to the routing strategy; and can meet the storage requirements of a large amount of data and improve the uniformity of data storage.

[0067] Figure 2 FIG. 1 is a schematic diagram of the main process of a method for storing data according to a reference embodiment of the present invention. Figure 2 As shown, the pre-configured grouping result is obtained by the following steps:

[0068] Step S201: determining the number of groups of the first user set based on the acquired data to be stored of the first user set.

[0069] Step S202: determining a predicted value of the data volume of each group based on the data volume and the number of groups of the data to be stored of the first user set.

[0070] Step S203 : evenly grouping the to-be-stored data amounts of the first users in the first user set according to the predicted value of the data amount of each group, to obtain a grouping result for the first users.

[0071] The first user set is a set of first users obtained by analyzing and screening the existing data. The data to be stored in the first user set refers to data information that needs to be stored in the first user set, such as basic information, account information, transaction records, behavior records, preference settings, service history information, or security information, etc. Specifically, the data to be stored can be adjusted and expanded according to actual business needs and system design, and the embodiments of the present invention do not specifically limit it.

[0072] First, all the data to be stored of the first user set are collected, including user information, transaction records, data volume, etc.; the collected data to be stored are analyzed to understand the data volume, data growth trend, transaction frequency and other characteristics of the data to be stored of the first user set, and the number of groups of the first user set is determined according to factors such as data volume, transaction frequency, data growth trend, and business needs. For example, the grouping can be based on the size of the data volume, or based on the number of users, etc.; based on the analysis results, the appropriate number of groups is determined to ensure that each group can carry and store an appropriate amount of data while ensuring balanced and efficient data storage.

[0073] According to the data volume and the number of groups of the data to be stored in the first user set, the data volume characteristics of each group are analyzed, including the average data volume, the maximum data volume, the minimum data volume, etc.; based on the data volume characteristics and historical data analysis, the future data volume of each group is predicted. For example, statistical methods, trend prediction, etc. can be used for prediction; based on the prediction results, the data volume prediction value of each group is determined as the basis for subsequent grouping operations.

[0074] The data volume of each first user in the first user set is calculated to obtain the data volume information of each user; the first users in the first user set are evenly grouped according to the data volume prediction value of each group determined in the above steps, and the principle of load balancing can be adopted to make the data volume carried by each group relatively uniform; after the grouping operation, the grouping result of each first user in the first user set is obtained, and each group carries an appropriate amount of first user data, thereby achieving balanced and efficient data storage.

[0075] The amount of data to be stored of the first users in the first user set is evenly grouped to obtain the grouping result of the first users; specifically, a hash function can be used to perform hash calculation on each first user in the first user set, and the users can be allocated to different groups according to the range of the hash value, and the number of first users in each group is roughly equal; the first users can also be evenly allocated to different groups according to the transaction volume of each first user, for example, the total transaction volume is divided by the number of databases, and then allocated to different databases in order of the transaction volume of the merchants, and the transaction volume carried by each database is relatively balanced; each first user in the first user set can also be grouped according to the amount of data, and the users with larger data volume can be allocated to the same group. The first users in the first user set may be grouped according to their number, and users with a large number of first users and a small number of first users may be placed in the same group, and users with a medium number of first users may be placed in other groups. The first users in the first user set may also be grouped according to their number, and users with a large number of first users and a small number of first users may be placed in the same group, and users with a medium number of first users may be placed in other groups. The first users may also be grouped according to the transaction frequency of each first user, and users with a high transaction frequency and a low transaction frequency may be placed in the same group, and users with a medium transaction frequency may be placed in other groups. The first users may also be grouped according to the data growth trend of each first user, and users with a faster data growth rate and users with a slower data growth rate may be placed in the same group, and users with a medium data growth rate may be placed in other groups. The embodiment of the present invention does not specifically limit the grouping form.

[0076] The steps of the embodiment of the present invention avoid the system burden caused by unbalanced data storage or excessive grouping by determining the number of groups of the first user set, and predict the data volume prediction value of each group by the data volume and the number of groups, effectively estimate the data volume that each group should carry, improve the efficiency and performance of data storage, and evenly group the data so that the data volume in each group is similar, which helps to balance the load and run the data processing smoothly, can effectively manage the storage and processing of large-scale data, and improve the stability and reliability of the system.

[0077] Figure 3 FIG. 1 is a schematic diagram of the main process of a method for storing data according to another reference embodiment of the present invention. Figure 3 As shown, before determining the number of groups of the first user set based on the acquired data to be stored of the first user set, the method further includes:

[0078] Step S301 , dividing users into a first user and a second user based on a preset division strategy; wherein the amount of data to be stored of the first user is greater than the amount of data to be stored of the second user.

[0079] Step S302: setting distinguishing marks for the first user and the second user respectively.

[0080] Step S303: in response to receiving the data query request, query the first user based on the distinguishing identifier and generate a first user set.

[0081] Step S304: obtaining the data to be stored of the first user set.

[0082] Among them, the preset segmentation strategy can be to segment users based on factors such as the size of their data, activity, transaction frequency, daily transaction volume, etc.; for example, the daily transaction volume of the first user is significantly greater than that of the second user. In the application scenario, users with daily transaction volumes greater than or equal to 100,000 are segmented into first users, and users with daily transaction volumes less than 100,000 are segmented into second users.

[0083] According to the preset division strategy, after the qualified users are divided into the first user and the second user, distinguishing identifiers are set for the first user and the second user respectively. Specific fields or attributes can be used to identify the first user and the second user. For example, a field is set with a value of 1 representing the first user and a value of 2 representing the second user.

[0084] After receiving the data query request, the data of the first user is queried according to the distinguishing identifier in the data query request, the queried data of the first user is formed into a first user set, the data to be stored is extracted from the generated first user set, and the data to be stored is preprocessed, such as format conversion, data cleaning, etc., and the preprocessed data to be stored is stored in a designated data storage medium, such as a database, file system, etc.

[0085] The embodiment of the present invention classifies and identifies users, distinguishes the first user from the second user, and can quickly query the data of the first user as needed to form a first user set. Data storage and management are optimized, and the efficiency and accuracy of data processing are improved; by dividing users and setting distinguishing identifiers, data processing and analysis of different user groups can be performed more accurately, improving the security and privacy protection of data storage.

[0086] Optionally, Figure 4 FIG. 1 is a schematic diagram of the main process of a method for storing data according to another reference embodiment of the present invention. Figure 4 As shown, the method also includes:

[0087] Step S401: Find out the second user based on the distinguishing identifier.

[0088] Step S402: perform hash calculation and modulus calculation on the second user using a preset hash function to obtain a calculation modulus result.

[0089] Step S403: Determine the target database and target table of the second user based on the calculation modulo result, and store the acquired data to be stored of the second user into the target table of the target database.

[0090] Specifically, after receiving a data query request, the distinguishing identifier of the data query request is parsed, and according to the distinguishing identifier, a query operation is performed to filter out the second user data that meets the conditions; a preset hash function is used to perform a hash calculation on the second user's data to obtain a hash value; the hash value is modulo-operated with a preset modulus to obtain a calculated modulo result; according to the calculated modulo result, a target database and a target table of the second user are determined for storing the second user's data; the acquired second user's data to be stored is stored in the target table of the target database to complete the data storage operation.

[0091] For example, merchant B is the second user, and merchant B's merchant number is 123456. There are 16 databases and 64 tables in total. Merchant B's merchant number is allocated to the corresponding database and table using a hash function. Merchant B's merchant number is 123456. The merchant number is hashed using a hash function. For example, the result of taking the hash value is 56789. The hash value 56789 is modulo-calculated with the total number of tables 64, that is, 56789% 64, and the calculated modulo result is 9. According to the calculated modulo result, merchant B should be saved in table No. 9. The specific database storage location depends on the rules and implementation methods of database sharding. It can be the result obtained by taking the hash value modulo, and then combined with the sharding rules to determine the final data storage location. For example, 64 tables are evenly distributed to 16 databases, and merchant B's data can eventually be stored in table No. 9 in database 1.

[0092] Optionally, determining the routing strategy of the target first user based on the group sequence number and date dimension includes:

[0093] If the date dimension is a day dimension, the group sequence number of the target first user and the transaction date of the data to be stored of the target first user are summed up on the nth day of the year to obtain a sum calculation result, where n is a positive integer;

[0094] The sum calculation result is modulo the total number of databases, and the routing strategy of the target first user is determined according to the modulo result.

[0095] If the date dimension is a week dimension, the group sequence number of the target first user and the transaction date of the data to be stored of the target first user are summed in the nth week of the year to obtain a sum calculation result, where n is a positive integer;

[0096] The sum calculation result is modulo the total number of databases, and the routing strategy of the target first user is determined according to the modulo result.

[0097] If the date dimension is a month dimension, the group sequence number of the target first user and the transaction date of the data to be stored of the target first user are summed in the nth month of the year to obtain a sum calculation result, where n is a positive integer;

[0098] The sum calculation result is modulo the total number of databases, and the routing strategy of the target first user is determined according to the modulo result.

[0099] For example, the date dimension is the day dimension. Take merchant A as an example. Merchant A's group number is 1. January 1, 2023 is the first day of 2023. The sum of 1 and 1 is 2. Assuming there are 16 databases from number 00 to number 15, that is, the total number of databases is 16, the sum calculation result modulo the total number of databases is 2%16=2, that is, merchant A is stored in database 02 on January 1, 2023. Among them, the sub-table naming strategy is the concatenation of group number, transaction date, and database number, that is, merchant A should be stored in database 02 on January 1, 2023, and according to the above sub-table naming strategy, the table name is "Group1_2023_01_01_DB02".

[0100] January 2, 2023 is the second day of 2023. The sum of 1 and 1 is 3. The sum modulo the total number of databases is 3%16=3. Merchant A is assigned to database 3 on January 2, 2023, and the table name is "Group1_2023_01_02_DB03".

[0101] By analogy, the massive data of merchant A can be effectively distributed evenly to various databases on a daily basis, thus avoiding the problem of data skew.

[0102] According to a second aspect of the embodiment of the present invention, Figure 5 As shown, a data storage device 500 is provided, comprising:

[0103] A group determination module 501 is used to obtain the data to be stored of the target first user and determine the group sequence number of the target first user based on a pre-configured grouping result;

[0104] A dimension determination module 502, configured to determine a date dimension according to a data volume growth rate of the target first user's data to be stored;

[0105] A policy determination module 503, configured to determine a routing policy for the target first user based on the group sequence number and date dimension;

[0106] The data storage module 504 is used to store the to-be-stored data of the target first user in separate libraries and tables according to the routing strategy.

[0107] Optionally, the data storage device 500 further includes a group configuration module, and the group configuration module is used to:

[0108] Determining the number of groups of the first user set based on the acquired data to be stored of the first user set;

[0109] Determine a predicted value of the data volume of each group based on the data volume and the number of groups of the data to be stored of the first user set;

[0110] The amounts of data to be stored of the first users in the first user set are evenly grouped according to the predicted value of the amount of data of each group, to obtain a grouping result for the first users.

[0111] Optionally, the data storage device 500 further includes a query module, and the query module is used to:

[0112] Dividing the user into a first user and a second user based on a preset division strategy; wherein the amount of data to be stored of the first user is greater than the amount of data to be stored of the second user;

[0113] Setting distinguishing marks for the first user and the second user respectively;

[0114] In response to receiving the data query request, querying the first user based on the distinguishing identifier and generating a first user set;

[0115] The data to be stored of the first user set is obtained.

[0116] Optionally, the query module is further used to:

[0117] Querying a second user based on the distinguishing identifier;

[0118] Performing hash calculation and modulo calculation on the second user using a preset hash function to obtain a calculation modulo result;

[0119] Based on the calculation modulo result, a target database and a target table of the second user are determined, and the acquired data to be stored of the second user is stored in the target table of the target database.

[0120] Optionally, the date dimension in the dimension determination module 502 includes a day dimension, a week dimension and a month dimension.

[0121] Optionally, the policy determination module 503 is further configured to:

[0122] If the date dimension is a day dimension, the group sequence number of the target first user and the transaction date of the data to be stored of the target first user are summed up on the nth day of the year to obtain a sum calculation result, where n is a positive integer;

[0123] The sum calculation result is modulo the total number of databases, and the routing strategy of the target first user is determined according to the modulo result.

[0124] It should be noted that the specific implementation content of the data storage device of the present invention has been described in detail in the data storage method described above, so the repeated content will not be described here.

[0125] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0126] one or more processors;

[0127] a storage device for storing one or more programs,

[0128] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiment of the present invention.

[0129] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0130] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0131] Figure 6 An exemplary system architecture 600 is shown to which the data storage method or data storage apparatus according to the embodiment of the present invention may be applied.

[0132] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0133] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0134] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0135] The server 605 may be a server that provides various services, such as a backend management server (only an example) that supports shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as the request for data storage, and feed back the processing results (such as the target sub-library and sub-table are only examples) to the terminal device.

[0136] It should be noted that the data storage method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data storage device is generally arranged in the server 605 .

[0137] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0138] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0139] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can execute various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0140] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 runs a communication process via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0141] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for running the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0142] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually run basically in parallel, and they can sometimes run in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that runs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a grouping determination module, a dimension determination module, a policy determination module and a data storage module. Among them, the names of these modules do not constitute a limitation on the modules themselves in certain circumstances, for example, the data storage module may also be described as "a module for storing the data to be stored by the target first user in separate libraries and tables according to the routing policy".

[0145] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining the data to be stored of the target first user, determining the grouping sequence number of the target first user based on the pre-configured grouping result; determining the date dimension according to the data volume growth rate of the data to be stored of the target first user; determining the routing strategy of the target first user based on the grouping sequence number and the date dimension; storing the data to be stored of the target first user in separate libraries and tables according to the routing strategy.

[0146] The computer program product provided by the embodiment of the present invention includes a computer program, and when the computer program is executed by a processor, the data storage method in the embodiment of the present invention is implemented.

[0147] The technical solution according to the embodiment of the present invention has the following advantages or beneficial effects: by acquiring the data to be stored of the target first user, determining the grouping sequence number of the target first user based on the pre-configured grouping result; determining the date dimension according to the data volume growth rate of the data to be stored of the target first user; determining the routing strategy of the target first user based on the grouping sequence number and the date dimension; storing the data to be stored of the target first user in separate libraries and tables according to the routing strategy; being able to meet the storage requirements of a large amount of data and improve the uniformity of data storage.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0149] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method for data storage, characterized in that: include: Acquire the data to be stored of the target first user, and determine the grouping sequence number of the target first user based on the pre-configured grouping result; Determining a date dimension according to a data volume growth rate of the target first user's data to be stored; Determining a routing strategy for the target first user based on the group sequence number and date dimension; The data to be stored of the target first user is stored in separate libraries and tables according to the routing strategy.

2. The method according to claim 1, characterized in that The pre-configured grouping result is obtained by the following steps: Determining the number of groups of the first user set based on the acquired data to be stored of the first user set; Determine a predicted value of the data volume of each group based on the data volume and the number of groups of the data to be stored of the first user set; The amounts of data to be stored of the first users in the first user set are evenly grouped according to the predicted value of the amount of data of each group, to obtain a grouping result for the first users.

3. The method according to claim 2, characterized in that Before determining the number of groups of the first user set based on the acquired data to be stored of the first user set, the method further includes: Dividing the user into a first user and a second user based on a preset division strategy; wherein the amount of data to be stored of the first user is greater than the amount of data to be stored of the second user; Setting distinguishing marks for the first user and the second user respectively; In response to receiving the data query request, querying the first user based on the distinguishing identifier and generating a first user set; The data to be stored of the first user set is obtained.

4. The method according to claim 3, characterized in that The method further comprises: Querying a second user based on the distinguishing identifier; Performing hash calculation and modulo calculation on the second user using a preset hash function to obtain a calculation modulo result; Based on the calculation modulo result, a target database and a target table of the second user are determined, and the acquired data to be stored of the second user is stored in the target table of the target database.

5. The method according to claim 1, characterized in that The date dimension includes a day dimension, a week dimension and a month dimension.

6. The method according to claim 5, characterized in that Determining a routing strategy for the target first user based on the group sequence number and date dimension includes: If the date dimension is a day dimension, the group sequence number of the target first user and the transaction date of the data to be stored of the target first user are summed up on the nth day of the year to obtain a sum calculation result, where n is a positive integer; The sum calculation result is modulo the total number of databases, and the routing strategy of the target first user is determined according to the modulo result.

7. A data storage device, characterized in that: include: A group determination module, used to obtain the data to be stored of the target first user, and determine the group sequence number of the target first user based on the pre-configured grouping result; A dimension determination module, used to determine a date dimension according to a data volume growth rate of the data to be stored of the target first user; A policy determination module, configured to determine a routing policy for the target first user based on the group sequence number and date dimension; The data storage module is used to store the to-be-stored data of the target first user in separate libraries and tables according to the routing strategy.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images