Data storage method and device

Abstract

The present invention discloses a method and device for data storage, and relates to the technical field of computers. A specific implementation of the method includes: receiving data to be stored; determining whether valid cached data corresponding to the user identifier is stored in a high-frequency cache according to a user identifier indicated by the data to be stored; if yes, storing the data to be stored in the high-frequency cache using Bitmaps; if no, storing the data to be stored in a database using an int field. This implementation reduces the amount of data stored and improves data access efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data storage. Background Art

[0002] With the development of computer technology, more and more users are shopping through e-commerce platforms. Therefore, the amount of data on e-commerce platforms (users' activity check-in data, shopping data, click data, etc.) is also increasing day by day.

[0003] For the storage of data on e-commerce platforms, the existing technology usually adopts a record-by-record method. For example, for user activity check-in data, each time a user checks in, a piece of data is generated, and the database saves the check-in data generated by a large number of users one by one.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] Each user has one record per day, and each user has 10 records in 10 days. Therefore, a large number of users will generate a huge amount of sign-in data during the sign-in process, and saving these sign-in data one by one requires a lot of storage space. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method and device for data storage, which can use Bitmaps to store data to be cached in a high-frequency cache, thereby reducing the amount of data storage in the high-frequency cache, and the cached data stored in the high-frequency cache can be easily accessed, thereby improving the access efficiency of the data. In addition, using an int field in the database to store the data to be stored can also reduce the amount of data in the database and reduce the storage space occupied by the stored data.

[0007] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for data storage is provided, comprising:

[0008] receiving data to be stored;

[0009] Determining, according to the user identifier indicated by the data to be stored, whether valid cached data corresponding to the user identifier is stored in the high-frequency cache;

[0010] If yes, storing the data to be stored in the high-frequency cache using Bitmaps;

[0011] If not, the data to be stored is stored in the database using an int field.

[0012] Optionally, before determining whether valid cached data corresponding to the user identifier is stored in the high-frequency cache, the method further includes:

[0013] It is determined whether the user identifier corresponds to a high-frequency user. If so, it is determined whether the cached data exists in the high-frequency cache according to the user identifier, and it is determined that the cached data is valid.

[0014] Optionally, when it is determined that the user identifier corresponds to a high-frequency user and the cached data corresponding to the user identifier does not exist in the high-frequency cache,

[0015] Determine the stored data corresponding to the user identifier in the database, and store the stored data in the high-frequency cache using Bitmaps.

[0016] Optionally, the determining whether the user identifier corresponds to a high-frequency user includes:

[0017] The frequency corresponding to the user identifier is determined according to any one or more of the following elements; the elements include:

[0018] The stored data corresponding to the user identifier within the preset time period, the total time period corresponding to the stored data, the first time period from the storage time of the first stored data in the stored data to the current time period, and the second time period from the preset time period to the current time period, wherein the preset time period is before the storage time period of the first stored data;

[0019] When the frequency is greater than a preset frequency threshold, it is determined that the user identifier corresponds to a high-frequency user.

[0020] Optionally, when it is determined that the user identifier does not correspond to a high-frequency user, the method further includes:

[0021] It is determined whether the cached data exists in the high-frequency cache according to the user identifier, and if so, it is determined whether the cached data is valid according to a preset validity period.

[0022] Optionally, after storing the data to be stored in the high-frequency cache using Bitmaps, the method further includes:

[0023] The cache duration of the stored data corresponding to the user identifier is updated in the high-frequency cache.

[0024] Optionally, when the cached data is invalid, the method further includes:

[0025] According to the stored data stored in the high-frequency cache using Bitmaps, the int field in the database is updated, and the stored data in the high-frequency cache is deleted after the update.

[0026] Optionally, the database uses an int field to store the data to be stored, including:

[0027] Each bit of the int field is used to store each piece of the data to be stored, and the storage identifier of the int field is set according to the user identifier.

[0028] Optionally, the int field is stored in a first data table; when the number of int fields stored in the data table is greater than a preset number threshold, the method further includes:

[0029] The multiple int fields in the first data table are grouped, and the grouped int fields are respectively stored in multiple second data tables.

[0030] Optionally, a hash algorithm is used to calculate a summary value of the storage identifier, the summary value is modulo the quantity of the second data table, and the multiple int fields are grouped according to the modulo result.

[0031] According to a second aspect of an embodiment of the present invention, a data storage device is provided, comprising: a receiving module, a determining module, a first storage module and a second storage module; wherein:

[0032] The receiving module is used to receive data to be stored;

[0033] The determination module is used to determine whether valid cached data corresponding to the user identifier is stored in the high-frequency cache according to the user identifier indicated by the data to be stored; if yes, trigger the first storage module; if not, trigger the second storage module;

[0034] The first storage module is used to store the data to be stored in the high-frequency cache using Bitmaps;

[0035] The second storage module is used to store the data to be stored in the database using an int field.

[0036] According to a third aspect of an embodiment of the present invention, there is provided an electronic device for storing data, comprising:

[0037] one or more processors;

[0038] a storage device for storing one or more programs,

[0039] When the one or more programs are executed by the one or more processors, the one or more processors implement any method described in the data storage method provided in the first aspect above.

[0040] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, any method in the data storage method provided in the first aspect is implemented.

[0041] One embodiment of the above invention has the following advantages or beneficial effects: when the high-frequency cache stores cached data corresponding to the data to be stored, Bitmaps are used to store the data to be cached in the high-frequency cache; thus, each bit of the Bitmaps can be used to store one data to be cached, thereby reducing the amount of data storage in the high-frequency cache; further, relative to the stored data stored in the database, the cached data stored in the high-frequency cache can be easily accessed, thereby improving the access efficiency of the data. In addition, when the high-frequency cache does not store the stored data corresponding to the data to be stored, the int field is used to store the data to be stored in the database, wherein each bit of the int field can also store one data to be stored, thereby reducing the amount of data storage in the database and reducing the storage space occupied by the stored data. In summary, caching data by combining the high-frequency cache with the database can reduce the amount of data stored and save storage space on the one hand, and on the other hand, it is conducive to improving the efficiency of data access.

[0042] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0044] Figure 1 is a schematic diagram of the main process of the data storage method according to an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of dynamically storing data using an int field according to an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of a data storage structure of an int field according to an embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of performing table partitioning according to a table partitioning strategy according to an embodiment of the present invention;

[0048] Figure 5 is a schematic diagram of using a hash algorithm to partition a table according to an embodiment of the present invention;

[0049] Figure 6 is a schematic diagram of main modules of a data storage device according to an embodiment of the present invention;

[0050] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0051] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0053] Figure 1 A data storage method according to an embodiment of the present invention is as follows: Figure 1 As shown, the method may include the following steps S101 to S104:

[0054] Step S101: receiving data to be stored.

[0055] When the data storage method provided by the embodiment of the present invention is applied to an e-commerce scenario, the data to be stored may be check-in data, purchase data, order data, etc. Among them, for different types of data, the subsequent storage process is the same. Therefore, in the embodiment of the present invention, the data storage method provided by the embodiment of the present invention is described in detail by taking check-in data as an example. When the data to be stored is check-in data, the data to be stored may include a user identifier (such as a UUID) and user check-in information.

[0056] Step S102: according to the user identifier indicated by the data to be stored, determine whether valid cached data corresponding to the user identifier is stored in the high-frequency cache; if yes, trigger step S103, otherwise execute step S104.

[0057] Before executing step S102, it can be determined whether the user identifier indicated by the data to be stored is a high-frequency user. If so, it is determined whether the cached data exists in the high-frequency cache according to the user identifier. If so, it is determined that the cached data is valid.

[0058] In an embodiment of the present invention, whether a user identifier is a high-frequency user can be determined in the following manner: the frequency corresponding to the user identifier is determined according to any one or more of the following factors; the factors include: stored data corresponding to the user identifier within a preset time length, the total time length corresponding to the stored data, a first time length from the storage moment of the first stored data in the stored data to the current moment, and a second time length from the preset moment to the current moment, wherein the preset moment is before the storage moment of the first storage; when the frequency is greater than a preset frequency threshold, it is determined that the user identifier corresponds to a high-frequency user.

[0059] Specifically, when the data to be stored is check-in data, high-frequency users correspond to high-frequency check-in users. Among the factors used to determine high-frequency users, the stored data corresponding to the user identifier within the preset duration can be the number of check-in days indicated by the user's check-in data within a certain period of time in the past; the total duration corresponding to the stored data represents the total number of check-in days of the user since the start of the check-in activity; the first duration represents the total number of days the user has experienced from the first check-in date to the present; the second duration represents the total number of days from the start date of the check-in activity to the present. In the process of determining whether a user is a high-frequency user, in order to improve the accuracy of determining high-frequency users, the weight values ​​corresponding to different factors can be set according to the importance of different factors, and the user's check-in frequency can be calculated according to the weight values.

[0060] In one embodiment of the present invention, the user's check-in frequency can be calculated by the following formula:

[0061]

[0062] Among them, a represents the number of check-in days indicated by the stored data corresponding to the user identification within the preset time period; A is the preset time period; b represents the total time period corresponding to the stored data, B represents the first time period from the storage moment of the first stored data in the stored data to the current moment, C represents the second time period from the preset moment to the current moment, and M1, M2 and M3 are weight values.

[0063] Still taking the above explanation of each factor in the sign-in scenario as an example, when judging whether a user is a high-frequency user based on data within one month, a is the number of sign-in days in the user's sign-in data in the past 30 days, A is 30 days, b is the total number of sign-in days since the start of the sign-in activity, B is the total number of days the user has experienced from the first sign-in date to the present, and C is the total number of days from the start of the sign-in activity to the present; M1 can be set according to actual needs, for example, M1 can be set to 50%, M2 can be set to 30%, and M3 can be set to 20%.

[0064] After the user's check-in frequency is calculated according to the above formula, whether it is a high-frequency user can be determined according to the preset frequency threshold, wherein the preset frequency threshold can be a number between 0 and 100%. For example, the preset frequency threshold is 60%. Then, if the calculated frequency value is greater than 60%, the user ID corresponds to a high-frequency user, otherwise the user ID corresponds to a low-frequency user.

[0065] In addition, the embodiment of the present invention may also determine whether a user is a high-frequency user by other methods, for example, determining whether a user is a high-frequency user based on the user's membership level, the user's total order amount, or the user's purchase frequency.

[0066] When it is determined that the user identifier corresponds to a high-frequency user, it is determined whether the cached data exists in the high-frequency cache according to the user identifier. If the cached data corresponding to the user identifier exists in the high-frequency cache, it means that the user's previous sign-in data has been stored in the high-frequency cache. At this time, it is directly determined that the cached data in the high-frequency cache is valid, and then the following step S103 can be continued, that is, the data to be stored (this sign-in data) is stored in the high-frequency cache using Bitmaps.

[0067] When it is determined that the user ID corresponds to a high-frequency user, but the high-frequency cache does not store cached data corresponding to the user ID, it means that the user was previously a low-frequency user, and after the last sign-in, it was converted into a high-frequency user based on the last sign-in data. In this case, the high-frequency cache is needed to store the sign-in data corresponding to the user ID. Therefore, the stored data corresponding to the user ID can be determined in the database, and the stored data can be stored in the high-frequency cache using Bitmaps. The stored data in the database is stored in the int field.

[0068] In the case where it is determined that the user identifier does not correspond to a high-frequency user, that is, the user identifier corresponds to a low-frequency user, in an embodiment of the present invention, it can be determined whether the cached data exists in the high-frequency cache based on the user identifier. If so, it is determined whether the cached data is valid based on a preset validity period.

[0069] Here, when the cached data in the high-frequency cache is still valid, the following step S103 can be executed, that is, the data to be stored (this sign-in data) is stored in the high-frequency cache using Bitmaps. When the cached data corresponding to the user identifier does not exist in the high-frequency cache or the cached data in the high-frequency cache is invalid, the following step S104 can be executed, that is, the data to be stored is stored in the database using an int field.

[0070] In order to improve the access efficiency of data, in an embodiment of the present invention, the data buffer period when a high-frequency user is converted to a low-frequency user is configured by updating the data cache duration in the high-frequency cache. Specifically, in one embodiment of the present invention, after the data to be stored is stored in the high-frequency cache using Bitmaps, the cache duration of the stored data corresponding to the user identifier is updated in the high-frequency cache.

[0071] For example, the preset data validity period is 7 days, and 4 days have passed since the last storage of the sign-in data to the current receipt of the data to be cached. Therefore, before the data to be stored is stored in the high-frequency cache, the remaining effective cache duration of the data stored in the high-frequency cache is 3 days. After the data to be stored (the sign-in data this time) is stored in the high-frequency cache, the validity period of the stored data corresponding to the user ID in the high-frequency cache can be updated to 7 days.

[0072] Therefore, for the stored data of a high-frequency user who has been converted to a low-frequency user in a short period of time, the corresponding stored data is not deleted in the high-frequency cache at the moment of conversion to a low-frequency user, but the stored data in the high-frequency cache is retained within the validity period. If new data to be stored corresponding to the user ID (i.e., new sign-in data) is received within this validity period, the cache duration of the stored data corresponding to the user ID will continue to be updated, so that the sign-in data converted to a low-frequency user can also continue to be stored in the high-frequency cache for a certain period of time, which is beneficial to improve the access efficiency of the data. Of course, for the data to be stored by high-frequency users, after it is stored in the high-frequency cache, its cache duration will also be updated accordingly.

[0073] In addition, when it is determined that the user identifier does not correspond to a high-frequency user and the cached data of the user identifier in the high-frequency cache is invalid, the int field in the database is updated according to the stored data stored in the high-frequency cache using Bitmaps, and the stored data in the high-frequency cache is deleted after the update. This is because when the user identifier was once determined to be a high-frequency user, its corresponding sign-in data was directly stored in the high-frequency cache using Bitmaps, and the database was not accessed once each time the sign-in data was received to avoid frequent database access. That is, when the user identifier was once determined to be a high-frequency user, only the corresponding sign-in data was stored in the high-frequency cache using Bitmaps. Therefore, when the user identifier is determined to be a low-frequency user and the cached data in the high-frequency cache is invalid, that is, when the user changes from a high-frequency user to a low-frequency user and the corresponding cached data is invalid, it is necessary to use the database to store its sign-in data. At this time, it is necessary to update the data stored in the int field in the data according to the cached data in the high-frequency cache to ensure that the data stored in the database is synchronized to the latest state. Furthermore, in order to save storage space in the high-frequency cache, the cached data corresponding to the user identifier in the high-frequency cache may be deleted after synchronization.

[0074] Step S103: storing the data to be stored in the high-frequency cache using Bitmaps.

[0075] In an embodiment of the present invention, Redis can be used to customize high-frequency user cache for high-frequency users, and the storage of the full amount of sign-in data of the high-frequency user life cycle can be realized through Redis Bitmaps, and high-performance query can be provided. The maximum number of bits supported by Redis Bitmaps is 2^32 (approximately 4.29 billion), which can fully support all sign-in data of each user's entire life cycle (100 years). In the above manner, this high-frequency storage solution is used to store the sign-in data of high-frequency customers, which can save Redis storage resources when it is necessary to store a large amount of user data.

[0076] For example, in a sign-in activity, the first day of the activity is used as the benchmark. Assuming that the activity starts on January 1, 2021, before the activity starts, all data bits of the bitmaps can be set to 0, and each bit corresponds to a sign-in day. When the user starts to sign in, the data bit of the date corresponding to the user's sign-in data is updated to 1. For example, when the user signs in on 20210101, the 0th position of the bitmaps is set to 1. After the customer signs in on 20210102, the 1st position of the bitmaps is set to 1. If the customer signs in on 20210101+n, the nth position is set to 1. The overall strategy is that the offset between the sign-in date and the start date of the activity is the position of the sign-in on that day. If the user signs in, the corresponding data position is set to 1, and if the user does not sign in, it is set to 0. Each user has only one copy of data in the cache, that is, the user's unique id (UUID) can be set as the key of redis.

[0077] Step S104: storing the data to be stored in the database using an int field.

[0078] In step S104, the int field can be used to store the sign-in data on a monthly basis. This is because the int field occupies 32 bits and can perfectly hold the sign-in records for the next month. The sign-in record storage method is the same as redis. In this way, both the sign-in activity cycle and the storage space of the database can be taken into account. In one embodiment of the present invention, each bit of the int field is used to store each of the data to be stored, and the storage identifier of the int field is set according to the user identifier.

[0079] When using int fields to store data, the storage method is basically the same as Bitmaps. Specifically, taking a certain sign-in activity as an example, using the first day of the activity as the benchmark, assuming that the activity starts on January 1, 2021, before the activity starts, all data bits of the int field can be set to 0, and each bit corresponds to a sign-in day. When the user starts to sign in, the data bit of the user's sign-in data corresponding to the date is updated to 1. For example, when the user signs in on 20210101, the 0th position of the int field is set to 1, and when the customer signs in on 20210102, the 1st position of the int field is set to 1. If the customer signs in on 20210101+n, the nth position is set to 1. The overall strategy is that the offset between the sign-in date and the start date of the activity is the position of the sign-in on that day. If the user signs in, the corresponding data position is set to 1, and if the user does not sign in, it is set to 0. When using int fields to store sign-in data by month, the storage identifier of the int field can be set to user id (UUID) + sign-in month. The schematic diagram of using int fields to dynamically store data can be as follows Figure 2 As shown, the storage result of the int field can be as follows Figure 3shown.

[0080] The embodiment of the present invention uses a data table to store int fields. It can be understood that each user's monthly sign-in data corresponds to an int field. Then, when the number of users participating in the sign-in activity is large and the users sign in continuously for a long time, the int fields stored in the data table will be very large, which may affect the access efficiency of the data table. Therefore, in one embodiment of the present invention, the int field is stored in a first data table; when the number of int fields stored in the first data table is greater than a preset number threshold, it also includes: grouping the multiple int fields in the first data table, and using multiple second data tables to store the grouped int fields respectively.

[0081] Here, the preset quantity threshold can be set according to the storage capacity of the first data table. For example, the first data table can store up to 100 int fields, then the preset quantity threshold can be set to the maximum storage capacity of the first data table × 95%, that is, 95. Of course, the preset quantity threshold can also be set to other values ​​according to actual needs. When the number of int fields stored in the first data table is greater than the preset quantity threshold, the multiple int fields stored can be stored in separate tables, that is, the grouped int fields are stored in multiple second data tables after the split tables. The schematic diagram of storing the int fields in the first data table in separate tables according to the preset split table strategy can be shown as follows: Figure 4 shown.

[0082] In one embodiment of the present invention, when partitioning the table, a hash algorithm may be used to calculate the summary value of the storage identifier, the summary value is modulo the number of the second data table, and the multiple int fields are grouped according to the modulo result.

[0083] Specifically, the user's unique ID (UUID) is first hashed, and the result of the operation is modulo the number of tables to be split (the number of second data tables), that is, hash (UUID) % n, where n represents the number of second data tables. The purpose of performing the hash operation first is to distribute the multiple int fields as evenly as possible in the multiple second data tables. When the second data table is about to reach a bottleneck, it can be dynamically split according to the month, that is, the multiple int fields stored in the same second data table are stored in the third data table according to the month indicated by their storage identifiers. The int fields of the same month are stored in different third data tables. The int fields corresponding to different months are stored in different third data tables. The process of splitting the tables can be as follows. Figure 5As shown, table-0-201901 stores the check-in data for January 2019 separated from the second data table with storage identification table0, table-0-201902 stores the check-in data for February 2019 separated from the second data table with storage identification table0, table-63-201901 stores the check-in data for January 2019 separated from the second data table with storage identification table63, and table-63-201902 stores the check-in data for February 2019 separated from the second data table with storage identification table63.

[0084] It is worth mentioning that when a low-frequency user becomes a high-frequency user, that is, when the high-frequency cache is used for the first time to store the check-in data corresponding to the user ID, the int fields can be assembled in sequence according to the check-in months corresponding to the multiple int fields corresponding to the user ID in the database, and the assembled stored data can be stored in the high-frequency cache.

[0085] According to the data storage method provided by the embodiment of the present invention, when the high-frequency cache stores cached data corresponding to the data to be stored, Bitmaps are used to store the data to be cached in the high-frequency cache; thus, each bit of the Bitmaps can be used to store one data to be cached, thereby reducing the amount of data storage in the high-frequency cache; further, relative to the stored data stored in the database, the cached data stored in the high-frequency cache can be easily accessed, thereby improving the access efficiency of the data. In addition, when the high-frequency cache does not store the stored data corresponding to the data to be stored, the int field is used to store the data to be stored in the database, wherein each bit of the int field can also store one data to be stored, thereby reducing the amount of data storage in the database and reducing the storage space occupied by the stored data. In summary, caching data by combining the high-frequency cache with the database can reduce the amount of data storage and save storage space on the one hand, and on the other hand, it is conducive to improving data access efficiency.

[0086] like Figure 6 As shown, an embodiment of the present invention provides a data storage device, including: a receiving module 601, a determining module 602, a first storage module 603 and a second storage module 604; wherein,

[0087] The receiving module 601 is used to receive data to be stored;

[0088] The determination module 602 is used to determine whether the high-frequency cache stores valid cached data corresponding to the user identifier according to the user identifier indicated by the data to be stored; if yes, trigger the first storage module 603; if not, trigger the second storage module 604;

[0089] The first storage module 603 stores the data to be stored in the high-frequency cache using Bitmaps;

[0090] The second storage module 604 is used to store the data to be stored in the database using an int field.

[0091] In one embodiment of the present invention, the determination module 602 is further used to determine whether the user identifier corresponds to a high-frequency user. If so, determine whether the cached data exists in the high-frequency cache based on the user identifier. If so, determine that the cached data is valid.

[0092] In one embodiment of the present invention, the determination module 602 is used to determine the stored data corresponding to the user identifier in the database when it is determined that the user identifier corresponds to a high-frequency user and the cached data corresponding to the user identifier does not exist in the high-frequency cache, and store the stored data in the high-frequency cache using Bitmaps.

[0093] In one embodiment of the present invention, the determination module 602 is used to determine the frequency corresponding to the user identifier based on any one or more of the following factors; the factors include: stored data corresponding to the user identifier within a preset time length, the total time length corresponding to the stored data, a first time length from the storage moment of the first stored data in the stored data to the current moment, and a second time length from the preset moment to the current moment, and the preset moment is before the storage moment of the first storage; when the frequency is greater than a preset frequency threshold, it is determined that the user identifier corresponds to a high-frequency user.

[0094] In one embodiment of the present invention, the determination module 602 is used to determine whether the cached data exists in the high-frequency cache according to the user identifier, and if so, determine whether the cached data is valid according to a preset validity period.

[0095] In one embodiment of the present invention, the storage module 603 is used to update the cache duration of the stored data corresponding to the user identifier in the high-frequency cache.

[0096] In one embodiment of the present invention, the first storage module 603 is used to update the int field in the database according to the stored data stored in the high-frequency cache using Bitmaps, and delete the stored data in the high-frequency cache after the update.

[0097] In one embodiment of the present invention, the second storage module 604 is used to update the int field in the database according to the stored data stored in the high-frequency cache using Bitmaps, and delete the stored data in the high-frequency cache after the update.

[0098] In one embodiment of the present invention, the second storage module 604 is used to store each of the to-be-stored data using each bit of the int field, and to set the storage identifier of the int field according to the user identifier.

[0099] In one embodiment of the present invention, the second storage module 604 is used to group the multiple int fields in the first data table, and use multiple second data tables to respectively store the grouped int fields.

[0100] In one embodiment of the present invention, the second storage module 604 is used to calculate the summary value of the storage identifier using a hash algorithm, take the modulus of the summary value and the number of the second data table, and group the multiple int fields according to the modulo result.

[0101] According to the data storage device provided by the embodiment of the present invention, when the high-frequency cache stores cached data corresponding to the data to be stored, Bitmaps are used to store the data to be cached in the high-frequency cache; thus, each bit of the Bitmaps can be used to store one data to be cached, thereby reducing the amount of data storage in the high-frequency cache; further, relative to the stored data stored in the database, the cached data stored in the high-frequency cache can be easily accessed, thereby improving the access efficiency of the data. In addition, when the high-frequency cache does not store the stored data corresponding to the data to be stored, the int field is used to store the data to be stored in the database, wherein each bit of the int field can also store one data to be stored, thereby reducing the amount of data storage in the database and reducing the storage space occupied by the stored data. In summary, caching data by combining the high-frequency cache with the database can reduce the amount of data stored and save storage space on the one hand, and on the other hand, it is conducive to improving data access efficiency.

[0102] An embodiment of the present invention also provides a server, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method provided in any of the above embodiments.

[0103] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the data storage method provided in any of the above embodiments is implemented.

[0104] Figure 7 An exemplary system architecture 700 is shown to which the data storage method or data storage device according to the embodiment of the present invention can be applied.

[0105] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0106] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0107] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0108] The server 705 may be a server that provides various services, such as a backend management server (only for example) that supports shopping websites browsed by users using the terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only for example) to the terminal device.

[0109] It should be noted that the data storage method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data storage device is generally arranged in the server 705 .

[0110] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0111] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0112] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0113] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0114] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0115] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0116] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a receiving module, a determining module, a first storage module, and a second storage module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the receiving module may also be described as a "module for receiving data to be stored".

[0118] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: receiving data to be stored; determining whether valid cached data corresponding to the user identifier is stored in the high-frequency cache according to the user identifier indicated by the data to be stored; if yes, storing the data to be stored in the high-frequency cache using Bitmaps; if not, storing the data to be stored in the database using int fields.

[0119] According to the technical solution of the embodiment of the present invention, when the high-frequency cache stores cached data corresponding to the data to be stored, Bitmaps are used to store the data to be cached in the high-frequency cache; thus, each bit of the Bitmaps can be used to store one data to be cached, thereby reducing the amount of data storage in the high-frequency cache; further, relative to the stored data stored in the database, the cached data stored in the high-frequency cache can be easily accessed, thereby improving the access efficiency of the data. In addition, when the high-frequency cache does not store the stored data corresponding to the data to be stored, the int field is used to store the data to be stored in the database, wherein each bit of the int field can also store one data to be stored, thereby reducing the amount of data storage in the database and reducing the storage space occupied by the stored data. In summary, caching data by combining the high-frequency cache with the database can reduce the amount of data storage and save storage space on the one hand, and on the other hand, it is conducive to improving data access efficiency.

[0120] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data storage, characterized in that: include: receiving data to be stored; Determining, according to the user identifier indicated by the data to be stored, whether valid cached data corresponding to the user identifier is stored in the high-frequency cache; If yes, storing the data to be stored in the high-frequency cache using Bitmaps, and updating the cache duration of the stored data corresponding to the user identifier in the high-frequency cache; If not, the data to be stored is stored in the database using an int field.

2. The method according to claim 1, characterized in that Before determining whether valid cached data corresponding to the user identifier is stored in the high-frequency cache, the method further includes: It is determined whether the user identifier corresponds to a high-frequency user. If so, it is determined whether the cached data exists in the high-frequency cache according to the user identifier. If so, it is determined that the cached data is valid.

3. The method according to claim 2, characterized in that When it is determined that the user identifier corresponds to a high-frequency user and the cached data corresponding to the user identifier does not exist in the high-frequency cache, Determine the stored data corresponding to the user identifier in the database, and store the stored data in the high-frequency cache using Bitmaps.

4. The method according to claim 2, characterized in that: The determining whether the user identifier corresponds to a high-frequency user includes: The frequency corresponding to the user identifier is determined according to any one or more of the following elements; the elements include: The stored data corresponding to the user identifier within the preset time period, the total time period corresponding to the stored data, the first time period from the storage time of the first stored data in the stored data to the current time period, and the second time period from the preset time period to the current time period, wherein the preset time period is before the storage time period of the first stored data; When the frequency is greater than a preset frequency threshold, it is determined that the user identifier corresponds to a high-frequency user.

5. The method according to claim 2, characterized in that: When it is determined that the user identifier does not correspond to a high-frequency user, the method further includes: It is determined whether the cached data exists in the high-frequency cache according to the user identifier, and if so, it is determined whether the cached data is valid according to a preset validity period.

6. The method according to claim 5, characterized in that When the cached data is invalid, the method further includes: According to the stored data stored in the high-frequency cache using Bitmaps, the int field in the database is updated, and the stored data in the high-frequency cache is deleted after the update.

7. The method according to claim 1, characterized in that The database uses an int field to store the data to be stored, including: Each bit of the int field is used to store each piece of the data to be stored, and a storage identifier of the int field is set according to the user identifier.

8. The method according to claim 7, characterized in that The int field is stored in the first data table; When the number of int fields stored in the first data table is greater than a preset number threshold, the method further includes: The multiple int fields in the first data table are grouped, and the grouped int fields are respectively stored in multiple second data tables.

9. The method according to claim 8, characterized in that A hash algorithm is used to calculate a summary value of the storage identifier, the summary value is modulo the quantity of the second data table, and the multiple int fields are grouped according to the modulo result.

10. A data storage device, characterized in that: include: A receiving module, a determining module, a first storage module and a second storage module; wherein, The receiving module is used to receive data to be stored; The determination module is used to determine whether valid cached data corresponding to the user identifier is stored in the high-frequency cache according to the user identifier indicated by the data to be stored; if yes, trigger the first storage module; if not, trigger the second storage module; The first storage module is used to store the data to be stored in the high-frequency cache using Bitmaps, and to update the cache duration of the stored data corresponding to the user identifier in the high-frequency cache; The second storage module is used to store the data to be stored in the database using an int field.

11. An electronic device for storing data, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images