Data storage method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data storage method, device, electronic equipment and storage medium, the data storage method comprises: obtaining at least one feature data to be stored in a target storage system, each feature data has a corresponding primary key; for each feature data, determine the target time partition to which the data occurrence time of the feature data belongs from the plurality of first time partitions corresponding to the target storage system; wherein each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space of the target storage system; the number of buckets corresponding to each first time partition is determined by the festival attribute information corresponding to the first time partition, or the number of buckets corresponding to each first time partition is determined by the data storage amount before the first time partition; based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, the corresponding bucket is allocated for the feature data, and the feature data is stored in the storage space associated with the corresponding bucket.

Description

Technical Field

[0001] This disclosure relates to the field of data storage technology, and more specifically, to a data storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of computer technology, the amount of data that needs to be stored is increasing. For example, in practical applications, it is necessary to store user behavior data during the use of applications for later use.

[0003] However, current data storage systems typically require global indexing of the data in the storage system during the data storage process to distinguish whether the data to be stored is new data being written or updated data. However, indexing services are usually limited by traffic, affecting the throughput performance of writing data and resulting in low data storage efficiency. Summary of the Invention

[0004] This disclosure provides at least one data storage method, apparatus, electronic device, and storage medium, which can improve data storage efficiency.

[0005] This disclosure provides a data storage method, including:

[0006] Obtain at least one piece of feature data to be stored in the target storage system, wherein each piece of feature data includes at least one piece of sub-feature data, and each piece of feature data has a corresponding primary key;

[0007] For each piece of feature data, the target time partition to which the data occurrence time of the feature data belongs is determined from multiple first time partitions corresponding to the target storage system; wherein, each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space of the target storage system; the number of buckets corresponding to each first time partition is determined by the holiday attribute information corresponding to the first time partition, or the number of buckets corresponding to each first time partition is determined by the amount of data stored before the first time partition;

[0008] Based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, a corresponding bucket is allocated to the feature data, and the feature data is stored in the storage space associated with its corresponding bucket.

[0009] In this embodiment of the disclosure, since the target storage system corresponds to multiple first time partitions, and each first time partition corresponds to a corresponding number of buckets, during the data storage process, the target time partition to which the feature data belongs can be determined first based on the data occurrence time of the feature data. Then, based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, a corresponding bucket can be allocated for each feature data. Thus, without the need for a global index, the feature data can be stored in the corresponding storage space within the target storage system. That is, the throughput performance of writing data can be improved by appending writes, thereby improving the efficiency of data storage.

[0010] Furthermore, since the number of buckets corresponding to each first time partition is determined by the holiday attribute information corresponding to the first time partition, or the number of buckets corresponding to each first time partition is determined by the amount of data stored before the first time partition, the number of buckets can be evolved. That is, the number of buckets can be adjusted according to the actual prediction needs, thereby reducing the occurrence of data overflow or large amounts of unused storage space caused by setting too many or too few buckets.

[0011] In one possible implementation, before acquiring at least one piece of feature data to be stored in the target storage system, the number of buckets corresponding to each first time partition is determined through the following steps:

[0012] If the current time meets the preset conditions, the first preset time period after the current time is divided into at least one second time partition;

[0013] Based on the holiday attribute information corresponding to each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition; or...

[0014] Based on the amount of data stored in the second preset time period preceding each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition.

[0015] In this embodiment of the disclosure, when the current time meets the preset conditions, the first preset time period after the current time can be automatically divided into at least one second time partition, and a corresponding number of buckets can be set for each second time partition. In this way, the number of buckets can be automatically configured, and the configured number of buckets can meet the corresponding data storage requirements. This not only improves the efficiency of setting the number of buckets, but also improves the applicability of the data storage method.

[0016] In one possible implementation, allocating corresponding buckets to the feature data based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data includes:

[0017] Based on the number of buckets corresponding to the target time partition, a modulo operation is performed on the primary key of the feature data to obtain the identity identifier of the bucket corresponding to the feature data, and a corresponding bucket is allocated to the feature data from the number of buckets corresponding to the target time partition based on the identity identifier.

[0018] In this embodiment of the disclosure, by performing a modulo operation on the primary key of the feature data and assigning a corresponding bucket to the feature data according to the operation result, feature data with the same primary key can be grouped into the same bucket, thereby improving the accuracy and efficiency of data storage.

[0019] In one possible implementation, the method further includes:

[0020] For each piece of feature data, if the feature data does not carry the time of occurrence of the data, a global index is performed on the buckets already configured in the target storage system based on the primary key of the feature data to determine whether there is a target bucket in the buckets already configured in the target storage system that corresponds to the primary key of the feature data;

[0021] If a target bucket corresponding to the feature data exists in the buckets configured in the target storage system, the feature data is stored in the storage space associated with the target bucket.

[0022] In this embodiment of the disclosure, when the feature data does not carry the time of data occurrence, data can also be stored using a global index. That is, the global index is used as a fallback method. In this way, even when the feature data does not carry the time of data occurrence, data storage can still be achieved, which helps to improve the applicability of the data storage method.

[0023] In one possible implementation, the method further includes:

[0024] If there is no target bucket corresponding to the feature data in the buckets already configured in the target storage system, the feature data is stored in the storage space associated with the first bucket; the first bucket includes buckets that do not contain data in the buckets already configured in the target storage system or newly created buckets.

[0025] In this embodiment of the disclosure, if there is no target bucket corresponding to the feature data in the buckets already configured in the target storage system, the feature data is stored in a new bucket, thus ensuring the accuracy of data storage.

[0026] In one possible implementation, after storing the feature data in the storage space associated with its corresponding bucket, the method further includes:

[0027] Obtain a data reading task, and determine at least one bucket containing the target sub-feature data to be read based on the data reading task;

[0028] Based on at least one bucket containing the target sub-feature data, the data reading task is divided into multiple sub-data reading tasks;

[0029] For each sub-data reading task, a multi-way merge algorithm is used to merge and read the data.

[0030] In this embodiment of the disclosure, the data reading task is divided according to buckets, and for each sub-data reading task, a multi-way merge algorithm is used to merge and read the data, which can improve the data reading efficiency.

[0031] In one possible implementation, feature data stored in the same bucket during the same time period falls into the same data file, and each data file has a corresponding primary key. The step of using a multi-way merge method to read data for each sub-data reading task includes:

[0032] For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined, and the at least one data file is sorted based on the primary key of the at least one data file. Then, a multi-way merge method is used to read data from the sorted at least one data file.

[0033] In this embodiment of the disclosure, sorting the at least one data file based on the primary key of the at least one data file corresponding to the sub-data reading task is beneficial for merging data files with the same primary key, thereby improving data reading efficiency.

[0034] In one possible implementation, sorting the at least one data file based on its primary key includes:

[0035] In the process of sorting the at least one data file, if there are two or more data files with the same primary key, the data files with the same primary key shall be sorted according to the order in which the data files were written.

[0036] In this embodiment of the disclosure, if two or more data files have the same primary key, the data files with the same primary key are sorted according to the order in which they were written. In this way, data files with the same primary key can also be sorted, which helps to improve the sorting accuracy.

[0037] In one possible implementation, feature data stored in the same bucket during the same time period falls into the same data file. Multiple sub-feature data corresponding to each data file are stored column-wise. For each sub-data reading task, a multi-way merge algorithm is used to merge and read the data, including:

[0038] For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined. Target column data is determined from the at least one data file respectively, and a multi-way merge algorithm is used to merge and read the target column data corresponding to the at least one data file respectively.

[0039] In this embodiment of the disclosure, since the feature data in the data file is stored column by column, that is, column clipping is supported, during the read-time merging process, the target sub-feature data that needs to be merged can be merged, thereby improving the efficiency of data reading.

[0040] In one possible implementation, feature data stored in the same bucket during the same time period falls into the same data file, and the method further includes:

[0041] Data files in the target storage system are merged according to a first range to obtain at least one first merged data file; and / or,

[0042] The data files in the target storage system are merged according to a second range to obtain at least one second merged data file; the second range is larger than the first range.

[0043] In this embodiment of the disclosure, after data storage, data merging can be performed according to a first range (small range) or a second range (large range) based on actual needs. In this way, data can be read directly in subsequent data reading processes without the need for read-time merging processing, thereby helping to improve the efficiency of subsequent data reading.

[0044] In one possible implementation, the feature data includes user behavior data during the use of the target application, the user behavior data including sample data and / or sample label data, the sample data and the sample label data being used to train the model.

[0045] In this embodiment of the disclosure, since the feature data includes user behavior data during the use of the target application, the user behavior data can be acquired and stored, and the model can be trained after reading the data. The trained model can then be used to predict user behavior, thereby providing better services to users and improving their user experience.

[0046] This disclosure provides a data storage device, including:

[0047] The data acquisition module is used to acquire at least one piece of feature data to be stored in the target storage system, wherein each piece of feature data includes at least one piece of sub-feature data, and each piece of feature data has a corresponding primary key;

[0048] The partition determination module is used to determine, for each piece of feature data, the target time partition to which the data occurrence time of the feature data belongs from multiple first time partitions corresponding to the target storage system; wherein, each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space of the target storage system; the number of buckets corresponding to each first time partition is determined by the holiday attribute information corresponding to the first time partition, or the number of buckets corresponding to each first time partition is determined by the amount of data stored before the first time partition;

[0049] The data storage module is used to allocate corresponding buckets to the feature data based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, and to store the feature data in the storage space associated with the corresponding bucket.

[0050] In one possible implementation, the feature data includes user behavior data during the use of the target application, the user behavior data including sample data and / or sample label data, the sample data and the sample label data being used to train the model.

[0051] In one possible implementation, the partition determination module is further configured to:

[0052] If the current time meets the preset conditions, the first preset time period after the current time is divided into at least one second time partition;

[0053] Based on the holiday attribute information corresponding to each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition; or...

[0054] Based on the amount of data stored in the second preset time period preceding each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition.

[0055] In one possible implementation, the data storage module is specifically used for:

[0056] Based on the number of buckets corresponding to the target time partition, a modulo operation is performed on the primary key of the feature data to obtain the identity identifier of the bucket corresponding to the feature data, and a corresponding bucket is allocated to the feature data from the number of buckets corresponding to the target time partition based on the identity identifier.

[0057] In one possible implementation, the data storage module is further configured to:

[0058] For each piece of feature data, if the feature data does not carry the time of occurrence of the data, a global index is performed on the buckets already configured in the target storage system based on the primary key of the feature data to determine whether there is a target bucket in the buckets already configured in the target storage system that corresponds to the primary key of the feature data;

[0059] If a target bucket corresponding to the feature data exists in the buckets configured in the target storage system, the feature data is stored in the storage space associated with the target bucket.

[0060] In one possible implementation, the data storage module is further configured to:

[0061] If there is no target bucket corresponding to the feature data in the buckets already configured in the target storage system, the feature data is stored in the storage space associated with the first bucket; the first bucket includes buckets that do not contain data in the buckets already configured in the target storage system or newly created buckets.

[0062] In one possible implementation, the device further includes:

[0063] The task acquisition module is used to acquire data reading tasks and determine at least one bucket containing the target sub-feature data to be read based on the data reading tasks.

[0064] The task partitioning module is used to divide the data reading task into multiple sub-data reading tasks based on at least one bucket where the target sub-feature data is located;

[0065] The data reading module is used to merge and read data for each sub-data reading task using a multi-way merge algorithm.

[0066] In one possible implementation, feature data stored in the same bucket during the same time period falls into the same data file, and each data file has a corresponding primary key. The data reading module is specifically used for:

[0067] For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined, and the at least one data file is sorted based on the primary key of the at least one data file. Then, a multi-way merge method is used to read data from the sorted at least one data file.

[0068] In one possible implementation, the data reading module is specifically used for:

[0069] In the process of sorting the at least one data file, if there are two or more data files with the same primary key, the data files with the same primary key shall be sorted according to the order in which the data files were written.

[0070] In one possible implementation, feature data stored in the same bucket during the same time period falls into the same data file, and multiple sub-feature data corresponding to each data file are stored column-wise. The data reading module is specifically used for:

[0071] For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined. Target column data is determined from the at least one data file respectively, and a multi-way merge algorithm is used to merge and read the target column data corresponding to the at least one data file respectively.

[0072] In one possible implementation, the apparatus further includes a data merging module, the data merging module being used for:

[0073] Data files in the target storage system are merged according to a first range to obtain at least one first merged data file; and / or,

[0074] The data files in the target storage system are merged according to a second range to obtain at least one second merged data file; the second range is larger than the first range.

[0075] This disclosure provides an electronic device, including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the data storage method described in any of the above embodiments is executed.

[0076] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the data storage method described in any of the above embodiments.

[0077] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0078] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0079] Figure 1 A flowchart of a data storage method provided by some embodiments of this disclosure is shown;

[0080] Figure 2 This illustration shows a process for allocating feature data to buckets, as provided in some embodiments of this disclosure.

[0081] Figure 3 A flowchart is shown illustrating another data storage method provided by some embodiments of this disclosure;

[0082] Figure 4 This illustration shows a schematic diagram of a data reading task partitioning process provided by some embodiments of the present disclosure;

[0083] Figure 5 The diagram shows a structural schematic of a data storage device provided in some embodiments of this disclosure;

[0084] Figure 6 This diagram illustrates the structure of another data storage device provided in some embodiments of the present disclosure;

[0085] Figure 7 A schematic diagram of an electronic device provided in some embodiments of the present disclosure is shown. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0087] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0088] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0089] Research has found that current data storage systems typically require global indexing of the data in the storage system during the data storage process to distinguish whether the data to be stored is new data being written or updated data. However, indexing services are usually limited by traffic, affecting the throughput performance of writing data and resulting in low data storage efficiency.

[0090] Based on the above research, this disclosure provides a data storage method. First, at least one piece of feature data to be stored in a target storage system is obtained, wherein each piece of feature data includes at least one piece of sub-feature data, and each piece of feature data has a corresponding primary key. Then, for each piece of feature data, a target time partition to which the data occurrence time belongs is determined from multiple first time partitions corresponding to the target storage system. Each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space in the target storage system. The number of buckets corresponding to each first time partition is determined by the holiday attribute information corresponding to the first time partition, or by the amount of data stored before the first time partition. Finally, based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, a corresponding bucket is allocated to the feature data, and the feature data is stored in the storage space associated with its corresponding bucket.

[0091] In this embodiment of the disclosure, since the target storage system corresponds to multiple first time partitions, and each first time partition corresponds to a corresponding number of buckets, during the data storage process, the target time partition to which the feature data belongs can be determined first based on the data occurrence time of the feature data. Then, based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, a corresponding bucket can be allocated for each feature data. Thus, without the need for a global index, the feature data can be stored in the corresponding storage space within the target storage system. That is, the throughput performance of writing data can be improved by appending writes, thereby improving the efficiency of data storage.

[0092] Furthermore, since the number of buckets corresponding to each first time partition is determined by the holiday attribute information corresponding to the first time partition, or the number of buckets corresponding to each first time partition is determined by the amount of data stored before the first time partition, the number of buckets can be evolved. That is, the number of buckets can be adjusted according to the actual prediction needs, thereby reducing the probability of data overflow or a large amount of storage space remaining due to setting too many or too few buckets.

[0093] To facilitate understanding of this embodiment, the executing entity of the data storage method provided in this disclosure will first be described in detail. The executing entity of the data storage method provided in this disclosure is an electronic device. In this embodiment, the electronic device is a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In other embodiments, the electronic device can also be a terminal device. This terminal device can be a mobile device, user terminal, handheld device, computing device, or wearable device, etc. Furthermore, this data storage method can also be implemented by a processor calling computer-readable instructions stored in memory.

[0094] The data storage method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. See also Figure 1 The diagram shown is a flowchart of a data storage method provided in an embodiment of this disclosure. The data storage method includes the following steps S101 to S102:

[0095] S101, obtain at least one piece of feature data to be stored in the target storage system, wherein each piece of feature data includes at least one sub-feature data, and each piece of feature data has a corresponding primary key.

[0096] In some implementations, a distributed framework can be used to load at least one piece of feature data that needs to be stored. When each piece of data includes multiple sub-feature data, there are pre-defined relationships between these sub-feature data. For example, these multiple sub-feature data may be usage behavior data of the same user within the same time frame. Specifically, these multiple sub-feature data may include data content type, username, user attributes, and user behavior, etc. The primary key, also known as the primary identifier, is a unique identifier for each piece of feature data; that is, different pieces of feature data can be distinguished by their primary key.

[0097] Optionally, the feature data can be user behavior data during the use of the target application. This behavior data includes sample data and / or sample label data. The sample data and sample label data are used to train a model. The trained model can then be used to predict the user's target usage behavior and, based on this target behavior, display content matching the target usage behavior to the user. In other embodiments, the feature data can also be user data that needs to be retained based on General Data Protection Regulation (GDPR) compliance requirements.

[0098] It should be noted that if the model performance is improved after training with the newly added sample data and sample label data, the newly added sample data and sample label data will be retained; if the model performance is reduced, the newly added sample data and sample label data will be deleted, thereby saving corresponding storage space.

[0099] For example, the target application can be a short video application, a news discovery application, or a shopping application, etc., without specific limitations. The sample data refers to data used to characterize user behavior, and the sample tag data refers to data used to characterize the results of user behavior. The content of the sample data and the content of the sample tag data differ for different types of applications.

[0100] For example, for news applications, the sample data could be data such as the content and duration of news browsing, and correspondingly, the sample tag data could be data on whether a news item was saved, liked, or disliked. For shopping applications, the sample data could be consumption data on whether a user searched for or watched a certain type of shopping content, and correspondingly, the sample tag data could be data on whether the user placed an order or added a product to their shopping cart. For short video applications, the sample data could be data on the duration of a user's viewing of a certain type or a specific video, while the sample tag data could be data on whether the user was interested in the video.

[0101] It should be noted that the sample data typically occurs earlier than the sample label data. Furthermore, the examples of sample data and sample label data above are merely illustrative; in other embodiments, the sample data and sample label data can be determined based on actual application requirements for different types of applications. Additionally, the classification of the target applications described above is also just an example; some applications may correspond to several different categories. For instance, some short video applications also have shopping functions and can be classified as shopping applications.

[0102] S102, for each piece of feature data, determine the target time partition to which the data occurrence time of the feature data belongs from multiple first time partitions corresponding to the target storage system; wherein, each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space of the target storage system; the number of buckets corresponding to each first time partition is determined by the holiday attribute information corresponding to the first time partition, or the number of buckets corresponding to each first time partition is determined by the amount of data stored before the first time partition.

[0103] Each first time partition corresponds to a time period, the length of which can be set according to actual needs. The lengths of the time periods corresponding to each first time partition can be the same or different. Furthermore, compared to the current time of acquiring at least one piece of feature data to be stored in the target storage system, the time period corresponding to each first time partition can include the time period before the current time, the time period to which the current time belongs, and the time period after the current time.

[0104] In this context, "bucket" is a logical concept used to logically divide the data that needs to be stored. Each bucket corresponds to a storage space in the target storage system, and the size of the storage space corresponding to each bucket is not specifically limited, but can be determined according to the requirements.

[0105] Specifically, the number of buckets corresponding to each first time partition can be determined through the following steps (1) to (2):

[0106] (1) If the current time meets the preset conditions, the first preset time period after the current time is divided into at least one second time partition;

[0107] (2) Set a corresponding number of buckets for each second time partition according to the festival attribute information corresponding to each second time partition, and determine the second time partition as the first time partition; or, set a corresponding number of buckets for each second time partition according to the data storage amount of the second preset time period before each second time partition, and determine the second time partition as the first time partition.

[0108] For example, a preset number of buckets can be set for each second time partition based on forecasting needs. For instance, the time period corresponding to the second time partition can be set to half a month, with the number of buckets for the second time partition (1-15) set on the 1st of each month, and the number of buckets for the second time partition (16-30) set on the 15th of each month. It should be noted that the second time partition refers to the time period after the current moment.

[0109] Of course, the time period corresponding to the second time partition can also be a day or even shorter, without specific limitations. For example, if the number of users using shopping applications increases significantly on a certain day (such as a shopping festival), the number of buckets corresponding to the second time partition for that date can be increased. In this way, when user traffic increases significantly, the concurrency can be improved by increasing the number of buckets, without having to spend a lot of resources and time migrating the data that has already been written.

[0110] In this embodiment of the disclosure, not only can the number of buckets be allocated to different time partitions based on the estimated data volume over time, but the number of buckets for the time partition to be allocated can also be determined based on the data volume over a previous period. In this way, the number of buckets can be automatically configured, and the number of configured buckets can meet the corresponding data storage requirements. This not only improves the efficiency of setting the number of buckets, but also enhances the applicability of the data storage method.

[0111] In addition, the data occurrence time refers to the time when the data is generated. For example, when a user is using a target application, the target application pushes an advertisement to the user. The time when the advertisement is pushed is the time when the advertisement occurs. Subsequent data such as whether the advertisement is clicked or viewed, or whether an order is placed, will carry the time when the advertisement occurs.

[0112] S103, based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, allocate corresponding buckets to the feature data and store the feature data in the storage space associated with its corresponding bucket.

[0113] In some implementations, see Figure 2 As shown, for each piece of feature data, a modulo operation can be performed on the primary key of the feature data to obtain the identifier of the bucket corresponding to the feature data, and the corresponding bucket can be assigned to the feature data based on the identifier. Specifically, the primary key of the feature data can be converted into a number, and the number can be divided by a preset number of buckets corresponding to the target time partition (e.g., 1000). The remainder can be used as the identifier of the bucket. For example, after the modulo operation, bucket 1 can be assigned to feature data 1, bucket 2 to feature data 2, bucket 3 to feature data 3, and so on.

[0114] In some embodiments, for each piece of feature data, if the feature data does not carry the time of occurrence of the data, a global index can be performed on the buckets configured in the target storage system based on the primary key of the feature data to determine whether there is a target bucket corresponding to the primary key of the feature data in the buckets configured in the target storage system; if there is a target bucket corresponding to the feature data in the buckets configured in the target storage system, the feature data is stored in the storage space associated with the target bucket.

[0115] In this embodiment of the disclosure, when the feature data does not carry the time of data occurrence, data can also be stored using a global index. That is, a global index can be used as a fallback method. In this way, even when the feature data does not carry the time of data occurrence, data storage can still be achieved, which helps to improve the applicability of the data storage method.

[0116] Furthermore, if no target bucket corresponding to the feature data exists in the buckets already configured in the target storage system, the feature data can be stored in the storage space associated with the first bucket. The first bucket includes buckets that do not contain data in the buckets already configured in the target storage system or newly created buckets. For example, if a global index determines that no target bucket corresponding to the feature data exists in the configured buckets, the feature data can be stored in an empty bucket, and if no empty bucket exists, a new bucket can be created. Here, an empty bucket means that the storage space associated with the bucket does not store any data, thus ensuring the accuracy of data storage.

[0117] Optionally, for some types of feature data (such as advertising feature data), if the feature data does not carry the data occurrence time, the data occurrence time of the feature data can be determined through the primary key of the feature data. For example, the data occurrence time corresponding to the feature data can be found based on the primary key of the feature data using a global indexing method of hash mapping.

[0118] It should be understood that after feature data is stored, feature data stored in the same bucket during the same time period falls into the same data file, and each data file has a corresponding primary key. When training the model, it is necessary to read the feature data required for training from the target storage system. Therefore, see [link to relevant documentation]. Figure 3 The diagram shown is a flowchart of another data storage method provided in an embodiment of this disclosure. Figure 1 The storage method described above differs from the previous one in that it includes the following steps S104 to S106 after step S103:

[0119] S104, obtain the data reading task, and determine at least one bucket where the task data file to be read is located based on the data reading task.

[0120] For example, after obtaining a data reading task, at least one bucket containing the target sub-feature data to be read can be determined based on the data reading task. The data reading task can be set by the user. For example, the sub-feature data to be read can be determined based on the predicate (the word corresponding to the relevant sub-feature data) or time partition determined by the user.

[0121] It should be understood that since multiple feature data written to the same bucket in the same time period (in the same batch) will fall into the same data file, and as the feature data is written to the target storage system at different times, feature data with the same primary key will fall into different data files, when reading data, after determining the target data file to which the target sub-feature data to be read belongs, the target sub-feature data with the same primary key in multiple target data files under the same bucket can be merged.

[0122] S105, based on at least one bucket where the task data file is located, the data reading task is divided into multiple sub-data reading tasks.

[0123] For example, see Figure 4 As shown, after determining at least one bucket where the target sub-feature data resides, the data reading task can be divided into multiple sub-data reading tasks based on the at least one bucket where the target sub-feature data resides. For example, if it is determined that the target sub-feature data to be read falls into bucket 1, bucket 2, and bucket 3 respectively, the data reading task can be divided by bucket. That is, the reading task corresponding to the target sub-feature data for each bucket is determined as a sub-data reading task. Therefore, bucket 1 corresponds to sub-task 1, bucket 2 corresponds to sub-task 2, and bucket 3 corresponds to sub-task 3.

[0124] Of course, in other embodiments, the data reading task can be divided in other ways, such as by dividing the data reading task of the target sub-feature data of multiple buckets in the same time partition into a sub-data reading task.

[0125] S106: For each sub-data reading task, a multi-way merge algorithm is used to merge and read the data.

[0126] For example, a multi-way merge algorithm can be used to merge reads for each sub-data reading task (such as sub-task 1). In this way, during data reading, it is not necessary to load all data files into memory at the same time, thus avoiding performance degradation caused by data overflowing to disk.

[0127] Specifically, for each sub-data reading task, the at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs can be determined first. Then, based on the primary key of the at least one data file, the at least one data file is sorted. Finally, the sorted at least one data file is read using a multi-way merge method.

[0128] Optionally, during the sorting process of the at least one data file, if two or more data files have the same primary key, the data files with the same primary key are sorted according to the order in which they were written. This ensures the accuracy of data reading.

[0129] In some embodiments, the multiple sub-feature data corresponding to each data file are stored column-wise. When performing data merging and reading using a multi-way merge algorithm for each sub-data reading task, the process may include: for each sub-data reading task, determining at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs; determining target column data from the at least one data file; and performing data merging and reading using a multi-way merge algorithm on the target column data corresponding to the at least one data file. That is, since the sub-feature data in the data file supports column pruning, during the read-time merging process, only the required target columns can be merged. For example, if the data file contains tens of thousands of columns of data, but only three hundred columns are needed during data reading, then only the sub-feature data of these three hundred target columns can be merged during read reading, thereby improving the efficiency of data reading.

[0130] It should be noted that during the read-time merging process of sub-feature data, if a sub-feature data is updated multiple times, the last updated sub-feature data will be used as the output (result), or the first sub-feature data written will be used as the output, depending on the actual situation.

[0131] It is understandable that after the feature data is stored in the target storage system, before reading the data, in order to improve the efficiency of data reading, the stored feature data can be merged. Therefore, in some embodiments, the method further includes: merging the feature data in the target storage system according to a first range to obtain at least one first merged data file. The first range is smaller than a preset range; for example, it can merge the feature data stored each day on a daily basis, or it can merge files on a bucket basis. That is, for the feature data in the target storage system, a small-scale merging can be performed to obtain at least one first merged data file. Thus, when reading data, if the target sub-feature data to be read belongs to the first merged data file, it can be read directly without merging, thereby improving the efficiency of data reading. It should be noted that if the target sub-feature data to be read is distributed across different first merged data files, then the different first merged data files need to be merged before reading.

[0132] Of course, in other embodiments, the feature data within the target storage system can also be merged according to a second range to obtain at least one second merged data file; the second range is larger than the first range. For example, the feature data files stored in each month or quarter can be merged on a monthly or quarterly basis to obtain at least one second merged data file.

[0133] In this embodiment of the disclosure, since the target storage system can also merge the stored feature data, it can improve the efficiency of subsequent data reading while providing more options for subsequent data reading. For example, for data warehouse queries, a large-scale merge can be performed after the data is stored for several hours or days, and the layout of the data files can be rewritten (such as using Z-Order to speed up queries). If the data files are no longer sorted by primary key, the data reading process is transformed from a multi-way merge algorithm to a Hash-Join algorithm. At this time, the data reading performance is still highly guaranteed because after the large-scale merge, the number of subsequent updates will be greatly reduced (assuming that the main updates all occurred recently and before the large-scale merge) and can be basically all placed in memory.

[0134] Those skilled in the art will understand that, in the methods described above in the specific embodiments, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. Furthermore, the various embodiments can be combined with each other to form different technical solutions.

[0135] Based on the same technical concept, this disclosure also provides a data storage device corresponding to the data storage method. Since the principle of the device in this disclosure for solving the problem is similar to the data storage method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0136] Reference Figure 5 The diagram shown is a schematic representation of a data storage device 500 provided in an embodiment of this disclosure. The device includes:

[0137] The data acquisition module 501 is used to acquire at least one piece of feature data to be stored in the target storage system, wherein each piece of feature data includes at least one piece of sub-feature data, and each piece of feature data has a corresponding primary key;

[0138] The partition determination module 502 is used to determine, for each piece of feature data, the target time partition to which the data occurrence time of the feature data belongs from multiple first time partitions corresponding to the target storage system; wherein, each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space of the target storage system; the number of buckets corresponding to each first time partition is determined by the holiday attribute information corresponding to the first time partition, or the number of buckets corresponding to each first time partition is determined by the amount of data stored before the first time partition;

[0139] The data storage module 503 is used to allocate corresponding buckets to the feature data based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, and to store the feature data in the storage space associated with the corresponding bucket.

[0140] In one possible implementation, the feature data includes user behavior data during the use of the target application, the user behavior data including sample data and / or sample label data, the sample data and the sample label data being used to train the model.

[0141] In one possible implementation, the partition determination module 502 is further configured to:

[0142] If the current time meets the preset conditions, the first preset time period after the current time is divided into at least one second time partition;

[0143] Based on the holiday attribute information corresponding to each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition; or...

[0144] Based on the amount of data stored in the second preset time period preceding each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition.

[0145] In one possible implementation, the data storage module 503 is specifically used for:

[0146] Based on the number of buckets corresponding to the target time partition, a modulo operation is performed on the primary key of the feature data to obtain the identity identifier of the bucket corresponding to the feature data, and a corresponding bucket is allocated to the feature data from the number of buckets corresponding to the target time partition based on the identity identifier.

[0147] In one possible implementation, the data storage module 503 is further configured to:

[0148] For each piece of feature data, if the feature data does not carry the time of occurrence of the data, a global index is performed on the buckets already configured in the target storage system based on the primary key of the feature data to determine whether there is a target bucket in the buckets already configured in the target storage system that corresponds to the primary key of the feature data;

[0149] If a target bucket corresponding to the feature data exists in the buckets configured in the target storage system, the feature data is stored in the storage space associated with the target bucket.

[0150] In one possible implementation, the data storage module 503 is further configured to:

[0151] If there is no target bucket corresponding to the feature data in the buckets already configured in the target storage system, the feature data is stored in the storage space associated with the first bucket; the first bucket includes buckets that do not contain data in the buckets already configured in the target storage system or newly created buckets.

[0152] In one possible implementation, see Figure 6 As shown, the device further includes:

[0153] The task acquisition module 504 is used to acquire data reading tasks and determine at least one bucket containing the target sub-feature data to be read based on the data reading tasks.

[0154] The task partitioning module 505 is used to partition the data reading task into multiple sub-data reading tasks based on at least one bucket where the target sub-feature data is located.

[0155] The data reading module 506 is used to merge and read data for each sub-data reading task using a multi-way merge algorithm.

[0156] In one possible implementation, feature data stored in the same bucket during the same time period falls into the same data file, and each data file has a corresponding primary key. The data reading module 504 is specifically used for:

[0157] For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined, and the at least one data file is sorted based on the primary key of the at least one data file. Then, a multi-way merge method is used to read data from the sorted at least one data file.

[0158] In one possible implementation, the data reading module 504 is specifically used for:

[0159] In the process of sorting the at least one data file, if there are two or more data files with the same primary key, the data files with the same primary key shall be sorted according to the order in which the data files were written.

[0160] In one possible implementation, feature data stored in the same bucket during the same time period fall into the same data file, and multiple sub-feature data corresponding to each data file are stored in columns. The data reading module 506 is specifically used for:

[0161] For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined. Target column data is determined from the at least one data file respectively, and a multi-way merge algorithm is used to merge and read the target column data corresponding to the at least one data file respectively.

[0162] In one possible implementation, the device further includes a data merging module 507, which is used for:

[0163] Data files in the target storage system are merged according to a first range to obtain at least one first merged data file; and / or,

[0164] The data files in the target storage system are merged according to a second range to obtain at least one second merged data file; the second range is larger than the first range.

[0165] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0166] Based on the same technical concept, this disclosure also provides an electronic device. (See also...) Figure 7 The diagram shown is a structural schematic of an electronic device 700 provided in an embodiment of this disclosure, including a processor 701, a memory 702, and a bus 703. The memory 702 is used to store execution instructions and includes a main memory 7021 and an external memory 7022. The main memory 7021, also called internal memory, is used to temporarily store computational data in the processor 701, as well as data exchanged with external memory 7022 such as a hard disk. The processor 701 exchanges data with the external memory 7022 through the main memory 7021.

[0167] In this embodiment, the memory 702 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 701. That is, when the electronic device 700 is running, the processor 701 communicates with the memory 702 through the bus 703, so that the processor 701 executes the application code stored in the memory 702, and then executes the method described in any of the foregoing embodiments.

[0168] The memory 702 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0169] Processor 701 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0170] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 700. In other embodiments of this application, the electronic device 700 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0171] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data storage method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0172] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the data storage method in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0173] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0174] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0177] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0178] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data storage method, characterized in that, include: Obtain at least one piece of feature data to be stored in the target storage system, wherein each piece of feature data includes at least one piece of sub-feature data, and each piece of feature data has a corresponding primary key; For each piece of feature data, the target time partition to which the data occurrence time of the feature data belongs is determined from multiple first time partitions corresponding to the target storage system; wherein, each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space of the target storage system; the number of buckets corresponding to each first time partition is positively correlated with the expected traffic of the first time partition, and the expected traffic is determined according to at least one of the following: determined by the holiday attribute information corresponding to the first time partition, or determined by the amount of data stored before the first time partition; Based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, a corresponding bucket is allocated to the feature data, and the feature data is stored in the storage space associated with its corresponding bucket; The step of allocating corresponding buckets to the feature data based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, and storing the feature data in the storage space associated with its corresponding bucket, includes: Based on the number of buckets corresponding to the target time partition, a modulo operation is performed on the primary key of the feature data to obtain the identity identifier of the bucket corresponding to the feature data; Based on the bucket's identity identifier, the feature data is stored in the storage space associated with the bucket corresponding to the identity identifier.

2. The method according to claim 1, characterized in that, Before acquiring at least one piece of feature data to be stored in the target storage system, the number of buckets corresponding to each first time partition is determined through the following steps: If the current time meets the preset conditions, the first preset time period after the current time is divided into at least one second time partition; Based on the holiday attribute information corresponding to each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition; or... Based on the amount of data stored in the second preset time period preceding each second time partition, a corresponding number of buckets are set for each second time partition, and the second time partition is determined as the first time partition.

3. The method according to claim 1, characterized in that, The process of allocating corresponding buckets to the feature data based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data includes: Based on the number of buckets corresponding to the target time partition, a modulo operation is performed on the primary key of the feature data to obtain the identity identifier of the bucket corresponding to the feature data, and a corresponding bucket is allocated to the feature data from the number of buckets corresponding to the target time partition based on the identity identifier.

4. The method according to claim 1, characterized in that, The method further includes: For each piece of feature data, if the feature data does not carry the time of occurrence of the data, a global index is performed on the buckets already configured in the target storage system based on the primary key of the feature data to determine whether there is a target bucket in the buckets already configured in the target storage system that corresponds to the primary key of the feature data; If a target bucket corresponding to the feature data exists in the buckets configured in the target storage system, the feature data is stored in the storage space associated with the target bucket.

5. The method according to claim 4, characterized in that, The method further includes: If there is no target bucket corresponding to the feature data in the buckets already configured in the target storage system, the feature data is stored in the storage space associated with the first bucket; the first bucket includes buckets that do not contain data in the buckets already configured in the target storage system or newly created buckets.

6. The method according to claim 1, characterized in that, After storing the feature data in the storage space associated with its corresponding bucket, the method further includes: Obtain a data reading task, and determine at least one bucket containing the target sub-feature data to be read based on the data reading task; Based on at least one bucket containing the target sub-feature data, the data reading task is divided into multiple sub-data reading tasks; For each sub-data reading task, a multi-way merge algorithm is used to merge and read the data.

7. The method according to claim 6, characterized in that, Feature data stored in the same bucket during the same time period falls into the same data file. Each data file has a corresponding primary key. For each sub-data reading task, a multi-way merge method is used for data reading, including: For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined, and the at least one data file is sorted based on the primary key of the at least one data file. Then, a multi-way merge method is used to read data from the sorted at least one data file.

8. The method according to claim 7, characterized in that, The sorting of the at least one data file based on its primary key includes: In the process of sorting the at least one data file, if there are two or more data files with the same primary key, the data files with the same primary key shall be sorted according to the order in which the data files were written.

9. The method according to claim 6, characterized in that, Feature data stored in the same bucket during the same time period falls into the same data file. Multiple sub-feature data corresponding to each data file are stored column-wise. For each sub-data reading task, a multi-way merge algorithm is used to merge and read the data, including: For each sub-data reading task, at least one data file to which the target sub-feature data corresponding to the sub-data reading task belongs is determined. Target column data is determined from the at least one data file respectively, and a multi-way merge algorithm is used to merge and read the target column data corresponding to the at least one data file respectively.

10. The method according to claim 1, characterized in that, Feature data stored in the same bucket during the same time period are included in the same data file. The method further includes: Data files in the target storage system are merged according to a first range to obtain at least one first merged data file; and / or, The data files in the target storage system are merged according to a second range to obtain at least one second merged data file; the second range is larger than the first range.

11. The method according to claim 1, characterized in that, The feature data includes user behavior data during the use of the target application, and the behavior data includes sample data and / or sample label data, which are used to train the model.

12. A data storage device, characterized in that, include: The data acquisition module is used to acquire at least one piece of feature data to be stored in the target storage system, wherein each piece of feature data includes at least one piece of sub-feature data, and each piece of feature data has a corresponding primary key; The partition determination module is used to determine, for each piece of feature data, the target time partition to which the data occurrence time of the feature data belongs from multiple first time partitions corresponding to the target storage system; wherein, each first time partition corresponds to a corresponding number of buckets, and each bucket is associated with a storage space of the target storage system; the number of buckets corresponding to each first time partition is positively correlated with the expected traffic of the first time partition, and the expected traffic is determined according to at least one of the following: determined by the holiday attribute information corresponding to the first time partition, or determined by the amount of data stored before the first time partition; The data storage module is used to allocate corresponding buckets to the feature data based on the number of buckets corresponding to the target time partition and the primary key corresponding to the feature data, and to store the feature data in the storage space associated with the corresponding bucket. The data storage module is used for: Based on the number of buckets corresponding to the target time partition, a modulo operation is performed on the primary key of the feature data to obtain the identity identifier of the bucket corresponding to the feature data; Based on the bucket's identity identifier, the feature data is stored in the storage space associated with the bucket corresponding to the identity identifier.

13. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the data storage method as described in any one of claims 1-11 is performed.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the data storage method as described in any one of claims 1-11.

Images