Label data storage method, computer equipment, readable storage medium and program product

Abstract

The invention relates to a tag data storage method and device, a computer readable storage medium and a computer program product, relates to the technical field of data storage, and can give consideration to tag data query efficiency and data storage cost. The method comprises the following steps: acquiring respective label data of a plurality of data objects; the object identifiers corresponding to the data objects form a first identifier sequence; obtaining a second identification sequence according to a random sampling result of the plurality of data objects; the identification distribution condition in the second identification sequence is matched with the identification distribution condition of the first identification sequence; determining a target object identifier corresponding to the partition position in the second identifier sequence, segmenting the plurality of data objects according to the target object identifier, and obtaining a plurality of data object partitions according to a segmentation result; and compressing label data, corresponding to the same label item, of the data object in the same data object partition to obtain a label data compression result corresponding to the label item, and storing each label data compression result to an online storage unit.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a tag data storage method, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of computer technology, massive amounts of label data are generated in daily business scenarios. Massive label data are generally produced by the business side and stored in offline storage units with low storage costs. However, offline storage units have problems such as poor real-time query performance, high latency, and low concurrency.

[0003] In the related art, in order to improve the efficiency of using massive label data, the label data is transferred to a high-performance storage unit in row storage to meet the demand for real-time use of label data. However, the inventors found in practice that the storage cost of the above storage method in a high-performance storage unit is much higher than that in an offline storage unit, making it difficult to import a large amount of label data in batches. It can be seen that the storage of label data in the related art is difficult to take into account both data query efficiency and data storage cost at the same time. Summary of the invention

[0004] Based on this, it is necessary to provide a tag data storage method, a computer device, a computer-readable storage medium and a computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a tag data storage method, comprising:

[0006] Acquire label data of each of a plurality of data objects; the object identifiers corresponding to each of the plurality of data objects constitute a first identifier sequence;

[0007] A second identification sequence is obtained according to the random sampling results of the plurality of data objects; the identification distribution in the second identification sequence matches the identification distribution in the first identification sequence;

[0008] Determine a target object identifier corresponding to a partition position in the second identifier sequence, and segment the plurality of data objects according to the target object identifier, and obtain a plurality of data object partitions according to the segmentation result;

[0009] The tag data corresponding to the same tag item of the data objects in the same data object partition are compressed to obtain the tag data compression results corresponding to the tag items, and each of the tag data compression results is stored in an online storage unit.

[0010] In one embodiment, obtaining the second identification sequence according to the random sampling results of the plurality of data objects includes:

[0011] Randomly sampling the multiple data objects according to a predetermined sampling ratio to obtain multiple sampled data objects;

[0012] The object identifier of each of the sampled data objects is obtained, and the object identifiers of each of the sampled data objects are sorted according to the order in which the object identifiers of each of the sampled data objects appear in the first identifier sequence to obtain a second identifier sequence.

[0013] In one embodiment, determining the target object identifier corresponding to the partition position in the second identifier sequence includes:

[0014] Determining an original partition step size of the plurality of data objects;

[0015] Reducing the original partition step size according to the sampling ratio to obtain an adjusted partition step size;

[0016] According to the adjusted partition step, a target object identifier corresponding to the partition position is determined from the second identifier sequence.

[0017] In one embodiment, storing the compression results of each tag data into an online storage unit includes:

[0018] Acquire first partition position information corresponding to a plurality of the tag items in the same data object partition; the first partition position information records the partition position of the tag data compression result of each of the tag items;

[0019] Acquire second partition location information corresponding to the plurality of data object partitions; the second partition location information records the data objects contained in each of the data object partitions;

[0020] Generate a data partition rule according to the first partition location information and the second partition location information;

[0021] The data partitioning rule and each of the label data compression results are associated and stored in an online storage unit.

[0022] In one embodiment, the step of associating the data partitioning rule with each of the label data compression results and storing them in an online storage unit includes:

[0023] Generating a data version number corresponding to the data partitioning rule for the tag data of the plurality of data objects;

[0024] The data version number, the data partitioning rule and each tag data compression result are associated and stored in a preset storage unit, so that when a tag data query event occurs, the data partitioning rule is determined according to the data version number, and the tag data is queried in the preset storage unit according to the data partitioning rule.

[0025] In one embodiment, compressing the tag data corresponding to the same tag item of the data objects in the same data object partition to obtain the tag data compression result corresponding to the tag item includes:

[0026] For each of the data objects in the same data object partition, determine data vacancy information of each of the data objects under the same tag item, and generate a null value bitmap of the tag item according to the data vacancy information; the data vacancy information indicates whether the data object has tag data under the tag item;

[0027] Convert all label data corresponding to the same label item in the same data object partition into arrays of known length respectively, and continuously splice the arrays to obtain a data splicing result;

[0028] The data concatenation result, the null value bitmap of the tag item and the array length information are compressed to obtain a tag data compression result corresponding to the tag item.

[0029] In one embodiment, converting all tag data corresponding to the same tag item in the same data object partition into arrays of known lengths includes:

[0030] Determine the tag data type of each tag item in the same data object partition;

[0031] If the tag data type is an integer, determining a first digit range according to all tag data corresponding to the tag item in the data object partition, and converting each tag data in all the tag data into a byte array of a preset length according to the first digit range;

[0032] If the tag data type is a floating point type, determining a second digit range of an integer part and a third digit range of a decimal part according to all tag data corresponding to the tag item in the data object partition, and converting each tag data in all the tag data into a byte array of a preset length according to the second digit range and the third digit range;

[0033] If the tag data type is character type, all tag data corresponding to the tag item in the data object partition are converted into a byte array.

[0034] In a second aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Acquire label data of each of a plurality of data objects; the object identifiers corresponding to each of the plurality of data objects constitute a first identifier sequence;

[0036] A second identification sequence is obtained according to the random sampling results of the plurality of data objects; the identification distribution in the second identification sequence matches the identification distribution in the first identification sequence;

[0037] Determine a target object identifier corresponding to a partition position in the second identifier sequence, and segment the plurality of data objects according to the target object identifier, and obtain a plurality of data object partitions according to the segmentation result;

[0038] The tag data corresponding to the same tag item of the data objects in the same data object partition are compressed to obtain the tag data compression results corresponding to the tag items, and each of the tag data compression results is stored in an online storage unit.

[0039] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] Acquire label data of each of a plurality of data objects; the object identifiers corresponding to each of the plurality of data objects constitute a first identifier sequence;

[0041] A second identification sequence is obtained according to the random sampling results of the plurality of data objects; the identification distribution in the second identification sequence matches the identification distribution in the first identification sequence;

[0042] Determine a target object identifier corresponding to a partition position in the second identifier sequence, and segment the plurality of data objects according to the target object identifier, and obtain a plurality of data object partitions according to the segmentation result;

[0043] The tag data corresponding to the same tag item of the data objects in the same data object partition are compressed to obtain the tag data compression results corresponding to the tag items, and each of the tag data compression results is stored in an online storage unit.

[0044] In a fourth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0045] Acquire label data of each of a plurality of data objects; the object identifiers corresponding to each of the plurality of data objects constitute a first identifier sequence;

[0046] A second identification sequence is obtained according to the random sampling results of the plurality of data objects; the identification distribution in the second identification sequence matches the identification distribution in the first identification sequence;

[0047] Determine a target object identifier corresponding to a partition position in the second identifier sequence, and segment the plurality of data objects according to the target object identifier, and obtain a plurality of data object partitions according to the segmentation result;

[0048] The tag data corresponding to the same tag item of the data objects in the same data object partition are compressed to obtain the tag data compression results corresponding to the tag items, and each of the tag data compression results is stored in an online storage unit.

[0049] The above-mentioned label data storage method, computer device, computer-readable storage medium and computer program product can obtain label data of each of multiple data objects, and the object identifiers corresponding to each of the multiple data objects constitute a first identifier sequence; then, based on the random sampling results of the multiple data objects, a second identifier sequence is obtained, and the identifier distribution in the second identifier sequence matches the identifier distribution in the first identifier sequence; then, the target object identifier corresponding to the partition position in the second identifier sequence is determined, and the multiple data objects are segmented according to the target object identifier, and multiple data object partitions are obtained according to the segmentation results; then, the label data corresponding to the same label item of the data objects in the same data object partition is compressed to obtain the label data compression result corresponding to the label item, and each label data compression result is stored in an online storage unit. In this embodiment, on the one hand, by obtaining a second identification sequence based on the random sampling results of multiple data objects, and using the target object identification determined from the second identification sequence to segment the multiple data objects, it is possible to reasonably partition the naturally sorted massive data objects, which helps to avoid data skew in different data object partitions after compression; on the other hand, by compressing the label data corresponding to the same label item of data objects in the same data object partition, it is possible to implement the storage of massive label data in online storage units based on the idea of ​​column-based compressed storage, so that the storage cost of high-performance storage units is effectively reduced while providing high-performance query and computing capabilities, thereby effectively solving the bottleneck of traditional label data storage methods that cannot take into account both high-performance query and low-cost import of large quantities of label data into online storage units. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram of a flow chart of a tag data storage method in one embodiment;

[0052] Figure 2 An architectural diagram of a label data calling system in one embodiment;

[0053] Figure 3 A schematic diagram of a flow chart of a step of storing a tag data compression result in one embodiment;

[0054] Figure 4 A schematic diagram of a data partitioning rule in an embodiment;

[0055] Figure 5 A schematic diagram of compression of different types of tag data in one embodiment;

[0056] Figure 6 A schematic diagram of an application interface of a label data calling system in one embodiment;

[0057] Figure 7 A schematic diagram of a process of storing and querying tag data in an embodiment;

[0058] Figure 8 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] In order to enable those skilled in the art to better understand the present application, the relevant technologies are introduced below.

[0061] With the development of computer technology, massive amounts of tagged data are generated in daily business scenarios. Massive amounts of tagged data are generally produced by the business side and stored in offline storage units with low storage costs. Offline storage units have the characteristics of low storage costs and support for rapid batch import of massive data, but they also have problems such as poor real-time query performance, high latency, and low concurrency. In actual applications, the business side often needs to use tagged data to generate real-time decision information to adapt to the flexibility of business status, such as pushing activity information, promotion information, and recommended music lists or products based on changes in user status.

[0062] In the related art, in order to provide high concurrency and low latency data query and computing capabilities, the label data is often transferred to a high-performance storage unit in row storage to meet the needs of real-time use of label data. This process is also called offline data online. However, the storage cost of traditional storage methods in high-performance storage units is much higher than that of offline storage units, and the batch import performance is poor, and it cannot support large-scale data writing. It can be seen that the storage of label data in the related art is difficult to take into account both data query efficiency and data storage costs at the same time.

[0063] Based on this, the present application provides a tag data storage method, a computer device, a computer-readable storage medium and a computer program product to at least solve the above technical problems.

[0064] In one embodiment, Figure 1 As shown, a tag data storage method is provided. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps S101 to S104:

[0065] S101, obtaining label data of each of a plurality of data objects; object identifiers corresponding to each of the plurality of data objects constitute a first identifier sequence.

[0066] Among them, the label data is the data value corresponding to the label item. Taking the label data stored when building a user portrait as an example, the label data of the label item "age" can be the data value 18, and the label data of the label item "gender" can be the data value "male" or "female".

[0067] The data object can be the basic collection unit when collecting label data, that is, in actual applications, for each data object, the label data of the data object under each label item can be collected. In some exemplary embodiments, the data object can be any of the following: account, media resource, device, business. It can be understood that when applying the method of the present application, various data objects can be set according to actual conditions, and the label data of the data object under each label item can be collected to obtain the label data of multiple data objects.

[0068] In practical applications, the label data of each of the multiple data objects can be obtained. In some exemplary embodiments, the label data of each of the multiple data objects can be obtained from an offline storage unit. For example, the following can be constructed: Figure 2The label data calling system shown will apply the label data storage method provided by this application. The system can be deployed in multiple servers, and there is no state between server nodes. In this system, offline storage units, online storage units, real-time interfaces, and online services can be included. Among them, the online storage unit is also called a high-performance storage unit. One or more types of online storage units can be set in the system, such as MongoDB storage units, Redis storage units, or other self-developed high-performance storage units. Figure 2 As shown, the offline storage unit can execute a data import task to import the tag data stored in the offline storage unit into the online storage unit.

[0069] After acquiring the tag data of multiple data objects, the import order or current storage order of the multiple data objects can be determined, and the identification sequence can be obtained according to the object identifications corresponding to the multiple data objects. For the convenience of distinction, the identification sequence composed of the object identifications of the multiple data objects is called the first identification sequence.

[0070] S102, obtaining a second identification sequence according to random sampling results of multiple data objects; the identification distribution in the second identification sequence matches the identification distribution in the first identification sequence.

[0071] In this step, all acquired data objects may be randomly sampled, for example, by a random sampling algorithm, and then a corresponding identification sequence may be determined based on the acquired random sampling results. The identification sequence includes object identifications corresponding to the multiple data objects obtained by random sampling. For the sake of distinction, the identification sequence is referred to as a second identification sequence. In some exemplary embodiments, all data objects may be randomly sampled, object identifications corresponding to the multiple data objects obtained by sampling are obtained, and the second identification sequence is obtained based on the multiple object identifications. For example, if the first identification sequence is obtained, all object identifications in the first identification sequence may be randomly sampled, and the second identification sequence may be obtained based on the multiple object identifications obtained by sampling.

[0072] It can be understood that, since the second identification sequence is obtained by randomly sampling multiple data objects, the number of object identifications included in the second identification sequence is less than the number of object identifications included in the first identification sequence, and at the same time, the identification distribution of multiple object identifications in the second identification sequence can match the identification distribution of the first identification sequence. The identification distribution can indicate the distribution position or interval of each object identification in the complete identification sequence.

[0073] S103, determining a target object identifier corresponding to a partition position in the second identifier sequence, and segmenting a plurality of data objects according to the target object identifier, and obtaining a plurality of data object partitions according to the segmentation result.

[0074] In a specific implementation, when the tag data is stored in the online storage unit, the stored tag data can be compressed, thereby improving the storage space utilization of the online storage unit and reducing the storage cost. Due to the large amount of stored tag data, in some exemplary examples, the tag data is pre-divided into multiple partitions and then compressed and stored separately.

[0075] However, the amount of data objects is huge, sometimes reaching tens of millions or even hundreds of millions. If all data objects are read one by one according to the partition step and then the partition position is determined, it will take a lot of time to perform partition processing. If the partition position is determined randomly among all data objects, it may cause data skew, resulting in a large difference in the file size of the compressed files, and the storage space of the online storage unit cannot be fully utilized.

[0076] In this regard, in this embodiment, after randomly sampling multiple data objects, a second identification sequence containing less object identification data can be obtained, and the object identification distribution in the second identification sequence can be close to the identification distribution of the first identification sequence. Therefore, by reasonably partitioning the second identification sequence and determining the corresponding partition position, the partition position can be transferred to all acquired data objects, thereby improving the partitioning efficiency while avoiding data skew in the partition, and avoiding causing some compressed blocks to be too large and affecting query performance.

[0077] Specifically, in this step, the second identification sequence can be segmented according to a predetermined partitioning rule, and the object identifier on the segmentation boundary can be determined as the target object identifier corresponding to the partition position. Since the first identification sequence also includes the target object identifier, after obtaining the target object identifier based on the second identification sequence, it can be directly divided into multiple data objects according to the target object identifier to obtain multiple data object partitions, and each data object partition can include multiple data objects.

[0078] S104, compressing the tag data corresponding to the same tag item of the data objects in the same data object partition to obtain the tag data compression results corresponding to the tag items, and storing each tag data compression result in an online storage unit.

[0079] After obtaining multiple data object partitions, each data object partition can be compressed separately. In some exemplary storage methods, the label data of the data object can be stored in the online storage unit in a row-based storage manner, that is, each label data under each data object is stored in units of data objects, and each row in the stored data corresponds to each label data under a data object. However, this method has a high storage cost and the amount of label data that can be stored is limited.

[0080] In this regard, after obtaining multiple data object partitions, this embodiment can perform column storage on the label data of the same label item in the same data object partition. Compared with traditional row storage, column storage has advantages in processing certain types of data query and analysis tasks. In column storage, data is organized on the storage medium in columns, which means that the same column of data is stored continuously. This storage method is very beneficial for big data analysis and processing because it allows the database system to read and process only the required data columns instead of the entire data row when querying.

[0081] Specifically, for each data object partition obtained by division, the label data corresponding to the same label item of the data object in the data object partition is compressed to obtain the label data compression result corresponding to the label item. If multiple label items are involved, the label data compression results corresponding to the multiple label items in the data object partition can be obtained. Then, the obtained label data compression results are stored in the online storage unit.

[0082] In the above-mentioned label data storage method, label data of each of the multiple data objects can be obtained, and the object identifiers corresponding to each of the multiple data objects constitute a first identifier sequence; then, based on the random sampling results of the multiple data objects, a second identifier sequence is obtained, and the identifier distribution in the second identifier sequence matches the identifier distribution in the first identifier sequence; then, the target object identifier corresponding to the partition position in the second identifier sequence is determined, and the multiple data objects are segmented according to the target object identifier, and multiple data object partitions are obtained according to the segmentation results; then, the label data corresponding to the same label item of the data objects in the same data object partition is compressed to obtain the label data compression result corresponding to the label item, and each label data compression result is stored in an online storage unit. In this embodiment, on the one hand, by obtaining a second identification sequence based on the random sampling results of multiple data objects, and using the target object identification determined from the second identification sequence to segment the multiple data objects, it is possible to reasonably partition the naturally sorted massive data objects, which helps to avoid data skew in different data object partitions after compression; on the other hand, by compressing the label data corresponding to the same label item of data objects in the same data object partition, it is possible to implement the storage of massive label data in online storage units based on the idea of ​​column-based compressed storage, so that the storage cost of high-performance storage units is effectively reduced while providing high-performance query and computing capabilities, thereby effectively solving the bottleneck of traditional label data storage methods that cannot take into account both high-performance query and low-cost import of large quantities of label data into online storage units.

[0083] In some exemplary embodiments, in step S102, obtaining the second identification sequence according to the random sampling results of the plurality of data objects may include the following steps:

[0084] A plurality of data objects are randomly sampled according to a predetermined sampling ratio to obtain a plurality of sampled data objects; an object identifier of each sampled data object is obtained, and the object identifiers of each sampled data object are sorted according to the order in which the object identifiers of each sampled data object appear in the first identifier sequence to obtain a second identifier sequence.

[0085] In practical applications, the sampling ratio of all acquired data objects can be determined. In some examples, the sampling ratio can be pre-set or determined based on the current available computing resources of the device. The sampling ratio can be positively correlated with the available computing resources, that is, the higher the available computing resources, the higher the sampling ratio, so that the identification distribution of the second identification sequence acquired subsequently can be closer to the identification distribution of the first identification sequence. Then, multiple data objects can be randomly sampled according to the predetermined sampling ratio to obtain multiple data objects. For ease of distinction, the sampled data objects are referred to as sampled data objects. Among them, the number of sampled data objects will be greater than the number of partitions of the data object partition to ensure that the partition position of each data object partition can be determined later based on the object identifier in the second identification sequence.

[0086] Then, the object identifier of each sampled data object is obtained, and the object identifiers of each sampled data object are sequentially sorted according to the order in which the object identifiers of each sampled data object appear in the first identifier sequence to obtain a second identifier sequence.

[0087] In this embodiment, by randomly sampling the multiple data objects according to the sampling ratio, and then sorting the object identifiers of each sampled data object according to the order of appearance of the object identifiers of each sampled data object in the first identifier sequence, a second identifier sequence whose identifier distribution is close to that of the first identifier sequence can be quickly obtained. Therefore, through the above-mentioned natural sorting dynamic partitioning algorithm, the partitioned data can be dynamically redistributed according to the distribution changes corresponding to the data objects collected daily, ensuring that the target object identifier identified from the second identifier sequence can be applied in the first identifier sequence, which helps to improve the flexibility and rationality of data object partitioning.

[0088] In some exemplary embodiments, in step S103, determining the target object identifier corresponding to the partition position in the second identifier sequence may include the following steps:

[0089] Determine the original partition step of multiple data objects; reduce the original partition step according to the sampling ratio to obtain an adjusted partition step; and determine the target object identifier corresponding to the partition position from the second identifier sequence according to the adjusted partition step.

[0090] In a specific implementation, the partition step size for all acquired data objects may be determined, wherein the partition step size may indicate the number of data objects included in each data object partition. For example, setting the partition step size to 1000 may be understood as dividing every 1000 data objects into one data object partition. For ease of distinction, the partition step size determined for all acquired data objects may be referred to as an original partition step size. In some examples, the original partition step size may be directly set, or the corresponding original partition step size may be calculated according to the number of data object partitions required.

[0091] Then, the original partition step can be reduced according to the sampling ratio to obtain the adjusted partition step. For example, if the original partition step is 10,000, and 10 million data objects are randomly sampled at a sampling ratio of 1 / 10, the number of sampled data objects obtained is 1 million. For these 1 million sampled data objects, the original partition step of 10,000 can be proportionally reduced from 10,000 to 1000 as the adjusted partition step.

[0092] After the adjusted partition step is obtained, the second identifier sequence can be divided according to the adjusted partition step, and an object identifier at each boundary position is determined as the target object identifier.

[0093] In this embodiment, the original partition step size is reduced according to the sampling ratio to obtain an adjusted partition step size, and then the target object identifier corresponding to the partition position is determined according to the adjusted partition step size. In this way, multiple target object identifiers with suitable distribution positions and matching the original number of partitions can be obtained, which is helpful for subsequent use of multiple target object identifiers to directly divide all acquired data objects.

[0094] In some exemplary embodiments, Figure 3 As shown, in step S104, storing the compression results of each tag data in an online storage unit may include the following steps:

[0095] S301, obtaining first partition position information corresponding to multiple tag items in the same data object partition; obtaining second partition position information corresponding to multiple data object partitions.

[0096] The first partition location information records the partition location of the tag data compression result of each tag item; the second partition location information records the data objects contained in each data object partition.

[0097] In practical applications, the storage of label data for each data object involves separately storing label data under different label items according to label items, and separately storing label data of multiple data objects according to the data object partitions to which the data objects belong.

[0098] In this regard, partition location information can be set for tag items and partition location information can be set for data objects. For easy distinction, the partition location information set for tag items is called first partition location information, and the partition location information set for data objects is called second partition location information. Figure 2 A schematic diagram of first partition location information and second partition location information is shown. In some examples, the second partition location information may be recorded via a null value bitmap, and each null value bitmap may record all data objects contained in a corresponding data object partition.

[0099] S302: Generate a data partition rule according to the first partition location information and the second partition location information.

[0100] S303: Store the data partitioning rules and the compression results of each tag data in an online storage unit in association with each other.

[0101] After obtaining the first partition location information and the second partition location information, a data partition rule may be generated accordingly, for example, by Figure 4 The pre-partition record table shown records the data partitioning rules. Then the acquired data partitioning rules and the compression results of each label data are associated and stored in the online storage unit.

[0102] In this embodiment, data partitioning rules are generated based on the first partition location information and the second partition location information, and are associated with each label data compression result and stored in an online storage unit. This helps to quickly retrieve the specified target data from the online storage unit according to the data partitioning rules when subsequently retrieving the label data, thereby improving the query efficiency of the label data in the online storage unit.

[0103] In some exemplary embodiments, in step S303, associating and storing the data partitioning rule and each tag data compression result in an online storage unit may include the following steps:

[0104] Generate a data version number corresponding to the data partitioning rule for the tag data of multiple data objects; store the data version number, the data partitioning rule and the compression results of each tag data in a preset storage unit in association, so that when a tag data query event occurs, the data partitioning rule is determined according to the data version number, and the tag data is queried in the preset storage unit according to the data partitioning rule.

[0105] In actual applications, label data of multiple data objects can be obtained from an offline storage unit according to a preset time interval, and the label data of the data objects can be dynamically partitioned and stored using the label data storage method provided by the present application. In other words, the storage method of the label data obtained each time may be different. For example, the label data partitioning method of multiple data objects obtained today is different from the label data partitioning method of multiple data objects obtained yesterday.

[0106] In this regard, for the label data of multiple data objects, a data version number corresponding to the data partitioning rules of the label data of multiple data objects is generated, and a data version is generated for each dynamic partitioning result of the label data. Then, the data version number, the data partitioning rules, and the label data compression results obtained after the compression and storage of each partition are associated and stored in a preset storage unit. When a label data query event occurs, the data partitioning rules are determined according to the data version number, and the label data is queried in the preset storage unit according to the data partitioning rules. For example, for the newly launched label data obtained by dynamic re-partitioning, after the data version number is generated, the data version number can be provided to each demand end that needs to call the label data. When requesting to call the label data, the demand end can determine whether the acquired data version number is consistent with the version number of the latest label data in the online storage unit. If they are consistent, the label data is updated to the latest version of the current data version number, and the old version of the label data is cleaned up as needed to save storage space; if they are inconsistent, an alarm is sent for manual investigation.

[0107] In this embodiment, by associating and storing the data version number, data partitioning rules and compression results of each tag data in a preset storage unit, the tag data partitions obtained by each redistribution can be switched stably and smoothly, thereby avoiding instability in the online tag data query results.

[0108] In some exemplary embodiments, in step S104, compressing the tag data corresponding to the same tag item of the data objects in the same data object partition to obtain the tag data compression result corresponding to the tag item may include the following steps:

[0109] For each data object in the same data object partition, determine the data missing information of each data object under the same tag item, and generate a null value bitmap of the tag item according to the data missing information; convert all tag data corresponding to the same tag item in the same data object partition into arrays of known length, and continuously splice the arrays to obtain a data splicing result; compress the data splicing result, the null value bitmap of the tag item and the array length information to obtain a compression result of the tag data corresponding to the tag item.

[0110] The data missing information indicates whether the data object has tag data under the tag item. In some exemplary embodiments, 0 or 1 may be used to represent whether the data object has tag data under the tag item.

[0111] In a specific implementation, some data objects may lack tag data under a certain tag item. In some methods, null values ​​can be used as placeholders when compressing tag data, but this method will occupy additional storage space.

[0112] In this regard, for the same tag item, in this embodiment, the data vacancy information of each data object under the tag item can be obtained during the process of compressing the tag data, and then the vacancy bitmap corresponding to the tag item is generated according to the data vacancy information of all data objects. Figure 3 Take the tag item Bucket0_fid2 shown as an example, where data object 1 and data object 3 have tag data value1 and tag data value3 under the tag item respectively, and data object 2 lacks tag data under the tag item, and the data missing from the tag data can be recorded through the null value bitmap (isNull bitmap). On the other hand, since the data vacancy information has been recorded through the null value bitmap, for each tag data corresponding to the same tag item in the same data object partition, each tag data can be converted into an array of known length, and each array is continuously spliced, without storing null values ​​or separators, to obtain the data splicing result.

[0113] Then, the array length information representing the array length, the data concatenation result, and the null value bitmap of the tag item can be compressed to obtain the tag data compression result corresponding to the tag item. When querying the tag data later, the data offset and position of each tag data can be directly calculated based on the null value bitmap and array length information, and the corresponding tag data can be accurately and quickly extracted.

[0114] In this embodiment, storing null values ​​in compressed data can be avoided. At the same time, since the null value bitmap supports efficient data compression, the data missing information can be recorded in the form of a null value bitmap, which can effectively save storage space. On the other hand, the corresponding label data can be accurately and quickly located in subsequent data queries, thereby improving the efficiency of label data queries.

[0115] In some exemplary embodiments, converting all tag data corresponding to the same tag item in the same data object partition into arrays of known lengths may include the following steps:

[0116] Determine the tag data type of each tag item in the same data object partition; if the tag data type is an integer type, determine a first digit range based on all tag data corresponding to the tag item in the data object partition, and convert each tag data in all tag data into a byte array of preset length based on the first digit range; if the tag data type is a floating point type, determine a second digit range of the integer part and a third digit range of the decimal part based on all tag data corresponding to the tag item in the data object partition, and convert each tag data in all tag data into a byte array of preset length based on the second digit range and the third digit range; if the tag data type is a character type, convert all tag data corresponding to the tag item in the data object partition into a byte array.

[0117] In practical applications, different tag data types can be converted into arrays in different ways.

[0118] In some exemplary embodiments, if the tag data type is an integer, the data object partition can be traversed in ascending order according to the object identifier of the data object to obtain the digit value of the tag data, and then the widest digit required for converting the tag data into a byte array is determined based on the maximum and minimum values ​​of the digit value. For easy distinction, the widest digit is called the first digit range. Then, each tag data is converted one by one into a byte array of a preset length corresponding to the first digit range according to the first digit range, and these byte arrays are subsequently spliced ​​together and compressed using a compression algorithm. Figure 5 An example of converting different tag data into byte arrays is shown. For the maximum and minimum values ​​within the byte type range, the digit range is 1, for the maximum and minimum values ​​within the short type range, the digit range is 2, for the maximum and minimum values ​​within the integer type range, the digit range is 4, and for the maximum and minimum values ​​within the long type range, the digit range is 8.

[0119] If the tag data type is a floating point type, the tag data is divided into an integer part and a decimal part, and then they are concatenated into byte arrays and compressed according to the strategy of integer tags. Specifically, according to all the tag data corresponding to the tag items in the data object partition, the digit range of the integer part and the digit range of the decimal part are determined. The method for determining the digit range is the same as the above method and will not be repeated here. Among them, the digit range of the integer part is called the second digit range, and the digit range of the decimal part is called the third digit range. Then, according to the second digit range and the third digit range, the integer part and the decimal part of each tag data in all the tag data are converted into byte arrays of preset lengths respectively. Figure 5As shown, the integer part of each tag data can be converted into a byte array with a length of "num width" and concatenated in sequence. Then, the decimal part of each tag data can be converted into a byte array with a length of "dec width" and concatenated in sequence after the integer part byte array.

[0120] If the tag data type is character type, all tag data corresponding to the tag item in the data object partition are converted into byte arrays, such as Figure 5 As shown, the string length str width of each tag data can be recorded, for example, the string lengths str1 width, str2 width ... strn width corresponding to the tag data str1, str2 ... strn are recorded, and then concatenated after being converted into a byte array.

[0121] In this embodiment, by converting different data types of tag data into corresponding byte arrays, data compression efficiency can be effectively improved, while ensuring that specified information can be accurately read from the compressed tag data according to the digit range.

[0122] In order to enable those skilled in the art to better understand the above steps, the embodiment of the present application is illustrated below by using an example, but it should be understood that the embodiment of the present application is not limited to this.

[0123] This example applies to Figure 2 The label data calling system shown in the figure is used as an example to illustrate that in this system, its underlying label data storage integrates the label data storage method involved in this application. In some examples, if the label data includes the label data of the account portrait, the business party can use the label data in real time to implement customized rules for the account or push relevant information. The system supports high concurrency and low latency online query and computing services, and the data is highly timely. In some embodiments, it can provide Figure 6 In the application interface shown, business personnel can set the label data grouping type, grouping name, and output ID type of the label data to be called, and can also set the update method of the label data distribution (including manual or automatic). In addition, label data extraction rules can also be set, such as setting the conditions that label data needs to meet or the conditions that data objects need to meet.

[0124] Figure 7 This paper provides a label data calling system for the online process of offline data based on dynamic partition columnar compression storage. This example takes the account portrait label data as an example. The relevant business parties collect label data every day and store it in the offline storage unit through reporting. The amount of label data with high frequency use is about 300 million, and the storage space occupied is about 32G. The information processing process of the entire offline data online process is as follows:

[0125] (1) The business party collects authorized legal portrait tag data through information collection points such as clients, web pages, and mini-programs, and reports it to the front-end message queue of the offline storage suite.

[0126] (2) The offline storage suite integrates and processes the data in its front-end message queue at preset time intervals, and obtains the portrait label data to be partitioned on the previous day at midnight of the next day.

[0127] (3) The offline data online processing unit obtains the full amount of data to be partitioned on the previous day of the portrait label data, uses the account identifier as the basis for data repartitioning, and uses the method provided in this application to generate a new data partitioning rule for further decomposing the portrait label data into smaller data sub-partitions, and records the data partitioning rule in the pre-partitioning record table. At the same time, a new data version number is generated, which can ensure that the production of new data does not affect the online process.

[0128] (4) The offline data online processing unit cleans and merges the portrait label data, and compresses the data columnarly based on each label.

[0129] (5) Store the column-compressed label data into an idle online storage unit.

[0130] (6) Then verify whether the new version of the label data to be obtained is consistent with the label data provided online. If they are consistent, update the data to the latest version corresponding to the current data version number, and clean up the old version of the label data as needed to save storage space. If they are inconsistent, send an alarm for manual investigation.

[0131] (7) The system online service queries the portrait label data through the query unit and decompresses and restores it to normal label data.

[0132] (8) To improve query performance, the system online service can add local cache units to cache the frequently used portrait label compression data in local memory, further improving query performance.

[0133] (9) The business party obtains the required portrait label data by calling the system's online services.

[0134] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0135] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store label data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a label data storage method is implemented.

[0136] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0138] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0139] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0141] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A tag data storage method, characterized in that: The method comprises: Acquire label data of each of a plurality of data objects; the object identifiers corresponding to each of the plurality of data objects constitute a first identifier sequence; A second identification sequence is obtained according to the random sampling results of the plurality of data objects; the identification distribution in the second identification sequence matches the identification distribution in the first identification sequence; Determine a target object identifier corresponding to a partition position in the second identifier sequence, and segment the plurality of data objects according to the target object identifier, and obtain a plurality of data object partitions according to the segmentation result; The tag data corresponding to the same tag item of the data objects in the same data object partition are compressed to obtain the tag data compression results corresponding to the tag items, and each of the tag data compression results is stored in an online storage unit.

2. The method according to claim 1, characterized in that The obtaining a second identification sequence according to the random sampling results of the plurality of data objects comprises: Randomly sampling the multiple data objects according to a predetermined sampling ratio to obtain multiple sampled data objects; The object identifier of each of the sampled data objects is obtained, and the object identifiers of each of the sampled data objects are sorted according to the order in which the object identifiers of each of the sampled data objects appear in the first identifier sequence to obtain a second identifier sequence.

3. The method according to claim 2, characterized in that The determining of the target object identifier corresponding to the partition position in the second identifier sequence includes: Determining an original partition step size of the plurality of data objects; Reducing the original partition step size according to the sampling ratio to obtain an adjusted partition step size; According to the adjusted partition step, a target object identifier corresponding to the partition position is determined from the second identifier sequence.

4. The method according to claim 1, characterized in that: The step of storing the compression results of each tag data in an online storage unit comprises: Acquire first partition position information corresponding to a plurality of the tag items in the same data object partition; the first partition position information records the partition position of the tag data compression result of each of the tag items; Acquire second partition location information corresponding to the plurality of data object partitions; the second partition location information records the data objects contained in each of the data object partitions; Generate a data partition rule according to the first partition location information and the second partition location information; The data partitioning rule and each of the label data compression results are associated and stored in an online storage unit.

5. The method according to claim 4, characterized in that The step of associating the data partitioning rule with each of the label data compression results and storing them in an online storage unit includes: Generating a data version number corresponding to the data partitioning rule for the tag data of the plurality of data objects; The data version number, the data partitioning rule and each tag data compression result are associated and stored in a preset storage unit, so that when a tag data query event occurs, the data partitioning rule is determined according to the data version number, and the tag data is queried in the preset storage unit according to the data partitioning rule.

6. The method according to claim 1, characterized in that The step of compressing the tag data corresponding to the same tag item of the data objects in the same data object partition to obtain a tag data compression result corresponding to the tag item includes: For each of the data objects in the same data object partition, determine data vacancy information of each of the data objects under the same tag item, and generate a null value bitmap of the tag item according to the data vacancy information; the data vacancy information indicates whether the data object has tag data under the tag item; Convert all label data corresponding to the same label item in the same data object partition into arrays of known length respectively, and continuously splice the arrays to obtain a data splicing result; The data concatenation result, the null value bitmap of the tag item and the array length information are compressed to obtain a tag data compression result corresponding to the tag item.

7. The method according to claim 6, characterized in that The step of converting all the tag data corresponding to the same tag item in the same data object partition into arrays of known lengths comprises: Determine the tag data type of each tag item in the same data object partition; If the tag data type is an integer, determining a first digit range according to all tag data corresponding to the tag item in the data object partition, and converting each tag data in all the tag data into a byte array of a preset length according to the first digit range; If the tag data type is a floating point type, determining a second digit range of an integer part and a third digit range of a decimal part according to all tag data corresponding to the tag item in the data object partition, and converting each tag data in all the tag data into a byte array of a preset length according to the second digit range and the third digit range; If the tag data type is character type, all tag data corresponding to the tag item in the data object partition are converted into a byte array.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images