Data sampling method, device and storage medium based on data modeling

Abstract

The application relates to the field of data analysis, and provides a data sampling method based on data modeling, a device and a storage medium.The method comprises the following steps: obtaining a data sampling requirement, the data sampling requirement comprising data attributes and a sampling quantity; comparing the data attributes with preset statistical attributes to obtain target attributes and non-statistical attributes; extracting target data corresponding to the sampling quantity from a preset bitmap index according to the target attributes; randomly generating virtual data of the sampling quantity according to the non-statistical attributes; and combining the target data and the virtual data to obtain sampling data, each piece of the sampling data comprising target data corresponding to the target attributes and virtual data corresponding to the non-statistical attributes. The target attribute data meeting the data sampling requirement is quickly generated in the form of the bitmap index, and the data of the non-statistical attributes is generated in a random manner, so that the data requirement and the quantity requirement of the data sampling are met, and the sampling efficiency is effectively improved, and the reliability of the sampling data is ensured.

Description

Technical Field

[0001] This application relates to the field of data analysis, and in particular to a data sampling method, device and storage medium based on data modeling. Background Technology

[0002] Big data computing and analysis have become the foundation of daily data statistics and analysis. When designing models for data analysis, each model requires multiple verifications and adjustments. Current technologies for querying, analyzing, and summarizing data rely on statistically significant columns. Especially when building analytical models for big data, it's typically unnecessary to perform frequent calculations on the entire dataset, effectively reducing resource consumption. Only a subset of data is needed for initial testing to verify the model's accuracy. However, existing big data sampling methods are all statistical sampling methods, including but not limited to simple random sampling, systematic sampling, stratified sampling, and cluster sampling. These sampling methods cannot guarantee the storage order and the discreteness of data extraction in the big data field, leading to significant uncertainty in the efficiency of data modeling. Summary of the Invention

[0003] This application provides a data sampling method, device, and storage medium based on data modeling to solve the technical problems of low efficiency and poor reliability of data sampling in the prior art.

[0004] Firstly, this application provides a data sampling method based on data modeling, including:

[0005] Obtain data sampling requirements, which include data attributes and sampling quantity;

[0006] The data attributes are compared with preset statistical attributes to obtain target attributes and non-statistical attributes;

[0007] Target data corresponding to the sampling quantity is extracted from a preset bitmap index based on the target attribute; the bitmap index is established by extracting data of multiple attributes from a database;

[0008] Virtual data for the sampling quantity is randomly generated based on the non-statistical attribute;

[0009] The target data and the virtual data are combined to obtain sampled data. Each sampled data includes target data corresponding to the target attribute and virtual data corresponding to the non-statistical attribute.

[0010] In one possible design, before extracting the target data corresponding to the sampling quantity from a preset bitmap index based on the target attribute, the method further includes:

[0011] The basic data in the database is classified to obtain multiple attribute sets, and each attribute set contains data of multiple categories of the same attribute.

[0012] Perform statistical filtering on the data in the attribute set corresponding to each attribute to obtain the statistically valid attributes that meet the preset statistical conditions;

[0013] A bitmap index is established based on the attribute set corresponding to the statistically significant attributes.

[0014] In one possible design, the step of statistically filtering the data in each attribute set to obtain statistically valid attributes that meet preset statistical conditions includes:

[0015] The number of categories is obtained by counting the categories of data in each attribute set;

[0016] Calculate the percentage of each category in the attribute set based on the basic data corresponding to each category in the attribute set;

[0017] Based on the number of categories, the proportion of categories, and the preset proportion threshold, statistically valid attributes that meet the preset statistical conditions are obtained.

[0018] In one possible design, obtaining the statistically significant attributes that satisfy preset statistical conditions based on the number of categories, the category proportion, and a preset proportion threshold includes:

[0019] The category percentage is compared with a preset percentage threshold to determine whether there are any categories whose category percentage is greater than the preset percentage threshold;

[0020] When there is a category whose proportion is greater than a preset proportion threshold, the attribute corresponding to the attribute set is divided into statistically significant attributes;

[0021] If there are no categories whose proportion is greater than a preset proportion threshold, determine whether the number of the categories is greater than a preset category threshold.

[0022] If the number of categories is greater than a preset category threshold, the attributes corresponding to the attribute set will be classified as non-statistical attributes;

[0023] If the number of categories is less than or equal to a preset category threshold, the attributes corresponding to the attribute set are divided into statistically significant attributes.

[0024] In one possible design, when there are categories with a proportion greater than a preset proportion threshold, after classifying the attributes corresponding to the attribute set into statistically significant attributes, the design further includes:

[0025] A category retention value is calculated based on the preset category threshold, and the category retention value is less than the preset category threshold and the difference between the preset category threshold and the category retention value is one.

[0026] The categories in the attribute set are sorted from largest to smallest according to the percentage of each category, to obtain a percentage sequence;

[0027] Extract the number of categories corresponding to the category retention value from the segment with the largest category percentage value in the percentage sequence, and then summarize the remaining categories into one retention category.

[0028] In one possible design, establishing a bitmap index based on the data set corresponding to the statistically significant attribute includes:

[0029] Create a corresponding bitmap for each category in the attribute set to obtain the type index corresponding to the attribute;

[0030] The type indices corresponding to multiple attributes are merged into a bitmap to obtain a bitmap index.

[0031] In one possible design, the step of randomly generating the virtual data for the sample size based on the non-statistical attribute includes:

[0032] Obtain multiple data categories corresponding to the non-statistical attributes;

[0033] Virtual data is randomly generated based on the data category, where the virtual data is one of the data categories.

[0034] One possible design also includes:

[0035] Obtain updated data from the database;

[0036] The updated data is analyzed based on the bitmap index to obtain difference data, which is data that does not exist in the bitmap index;

[0037] The attributes and categories of the difference data are analyzed, and a new bitmap is generated and added to the bitmap index based on the attributes and categories corresponding to the difference data.

[0038] Secondly, this application provides a data sampling device based on data modeling, comprising:

[0039] The acquisition module is used to acquire data sampling requirements, which include data attributes and sampling quantity.

[0040] The comparison module is used to compare the data attributes with preset statistical attributes to obtain target attributes and non-statistical attributes;

[0041] The extraction module is used to extract target data corresponding to the sampling quantity from a preset bitmap index based on the target attribute; the bitmap index is established by extracting basic data of multiple attributes from a database, and the database stores data of all attributes.

[0042] The generation module is used to randomly generate virtual data of the sampling quantity based on the non-statistical attribute;

[0043] The combination module is used to combine the target data and the virtual data to obtain sampled data. Each sampled data includes target data corresponding to the target attribute and virtual data corresponding to the non-statistical attribute.

[0044] In one possible design, the extraction module is also used for:

[0045] The basic data in the database is classified to obtain multiple attribute sets, and each attribute set contains data of multiple categories of the same attribute.

[0046] Perform statistical filtering on the data in the attribute set corresponding to each attribute to obtain the statistically valid attributes that meet the preset statistical conditions;

[0047] A bitmap index is established based on the attribute set corresponding to the statistically significant attributes.

[0048] In one possible design, the extraction module is also used for:

[0049] The number of categories is obtained by counting the categories of data in each attribute set;

[0050] Calculate the percentage of each category in the attribute set based on the basic data corresponding to each category in the attribute set;

[0051] Based on the number of categories, the proportion of categories, and the preset proportion threshold, statistically valid attributes that meet the preset statistical conditions are obtained.

[0052] In one possible design, the extraction module is also used for:

[0053] The category percentage is compared with a preset percentage threshold to determine whether there are any categories whose category percentage is greater than the preset percentage threshold;

[0054] When there is a category whose proportion is greater than a preset proportion threshold, the attribute corresponding to the attribute set is divided into statistically significant attributes;

[0055] If there are no categories whose proportion is greater than a preset proportion threshold, determine whether the number of the categories is greater than a preset category threshold.

[0056] If the number of categories is greater than a preset category threshold, the attributes corresponding to the attribute set will be classified as non-statistical attributes;

[0057] If the number of categories is less than or equal to a preset category threshold, the attributes corresponding to the attribute set are divided into statistically significant attributes.

[0058] In one possible design, the extraction module is also used for:

[0059] A category retention value is calculated based on the preset category threshold, and the category retention value is less than the preset category threshold and the difference between the preset category threshold and the category retention value is one.

[0060] The categories in the attribute set are sorted from largest to smallest according to the percentage of each category, to obtain a percentage sequence;

[0061] Extract the number of categories corresponding to the category retention value from the segment with the largest category percentage value in the percentage sequence, and then summarize the remaining categories into one retention category.

[0062] In one possible design, the extraction module is also used for:

[0063] Create a corresponding bitmap for each category in the attribute set to obtain the type index corresponding to the attribute;

[0064] The type indices corresponding to multiple attributes are merged into a bitmap to obtain a bitmap index.

[0065] In one possible design, a generation module is specifically used for:

[0066] Obtain multiple data categories corresponding to the non-statistical attributes;

[0067] Virtual data is randomly generated based on the data category, where the virtual data is one of the data categories.

[0068] In one possible design, the extraction module is also used for:

[0069] Obtain updated data from the database;

[0070] The updated data is analyzed based on the bitmap index to obtain difference data, which is data that does not exist in the bitmap index;

[0071] The attributes and categories of the difference data are analyzed, and a new bitmap is generated and added to the bitmap index based on the attributes and categories corresponding to the difference data.

[0072] Thirdly, this application provides a data sampling device based on data modeling, including: a processor, and a memory communicatively connected to the processor;

[0073] The memory stores the instructions that the computer executes;

[0074] The processor executes computer execution instructions stored in memory, causing the path query device accessing network resources to execute any of the data modeling-based data sampling methods in the first aspect.

[0075] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a data sampling method based on data modeling as described in any of the first aspects.

[0076] The data sampling method, device, and storage medium based on data modeling provided in this application obtain data sampling requirements, including data attributes and sampling quantity; compare the data attributes with preset statistical attributes to obtain target attributes and non-statistical attributes; extract target data corresponding to the sampling quantity from a preset bitmap index based on the target attributes; the bitmap index is established by extracting data of multiple attributes from a database; generate virtual data for the sampling quantity randomly based on the non-statistical attributes; combine the target data and virtual data to obtain the sampled data, with each sampled data including target data corresponding to the target attribute and virtual data corresponding to the non-statistical attribute. By rapidly generating target attribute data that meets the data sampling requirements through bitmap indexing, and simultaneously generating non-statistical attribute data through random generation, the data and quantity requirements for data sampling are met, effectively improving sampling efficiency and ensuring the reliability of the sampled data. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0078] Figure 1 This is a flowchart illustrating the data sampling method based on data modeling provided in the embodiments of this application. Figure 1 ;

[0079] Figure 2 This is a schematic diagram of a bitmap index provided in an embodiment of this application;

[0080] Figure 3 This is a flowchart illustrating the data sampling method based on data modeling provided in the embodiments of this application. Figure 2 ;

[0081] Figure 4 This is a flowchart illustrating the data sampling method based on data modeling provided in the embodiments of this application. Figure 3 ;

[0082] Figure 5 This is a flowchart illustrating the data sampling method based on data modeling provided in the embodiments of this application. Figure 4 ;

[0083] Figure 6 This is a flowchart illustrating the data sampling method based on data modeling provided in the embodiments of this application. Figure 5 ;

[0084] Figure 7 This is a schematic diagram of the structure of the data sampling device based on data modeling provided in the embodiments of this application;

[0085] Figure 8 This is a schematic diagram of the hardware structure of the data sampling device based on data modeling provided in the embodiments of this application.

[0086] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] In existing technologies, big data databases such as HBase and Hive store massive amounts of data. Their architecture is based on NoSQL, and all operations rely on big data computing platforms like Spark and Flink. All computations, statistics, and analysis depend on substantial computing resources. Taking telecommunications services as an example, most data in HBase and Hive is stored in a single table, uniformly recording various user information columns, including but not limited to mobile phone number, province / city, caller ID, call duration, and call charges. These information columns can be categorized into statistically significant and non-statistically significant attributes. Statistically significant attributes, such as city / city, province / city, and gender, are clearly defined and frequently used as query and aggregation conditions, exhibiting good aggregation characteristics. Non-statistically significant attributes, such as called number, owner name, call duration, and call charges, may have many values ​​for a single value due to name duplication or similarity, but lack good aggregation characteristics and are not particularly meaningful for routine data queries, statistics, and analysis. Furthermore, existing big data sampling methods are all statistical sampling methods, including but not limited to simple random sampling, systematic sampling, stratified sampling, and cluster sampling. These sampling methods cannot guarantee the storage order of data and the discreteness of data extraction in the field of big data, resulting in significant uncertainty in the efficiency of data modeling.

[0089] The data sampling method based on data modeling provided in this application aims to solve the above-mentioned technical problems in the prior art.

[0090] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0091] Figure 1 This is a flowchart illustrating the data sampling method based on data modeling according to an embodiment of the present invention. Figure 1 ,like Figure 1 As shown, this embodiment provides a data sampling method based on data modeling, which includes the following steps:

[0092] Step S101: Obtain data sampling requirements, which include data attributes and sampling quantity.

[0093] Specifically, when data modeling is required, data of the corresponding attributes need to be extracted as training data according to the data requirements of the model.

[0094] Step S102: Compare the data attributes with the preset statistical attributes to obtain the target attributes and non-statistical attributes.

[0095] Specifically, taking telecommunications services as an example, databases store a large amount of attribute data, which can be categorized into statistical attributes and non-statistical attributes. Statistical attributes, such as city / province of origin, gender, etc., have clear characteristics and are frequently used as query and summary conditions. Non-statistical attributes, such as called number, owner's name, call duration, and call charges, may contain a lot of data for a single value due to name duplication or similarity, but the data lacks aggregation and has poor analytical value for routine data queries and statistics. However, some models require extracting the entire dataset for modeling, which necessitates a variety of data types, including both statistical and non-statistical attributes.

[0096] Step S103: Extract target data corresponding to the sampling quantity from the preset bitmap index according to the target attribute; the bitmap index is established by extracting data of multiple attributes from the database.

[0097] Specifically, a bitmap index stores a sequence of bits, each bit representing a different attribute category. Each bit has only two states: 0 and 1, representing whether the attribute exists. Figure 2The diagram shows a bitmap index corresponding to a single piece of user data. One segment of bits represents the service attribute; if a user has subscribed to that service, the corresponding bit is 1. Another segment represents the province attribute; the bit for the user's province is 1, and the bits for other provinces are 0. When sampling data, only a sample number of data points corresponding to the target attribute need to be extracted from the pre-defined bitmap index. For example, to analyze the relationship between a user's province and service subscriptions and build a data model, a certain number of user data points, including both service and province attributes, need to be extracted.

[0098] Step S104: Randomly generate virtual data of the sample size based on non-statistical attributes.

[0099] Specifically, since some models require attributes that lack analytical value to correlate with other attributes in the model, virtual data is generated randomly to meet the necessary data attribute requirements. Simultaneously, generating virtual data provides a degree of data encryption. Non-statistical attributes can be added in a pre-defined manner, such as names and ID numbers. Randomly generating this data protects user privacy without affecting the modeling of the sampled data.

[0100] Step S105: Combine the target data and virtual data to obtain sampled data. Each sampled data includes target data corresponding to the target attribute and virtual data corresponding to the non-statistical attribute.

[0101] Specifically, each sampled data point includes data on the actual target attributes and randomly generated data on non-statistical attributes. While meeting the data sampling requirements, the sampling efficiency is effectively improved and the reliability of the sampled data is guaranteed through bitmap indexing and random generation.

[0102] The data sampling method based on data modeling provided in this invention quickly generates target attribute data that meets the data sampling requirements through bitmap indexing, and at the same time generates non-statistical attribute data through random generation, which meets the data and quantity requirements of data sampling, effectively improving sampling efficiency and ensuring the reliability of sampling data.

[0103] Figure 3 A flowchart illustrating the data sampling method based on data modeling provided in this application embodiment. Figure 2 .like Figure 3 As shown, this embodiment provides a detailed description of the bitmap index creation process described in the above embodiment. The specific implementation of this process includes the following steps:

[0104] Step S301: Classify the basic data in the database to obtain multiple attribute sets, each of which contains data of the same attribute in multiple categories.

[0105] Specifically, in the field of communications, user information is classified by attributes, such as province, city, service, gender, call duration, service cost, etc. Each attribute contains multiple categories, such as the attribute set corresponding to a province containing data for each province.

[0106] Step S302: Count the number of categories in the data of each attribute set.

[0107] Specifically, since different attributes occupy different numbers and positions of bits in the bit sequence, we can check the bit requirement for each attribute by counting the number of categories. We can also check whether the data is scattered and whether it meets preset statistical conditions based on the number of categories. For example, the gender attribute has two categories, each requiring one bit in the bit sequence.

[0108] Step S303: Calculate the proportion of each category in the attribute set based on the basic data corresponding to each category in the attribute set.

[0109] Specifically, for example, statistics can be collected on user information from different provinces in the province attribute set to calculate the proportion of users from each province to the total number of users, which will facilitate subsequent analysis to determine whether the attribute is statistically significant.

[0110] Step S304: Based on the number of categories, the proportion of categories, and the preset proportion threshold, obtain the statistically available attributes that meet the preset statistical conditions.

[0111] Specifically, statistically significant attributes are those with obvious statistical characteristics, such as province and gender. Some statistically significant attributes also have a large number of categories, but one category has a large proportion, exhibiting obvious clustering characteristics. To best meet data sampling requirements, we analyze the number of categories and their proportions to obtain statistically significant attributes.

[0112] Step S305: Create a corresponding bitmap for each category in the attribute set to obtain the type index corresponding to the attribute.

[0113] Specifically, such as Figure 2 The bitmap index diagram shown has each attribute set corresponding to a segment of bits, and each bit represents a different category.

[0114] Step S306: Merge the type indices corresponding to multiple attributes into a bitmap to obtain a bitmap index.

[0115] Specifically, a bitmap index is a bit sequence that stores a large number of bits, each bit representing a different attribute category. By creating bit segments corresponding to different attributes and then merging the segments into a bitmap, a bitmap index is obtained.

[0116] This invention employs a full-data statistical analysis approach to analyze massive datasets, selecting statistically significant attributes to create a bitmap index. This method is space-efficient and extremely fast to create and use. When querying based on data sampling requirements, data can be quickly located based on the bitmap status, and target attribute data matching the specified data attributes and sampling quantity can be extracted, resulting in high speed and accuracy.

[0117] Figure 4 A flowchart illustrating the data sampling method based on data modeling provided in this application embodiment. Figure 3 .like Figure 4 As shown, this embodiment provides a detailed explanation of the process for obtaining statistically significant attributes that meet preset statistical conditions in the above embodiments. The specific implementation of this process includes the following steps:

[0118] Step S401: Compare the category percentage with the preset percentage threshold to determine whether there are any categories whose category percentage is greater than the preset percentage threshold.

[0119] Specifically, the preset percentage threshold can be 40%, 50%, etc., and can be adjusted according to actual needs. Comparing the category percentage with the preset percentage threshold is to filter out attributes where a certain category has an excessively large percentage and exhibits obvious clustering characteristics.

[0120] Step S402: When there is a category with a proportion greater than the preset proportion threshold, the attribute corresponding to the attribute set is divided into statistical attributes.

[0121] Specifically, for example, if a village has users with multiple surnames, and one surname accounts for more than 50% of the users, then the surnames in that village are classified as statistical attributes.

[0122] Step S403: Calculate the category retention value based on the preset category threshold. The category retention value is less than the preset category threshold and the difference between the category retention value and the preset category threshold is one.

[0123] Specifically, the preset category threshold is the number of bits allocated to the attribute in the bitmap index. For example, if the attribute occupies 10 bits, the preset category threshold is 10, and the village in the example above contains 20 surnames, then the category retention value is 9.

[0124] Step S404: Sort the categories in the attribute set from largest to smallest according to the category percentage value to obtain the percentage sequence.

[0125] Specifically, the purpose of sorting is to see which categories need to be retained, and to filter out the data with the highest proportion based on the category percentage.

[0126] Step S405: Extract the number of categories corresponding to the category retention values ​​from the segment with the largest category proportion values ​​in the proportion sequence, and summarize the remaining categories into one retention category.

[0127] Specifically, taking the aforementioned village as an example, the top nine surnames with the highest percentage of people with these surnames are retained, while the remaining surnames have lower reference value and are grouped into one category, which can be replaced by "other surnames".

[0128] Step S406: When there is no category with a proportion greater than the preset proportion threshold, determine whether the number of categories is greater than the preset category threshold.

[0129] Specifically, taking provinces and user names as examples, there are no categories with a proportion greater than the preset proportion threshold. The determination of whether a category is a statistically significant attribute is made by checking whether it exceeds the preset category threshold.

[0130] Step S407: If the number of categories is greater than the preset category threshold, the attributes corresponding to the attribute set are classified as non-statistical attributes.

[0131] Specifically, for example, if there are hundreds of thousands of categories for names, and the number of categories exceeds a preset category threshold, then they are not statistically valuable and are classified as non-statistical attributes.

[0132] Step S408: If the number of categories is less than or equal to the preset category threshold, the attributes corresponding to the attribute set are divided into statistical attributes.

[0133] Specifically, if the number of preset category thresholds is greater than the number of provinces, then provinces are considered as statistical attributes.

[0134] In order to meet data sampling requirements as much as possible and improve the reliability and accuracy of bitmap index data, this embodiment of the invention analyzes the number and proportion of categories to obtain statistically significant attributes. These statistically significant attributes are those with obvious statistical characteristics, such as province and gender. Some statistically significant attributes are those with a large number of categories but a particularly large proportion of a specific category, exhibiting clear clustering characteristics.

[0135] Figure 5 A flowchart illustrating the data sampling method based on data modeling provided in this application embodiment. Figure 4 .like Figure 5 As shown, this embodiment provides a detailed explanation of the process of generating virtual data in the above embodiments. The specific implementation of this process includes the following steps:

[0136] Step S501: Obtain multiple data categories corresponding to non-statistical attributes.

[0137] Specifically, non-statistical attributes include various data formats, such as telephone numbers and ID card numbers, which are fixed-length numbers; for example, villages are village names in a pre-defined regional name database.

[0138] Step S502: Randomly generate virtual data according to the data category, where the virtual data is one of the data categories.

[0139] Specifically, virtual data is randomly generated based on data categories, generating data that meets the requirements without affecting data sampling.

[0140] This invention generates virtual data through random generation to meet the required data attribute requirements. Simultaneously, generating virtual data provides a certain degree of data encryption. Non-statistical attributes, such as names and ID numbers, can be added in a pre-defined manner. The random generation of these attributes protects user privacy without affecting the sampling data modeling. Furthermore, all big data-based queries and statistical operations can be processed using bitmap-based simulated data, improving the efficiency of data modeling.

[0141] Figure 6 A flowchart illustrating the data sampling method based on data modeling provided in this application embodiment. Figure 5 .like Figure 6 As shown, this embodiment provides a detailed explanation of the bitmap index update process in the above embodiment. The specific implementation of this process includes the following steps:

[0142] Step S601: Obtain updated data from the database.

[0143] Specifically, to ensure the integrity and reliability of bitmap index data, the bitmap index needs to be updated according to the data updates in the database. This can be achieved by setting up periodic updates.

[0144] Step S602: Analyze the updated data based on the bitmap index to obtain the difference data, which is data that does not exist in the bitmap index.

[0145] Specifically, by analyzing the discrepancies in the data, the bitmap index is supplemented in a timely manner to meet the needs of data sampling.

[0146] Step S603: Analyze the attributes and categories of the differential data, and generate a new bitmap based on the attributes and categories corresponding to the differential data and add it to the bitmap index.

[0147] Specifically, the indexed data is stored in memory as a binary tree. Whenever new data is added, it is necessary to compare whether the content of its statistical attributes is in the bitmap. If not, new bitmap information needs to be added to the binary tree to ensure the integrity and reliability of the statistical attribute data information in the bitmap.

[0148] This invention updates the bitmap index to ensure the integrity and reliability of the bitmap index data. When data sampling is required, the data extracted based on the bitmap index is accurate and reliable, which helps to improve the efficiency of data modeling.

[0149] Figure 7 This is a schematic diagram of the structure of a data sampling device based on data modeling provided in an embodiment of the present invention. Figure 7 As shown, the data sampling device 70 based on data modeling includes: an acquisition module 701, a comparison module 702, an extraction module 703, a generation module 704, and a combination module 705.

[0150] The acquisition module 701 is used to acquire data sampling requirements, which include data attributes and sampling quantity.

[0151] The comparison module 702 is used to compare data attributes with preset statistical attributes to obtain target attributes and non-statistical attributes;

[0152] The extraction module 703 is used to extract target data corresponding to the sampling quantity from a preset bitmap index based on the target attribute; the bitmap index is established by extracting basic data of multiple attributes from the database, and the database stores data of all attributes;

[0153] The generation module 704 is used to randomly generate virtual data of sample sizes based on non-statistical attributes;

[0154] The combination module 705 is used to combine target data and virtual data to obtain sampled data. Each sampled data includes target data corresponding to the target attribute and virtual data corresponding to the non-statistical attribute.

[0155] In one possible design, the extraction module 703 is also used for:

[0156] The basic data in the database is classified to obtain multiple attribute sets, and each attribute set contains data of multiple categories of the same attribute.

[0157] Perform statistical filtering on the data in the attribute set corresponding to each attribute to obtain the statistically valid attributes that meet the preset statistical conditions;

[0158] A bitmap index is created based on the set of attributes corresponding to the statistically significant attributes.

[0159] In one possible design, the extraction module 703 is also used for:

[0160] The number of categories is obtained by counting the categories of data in each attribute set;

[0161] Calculate the percentage of each category in the attribute set based on the basic data corresponding to each category in the attribute set;

[0162] Based on the number of categories, the proportion of categories, and the preset proportion threshold, statistically valid attributes that meet the preset statistical conditions are obtained.

[0163] In one possible design, the extraction module 703 is also used for:

[0164] The category percentage is compared with a preset percentage threshold to determine whether there are any categories whose category percentage is greater than the preset percentage threshold.

[0165] When there are categories whose proportions exceed a preset proportion threshold, the attributes corresponding to the attribute set are classified as statistically significant attributes.

[0166] If there are no categories whose proportion is greater than the preset proportion threshold, determine whether the number of categories is greater than the preset category threshold.

[0167] If the number of categories exceeds the preset category threshold, the attributes corresponding to the attribute set will be classified as non-statistical attributes.

[0168] If the number of categories is less than or equal to the preset category threshold, the attributes corresponding to the attribute set will be classified as statistical attributes.

[0169] In one possible design, the extraction module 703 is also used for:

[0170] The category retention value is calculated based on the preset category threshold. The category retention value is less than the preset category threshold and the difference between the category retention value and the preset category threshold is one.

[0171] Sort the categories in the attribute set from largest to smallest according to the percentage of each category to obtain the percentage sequence;

[0172] Extract the number of categories corresponding to the category retention value from the segment with the largest category percentage in the percentage sequence, and then combine the remaining categories into one retention category.

[0173] In one possible design, the extraction module 703 is also used for:

[0174] Create a corresponding bitmap for each category in the attribute set to obtain the type index corresponding to the attribute;

[0175] The type indices corresponding to multiple attributes are merged into a bitmap to obtain a bitmap index.

[0176] In one possible design, module 704 is specifically used for:

[0177] Retrieve multiple data categories corresponding to non-statistical attributes;

[0178] Virtual data is randomly generated based on the data category, where virtual data is one of the data categories.

[0179] In one possible design, the extraction module 703 is also used for:

[0180] Retrieve updated data from the database;

[0181] The updated data is analyzed based on the bitmap index to obtain the difference data, which is data that does not exist in the bitmap index;

[0182] Analyze the attributes and categories of the differential data, and generate new bitmaps based on the corresponding attributes and categories of the differential data, adding them to the bitmap index.

[0183] The data sampling device based on data modeling provided in this embodiment can be used to execute the data sampling method based on data modeling described above. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0184] Figure 8 A schematic diagram of the hardware structure of the data sampling device based on data modeling provided in an embodiment of the present invention is shown below. Figure 8 As shown, the data sampling device 80 based on data modeling includes at least one processor 801 and a memory 802. Optionally, the data sampling device 80 based on data modeling also includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus 804.

[0185] In the specific implementation process, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to execute the data sampling method based on data modeling as described above.

[0186] The communication component 803 can interact with the server.

[0187] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0188] In the embodiments shown in Figure 8 above, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0189] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0190] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0191] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data sampling method based on data modeling as described above.

[0192] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0193] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0194] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0196] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0197] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data sampling method based on data modeling, characterized in that, include: Obtain data sampling requirements, which include data attributes and sampling quantity; The data attributes are compared with preset statistical attributes to obtain target attributes and non-statistical attributes; The basic data in the database is classified to obtain multiple attribute sets, and each attribute set contains data of multiple categories of the same attribute. Perform statistical filtering on the data in the attribute set corresponding to each attribute to obtain the statistically valid attributes that meet the preset statistical conditions; Create a corresponding bitmap for each category in the attribute set to obtain the type index corresponding to the attribute; Merge the type indices corresponding to multiple attributes into a bitmap to obtain a bitmap index; Based on the target attribute, target data corresponding to the sampling quantity is extracted from the preset bitmap index; The bitmap index is built by extracting data from multiple attributes from the database; Virtual data for the sampling quantity is randomly generated based on the non-statistical attribute; The target data and the virtual data are combined to obtain sampled data. Each sampled data includes target data corresponding to the target attribute and virtual data corresponding to the non-statistical attribute.

2. The method according to claim 1, characterized in that, The step of statistically filtering the data in the attribute set corresponding to each attribute to obtain statistically valid attributes that meet preset statistical conditions includes: The number of categories is obtained by counting the categories of data in each attribute set; Calculate the percentage of each category in the attribute set based on the basic data corresponding to each category in the attribute set; Based on the number of categories, the proportion of categories, and the preset proportion threshold, statistically valid attributes that meet the preset statistical conditions are obtained.

3. The method according to claim 2, characterized in that, The step of obtaining statistically significant attributes that satisfy preset statistical conditions based on the number of categories, the proportion of categories, and a preset proportion threshold includes: The category percentage is compared with a preset percentage threshold to determine whether there are any categories whose category percentage is greater than the preset percentage threshold; When there is a category whose proportion is greater than a preset proportion threshold, the attribute corresponding to the attribute set is divided into statistically significant attributes; If there are no categories whose proportion is greater than a preset proportion threshold, determine whether the number of the categories is greater than a preset category threshold. If the number of categories is greater than a preset category threshold, the attributes corresponding to the attribute set will be classified as non-statistical attributes; If the number of categories is less than or equal to a preset category threshold, the attributes corresponding to the attribute set are divided into statistically significant attributes.

4. The method according to claim 3, characterized in that, When there are categories with a proportion greater than a preset proportion threshold, after classifying the attributes corresponding to the attribute set into statistically significant attributes, the following further steps are taken: A category retention value is calculated based on the preset category threshold, and the category retention value is less than the preset category threshold and the difference between the preset category threshold and the category retention value is one. The categories in the attribute set are sorted from largest to smallest according to the percentage of each category, to obtain a percentage sequence; Extract the number of categories corresponding to the category retention value from the segment with the largest category percentage value in the percentage sequence, and then summarize the remaining categories into one retention category.

5. The method according to claim 1, characterized in that, The step of randomly generating virtual data for the sampling size based on the non-statistical attribute includes: Obtain multiple data categories corresponding to the non-statistical attributes; Virtual data is randomly generated based on the data category, where the virtual data is one of the data categories.

6. The method according to claim 1, characterized in that, Also includes: Obtain updated data from the database; The updated data is analyzed based on the bitmap index to obtain difference data, which is data that does not exist in the bitmap index; The attributes and categories of the difference data are analyzed, and a new bitmap is generated and added to the bitmap index based on the attributes and categories corresponding to the difference data.

7. A data sampling device based on data modeling, characterized in that, include: The acquisition module is used to acquire data sampling requirements, which include data attributes and sampling quantity. The comparison module is used to compare the data attributes with preset statistical attributes to obtain target attributes and non-statistical attributes; The extraction module is used to classify the basic data in the database to obtain multiple attribute sets, and each attribute set contains data of multiple categories of the same attribute; Perform statistical filtering on the data in the attribute set corresponding to each attribute to obtain the statistically valid attributes that meet the preset statistical conditions; Create a corresponding bitmap for each category in the attribute set to obtain the type index corresponding to the attribute; merge the type indices corresponding to multiple attributes into a bitmap index; extract the target data corresponding to the sampling quantity from the preset bitmap index according to the target attribute. The bitmap index is built by extracting base data of multiple attributes from a database, which stores data for all attributes. The generation module is used to randomly generate virtual data of the sampling quantity based on the non-statistical attribute; The combination module is used to combine the target data and the virtual data to obtain sampled data. Each sampled data includes target data corresponding to the target attribute and virtual data corresponding to the non-statistical attribute.

8. A data sampling device based on data modeling, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

Images