Data sampling method and device, equipment and medium
By using dimension trees to sample data in the case of large data volumes, we ensure that each data only corresponds to one leaf node, solving the problems of uneven data distribution and duplicate data caused by random number method, and achieving scientific, accurate and efficient data sampling .
Patent Information
- Application Number
- CN202310728934.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-19
- Publication Date
- 2024-12-20
AI Technical Summary
In the case of large data volume, random number method can easily lead to uneven data distribution when sampling, and the results are unscientific. When there are many dimensions and large data volumes, duplicate data often appear in the sampling results, which requires additional processing and cumbersome process. .
By obtaining the dimension tree corresponding to the original data, select the corresponding number of data from the original data of the leaf node according to the sampling ratio of each leaf node of the dimension tree, ensuring that each data corresponds to only one leaf node, avoid duplicate data, and use Parallel processing and hash partitioning methods for data sampling.
The scientificity and accuracy of data sampling results are achieved, duplicate data is avoided, the process is simplified, and the data sampling efficiency is improved.
Smart Images
Figure CN119166685A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to information technology, and in particular to a data sampling method, device, equipment and medium. Background Art
[0002] Big data has the characteristics of large volume, rich variety, and fast growth rate. At the same time, it also has problems such as low value density and poor representativeness, which brings opportunities and challenges to data sampling. In daily life, in order to meet the needs of certain operational activities, data sampling is often used to analyze and process big data. A part of the data is randomly extracted from a large amount of data as a control group, and the extracted control group is subsequently processed.
[0003] At present, when data personnel conduct data sampling, they usually use more convenient methods such as random numbers for sampling. However, when the amount of data is large, the random number method will lead to uneven data distribution and unscientific control group results. In order to sample data more scientifically, data processing personnel generally determine the dimensions of data sampling before sampling. However, when the number of dimensions is too large and the amount of data is large, this sampling method will lead to duplicate data in the obtained sampling results, which requires further secondary processing and the process is cumbersome. Summary of the invention
[0004] The present application provides a data sampling method, device, equipment and medium to solve the problem of duplicate data in sampling results.
[0005] On the one hand, the present application provides a data sampling method, including: obtaining original data, a dimension tree corresponding to the original data, and a sampling ratio; wherein the dimension tree represents the hierarchical relationship between the data dimensions of the original data; for the original data corresponding to each leaf node of the dimension tree, according to the sampling ratio, a corresponding number of data are selected from the original data corresponding to the leaf node to obtain the sampled data corresponding to each leaf node; the sampled data corresponding to all leaf nodes of the dimension tree are integrated to obtain the data sampling result.
[0006] In a possible implementation, obtaining a dimension tree corresponding to the original data includes: obtaining field values of data dimension fields based on the original data; combining all field values of each data dimension using an aggregate query to obtain a two-dimensional table; each combination includes the field value of at least one dimension field; and traversing the two-dimensional table to generate a dimension tree corresponding to the original data.
[0007] In a possible implementation, for the original data corresponding to each leaf node of the dimension tree, a corresponding number of data is selected from the original data corresponding to the leaf node according to a sampling ratio to obtain sampled data corresponding to each leaf node, including: for the original data corresponding to each leaf node of the dimension tree, the original data corresponding to the leaf node is evenly divided into multiple subsets; according to the number of multiple subsets and the hash factor of the original data corresponding to the leaf node, a hash value of the original data corresponding to the leaf node is calculated, and a label of each subset corresponding to the leaf node is obtained by performing a remainder calculation; wherein all the original data under each subset have the same label; according to the sampling ratio and the number of multiple subsets, a sampling label range is calculated, and a subset whose label is within the sampling label range is selected from the multiple subsets as the sampled data corresponding to the leaf node.
[0008] In a possible implementation, the method further includes: if the data dimension of the original data is empty, treating the original data as a leaf node.
[0009] In a possible implementation, for the original data corresponding to each leaf node of the dimension tree, a corresponding number of data is selected from the original data corresponding to the leaf node according to a sampling ratio to obtain the sampled data corresponding to each leaf node, including: establishing multiple threads, where the multiple threads correspond one-to-one to the leaf nodes; using a parallel processing method, for the original data corresponding to each leaf node, based on the thread corresponding to the leaf node, a corresponding number of data is selected from the original data corresponding to the leaf node according to a sampling ratio to obtain the sampled data corresponding to each leaf node; wherein each thread corresponds to a leaf node.
[0010] On the other hand, the present application provides a data sampling device, including: an acquisition module, used to acquire original data, a dimension tree corresponding to the original data, and a sampling ratio; wherein the dimension tree represents the hierarchical relationship between the data dimensions of the original data; a processing module, used to select a corresponding number of data from the original data corresponding to each leaf node of the dimension tree according to the sampling ratio, to obtain the sampling data corresponding to each leaf node; an integration module, used to integrate the sampling data corresponding to all leaf nodes of the dimension tree to obtain a data sampling result.
[0011] In one possible implementation, the acquisition module is specifically used to: obtain the field value of the data dimension field based on the original data; use an aggregate query to combine all the field values of each data dimension to obtain a two-dimensional table; each combination includes the field value of at least one dimension field; traverse the two-dimensional table to generate a dimension tree corresponding to the original data.
[0012] In one possible implementation, the processing module is specifically used to: for each leaf node of the dimension tree, divide the original data corresponding to the leaf node into multiple subsets on average; calculate the hash value of the original data corresponding to the leaf node according to the number of the multiple subsets and the hash factor of the original data corresponding to the leaf node, and obtain the label of each subset corresponding to the leaf node by performing a remainder calculation; wherein all the original data under each subset have the same label; calculate the sampling label range according to the sampling ratio and the number of the multiple subsets, and select the subset whose label is within the sampling label range from the multiple subsets as the sampling data corresponding to the leaf node.
[0013] In a possible implementation, the acquisition module is further used to: if the data dimension of the original data is empty, then treat the original data as a leaf node.
[0014] In one possible implementation, the processing module is specifically used to: establish multiple threads, where the multiple threads correspond one-to-one to the leaf nodes; adopt a parallel processing method, for the original data corresponding to each leaf node, based on the thread corresponding to the leaf node, and according to the sampling ratio, select a corresponding number of data from the original data corresponding to the leaf node to obtain the sampled data corresponding to each leaf node; wherein each thread corresponds to a leaf node.
[0015] On the other hand, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the above method.
[0016] On the other hand, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.
[0017] In the data sampling method, device, equipment and medium provided by the present application, the original data and the dimension tree and sampling ratio corresponding to the original data are obtained, and according to the original data corresponding to each leaf node of the dimension tree, a corresponding amount of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node, and the sampled data corresponding to all leaf nodes of the dimension tree are integrated to obtain the result of data sampling. The scheme of the present application performs data sampling based on the leaf nodes of the dimension tree, wherein the dimension tree represents the hierarchical relationship between the data dimensions of the data sampling, so as to ensure that each data in the original data will only correspond to one of the leaf nodes under the dimension tree, and subsequently perform data sampling based on the original data under each leaf node, and the obtained sampling result can effectively avoid the occurrence of duplicate data, and realize convenient and accurate data sampling. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0019] Figure 1 Schematic diagram of the process of the data sampling method provided in the first embodiment of the present application is shown in FIG.
[0020] Figure 2 A schematic diagram of a dimension tree of an example;
[0021] Figure 3 This is a specific flowchart for data sampling using Airflow;
[0022] Figure 4 The structural diagram of the data sampling device provided in the second embodiment of the present application is exemplarily shown in FIG.
[0023] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Example 3 of the present application.
[0024] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0026] The terms "including" and "having" in this application are used to express an open-ended inclusion, and mean that in addition to the listed elements / components / etc., there may be additional elements / components / etc.; the terms "first" and "second" etc. are only used as marks or distinctions, and are not intended to limit the order or quantity of their objects. In addition, the different elements and areas in the drawings are only shown schematically, and are therefore not limited to the sizes or distances shown in the drawings. The technical solution is described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0027] In the context of big data, with the advancement of information technology, a large amount of data resources have emerged and are prevalent in various fields. It not only brings new ideas and concepts, but also provides opportunities for the process of data sampling. Data sampling is a systematic and scientific data processing method. Even in the context of big data, data sampling still has strong vitality, and data sampling methods are still needed to process data in many scenarios in life.
[0028] As an example, in actual applications, in order to meet the needs of certain operational activities, a certain proportion of users are usually extracted from the target user data as a control group. Due to the diversity of scenarios and data in actual applications, in order to make the data sampling results more realistically and accurately reflect the characteristics of the original user group, it is usually necessary to consider the proportion of the sampled data volume in different dimensions in the entire sampling results, and whether it is consistent with the distribution in the original data. For example, if you need to extract 10% of the users from the entire national user base as a control group, you may need to consider whether the proportion of the control group users in various dimensions, such as province, gender, age group, etc., is consistent with the proportion of these dimensions in the original data to ensure that the composition structure of the data is similar.
[0029] In actual operations, data personnel usually use simpler methods to sample data, such as the random number method. However, when the amount of data is large, the random number method will lead to uneven distribution of control group data and unscientific results. In order to make the sampling results more scientific and effective, data personnel generally determine the dimensions of data sampling before sampling, and then perform subsequent processing. However, when the number of dimensions is too large and the amount of data is large, this sampling method will lead to duplication of data in the sampling results, requiring additional secondary processing, and the process is cumbersome.
[0030] Some aspects of the embodiments of the present application relate to the above considerations. The following is an example introduction of the solution in combination with some embodiments.
[0031] Figure 1 The flowchart of the data sampling method provided in the first embodiment of the present application is shown in FIG. 1 . The execution subject of this embodiment may be a data sampling device, such as Figure 1 As shown, the method includes:
[0032] Step 101: Obtain original data, a dimension tree corresponding to the original data, and a sampling ratio;
[0033] Step 102: for the original data corresponding to each leaf node of the dimension tree, select a corresponding amount of data from the original data corresponding to the leaf node according to the sampling ratio to obtain sampled data corresponding to each leaf node;
[0034] Step 103: Integrate the sampled data corresponding to all leaf nodes of the dimension tree to obtain data sampling results.
[0035] In practical applications, the executor of the method can be a data sampling device. There are many ways to implement the data sampling device. For example, it can be implemented through a computer program, such as application software, etc.; or it can be implemented as a medium storing relevant computer programs, such as a USB flash drive, a cloud disk, etc.; or it can be implemented through a physical device that integrates or installs relevant computer programs, such as a chip, etc.
[0036] Specifically, the original data refers to the data source from which data sampling is required. The type of original data can be determined according to the needs of data sampling. As an example, assuming that the original data is user data, for example, the original data may include but is not limited to the hash factor corresponding to the original data, user personal information, user communication data, user traffic data, etc., and the original data can be obtained from the user database. Among them, the sampling ratio can be set as needed. For example, when the amount of data to be sampled is large, a higher sampling ratio can be set, and when the amount of sampled data is small, a lower sampling ratio can be set. Optionally, the form of the sampling ratio can also be pre-set, for example, it can be in the form of a percentage.
[0037] Among them, the dimension tree represents the hierarchical relationship between the data dimensions of the original data. The hierarchical relationship between the data dimensions can reflect clear logic in the tree. As an example, if there is only one dimension, then the tree has only one layer; if there are two dimensions, then the tree has only two layers; if there are multiple dimensions, then the tree has multiple layers. Specifically, the dimension tree can clearly reflect the dimensional fields of the data and the hierarchical relationship of the data dimensions. All leaf nodes in the dimension tree reflect the grouping of the original data. Among them, the leaf node here refers to the node at the end of the dimension tree, that is, the node that is not the parent node of any node. Based on the establishment of the dimension tree, the number of all leaf nodes is the number of groups of the original data, and each original data will only be divided into one leaf node, so as to ensure that the subsequent data sampling based on the original data under the leaf node will not have duplicate data in the obtained sampling results.
[0038] For a better understanding, combine Figure 2 For example, Figure 2 As shown in the figure, it is necessary to sample data from the original users according to the three data dimensions of province (Beijing / Tianjin / Shanghai), gender (male / female), and age (elderly / middle-aged / young), then the dimension tree corresponding to the original data can be established in advance. Figure 2As shown in the figure, in order from top to bottom, the root node of the first layer of the dimension tree corresponds to all original users; the second layer node is the first grouping of the original users according to the dimension of province, and the user groups corresponding to different provinces under the province dimension are obtained, for example, including the user grouping under the province dimension of Beijing, the user grouping under the province dimension of Tianjin, and the user grouping under the province dimension of Shanghai; the third layer further groups the user groups under each node in the second layer according to the dimension of gender, and obtains the user groups corresponding to different province dimensions and different gender dimensions, for example, including the user grouping under the province dimension of Beijing and the user grouping under the province dimension of gender of male, and the user grouping under the province dimension of Beijing and the user grouping under the province dimension of male. The fourth layer further groups the user groups under each node in the third layer according to the dimension of age, and obtains the corresponding user groups under different provinces, different genders, and different age dimensions. For example, the user group with the province dimension of Beijing is the male gender dimension and the age dimension is the elderly, the user group with the province dimension of Beijing is the male gender dimension and the age dimension is the middle-aged user group, and the user group with the province dimension of The user grouping for male gender and young age in Beijing, the user grouping for female gender and old age in Beijing, the user grouping for female gender and middle-aged in Beijing, the user grouping for female gender and young age in Beijing, the user grouping for male gender and old age in Tianjin, the user grouping for male gender and middle-aged in Tianjin, the user grouping for male gender and young age in Tianjin, the user grouping for female gender and old age in Tianjin, the user grouping for male gender and female gender in Tianjin, the user grouping for The user grouping with gender dimension of female and age dimension of middle-aged under Tianjin dimension, the user grouping with gender dimension of female and age dimension of young under Tianjin province dimension, the user grouping with gender dimension of male and age dimension of old under Shanghai province dimension, the user grouping with gender dimension of male and middle-aged under Shanghai province dimension, the user grouping with gender dimension of male and age dimension of young under Shanghai province dimension, the user grouping with gender dimension of female and age dimension of old under Shanghai province dimension, the user grouping with gender dimension of female and age dimension of middle-aged under Shanghai province dimension, and the user grouping with gender dimension of female and age dimension of young under Shanghai province dimension. Finally, the original user data is divided into 18 groups of data, corresponding to 18 leaf nodes, each group of data belongs to only one of the leaf nodes, and there is no duplicate data between each group of data.
[0039] Combination Figure 2From the example, it can be seen that the hierarchical relationship between dimensions reflected in the figure is that the next level of the province dimension is the gender dimension, and the next level of the gender dimension is the age dimension. It should be noted that the figure is only an example. In actual applications, there is no restriction on the hierarchical relationship between each data dimension. For example, the gender dimension can be used as the highest-level dimension, or the age dimension can be used as the highest-level dimension. The hierarchical relationship between each data dimension can be pre-set, as long as all sampled data dimensions are reflected in the dimension tree. Subsequent sampling is performed based on the leaf nodes of the dimension tree, and the sampled data obtained can avoid duplicate data. In actual applications, the dimension tree can be established in advance and stored in the form of a two-dimensional table. When data sampling is required from the original data again in the future, the dimension tree can be obtained directly by extracting the content in the two-dimensional table.
[0040] Specifically, after obtaining the dimension tree, the original data under each leaf node is sampled according to the sampling ratio to obtain the sampled data of each leaf node, and finally the sampled data of all leaf nodes are integrated to obtain the data sampling result. The data sampling result obtained by the above scheme avoids duplicate data, thereby achieving convenient and accurate data sampling.
[0041] Combined with the scenario example, in actual application, the execution code can be called through open source scheduling tools, such as Airflow, to implement the process of this embodiment. For example, the configuration file of the scheduling system can be written in advance and referenced PythonOperator. In Airflow, Python Operator can be used to execute Python code. Specifically, in this scenario, the dimension and sampling ratio can be set by configuring the parameters of abdata under the open source scheduling tool. According to the actual usage, the execution code is called, and finally the result file path or table name is returned to the scheduling system, so as to realize data sampling in Airflow. The specific implementation method flow is as follows. Figure 3 As shown, Figure 3 This is a specific flowchart for using Airflow for data sampling.
[0042] Optionally, the method for establishing the dimension tree is not limited. In one example, the dimension tree can be generated according to the two-dimensional table. Specifically, in step 101, the dimension tree corresponding to the original data is obtained, which specifically includes:
[0043] Based on the original data, the field values of the data dimension fields are obtained; all field values of each data dimension are combined using an aggregate query to obtain a two-dimensional table; each combination includes the field value of at least one dimension field; the two-dimensional table is traversed to generate a dimension tree corresponding to the original data.
[0044] In this example, after obtaining the original data, in order to use the aggregation query method to build a two-dimensional table later, the field values of the data dimension fields can be obtained based on the user personal information analysis in the original data. For example, the original data itself has dimension fields, including province, gender, age, etc., then the corresponding field values are the specific values under the above dimension fields, such as the field values under the province dimension field can be Beijing, Tianjin, Shanghai; the field values under the gender dimension field can be male or female; the field values under the age dimension field can be old, middle-aged, young, etc. At this time, the dimension field values for data sampling can be obtained based on the dimension field values contained in the original data.
[0045] After analyzing and obtaining the field values of the original data dimension fields, you can use the aggregation query method to combine all the field values of each data dimension according to the different field values and the hierarchical relationship of the data dimensions to obtain a two-dimensional table. Each combination includes the field value of at least one dimension field. For example, the above-mentioned province dimension fields are combined to obtain three combinations of province dimensions of Beijing, Tianjin, and Shanghai, which are put into the two-dimensional table to obtain the first dimension level, corresponding to the second-level node of the dimension tree. The gender dimension field value is combined with the above-mentioned three combinations again to obtain six combinations of province dimension of Beijing and gender dimension of male; province dimension of Beijing and gender dimension of female; province dimension of Tianjin and gender dimension of male; province dimension of Tianjin and gender dimension of female; province dimension of Shanghai and gender dimension of male; province dimension of Shanghai and gender dimension of female, which are put into the two-dimensional table to obtain the second dimension level, corresponding to the third-level node of the dimension tree. The age dimension field value is further combined with the above 6 combinations to obtain the province dimension of Beijing, gender dimension of male, and age dimension of old age, the province dimension of Beijing, gender dimension of male, and age dimension of middle age, the province dimension of Beijing, gender dimension of male, and age dimension of youth, the province dimension of Beijing, gender dimension of female, and age dimension of old age, the province dimension of Beijing, gender dimension of female, and age dimension of middle age, the province dimension of Beijing, gender dimension of female, and age dimension of youth, the province dimension of Tianjin, gender dimension of male, and age dimension of old age, the province dimension of Tianjin, gender dimension of male, and age dimension of middle age, the province dimension of Tianjin, gender dimension of male, and age dimension of youth, and the province dimension of Tianjin , gender dimension is female, age dimension is old, province dimension is Tianjin, gender dimension is female, age dimension is middle-aged, province dimension is Tianjin, gender dimension is female, age dimension is youth, province dimension is Shanghai, gender dimension is male, age dimension is old, province dimension is Shanghai, gender dimension is male, age dimension is middle-aged, province dimension is Shanghai, gender dimension is male, age dimension is youth, province dimension is Shanghai, gender dimension is female, age dimension is old, province dimension is Shanghai, gender dimension is female, age dimension is middle-aged, province dimension is Shanghai, gender dimension is female, age dimension is youth, province dimension is Shanghai, gender dimension is female, age dimension is old, province dimension is Shanghai, gender dimension is female, age dimension is middle-aged, province dimension is Shanghai, gender dimension is female, age dimension is youth, put into the two-dimensional table, get the third dimension level, corresponding to the 18 leaf nodes of the fourth layer of the dimension tree. It can be obtained that the fewer the dimension field values in the combination, the higher the dimension level. Among them, the two-dimensional table includes but is not limited to dimension field values, dimension combinations, etc.
[0046] The dimension level can be random and has no effect on the sampling results. For example, the original data can be grouped by province first, then by gender, and finally by age. It can also be grouped by gender first, then by age, and finally by province. As long as the final grouping result belongs to only one dimension combination, the sampling can be completed in one go, and there is no need to process duplicate data between groups after the first grouping. In practical applications, although dimensions are enumerable, the value combinations of dimensions are indeed extremely diverse. Users do not need to list the dimension combinations corresponding to the original data one by one before sampling. All dimension combinations will be clearly reflected in the two-dimensional table. Traverse the two-dimensional table to generate the dimension tree corresponding to the original data.
[0047] Through the solution of this example, the field values of the dimension fields of the original data can be obtained for the user personal information in the original data, and the field values of the dimension fields can be combined by means of aggregate query to obtain a two-dimensional table of dimension combinations and generate a dimension tree. The above solution can quickly and accurately group the original data so that there is no duplicate data in each dimension combination, thereby achieving convenient and accurate data sampling.
[0048] In actual applications, after obtaining the original data, when using the aggregation query method to build a two-dimensional table, since the dimension relationship in the original data is not close, after the aggregation query, the field values of the data dimensions obtained are different and the field values cannot be combined. In one example, when the original data type is relatively divergent, the method also includes: if the data dimension of the original data is empty, the original data is used as a leaf node.
[0049] In this example, when the data dimension relationship in the original data is not close or there is no data dimension relationship, the combination of dimension field values cannot be obtained using the aggregation query method. At this time, the data dimension in the original data is empty. Then the original data is taken as a leaf node, and sampling is performed in the original data corresponding to the above leaf node as long as the set sampling ratio is met.
[0050] Through the solution of this example, when the original data dimension is empty, the original data is used as a leaf node for subsequent proportional sampling, which is applicable to more abundant sampling scenarios, takes into account comprehensive sampling conditions, and improves the reliability and accuracy of data sampling.
[0051] Optionally, after the dimension tree is established, the original data corresponding to each leaf node is sampled. There can be many specific sampling methods, and sampling only needs to be implemented according to the set sampling ratio. In one example, the original data under each leaf node can be evenly divided into multiple subsets, the subsets are numbered, and the subsets within the number range are selected as the sampled data corresponding to each leaf node. In step 102, for the original data corresponding to each leaf node of the dimension tree, a corresponding amount of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node, specifically including:
[0052] For the original data corresponding to each leaf node of the dimension tree, the original data corresponding to the leaf node is evenly divided into multiple subsets; according to the number of multiple subsets and the hash factor of the original data corresponding to the leaf node, the hash value of the original data corresponding to the leaf node is calculated, and the label of each subset corresponding to the leaf node is obtained by performing a remainder calculation; wherein, the labels of all original data under each subset are the same; according to the sampling ratio and the number of multiple subsets, the sampling label range is calculated, and the subset whose label is within the sampling label range is selected from the multiple subsets as the sampling data corresponding to the leaf node.
[0053] In this example, for the original data corresponding to each leaf node, the original data corresponding to each leaf node is evenly divided into 100 subsets, and the number of original data in each subset is equal or nearly equal. According to the hash factor of the original data corresponding to each leaf node, the hash value of the original data corresponding to each leaf node is calculated. According to the number of subsets and the hash value of the original data corresponding to the leaf node, by performing a remainder calculation, the remainder of the original data corresponding to each leaf node is obtained as one of the integers in 0-99, and the remainder is added by 1 to obtain one of the integers in 0-100 as the label corresponding to the original data, and the original data with the same label is divided into a subset, and the label of the original data in the subset is used as the label of the subset.
[0054] There is no limit to the method of labeling the subsets. You can also use a random number method to randomly label the divided subsets to ensure that each subset corresponds to a random integer from 1 to 100, and the labels of each subset are different. According to the sampling ratio and the number of subsets, the sampling label range is calculated, and the subsets with labels within the sampling label range are selected from the 100 subsets as the sampling data corresponding to the leaf node.
[0055] For example, 10% of the target users with the three characteristics of Beijing, male, and youth are randomly selected from the original user data for activity A, 20% of the target users are used for activity B, and 70% of the target users are used for activity C. According to the sampling ratio of 10% and the number of subsets of 100, 10 groups of subsets are multiplied together, and the data with subset numbers 1-10 under the leaf node corresponding to the dimension combination are selected as the target users for activity A; according to the sampling ratio of 20% and the number of subsets of 100, 20 groups of subsets are multiplied together, and the data with subset numbers 11-30 under the leaf node corresponding to the dimension combination are selected as the target users for activity B; according to the sampling ratio of 70% and the number of subsets of 100, 70 groups of subsets are multiplied together, and the data with subset numbers 31-100 under the leaf node corresponding to the dimension combination are selected as the target users for activity C, and the sampling data corresponding to the leaf node is obtained.
[0056] In the above example, for the original data corresponding to each leaf node, the original data corresponding to the leaf node is evenly divided into multiple subsets, and the hash value of the original data corresponding to the leaf node is calculated according to the number of multiple subsets and the hash factor of the original data corresponding to the leaf node. The label of each subset corresponding to the leaf node is obtained by performing a remainder calculation or a random number method. According to the sampling ratio and the number of multiple subsets, the sampling label range is calculated, and a subset whose label is within the sampling label range is selected from the multiple subsets as the sampling data corresponding to the leaf node. In the sampling process, combined with the actual sampling needs, the accuracy and practicality of the sampling results are improved by dividing the original data corresponding to each leaf node into subsets and labeling them.
[0057] In another example, in step 102, for the original data corresponding to each leaf node of the dimension tree, a corresponding amount of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node. The following method can also be used: the product of the amount of data of the original data corresponding to the leaf node and the sampling ratio is calculated to obtain a first quantity; and a first amount of data is selected from the original data corresponding to the leaf node as the sampled data corresponding to the leaf node.
[0058] In this example, the first quantity is obtained by multiplying the amount of original data by the set sampling ratio, and the first quantity of data is selected from the original data corresponding to the leaf node as the sampling result. For example, the original data corresponding to the leaf node is 1000 user data, and the set sampling ratio is 12%. Then, the amount of user data 1000 and the sampling ratio 12% are multiplied to obtain 120 users, and 120 users are randomly selected from the user data corresponding to the leaf node as the data sampling result.
[0059] In the above example, the method of obtaining the first quantity by multiplying the amount of original data by the set sampling ratio and randomly selecting the first quantity of data as the sampling result of the data makes the sampling process easier to operate, more convenient and faster.
[0060] Based on any of the above examples, in order to improve the processing efficiency of sampling, a parallel processing method can be used to sample data. In step 102, for the original data corresponding to each leaf node of the dimension tree, a corresponding amount of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node, specifically including:
[0061] Establish multiple threads, and the multiple threads correspond to the leaf nodes one by one; adopt parallel processing mode, for the original data corresponding to each leaf node, based on the thread corresponding to the leaf node, select the corresponding number of data from the original data corresponding to the leaf node according to the sampling ratio, and obtain the sampling data corresponding to each leaf node; wherein each thread corresponds to a leaf node.
[0062] Specifically, according to the original data corresponding to each leaf node, multiple threads are established, and the multiple threads correspond to the leaf nodes one by one, traverse the leaf nodes, and sample the user groups corresponding to all leaf nodes in parallel. Each leaf node corresponds to a thread, and each thread processes a group separately. According to the sampling ratio, a corresponding amount of data is selected from the original data corresponding to the leaf node as the sampled data corresponding to each leaf node. In the above example, according to the establishment of multiple threads and the use of parallel processing methods to select data, the sampling results can be obtained more quickly, which improves the processing efficiency of the sampling.
[0063] Furthermore, the sampling data corresponding to each leaf node is integrated to obtain the final data sampling result.
[0064] In the data sampling method provided in this embodiment, the original data is obtained, the field value of the data dimension field is obtained based on the original data, all the field values are combined to obtain a two-dimensional table, the two-dimensional table is traversed, and a dimension tree corresponding to the original data is generated. Then, for the original data corresponding to each leaf node of the dimension tree, a corresponding number of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node, and the sampled data is integrated to obtain the result of data sampling. In the scheme of the present application, data sampling is performed based on the leaf nodes of the dimension tree during the data sampling process, so as to ensure that each data in the original data will only correspond to one of the leaf nodes under the dimension tree, and data sampling is subsequently performed based on the original data under each leaf node. The sampling result obtained can effectively avoid the occurrence of duplicate data and realize convenient and accurate data sampling.
[0065] Figure 4 The structural diagram of the data sampling device provided in the second embodiment of the present application is exemplarily shown in FIG. Figure 4 As shown, the device comprises:
[0066] An acquisition module 41 is used to acquire original data, a dimension tree corresponding to the original data, and a sampling ratio;
[0067] The processing module 42 is used to select a corresponding amount of data from the original data corresponding to each leaf node of the dimension tree according to the sampling ratio to obtain sampled data corresponding to each leaf node;
[0068] The integration module 43 is used to integrate the sampled data corresponding to all leaf nodes of the dimension tree to obtain data sampling results.
[0069] In practical applications, there are many ways to implement the data sampling device. For example, it can be implemented through a computer program, such as application software, etc.; or it can be implemented as a medium storing relevant computer programs, such as a USB flash drive, a cloud disk, etc.; or it can be implemented through a physical device integrated or installed with a relevant computer program, such as a chip, etc.
[0070] Specifically, the raw data refers to the data source from which data sampling is required. The type of raw data can be determined according to the needs of data sampling. Among them, the sampling ratio can be set as needed. Optionally, the form of the sampling ratio can also be pre-set.
[0071] Among them, the dimension tree represents the hierarchical relationship between the data dimensions of the original data. The hierarchical relationship between the data dimensions can reflect clear logic in the tree. Specifically, the dimension tree can clearly reflect the dimensional fields of the data and the hierarchical relationship of the data dimensions. All leaf nodes in the dimension tree reflect the grouping of the original data. Among them, the leaf node here refers to the node at the end of the dimension tree, that is, the node that is not the parent node of any node. Based on the establishment of the dimension tree, the number of all leaf nodes is the number of groups of the original data, and each original data will only be divided into one leaf node, so as to ensure that the subsequent data sampling based on the original data under the leaf node will not have duplicate data in the obtained sampling results.
[0072] For a better understanding, suppose that data sampling is required from the original users according to the three data dimensions of province (Beijing / Tianjin / Shanghai), gender (male / female), and age (elderly / middle-aged / young), then the dimension tree corresponding to the original data can be established in advance. The original user data is finally divided into 18 groups of data, corresponding to 18 leaf nodes, each group of data belongs to only one of the leaf nodes, and there is no duplicate data between each group of data.
[0073] It should be noted that in actual applications, there is no restriction on the hierarchical relationship of each data dimension, and the hierarchical relationship between each data dimension can be preset, as long as all sampled data dimensions are reflected in the dimension tree. Subsequent sampling is performed based on the leaf nodes of the dimension tree, and the sampled data obtained can avoid duplicate data. In actual applications, the dimension tree can be established in advance and stored in the form of a two-dimensional table. When data sampling is required from the original data again in the future, the dimension tree can be obtained directly by extracting the content in the two-dimensional table.
[0074] Specifically, after the acquisition module 41 acquires the dimension tree, the processing module 42 samples the original data under each leaf node according to the sampling ratio to obtain the sampled data of each leaf node, and finally the integration module 43 integrates the sampled data of all leaf nodes to obtain the data sampling result. The data sampling result obtained by the above scheme avoids duplicate data, thereby achieving convenient and accurate data sampling.
[0075] In combination with the scenario example, in actual applications, the process of this embodiment can be implemented by calling the execution code through an open source scheduling tool.
[0076] Optionally, the method for establishing the dimension tree is not limited. In one example, the acquisition module acquires the dimension tree corresponding to the original data, specifically including:
[0077] Based on the original data, the field values of the data dimension fields are obtained; all field values of each data dimension are combined using an aggregate query to obtain a two-dimensional table; each combination includes the field value of at least one dimension field; the two-dimensional table is traversed to generate a dimension tree corresponding to the original data.
[0078] In this example, after obtaining the original data, in order to use the aggregation query method to build a two-dimensional table later, the field value of the data dimension field can be obtained based on the user personal information analysis in the original data. At this time, the dimension field value for data sampling can be obtained based on the dimension field value contained in the original data.
[0079] After analyzing and obtaining the field values of the original data dimension fields, you can use the aggregation query method to combine all the field values of each data dimension according to the different field values and the hierarchical relationship of the data dimensions to obtain a two-dimensional table. Each combination includes the field value of at least one dimension field. The fewer the dimension field values in the combination, the higher the dimension level. The two-dimensional table includes but is not limited to dimension field values, dimension combinations, etc.
[0080] The dimension level can be random and has no effect on the sampling results. As long as the final grouping result belongs to only one dimension combination, the sampling can be completed in one go, and there is no need to process duplicate data between groups after the first grouping. In practical applications, although the dimensions are enumerable, the value combinations of the dimensions are indeed very diverse. Users do not need to list the dimension combinations corresponding to the original data one by one before sampling. All dimension combinations will be clearly reflected in the two-dimensional table. Traverse the two-dimensional table to generate the dimension tree corresponding to the original data.
[0081] Through the solution of this example, the field values of the dimension fields of the original data can be obtained for the user personal information in the original data, and the field values of the dimension fields can be combined by means of aggregate query to obtain a two-dimensional table of dimension combinations and generate a dimension tree. The above solution can quickly and accurately group the original data so that there is no duplicate data in each dimension combination, thereby achieving convenient and accurate data sampling.
[0082] In actual applications, after obtaining the original data, when the aggregation query method is used to build a two-dimensional table, since the dimension relationship in the original data is not close, after the aggregation query, the field values of the data dimensions obtained are different and the field values cannot be combined. In one example, when the original data type is divergent, the acquisition module is also used to: if the data dimension of the original data is empty, the original data is used as a leaf node.
[0083] In this example, when the data dimension relationship in the original data is not close or there is no data dimension relationship, the combination of dimension field values cannot be obtained using the aggregation query method. At this time, the data dimension in the original data is empty. Then the original data is taken as a leaf node, and sampling is performed in the original data corresponding to the above leaf node as long as the set sampling ratio is met.
[0084] Through the solution of this example, when the original data dimension is empty, the original data is used as a leaf node for subsequent proportional sampling, which is applicable to more abundant sampling scenarios, takes into account comprehensive sampling conditions, and improves the reliability and accuracy of data sampling.
[0085] Optionally, after the dimension tree is established, the original data corresponding to each leaf node is sampled. There can be many specific sampling methods, and sampling only needs to be implemented according to the set sampling ratio. In one example, the original data under each leaf node can be evenly divided into multiple subsets, the subsets are labeled, and the subsets within the labeled range are selected as the sampled data corresponding to each leaf node. The processing module can select a corresponding amount of data from the original data corresponding to each leaf node of the dimension tree according to the sampling ratio, and obtain the sampled data corresponding to each leaf node, specifically including:
[0086] For the original data corresponding to each leaf node of the dimension tree, the original data corresponding to the leaf node is evenly divided into multiple subsets; according to the number of multiple subsets and the hash factor of the original data corresponding to the leaf node, the hash value of the original data corresponding to the leaf node is calculated, and the label of each subset corresponding to the leaf node is obtained by performing a remainder calculation; wherein, the labels of all original data under each subset are the same; according to the sampling ratio and the number of multiple subsets, the sampling label range is calculated, and the subset whose label is within the sampling label range is selected from the multiple subsets as the sampling data corresponding to the leaf node.
[0087] In this example, for the original data corresponding to each leaf node, the original data corresponding to each leaf node is evenly divided into 100 subsets, and the number of original data in each subset is equal or nearly equal. According to the hash factor of the original data corresponding to each leaf node, the hash value of the original data corresponding to each leaf node is calculated. According to the number of subsets and the hash value of the original data corresponding to the leaf node, by performing a remainder calculation, the remainder of the original data corresponding to each leaf node is obtained as one of the integers in 0-99, and the remainder is added by 1 to obtain one of the integers in 0-100 as the label corresponding to the original data, and the original data with the same label is divided into a subset, and the label of the original data in the subset is used as the label of the subset.
[0088] There is no limit to the method of labeling the subsets. You can also use a random number method to randomly label the divided subsets to ensure that each subset corresponds to a random integer from 1 to 100, and the labels of each subset are different. According to the sampling ratio and the number of subsets, the sampling label range is calculated, and the subsets with labels within the sampling label range are selected from the 100 subsets as the sampling data corresponding to the leaf node.
[0089] In the above example, the processing module divides the original data corresponding to each leaf node into multiple subsets on average; calculates the hash value of the original data corresponding to the leaf node according to the number of multiple subsets and the hash factor of the original data corresponding to the leaf node, and obtains the label of each subset corresponding to the leaf node by performing remainder calculation or random number method; calculates the sampling label range according to the sampling ratio and the number of multiple subsets, and selects the subset whose label is within the sampling label range from the multiple subsets as the sampling data corresponding to the leaf node. In the sampling process, combined with the actual sampling needs, the accuracy and practicality of the sampling results are improved by dividing the original data corresponding to each leaf node into subsets and labeling them.
[0090] In another example, the processing module selects a corresponding amount of data from the original data corresponding to each leaf node of the dimension tree according to a sampling ratio to obtain the sampled data corresponding to each leaf node. The following device can also be used: the product of the amount of data of the original data corresponding to the leaf node and the sampling ratio is calculated to obtain a first quantity; and the first amount of data is selected from the original data corresponding to the leaf node as the sampled data corresponding to the leaf node.
[0091] In this example, the first quantity is obtained by multiplying the amount of original data by the set sampling ratio, and the first quantity of data is selected from the original data corresponding to the leaf node as the sampling result. In the above example, the method of multiplying the amount of original data by the set sampling ratio to obtain the first quantity and randomly selecting the first quantity of data as the sampling result of the data makes the sampling process easier to operate, convenient and quick.
[0092] Based on any of the above examples, in order to improve the processing efficiency of sampling, a parallel processing method can be used to sample data. The processing module selects a corresponding amount of data from the original data corresponding to each leaf node of the dimension tree according to the sampling ratio to obtain the sampled data corresponding to each leaf node, specifically including:
[0093] Establish multiple threads, and the multiple threads correspond to the leaf nodes one by one; adopt parallel processing mode, for the original data corresponding to each leaf node, based on the thread corresponding to the leaf node, select the corresponding number of data from the original data corresponding to the leaf node according to the sampling ratio, and obtain the sampling data corresponding to each leaf node; wherein each thread corresponds to a leaf node.
[0094] Specifically, according to the original data corresponding to each leaf node, multiple threads are established, and the multiple threads correspond to the leaf nodes one by one, traverse the leaf nodes, and sample the user groups corresponding to all leaf nodes in parallel. Each leaf node corresponds to a thread, and each thread processes a group separately. According to the sampling ratio, a corresponding amount of data is selected from the original data corresponding to the leaf node as the sampled data corresponding to each leaf node. In the above example, according to the establishment of multiple threads and the use of parallel processing methods to select data, the sampling results can be obtained more quickly, which improves the processing efficiency of the sampling.
[0095] Furthermore, the integration module integrates the sampling data corresponding to each leaf node to obtain the final data sampling result.
[0096] In the data sampling device provided in this embodiment, the acquisition module acquires the original data, obtains the field value of the data dimension field based on the original data, combines all the field values to obtain a two-dimensional table, traverses the two-dimensional table, and generates a dimension tree corresponding to the original data. Then, the processing module selects a corresponding number of data from the original data corresponding to each leaf node of the dimension tree according to the sampling ratio, obtains the sampled data corresponding to each leaf node, and the integration module integrates the sampled data to obtain the result of data sampling. In the scheme of the present application, data sampling is performed based on the leaf nodes of the dimension tree during the data sampling process, so as to ensure that each data in the original data will only correspond to one of the leaf nodes under the dimension tree, and subsequently data sampling is performed based on the original data under each leaf node. The sampling result obtained can effectively avoid the occurrence of duplicate data and realize convenient and accurate data sampling.
[0097] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Example 3 of the present application, such as Figure 5 As shown, the electronic device includes:
[0098] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, the memory 292, and the communication interface 293 may communicate with each other through the bus 294. The communication interface 293 may be used for information transmission. The processor 291 may call the logic instructions in the memory 292 to execute the above-mentioned example method.
[0099] In addition, the logic instructions in the above-mentioned memory 292 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.
[0100] The memory 292 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, that is, implementing the methods in the above method examples.
[0101] The memory 292 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 292 may include a high-speed random access memory and may also include a non-volatile memory.
[0102] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method in any example.
[0103] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.
[0104] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.
[0105] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A data sampling method, characterized in that: include: Obtaining original data, a dimension tree corresponding to the original data, and a sampling ratio; wherein the dimension tree represents a hierarchical relationship between data dimensions of the original data; For the original data corresponding to each leaf node of the dimension tree, a corresponding amount of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node; The sampled data corresponding to all leaf nodes of the dimension tree are integrated to obtain a data sampling result.
2. The method according to claim 1, characterized in that The obtaining of the dimension tree corresponding to the original data includes: Based on the original data, obtain the field value of the data dimension field; Using an aggregate query to combine all field values of each data dimension to obtain a two-dimensional table; each combination includes a field value of at least one dimension field; The two-dimensional table is traversed to generate a dimension tree corresponding to the original data.
3. The method according to claim 1, characterized in that: The raw data corresponding to each leaf node of the dimension tree, selecting a corresponding amount of data from the raw data corresponding to the leaf node according to the sampling ratio, to obtain the sampled data corresponding to each leaf node, includes: For the original data corresponding to each leaf node of the dimension tree, the original data corresponding to the leaf node is evenly divided into multiple subsets; According to the number of the plurality of subsets and the hash factor of the original data corresponding to the leaf node, a hash value of the original data corresponding to the leaf node is calculated, and a label of each subset corresponding to the leaf node is obtained by performing a remainder calculation; wherein the labels of all the original data under each subset are the same; A sampling number range is calculated according to the sampling ratio and the number of the multiple subsets, and a subset whose number is within the sampling number range is selected from the multiple subsets as the sampling data corresponding to the leaf node.
4. The method according to claim 2, characterized in that: The method further comprises: If the data dimension of the original data is empty, the original data is taken as a leaf node.
5. The method according to any one of claims 1 to 4, characterized in that The raw data corresponding to each leaf node of the dimension tree, selecting a corresponding amount of data from the raw data corresponding to the leaf node according to the sampling ratio, to obtain the sampled data corresponding to each leaf node, includes: Establishing multiple threads, wherein the multiple threads correspond to the leaf nodes one by one; A parallel processing method is adopted. For the original data corresponding to each leaf node, based on the thread corresponding to the leaf node, a corresponding amount of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node; wherein each thread corresponds to a leaf node.
6. A data sampling device, characterized in that: include: An acquisition module, used to acquire original data, a dimension tree corresponding to the original data, and a sampling ratio; wherein the dimension tree represents a hierarchical relationship between data dimensions of the original data; A processing module, configured to select a corresponding amount of data from the original data corresponding to each leaf node of the dimension tree according to the sampling ratio, so as to obtain sampled data corresponding to each leaf node; The integration module is used to integrate the sampled data corresponding to all leaf nodes of the dimension tree to obtain data sampling results.
7. The device according to claim 6, characterized in that The acquisition module is specifically used for: Based on the original data, obtain the field value of the data dimension field; Using an aggregate query to combine all field values of each data dimension to obtain a two-dimensional table; each combination includes a field value of at least one dimension field; The two-dimensional table is traversed to generate a dimension tree corresponding to the original data.
8. The device according to claim 6, characterized in that The processing module is specifically used for: For the original data corresponding to each leaf node of the dimension tree, the original data corresponding to the leaf node is evenly divided into multiple subsets; According to the number of the plurality of subsets and the hash factor of the original data corresponding to the leaf node, a hash value of the original data corresponding to the leaf node is calculated, and a label of each subset corresponding to the leaf node is obtained by performing a remainder calculation; wherein the labels of all the original data under each subset are the same; A sampling number range is calculated according to the sampling ratio and the number of the multiple subsets, and a subset whose number is within the sampling number range is selected from the multiple subsets as the sampling data corresponding to the leaf node.
9. The device according to claim 7, characterized in that The acquisition module is further used for: If the data dimension of the original data is empty, the original data is taken as a leaf node.
10. The device according to any one of claims 6 to 9, characterized in that: The processing module is specifically used for: Establishing multiple threads, wherein the multiple threads correspond to the leaf nodes one by one; A parallel processing method is adopted. For the original data corresponding to each leaf node, based on the thread corresponding to the leaf node, a corresponding amount of data is selected from the original data corresponding to the leaf node according to the sampling ratio to obtain the sampled data corresponding to each leaf node; wherein each thread corresponds to a leaf node.
11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 5.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.