Multi-dimensional feature channel grouping sampling algorithm based on priori
By designing a priori-based multi-dimensional feature channel grouping sampling algorithm in multi-feature large-scale data, the traditional sampling method lacks accuracy and representativeness in multi-feature large-scale data, and an efficient and accurate sampling effect is achieved.
Patent Information
- Application Number
- CN202411416094.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2025-02-14
AI Technical Summary
Existing sampling algorithms are difficult to ensure sampling accuracy and representativeness in multi-feature large-scale data, and traditional probability sampling methods cannot meet the needs of high efficiency and precision.
A multi-dimensional feature channel grouping sampling algorithm based on priorities is designed. By calculating the priorities and minimum sampling numbers of feature channels, the corrected priorities and corrected sampling numbers are calculated, and the cycle optimization is achieved until the desired sampling accuracy is achieved and Sample size.
This algorithm can improve sampling accuracy and representativeness in multi-feature large-scale data, while reducing computational complexity and meeting real-time requirements.
Smart Images
Figure CN119441821A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data sampling based on machine learning, and in particular to a multi-dimensional feature channel grouping sampling algorithm based on a priori. Background Art
[0002] With the rapid development of science and technology, the scale of data is growing at a geometric rate. However, it is often impractical and sometimes even impossible to process the entire data set. Therefore, sampling becomes an effective way to process data. For example, in inferential statistics, we often use sampled data to infer the estimated sample of the population. The advantage of sampling is that it can reduce the load on the computer and improve computing efficiency, and the quality of sampling directly affects the upper limit of the model accuracy. An effective sampling method needs to take into account the multi-features and scale of the data to ensure the representativeness of the sample. Unreasonable or skewed sampling may cause the model to have large errors in the overall. Therefore, it is crucial to design an efficient and representative sampling method for multi-feature large-scale data.
[0003] At present, traditional sampling algorithms are generally divided into probability sampling and non-probability sampling. Probability sampling includes random sampling, systematic sampling, stratified sampling, cluster sampling and multi-level sampling. Random sampling is the simplest type of sampling. The probability of each sample in the population being selected is equal. No factors are considered and it depends entirely on probability. Systematic sampling is sampling from the population at fixed and periodic intervals. Random sampling and systematic sampling are not suitable for data sets with multiple features. Both types of sampling cannot guarantee sampling accuracy and representativeness of samples. Stratified sampling is to divide the population into non-overlapping layers, and then independently extract a certain number of individuals from each layer according to a certain proportion, and combine the individuals taken from each layer as samples. Stratified sampling is stratified according to a certain feature and is suitable for sampling of single-feature data sets. Cluster sampling is to divide the population into different clusters using clustering algorithms, independently extract a certain number of individuals from each cluster, and combine the individuals taken from each cluster as samples. However, when processing large-scale data with multiple features, using clustering algorithms will take a lot of time and cannot guarantee sampling accuracy. Multistage sampling refers to dividing the sampling process into stages, and the sampling methods used in each stage are often different, that is, various sampling methods are combined and used in large-scale epidemiological surveys. Multistage sampling is suitable for data sets with easily distinguishable levels, and there is also uncertainty in sampling accuracy on multi-feature large-scale data sets. Non-probability sampling includes simple sampling, voluntary sampling, and snowball sampling. These methods are often conditional choices and cannot be implemented through statistical probability and programming. Therefore, non-probability sampling is not suitable for sampling large-scale data sets.
[0004] For the sampling of multi-feature large-scale data, we must ensure both sampling efficiency and accuracy, as well as sample representativeness. Traditional probability sampling algorithms cannot meet these requirements. Therefore, we designed a multi-dimensional feature channel grouping sampling algorithm based on priors. Through this multi-dimensional feature channel grouping sampling algorithm based on modified priors, we can more flexibly adapt to the characteristics of multi-feature large-scale data, thereby better responding to sampling needs in real application scenarios. Summary of the invention
[0005] The purpose of the present invention is to provide a multi-dimensional feature channel grouping sampling algorithm based on a priori to solve the problem of unbalanced sampling of multi-feature large-scale data, while ensuring the sampling accuracy and high efficiency of the calculation.
[0006] The technical solution of the present invention is as follows:
[0007] A multi-dimensional feature channel grouping sampling algorithm based on priori, comprising the following steps:
[0008] S1. Select sampling features and establish the original sample feature channel set based on the sampling features And the sample feature channel set
[0009] S2. Calculate the original feature channel set Priority of feature channels in and the minimum number of samples Multiply the prior and the minimum number of samples to get the initial number of samples for each feature attribute
[0010] S3, randomly select an initial feature channel for initial sampling, and select the feature channel according to Each sampling number is randomly selected without replacement from each feature attribute to obtain the sampled sample feature channel set Calculate the deviation ε of each feature channel j , find the maximum deviation The corresponding feature channel, and calculate the number of feature attributes in the feature channel and a priori
[0011] S4, at maximum deviation In the corresponding feature channel, the modified prior is calculated according to the formula and corrected sampling number
[0012] S5, at maximum deviation In the corresponding feature channel, Each sampling number is randomly selected without replacement to obtain a new sample channel set of sample features. Calculate the deviation ε of each feature channel j , find the maximum deviation The corresponding feature channel, and calculate the number of feature attributes in the feature channel and a priori
[0013] S6, looping steps S4 and S5, when the sampling sample feature channel set When the average deviation in or the sampling sample size reaches the expected value, sampling is stopped.
[0014] Furthermore, the S1 specifically includes the following steps:
[0015] Select data sampling features and establish the original sample feature channel set based on the sampling features The original feature channel set contains n feature channels, one feature channel corresponds to one feature, each feature channel contains m groups, one group corresponds to one feature attribute, and a sample feature channel set is established based on the sampling features. Original sample feature channel set And the sample feature channel set The structure of the feature channel set is consistent;
[0016] Where j = 0, 1, ..., n-1, n represents the number of features, i = 0, 1, ..., m-1, m represents the number of feature attributes;
[0017] Furthermore, the prior P in the feature channel set in S2 i j It is the proportion of the feature attribute in the feature channel set, that is, the number of samples with this feature attribute divided by the number of all samples in the feature channel.
[0018] Furthermore, the minimum number of samples in S2 is It is the minimum number of samples that ensures that the priori of the feature channel of the sampled sample is closest to the priori of the feature channel of the original sample; the initial sampling number of the feature attribute The specific calculation is:
[0019] The minimum sampling sample size M is estimated within the allowable error range, and the expression is as follows:
[0020]
[0021] Where N represents the total number of samples, z represents the z score, and ε represents the error margin. represents the sample ratio;
[0022] when When the value of is less than M,
[0023] when When the value is equal to M, the minimum sampling number of the feature channel Just take M;
[0024] when When the value of is greater than M, traverse each probability in the feature channel and find the probability that just satisfies The probability P j , take the probability P j The reciprocal of is the minimum sampling number of the feature channel
[0025] Minimum number of samples per feature Multiply by the prior P i j Get the minimum number of samples corresponding to each feature attribute in each initial feature channel Minimum number of samples The expression is as follows:
[0026]
[0027] Furthermore, the deviation ε in S3 j is the weighted Manhattan distance deviation, which is calculated by the difference between the prior of the original feature channel set and the prior of the sampled feature channel set. The prior P i j As the deviation weight of the corresponding feature attribute, the influence of each feature attribute on the deviation is distinguished, and its derived expression is:
[0028]
[0029] Furthermore, the modified prior RP in S4 i j The priori of the original sample feature channel set is taken as the target, and the modified prior is calculated to compensate the priori of the sample group, so that the priori of the sample set SP i j Closer to the prior P of the original sample set i j .
[0030] Furthermore, the modified sampling number in S4 is By modifying the prior RP i j Multiply by the minimum number of samples of each feature attribute in the feature channel The calculation expression is as follows:
[0031]
[0032]
[0033]
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] A multi-dimensional feature channel grouping sampling algorithm based on a priori, using a sampling technique based on a priori multi-dimensional feature channel grouping, is suitable for sampling large-scale data with multi-dimensional features; secondly, the present invention can comprehensively consider the characteristic attribute distribution of each feature channel, while ensuring the sampling quantity, it can meet the high-precision requirements, and make the sampled samples more representative; finally, for the sampling of large-scale data with multi-dimensional features, the present invention can ensure the sampling accuracy and sampling quantity while having higher calculation efficiency and meeting the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a flow chart of the present invention.
[0037] Figure 2 Visualization of experimental results of the present invention Figure 1 .
[0038] Figure 3 Visualization of experimental results of the present invention Figure 2 .
[0039] Figure 4 Visualization of experimental results of the present invention Figure 3 .
[0040] Figure 5 Visualize the randomized experimental results of the present invention Figure 1 .
[0041] Figure 6 Visualize the randomized experimental results of the present invention Figure 2 .
[0042] Figure 7 Visualize the randomized experimental results of the present invention Figure 3 . DETAILED DESCRIPTION
[0043] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0044] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.
[0045] See also Figure 1-7 , a multi-dimensional feature channel grouping sampling algorithm based on priors, such as Figure 1 As shown, the following steps are included:
[0046] S1. Select data sampling features and establish the original sample feature channel set based on the sampling features And the sample feature channel set
[0047] S2. Calculate each original feature channel set The prior P i j and the minimum number of samples Multiply the prior and the minimum number of samples to get the initial number of samples for each feature attribute The specific calculation is:
[0048] The minimum sampling sample size M is estimated within the allowable error range, and the expression is as follows:
[0049]
[0050] Where N represents the total number of samples, z represents the z score, and ε represents the error margin. represents the sample ratio;
[0051] when When the value of is less than M,
[0052] when When the value of is equal to M, the minimum sampling number of the feature channel is M.
[0053] when When the value of is greater than M, traverse each probability in the feature channel and find the probability that just satisfies The probability P j , take the probability P j The reciprocal of is the minimum sampling number of the feature channel;
[0054] Minimum number of samples per feature Multiply by the prior P i j Get the minimum number of samples corresponding to each feature attribute in each initial feature channel Minimum number of samples The expression is as follows:
[0055]
[0056] S3, select an initial feature channel for initial sampling, and select the feature channel according to Each sampling number is randomly selected without replacement from each feature attribute to obtain the sampled sample feature channel set Calculate the deviation ε of each feature channel j , find the maximum deviation The corresponding feature channel, and calculate the number of feature attributes in the feature channel and a priori SP i j The weighted Manhattan distance deviation formula is used to calculate the deviation ε of each feature channel. j , the expression is as follows:
[0057]
[0058] S4, at maximum deviation In the corresponding feature channel, the modified prior RP of the feature channel is calculated according to the formula i j and the number of samples The calculation expression is as follows:
[0059]
[0060]
[0061]
[0062] S5, at maximum deviation In the corresponding feature channel, Each sampling number is randomly selected without replacement to obtain a new sample channel set of sample features. Calculate the deviation ε of each feature channel j , find the maximum deviation The corresponding feature channel, and calculate the number of feature attributes in the feature channel and a priori
[0063] S6, looping steps S4 and S5, when the sampling sample feature channel set When the maximum deviation or the sample size reaches the expected value, sampling is stopped.
[0064] The core of the invention is to provide a multi-dimensional feature channel grouping sampling algorithm based on a priori. First, it can be well applied to the sampling of multi-feature large-scale data sets. Second, it can meet the sampling efficiency and sampling accuracy, so that the sampled samples have better representativeness.
[0065] Experimental verification
[0066] This experimental data set comes from the customer data of a tobacco company in China, with a total of 27,119 customer samples. For this experiment, 11 important features were selected from the samples, namely: customer level, settlement method, market type, business type, marketing department, delivery route, administrative division, economic nature, business district type, special group, whether the goods are not needed for a long time, and the original data set was cleaned, and the samples with missing 11 feature dimensions and order quantity were deleted, leaving 26,885 samples. In the cleaned data, the 11 features and order quantity were analyzed for correlation, which is the correlation analysis of categorical variables and continuous variables. The features with strong and medium correlation were selected as the feature channels for sampling, namely: customer level, market type, delivery route, business type and administrative division.
[0067] Experimental protocol
[0068] In order to verify the superiority of the sampling method of the present invention, we used the most common random sampling method to compare the experimental results.
[0069] In the experiment of the present invention, the confidence levels are set to 99%, 95% and 90%, respectively. As shown in Table 1, the z scores are 2.58, 1.96 and 1.64, respectively. The corresponding sample numbers are 1731, 1027 and 736, respectively. The error range ε is 3%, and the sample ratio is The default value is 50%. The five features are traversed and used as the initial feature channels for sampling to increase randomness. A group of samples with the smallest average deviation is selected as the sampling result. The sampling termination condition is set when the number of samples reaches 1731, 1027 and 736. The experimental results are shown in Table 1.
[0070] Table 1: z-score table
[0071] Confidence Level Z-Score(±) 0.70 1.04 0.75 1.15 0.80 1.28 0.85 1.44 0.90 1.64 0.92 1.75 0.95 1.96 0.96 2.05 0.98 2.33 0.99 2.58 0.999 3.29 0.9999 3.89 0.99999 4.42
[0072] Table 2: Experimental results of multi-dimensional feature channel grouping sampling based on prior
[0073]
[0074] In the random sampling experiment, the confidence levels are set to 99%, 95%, and 90%, the error range ε is 3%, and the sample ratio is The default value is 50%, and the corresponding sampling numbers are 1731, 1027, and 736, respectively. Random sampling is performed 5 times in a cycle to increase randomness. The group of samples with the smallest average deviation is selected as the sampling result. The experimental results are shown in Table II.
[0075] Table 3: Random sampling experiment results
[0076]
[0077] The results of the two groups of experiments are shown in Tables 2 and 3. It can be seen that when the number of samples is similar, the accuracy of random sampling is far less than the accuracy of the sampling of the present invention. The experimental results of the two groups of experiments with a sample number of 736 are visualized as follows: Figure 2-Figure 4 , Figure 5-Figure 7 As shown, it can be obtained from the experimental results that the present invention is well applicable to the sampling of multi-feature large-scale data, can ensure a certain sampling quantity while ensuring the sampling accuracy, and can control the sampling accuracy and sampling quantity required in practice.
[0078] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.
Claims
1. A multi-dimensional feature channel grouping sampling algorithm based on priori, characterized in that: The following steps are involved: S1. Select sampling features and establish the original sample feature channel set based on the sampling features And the sample feature channel set S2. Calculate the original feature channel set Priority of feature channels in and the minimum number of samples Multiply the prior and the minimum number of samples to get the initial number of samples for each feature attribute S3, randomly select an initial feature channel for initial sampling, and select the feature channel according to Each sampling number is randomly selected without replacement from each feature attribute to obtain the sampled sample feature channel set Calculate the deviation ε of each feature channel j , find the maximum deviation The corresponding feature channel, and calculate the number of feature attributes in the feature channel and a priori S4, at maximum deviation In the corresponding feature channel, the modified prior is calculated according to the formula and corrected sampling number S5, at maximum deviation In the corresponding feature channel, Each sampling number is randomly selected without replacement to obtain a new sample channel set of sample features. Calculate the deviation ε of each feature channel j , find the maximum deviation The corresponding feature channel, and calculate the number of feature attributes in the feature channel and a priori S6, looping steps S4 and S5, when the sampling sample feature channel set When the average deviation in or the sampling sample size reaches the expected value, sampling is stopped.
2. The a priori-based multi-dimensional feature channel grouping sampling algorithm according to claim 1 is characterized in that: The S1 specifically includes the following steps: Select data sampling features and establish the original sample feature channel set based on the sampling features The original feature channel set contains n feature channels, one feature channel corresponds to one feature, each feature channel contains m groups, one group corresponds to one feature attribute, and a sample feature channel set is established based on the sampling features. Original sample feature channel set And the sample feature channel set The structure of the feature channel set is consistent; Among them, j = 0, 1, ..., n-1, n represents the number of features, i = 0, 1, ..., m-1, m represents the number of feature attributes.
3. The a priori-based multi-dimensional feature channel grouping sampling algorithm according to claim 2 is characterized in that: The prior on the feature channel concentration in S2 It is the proportion of the feature attribute in the feature channel set, that is, the number of samples with this feature attribute divided by the number of all samples in the feature channel.
4. The a priori-based multi-dimensional feature channel grouping sampling algorithm according to claim 3 is characterized in that: The minimum number of samples in S2 It is the minimum number of samples that ensures that the priori of the feature channel of the sampled sample is closest to the priori of the feature channel of the original sample; the initial sampling number of the feature attribute The specific calculation is: The minimum sampling sample size M is estimated within the allowable error range, and the expression is as follows: Where N represents the total number of samples, z represents the z score, and ε represents the error margin. represents the sample ratio; when When the value of is less than M, when When the value is equal to M, the minimum sampling number of the feature channel Just take M; when When the value of is greater than M, traverse each probability in the feature channel and find the probability that just satisfies The probability P j , take the probability P j The reciprocal of is the minimum sampling number of the feature channel Minimum number of samples per feature Multiply by the prior Get the minimum number of samples corresponding to each feature attribute in each initial feature channel Minimum number of samples The expression is as follows:
5. The multi-dimensional feature channel grouping sampling algorithm based on a priori according to claim 4 is characterized in that: The deviation ε in S3 j is the weighted Manhattan distance deviation, which is calculated by the difference between the prior of the original feature channel set and the prior of the sampled feature channel set. A priori As the deviation weight of the corresponding feature attribute, the influence of each feature attribute on the deviation is distinguished, and its derived expression is:
6. The a priori-based multi-dimensional feature channel grouping sampling algorithm according to claim 5 is characterized in that: The modified priors in S4 The method is to take the prior of the original sample feature channel set as the target, calculate the modified prior to compensate the prior of the sample group, so that the prior of the sample set is Closer to the prior of the original sample set 7. The a priori-based multi-dimensional feature channel grouping sampling algorithm according to claim 6 is characterized in that: The corrected sampling number in S4 By modifying the prior Multiply by the minimum number of samples of each feature attribute in the feature channel The calculation expression is as follows: