Data processing method and device, electronic equipment and computer storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and computer storage medium. The data processing method comprises: determining at least two sets of data to be analyzed and corresponding set logical operation operations according to a received data analysis request; obtaining at least two sampling sets corresponding to the sets of data to be analyzed, the sampling sets being obtained by uniformly stratified sampling and KMV sampling of mapping values of the data to be analyzed in the sets of data to be analyzed; determining a target mapping value that meets a set condition from the maximum mapping values of the sampling sets; performing the set logical operation operations on the sampling sets, and determining an analysis result according to the set logical operation result, the target mapping value and a sampling ratio of the uniformly stratified sampling. The method is fast in analysis and small in computing power burden.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data computing, and in particular, to a data processing method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] In a data insight use scenario, more attention is usually paid to data trends, such as insight analysis of target objects and media, and the desired analysis result can be obtained through the trends. Meanwhile, considering that accurate calculation takes too long to query and causes too much data storage overhead when processing a large data set, the scheme of accurate calculation cannot meet people's demand for fast query results. Based on this situation, AQP (Approximate Query Process) technology is generated.

[0003] AQP is a technology for estimating the cardinality of an overall set based on different probability algorithms. Unlike accurate calculation, which calculates each element in the set accurately, approximate calculation provides a possibility of quickly querying the estimated cardinality of the set within a controllable error range.

[0004] However, the existing approximate calculation has the problem that if the amount of sampled data is small, the precision is poor and it is difficult to accurately estimate, and if the amount of data is large, it will cause excessive overhead of calculation, storage and the like and affect the analysis efficiency. SUMMARY

[0005] Therefore, embodiments of the present application provide a data processing scheme to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a data processing method is provided, including determining at least two sets of data to be analyzed and corresponding set logical operation operations according to a received data analysis request; obtaining a sampling set corresponding to each of the at least two sets of data to be analyzed, the sampling set being obtained by uniformly stratified sampling and KMV sampling of a mapping value of data to be analyzed in the set of data to be analyzed; determining a target mapping value that meets a set condition from a maximum mapping value of each of the sampling sets; performing the set logical operation operation on the sampling sets, and determining an analysis result according to the set logical operation result, the target mapping value and a sampling ratio of the uniformly stratified sampling.

[0007] According to a second aspect of the embodiments of the present application, a data processing apparatus is provided, comprising: a first determining module configured to determine at least two data sets to be analyzed and corresponding set logical operation operations according to a received data analysis request; an obtaining module configured to obtain sampling sets corresponding to the at least two data sets to be analyzed, the sampling sets being obtained by performing uniform stratified sampling and KMV sampling on mapping values of data to be analyzed in the data sets to be analyzed; a second determining module configured to determine target mapping values meeting a set condition from maximum mapping values of the sampling sets; and a third determining module configured to perform the set logical operation operations on the sampling sets, and determine an analysis result according to a result of the set logical operation, the target mapping values and a sampling ratio of the uniform stratified sampling.

[0008] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, the executable instruction causes the processor to perform operations corresponding to the data processing method of the first aspect.

[0009] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program, the program is executed by a processor to implement the data processing method of the first aspect.

[0010] According to the data processing scheme provided by the embodiments of the present application, the data sets to be analyzed and the corresponding set logical operation operations are determined according to the data analysis request, and the sampling sets corresponding to the data sets to be analyzed are obtained by twice sampling of uniform stratified sampling and KMV sampling. Since the uniform stratified sampling can make the mapping values of the data to be analyzed uniformly distributed in a set interval, the data sparsity problem can be solved, so that the data to be analyzed in the sampling sets obtained by the subsequent KMV sampling meet the uniform distribution. The data amount in the sampling sets can be limited by the KMV sampling, so that the uniform stratified sampling and the KMV sampling not only reduce the data amount and the computing load, improve the operation speed, but also solve the data sparsity problem, avoid errors caused by data sparsity, so as to realize the effect of controlling the error within the allowable range. In this way, the problem that the error is easily amplified due to factors such as data sparsity and sampling amount in the existing sampling and calculation method is effectively solved. The target mapping values are determined based on the sampling sets, and the set logical operation operations are performed on the sampling sets. The analysis result is determined according to the result of the set logical operation, the target mapping values and the sampling ratio. The data processing scheme makes the analysis response speed better and the error smaller. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings.

[0012] Figure 1 A step flow chart of a data processing method according to the first embodiment of the present application;

[0013] Figure 2 A step flow chart of a data processing method according to the second embodiment of the present application;

[0014] Figure 3A A step flow chart of a data processing method according to the third embodiment of the present application;

[0015] Figure 3B A structural block diagram of an ACS system in a use scenario according to the third embodiment of the present application;

[0016] Figure 3C A flow diagram of a data processing method in a use scenario according to the third embodiment of the present application;

[0017] Figure 3D A flow diagram of sampling of a sampling unit in a use scenario according to the third embodiment of the present application;

[0018] Figure 3E A flow diagram of storage and logical calculation in a use scenario according to the third embodiment of the present application;

[0019] Figure 4 A structural block diagram of a data processing device according to the fourth embodiment of the present application;

[0020] Figure 5 A structural diagram of an electronic device according to the fifth embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the technical solutions in the embodiments of the present application better understood, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art should belong to the scope of protection of the embodiments of the present application.

[0022] The specific implementation of the embodiments of the present application will be further described below with reference to the drawings of the embodiments of the present application.

[0023] Embodiment One

[0024] Referring to Figure 1 FIG. 1 shows a schematic diagram of a step flow of a data processing method according to an embodiment of the present application.

[0025] In this embodiment, the method can be applied to approximate calculation on independent target object data, so as to determine the distribution of the independent target object data in different intervals. Of course, in other embodiments, the method can also be applied to other data analysis scenarios requiring approximate calculation, without limitation.

[0026] The method comprises the following steps:

[0027] Step S102: According to the received data analysis request, at least two sets of to-be-analyzed data and corresponding set logical operation operations are determined.

[0028] Different application scenarios of data analysis requests can be different. For example, the data analysis request indicates the distribution of the target object in different age interval, or the data analysis request indicates the distribution of the target object in different geographical interval, etc.

[0029] Different data analysis requests involve different sets of to-be-analyzed data and set logical operation operations. For example, the data analysis request indicates the distribution of the target object in different age interval, and the sets of to-be-analyzed data involved include but are not limited to the target object information set, the target object age information set, etc. Among them, the target object age information set can be multiple, and each age interval has a corresponding target object age information set. For example, the target object age information set C1 contains to-be-analyzed data of age interval 1-17 years old (which includes user information and age information), the target object age information set C2 contains to-be-analyzed data of age interval 18-25 years old, etc.

[0030] The set logical operation operation includes but is not limited to the set intersection, and can also be the set union, complement set and difference set, etc.

[0031] Step S104: Obtain the sampling set corresponding to at least two sets of to-be-analyzed data.

[0032] In order to reduce the calculation amount, improve the estimation speed and the response time of the data analysis request, the cardinality of the target object distributed in different age interval is estimated by sampling, so as to obtain the distribution of the target object in different age interval.

[0033] In one case, the sampling set can be pre-sampled from the data set to be analyzed, and the obtained sampling set is stored in the database or pre-loaded into the cache for direct use in analysis, thereby reducing processing time and improving response speed.

[0034] Alternatively, in another case, the data set to be analyzed is not pre-sampled, i.e., there is no sampling set in the database and the cache, and the data set to be analyzed can be sampled in real time to obtain a sampling set.

[0035] In the estimation of the cardinality based on the sampling set, the uniformity, sparsity, and sampling quantity of the data to be analyzed in the data set to be analyzed all affect the error of the estimated cardinality. Therefore, in order to ensure the accuracy of the estimation, avoid the sparsity of the data to be analyzed causing too large an error, and achieve the purpose of reducing the sampling quantity as much as possible while ensuring a small error to reduce storage and calculation pressure, the sampling set is obtained by uniformly stratified sampling and KMV (k minimum values) sampling of the mapping values of the data to be analyzed in the data set to be analyzed. The mapping value is a mapping value that can convert the sparsely distributed data to be analyzed into a uniformly distributed mapping value to ensure the accuracy of subsequent analysis.

[0036] The uniform stratified sampling can be understood as uniformly processing the mapping values of the data to be analyzed in the data set to be analyzed to satisfy the uniform distribution, so that the data to be analyzed is uniformly distributed in the interval corresponding to the set (it should be noted that the uniform distribution should be understood as uniformly distributed within the allowable error range), and then the data to be analyzed is sampled at equal intervals, which can avoid the problem of increased error caused by data sparsity.

[0037] For the stratified sampling set obtained by uniform stratified sampling, KMV sampling can be performed to obtain a sampling set. KMV sampling can further reduce the number of mapping values of the data to be analyzed contained in the sampling set based on uniform stratified sampling, thereby reducing the calculation amount and improving the response speed, and ensuring that the error of the estimated cardinality is within the allowable range.

[0038] Based on the obtained sampling set, the cardinality can be estimated by using the principle of the KMV algorithm in subsequent steps.

[0039] Step S106: determining a target mapping value that satisfies a set condition from the maximum mapping values of each sampling set.

[0040] The sampling set corresponds to the data set to be analyzed involved in the data analysis request.

[0041] Taking the data analysis request indicating the distribution of the target objects in different age interval as an example, if the age interval is divided into 7 segments, the data set to be analyzed involved in the data analysis request respectively includes a target object information set (for example, set B), and 7 target object age information sets (for example, sets C1-C7) corresponding to the 7 age intervals. Each of the 8 sets has a corresponding sampling set, which is denoted as sampling sets 1-8.

[0042] The maximum mapping value is selected from each of the sampling sets 1-8.

[0043] In this embodiment, the condition is set to be the minimum value, i.e., the target mapping value satisfying the condition is the minimum value selected from all the maximum mapping values. Subsequently, the cardinality can be estimated according to the target mapping value.

[0044] Step S108: performing the set logical operation on the sampling sets, and determining an analysis result according to the set logical operation result, the target mapping value, and the sampling ratio of the uniform stratified sampling.

[0045] Different data analysis requests indicate different set logical operations. For example, when analyzing the distribution of the target objects in different age intervals, the set intersection operation can be performed between the target object information set (i.e., set B) and the target object age information sets (i.e., sets C1-C7), thereby obtaining 7 set logical operation results, each of which indicates the cardinality of the target objects in the age interval.

[0046] Further, for each target object age information set, the cardinality of the target objects in the age interval is estimated according to the corresponding set logical operation result, the target mapping value, and the sampling ratio, and the analysis result is determined according to the estimated 7 cardinalities.

[0047] The method adopts the core idea of the KMV algorithm to perform two-stage sampling, i.e., stratified sampling and KMV sampling, on the data set to be analyzed, thereby overcoming the problem that the error of direct sampling is greatly affected by the sampling ratio, the sampling method, and the sparsity of the data set to be analyzed, and solving the problem that the error is excessively magnified and the estimated cardinality is inaccurate due to the insufficient sampling number caused by the limitation of the cache space.

[0048] The implementation process of the method is described below by taking the offline big data analysis in the independent visitor scenario as an example.

[0049] In the present use scenario, the data analysis request indicates to analyze the distribution of the population in a certain region in different age interval. According to the analysis requirement, the age interval is divided into 7 different intervals, and each age interval has a corresponding population age information set (denoted as sets C1~C7). The data to be analyzed in the population age information set includes the population identifier and its age.

[0050] Based on the data analysis request, it is determined that the data set to be analyzed involved in the request includes the population information set of the region (denoted as set B) and the population age information set corresponding to the 7 age intervals (denoted as sets C1~C7), and the corresponding set logical operation is set intersection. It should be noted that according to different data analysis requests, the data set to be analyzed and the set logical operation may be different, and the present scenario is only an example.

[0051] In the first case, if it is determined that the required sample set exists in the cache or database, the set logical operation can be performed on the sample set, and the target mapping value can be determined according to the sample set, so as to determine the analysis result according to the set logical operation result, the target mapping value and the sampling ratio of the sample set. In this way, the analysis result can be quickly obtained, thereby reducing the response waiting time.

[0052] In the second case, if the sample set corresponding to the data set to be analyzed does not exist in the cache or database, the sample set can be obtained in the following manner.

[0053] The processing manner of set B and sets C1~C7 can be similar or the same, so the process of obtaining the sample set is briefly described below with set B as an example:

[0054] The data to be analyzed in set B is uniformly processed to obtain a mapping value that meets the uniform distribution, so as to be evenly distributed in the set interval. Then, the mapping value obtained by uniform processing is stratified sampled according to the set sampling ratio (such as 16), so as to solve the sparsity problem and ensure that the data in the stratified sample set still reflects the real data situation and meets the uniform distribution. The stratified sample set is subjected to KMV sampling (minimum k value sampling), that is, the minimum k data is taken from the stratified sample set to form the sample set corresponding to set B. In this way, the data amount is reduced by sampling, but the real data situation in the data set to be analyzed is retained, so that the error between the estimated analysis result and the real value is within the allowable range.

[0055] The sampling sets of the sets C1-C7 can be obtained in a similar manner. According to the obtained sampling sets, the maximum mapping value in each sampling set can be determined, and then the minimum value of the maximum mapping values of the eight sets is selected as the target mapping value. Of course, in other use scenarios, the target mapping value can be determined according to other conditions.

[0056] To analyze the distribution of the population in different age interval, the set B is subjected to set intersection logical operation with the sets C1-C7. Since the operation process of the set B with the sets C1-C7 is similar, the set B and the set C1 are taken as an example for illustration.

[0057] The data in the intersection of the set B and the set C1 is the population in a certain region whose age is in the age interval indicated by the set C1. The intersection of the set B and the set C1 is denoted as the set logical operation result R1. The intersections of the set B with the sets C2-C7 can be obtained in a similar manner and are denoted as the results R2-R7, respectively.

[0058] According to the set logical operation result R1, the target mapping value, and the sampling ratio, the cardinality A1 in the age interval corresponding to the set C1 can be estimated. Similarly, according to the set logical operation results R2-R7, the cardinalities A2-A7 in the remaining age intervals can be estimated, respectively. Then, the distribution in the seven age intervals is determined as the analysis result according to the cardinalities A1-A7.

[0059] According to the method of the embodiment, the data set to be analyzed and the corresponding set logical operation operation are determined according to the data analysis request, and the sampling set corresponding to the data set to be analyzed is obtained through uniform stratified sampling and KMV sampling twice. Since the uniform stratified sampling can make the mapping values of the data to be analyzed uniformly distributed in the set interval, the problem of data sparsity can be solved, so that the data to be analyzed in the sampling set obtained through the subsequent KMV sampling satisfies the uniform distribution. Through the KMV sampling, the amount of data in the sampling set can be limited. Therefore, through the uniform stratified sampling and the KMV sampling, the amount of data and the computing load are reduced, the operation speed is improved, the problem of data sparsity is solved, the error caused by the data sparsity is avoided, and the effect of controlling the error within the allowable range is achieved. In this way, the problem that the error is easily amplified due to factors such as data sparsity and sampling quantity in the existing sampling and calculation method is effectively solved. The target mapping value is determined based on the sampling set, the set logical operation operation is performed on the sampling set, and the analysis result is determined according to the set logical operation result, the target mapping value, and the sampling ratio. In this way, the analysis response speed is better and the error is smaller.

[0060] The data processing method can be executed by any appropriate electronic device with data processing capability, including but not limited to servers, mobile terminals (such as mobile phones, PADs, etc.), and PC machines, etc.

[0061] Embodiment Two

[0062] Reference Figure 2 Fig. 2 shows a schematic diagram of the step flow of the data processing method of this embodiment two.

[0063] In this embodiment, the method comprises the aforementioned steps S102-S108. In step S104, there are two cases when the sampling set is acquired. In the first case, if the database does not contain the sampling set corresponding to the at least two data sets to be analyzed, and the mapping value is a hash value, then step S104 can be implemented through sub-steps S1041-S1044.

[0064] Alternatively, in the second case, if the database contains the sampling set corresponding to the at least two data sets to be analyzed, then step S104 is implemented by reading the sampling set from the database and obtaining the sampling ratio used for uniform stratified sampling of the sampling set. The cardinality can be estimated according to the sampling set and the sampling ratio subsequently.

[0065] For the first case:

[0066] Sub-step S1041: Determine the sampling ratio and the sampling number of KMV sampling.

[0067] The sampling ratio can be determined as needed. In this embodiment, the sampling ratio is mainly affected by two aspects, which are the estimated cardinality of the set and the expected error range. The theoretical error rate can be used to derive the corresponding maximum cardinality of the set, and the theoretical maximum cardinality of the set / the estimated cardinality of the set is the final sampling ratio 1 / n.

[0068] For example, it is desired to control the error rate at about 2%, and the estimated cardinality of the set is about 3.2 billion. According to the error rate, the theoretical maximum cardinality of the set is about 200 million, so the sampling ratio 1 / n should be set to 2 / 32 = 1 / 16. The sampling ratio and the sampling coefficient are reciprocal of each other.

[0069] Under normal circumstances, if the data to be analyzed is strictly uniformly distributed in the data set to be analyzed, the value of the sampling number will not affect the accuracy of the analysis result. However, in actual use, the data to be analyzed will have a certain sparsity in the data set to be analyzed, and after uniform distribution processing, it cannot be guaranteed to be strictly uniform, so the value of the sampling number will still affect the accuracy of the analysis result. Therefore, the value of the sampling number can be determined according to the data storage overhead and the estimated accuracy.

[0070] The greater the value of the sampling number, the greater the data storage overhead, but the higher the estimation accuracy. Conversely, the smaller the value of the sampling number, the smaller the data storage overhead, but the lower the estimation accuracy. Therefore, the value of the sampling number is to be as large as possible within the allowable data storage overhead.

[0071] Sub-step S1042: A uniform distribution hash function is used to perform hash mapping on the to-be-analyzed data set, so that the hash values of the to-be-analyzed data in the to-be-analyzed data set are uniformly distributed within a set interval.

[0072] It should be noted that using a uniform distribution hash function to map the to-be-analyzed data in the to-be-analyzed data set is only an exemplary uniform distribution processing manner, which is not limited to this manner.

[0073] The uniform distribution hash function may be, for example, MurmurHash3. The hash value of the to-be-analyzed data calculated by the uniform distribution hash function is used as a mapping value, so that the to-be-analyzed data is mapped into the set interval, and the hash value corresponding to the to-be-analyzed data is uniformly distributed within the set interval.

[0074] In other embodiments, the mapping value can be other values, as long as the to-be-analyzed data can be uniformly distributed within the set interval. It should be noted that the uniform distribution in this embodiment is not limited to strict uniform distribution, as long as the overall error rate is controllable.

[0075] In this embodiment, the number type of the hash value is Long value, and the bit number is 64 bits. The set interval can be [0-Long.MAX], and Long.MAX= .

[0076] Since the uniformity of the to-be-analyzed data has a greater impact on the error during subsequent analysis based on the sampling set, the to-be-analyzed data can be uniformly distributed and mapped by the uniform distribution hash function, and then uniform distribution verification can be performed according to the calculated hash value.

[0077] A feasible uniform distribution verification manner can be to calculate the ratio of the hash value (denoted as long) and Long.MAX, which is represented as: long / Long.MAX. In this way, the hash value can be mapped into the interval [0-1], and whether it is uniformly distributed can be verified.

[0078] Sub-step S1043: According to the sampling ratio, the hash values of the uniformly distributed to-be-analyzed data are stratified sampled to obtain a stratified sampling set.

[0079] The sampling ratio indicates that one is extracted from n data to be analyzed. In one possible implementation, equidistant extraction can be achieved by taking the modulus of the hash value with the sampling coefficient corresponding to the sampling ratio. If the sampling ratio is 1 / n, the sampling coefficient is n, and if the result of the modulus of the hash value with respect to n is 1, the hash value is extracted, otherwise it is discarded.

[0080] The hierarchical sampling set is formed according to the sampled hash values.

[0081] Sub-step S1044: KMV sampling is performed on the hash values in the hierarchical sampling set according to the sampling number of the KMV sampling, to obtain a sampling set corresponding to the data set to be analyzed.

[0082] In one possible implementation, sub-step S1044 can be implemented based on the idea of the KMV algorithm as follows:

[0083] Process A1: The hash values in the hierarchical sampling set are sorted in ascending order.

[0084] Since the idea of the KMV algorithm is to estimate the cardinality according to the smallest k values in the sampling set, when performing KMV sampling, the hash values in the hierarchical sampling set can be sorted in ascending order, i.e., the hash values are sorted from small to large.

[0085] Process B1 selects the first k hash values from the hierarchical sampling set sorted in ascending order to form the sampling set corresponding to the data set to be analyzed, where k is the sampling number.

[0086] According to the sampling number, the first k (i.e., the smallest k) hash values are selected from the hierarchical sampling set sorted in ascending order to form the sampling set.

[0087] Optionally, to reduce the storage space, the method further includes steps S105a and S105b.

[0088] Step S105a: The hash values in the sampling set are adjusted to integers, and the data length of the hash values is compressed to a length satisfying the data compression condition.

[0089] It should be noted that the execution order between step S105a and sub-step S1044 is not strictly limited, and step S105a can be executed before or after sub-step S1044.

[0090] If step S105a is executed before sub-step S1044, the hash values in the hierarchical sampling set are converted from long type to int type (integer type), and the data length is compressed to an appropriate number of bits. For example, the 64-bit long type hash value is converted to a 32-bit int type hash value. Then the converted hash value is subjected to KMV sampling.

[0091] Alternatively, if executed after sub-step S1044, the hash values in the sampling set obtained after KMV sampling are converted into 32-bit int types.

[0092] In this way, the data amount can be compressed, the storage space occupied by the sampling set can be reduced, the number of mapping values in the sampling set can be increased, and thus the error can be reduced as much as possible.

[0093] Step S105b: data compression is performed on the hash values in the adjusted sampling set, and the data-compressed sampling set is saved to the database.

[0094] The data compression can be performed in an appropriate manner, for example, using a structure storage with a data compression function.

[0095] In this embodiment, step S105b includes the following sub-steps:

[0096] Sub-step S105b1: a data compression bitmap structure for compressing the sampling set is created, and the hash values in the sampling set are written into the data compression bitmap structure.

[0097] The data compression bitmap structure is, for example, a RoaringBitmap structure, and of course, can also be other structures capable of realizing data compression. Each sampling set corresponds to one data compression bitmap structure.

[0098] Since the RoaringBitmap structure can store data with a data length of 32 bits, the hash values in the sampling set are converted into 32-bit integers, so that they can be successfully stored in the RoaringBitmap structure and efficiently compressed.

[0099] Of course, in other embodiments, if other structures are used for data compression, the hash values can be converted into corresponding types and data lengths.

[0100] Because the data compression bitmap structure can compress and store the hash values in the sampling set, the storage space occupied can be reduced, and thus more hash values can be stored in a certain storage space, which can increase the number of samples and thus reduce the error and improve the accuracy.

[0101] Sub-step S105b2: according to a set number of segments, the data compression bitmap structure storing the hash values is split in the length direction of the hash values to obtain m bitmap segments corresponding to the hash values, where m is the value of the number of segments.

[0102] In order to improve parallel computing capability, the data compression bitmap structure can be split horizontally (i.e. in the length direction of the hash value) to obtain m bitmap segments, m being a positive integer and being the number of segments. In this way, different bitmap segments can be calculated in parallel in subsequent calculations, thereby improving the calculation efficiency and reducing the response time.

[0103] For example, the data length of the hash value stored in the RoaringBitmap structure is 32 bits, and m is 4. Then, one hash value can be divided into 4 segments, each of which is 8 bits, to form a bitmap segment. Specifically, the 0-7 bits of the hash value form the first bitmap segment, the 8-15 bits form the second bitmap segment, the 16-23 bits form the third bitmap segment, and the 24-31 bits form the fourth bitmap segment.

[0104] Sub-step S105b3: store the m bitmap segments into the database

[0105] By storing the bitmap segments into the database, not only the storage of the sample set can be realized, but also the corresponding index of the sample set can be generated through the index generation function of the database, thereby improving the efficiency of subsequent calculations.

[0106] Through the method of the embodiment, the data set to be analyzed and the corresponding set logical operation operation are determined according to the data analysis request, and the sample set corresponding to the data set to be analyzed is obtained through uniform stratified sampling and KMV sampling twice. Since the uniform stratified sampling can make the mapping values of the data to be analyzed uniformly distributed in the set interval, the problem of data sparsity can be solved, so that the data to be analyzed in the sample set obtained through the subsequent KMV sampling satisfies the uniform distribution. Through the KMV sampling, the amount of data in the sample set can be limited. Therefore, through the uniform stratified sampling and the KMV sampling, not only the amount of data and the calculation load are reduced, and the operation speed is improved, but also the problem of data sparsity is solved, and the error caused by data sparsity is avoided, thereby realizing the effect of controlling the error within the allowable range. In this way, the problem that the error is easily amplified due to factors such as data sparsity and sample quantity in the existing sampling and calculation method is effectively solved. The target mapping value is determined based on the sample set, and the set logical operation operation is performed on the sample set. The analysis result is determined according to the set logical operation result, the target mapping value and the sampling ratio. This way makes the analysis response speed better and the error smaller.

[0107] The data processing method can be executed by any appropriate electronic device with data processing capability, including but not limited to servers, mobile terminals (such as mobile phones, PADs, etc.) and PC machines, etc.

[0108] Embodiment three

[0109] Reference Figure 3AFIG. 3 shows a schematic diagram of a step flow of the data processing method according to Embodiment Three of the present application.

[0110] In the present embodiment, the method comprises the aforementioned steps S102-S108, wherein step S104 can be implemented in any of the aforementioned embodiments. Step S106 comprises the following sub-steps:

[0111] Sub-step S1061: Determine the maximum hash value corresponding to each sample set.

[0112] As in the aforementioned example, set B and sets C1-C7 each have a corresponding sample set, and the maximum hash value of the sample set corresponding to each of these sets is determined.

[0113] Sub-step S1062: Select the minimum maximum hash value from the maximum hash values of at least two sample sets as the target mapping value.

[0114] Select the minimum maximum hash value from the maximum hash values of the aforementioned eight sample sets as the target mapping value.

[0115] On this basis, step S108 comprises the following sub-steps:

[0116] Sub-step S1081: For each set logical operation, determine the different sample sets involved in the current set logical operation, perform logical operation on the bitmap segments corresponding to the different sample sets according to the corresponding relationship between the bitmap segments, and accumulate the logical operation results corresponding to each bitmap segment to obtain the set logical operation result corresponding to the current set logical operation.

[0117] For example, in the aforementioned scenario of determining the distribution of the population in different age interval, the logical set operation includes the set intersection operation between set B and set C1, the set intersection operation between set B and set C2, and so on until the set intersection operation between set B and set C7. The processes of these seven set intersection operations are similar, with the difference being the data used for the operation. Therefore, taking the set intersection operation between set B and set C1 as an example, the sample sets involved in this set logical operation are sample set 1 corresponding to set B and sample set 2 corresponding to set C1.

[0118] Since the hash values of sample set 1 and sample set 2 are stored through the data compression bitmap structure, and the data compression bitmap structure comprises m bitmap segments, the corresponding bitmap segments of the two sample sets can be calculated respectively when performing the set logical operation, which can realize parallel calculation and thus improve the calculation efficiency.

[0119] For example, the first bitmap segment of the sample set 1 is intersected with the first bitmap segment of the sample set 2, the second bitmap segment of the sample set 1 is intersected with the second bitmap segment of the sample set 2, and so on until the mth bitmap segment of the sample set 1 is intersected with the mth bitmap segment of the sample set 2.

[0120] The set logical operation result of the set B and the set C1 is obtained by accumulating the logical operation results of the bitmap segments.

[0121] The sub-step S1082: determining the analysis result according to the set logical operation result corresponding to each set logical operation, the target mapping value and the sampling ratio of the uniform stratified sampling.

[0122] In one possible implementation, for each age interval, the number of the population in the age interval, i.e. the base, can be estimated according to the set logical operation result, the target mapping value and the sampling ratio of the uniform stratified sampling. Then, the distribution in each age interval is determined according to the estimated base of each age interval, as the analysis result.

[0123] For example, according to the set logical operation result of the set B and the set C1, the target mapping value and the sampling coefficient corresponding to the sampling ratio (the sampling ratio is 1 / n and the sampling coefficient is n), the base in the age interval corresponding to the set C1 is estimated. The specific calculation formula can be: wherein, is the set logical operation result, is the logical operation result of the first bitmap segment. is the logical operation result of the mth bitmap segment. is the target mapping value, is the aforementioned value, and n is the sampling coefficient, which is 16 in this embodiment.

[0124] It should be noted that, since the ratio of the hash value to Long.MAX is calculated in the foregoing process for verifying the uniformity of the hash value distribution, the calculation result stored in the foregoing process can be directly used for calculation here, so as to improve the efficiency.

[0125] According to the formula, the base of the population in the age interval corresponding to the set C1 can be estimated, and the bases of the age intervals corresponding to the remaining sets C2-C7 can be calculated in the same way, so as to determine the distribution according to the bases corresponding to the sets C1-C7.

[0126] The implementation process will be described below in combination with a specific scenario.

[0127] For example, Figure 3BAs shown, the method can be integrated in the ACS system to realize the provision of unified approximate calculation services. In the use scenario, the ACS system includes a sampling unit, a data storage unit, a logical calculation unit, and a system service unit. The sampling unit can be a two-stage sampling designed in combination with stratified sampling and KMV algorithm thinking. The data storage unit can use a 32-bit RoaringBitMap structure for underlying data storage, while realizing the horizontal expansion of the RoaringBitmap structure. The logical calculation unit can provide set calculation capabilities for bitmap segments of the RoaringBitMap structure. The system service unit can provide encapsulated database calls, memory management, and concurrent query functions, and provide unified approximate calculation services to the outside.

[0128] As shown in Figure 3C When providing services to the outside through the ACS system, the system service unit encapsulates and integrates each functional unit, while adding database calls, memory management, and concurrent query functions.

[0129] The method includes:

[0130] S1: receiving a data analysis request.

[0131] Upon receiving the data analysis request, it is determined whether the analysis result corresponding to the data analysis request exists in the cache. If it exists, S6 is executed.

[0132] Or if it does not exist, it is further determined whether the sampling set of the data set to be analyzed involved in the data analysis request exists in the memory. If it exists in the memory, the logical calculation unit can directly perform calculation, and S4 is executed.

[0133] Or, if it does not exist in the memory but exists in the database, S3 is executed.

[0134] Or, if it does not exist in the memory and the database, the sampling unit and the data storage unit are called to execute the process corresponding to S2 to obtain the sampling set and the KMV index. After obtaining the sampling set and the KMV index, S3 is executed.

[0135] S2: The system service unit receives the data to be analyzed of the data set to be analyzed, samples it through the sampling unit to obtain a sampling set, and stores the sampling set through the data storage unit to form a complete KMV index structure. The index structure is batch written into the MongoDB database, and a database index is added.

[0136] As shown in Figure 3D The sampling process of the sampling unit is as follows:

[0137] S211: Set the sampling ratio 1 / n of stratified sampling.

[0138] Since the subsequent uniform distribution mapping of the data to be analyzed by the hash function, the sampling ratio can be further derived error rate according to the hash collision data expected formula.

[0139] Wherein the error rate ∂ can be expressed as: . Wherein, u is the number of hash collisions, n is the sampling ratio, m is the bit number of bitmap (take 2^32). The theoretical error rate can be deduced to the corresponding maximum set cardinality, and the theoretical maximum set cardinality / set estimated cardinality is the final sampling ratio 1 / n.

[0140] S212: Set the number of KMV sampling.

[0141] For example, 4 million. It can be selected according to the storage overhead and accuracy.

[0142] S213: Set variable i=0.

[0143] S214: Determine whether the value of i is less than the total number of data to be analyzed in the data set to be analyzed.

[0144] If less, execute S215, otherwise, execute S218.

[0145] S215: Input the i-th data to be analyzed in the data set to be analyzed into the hash function to obtain the hash value.

[0146] The hash function is MurmurHash3, and the result after hash is a 64-bit Long value. The MurmurHash3 function can realize the uniform distribution of hash value.

[0147] The data set to be analyzed can be the aforementioned set B, set C1~C7. After calculating the hash value of all data in the data set to be analyzed, the Long value / Long.MAX after hash can be calculated to verify the uniformity of the Long value. Generally, the calculated Long value / Long.MAX is uniformly distributed in the [0-1] interval.

[0148] Long.MAX=2 64 -1=9223372036854775807.

[0149] S216: Determine whether the hash value modulo the sampling coefficient n is equal to 1.

[0150] If equal to 1, execute S217; otherwise, discard.

[0151] S217: Write the hash value modulo n equal to 1 into the stratified sampling set. Add 1 to the value of i, and return to S214 for continuous execution until the result of S214 is false.

[0152] Through such a cycle process, the to-be-analyzed data in the to-be-analyzed data set can be uniformly hashed, and the processed to-be-analyzed data corresponds to a hash value distributed in the [0-Long.MAX] interval. Moreover, by selecting a hash value that is 1 modulo n, the [0-Long.MAX] interval can be divided into n parts (n is a sampling coefficient, which is the reciprocal of the sampling ratio) in proportion and equidistance, and one hash value is selected from each part for stratified sampling effect.

[0153] Since the hash value after hashing is uniformly distributed, the stratified sampling set after stratified sampling also obeys uniform distribution.

[0154] S218: Obtain the stratified sampling set.

[0155] For the hash value in the stratified sampling set, it can be converted from a Long value (64 bits) to an Int value (32 bits). The converted Int value / Int.MAX is also in the range of [0-1], int.MAX=2 32 -1.

[0156] The purpose of such conversion is to write into the 32-bit RoaringBitmap structure more efficiently and conveniently.

[0157] S219: Arrange the hash values in the stratified sampling set in ascending order.

[0158] S220: Set variable i=0.

[0159] S221: Determine whether the value of i is less than the sampling number.

[0160] In this example, the sampling number in the sampling set is 4 million.

[0161] If yes, execute S222; otherwise, execute S223.

[0162] S222: Write the i-th hash value in the stratified sampling set into the sampling set. Increase the value of i by 1, and return to S221 for continuous execution until S221 determines no.

[0163] In this way, the KMV sampling method is used for sampling, only the smallest k hash values in the stratified sampling set are retained, and a sampling set is formed. The data storage process is as follows:

[0164] S224: Write the data in the sampling set into the RoaringBitmap structure, and each sampling set corresponds to one RoaringBitmap structure.

[0165] For example, the sampling sets of set B and sets C1~C7 all have corresponding RoaringBitmap structures.

[0166] S225: Horizontally expand the RoaringBitmap structure.

[0167] For example, a 32-bit RoaringBitmap can be divided into m segments according to the [0-31] bit range, and each segment can independently form a RoaringBitmap structure (denoted as bitmap segmentation), thereby horizontally expanding one RoaringBitmap into m.

[0168] S226: Store the RoaringBitmap structure in the database.

[0169] By storing the RoaringBitmap structure in a database, a KMV index can also be generated from the database for later use.

[0170] S3: Read the database to obtain the KMV index and determine whether to perform hot reloading.

[0171] If hot reloading is performed, the KMV index is written into memory. This enables hot reloading of frequently used sampling sets and KMV indexes through memory management, allowing the KMV index to be pre-loaded into memory.

[0172] Alternatively, if hot reloading is not performed, then execute S4.

[0173] S4: Call the logic calculation unit.

[0174] S5: Parallel computation of bitmap segments using the RoaringBitmap structure.

[0175] like Figure 3E As shown, the calculation process of the logic calculation unit is as follows:

[0176] S51: Perform logical operations on each segment of the split bitmap separately according to the segmentation order.

[0177] Logical operations include finding the intersection, difference, union, and complement of sets. Performing a logical operation separately refers to performing a logical operation on the first bitmap segment corresponding to sample set 1 and the first bitmap segment corresponding to sample set 2.

[0178] S52: Accumulate the results of the bit graph segmentation operation and calculate the target hash value corresponding to multiple sampling sets.

[0179] For example, the maximum hash value is selected from each sampling set, and then the smallest of these maximum hash values ​​is selected as the target mapping value. The target mapping value can be represented as: , p represents the total number of sets.

[0180] The accumulation of the operation results of each bitmap segment can be expressed as: , r1 is the logical operation result of the first bitmap segment.

[0181] S53: Estimate the base.

[0182] For example, the estimated base = , where n is the sampling coefficient.

[0183] The aforementioned S5 can be performed multiple times to obtain all the required estimated bases, and subsequent analysis results are obtained according to the estimated bases.

[0184] S6: Return the analysis result and cache the analysis result queried by the ACS service.

[0185] This can avoid resource overhead caused by repeated queries.

[0186] Through the above process, the estimated base of multiple set operations can be quickly queried using the ACS system, thereby achieving the purpose of understanding the population distribution in each age group.

[0187] It adopts the core idea of KMV algorithm, designs two-stage sampling based on stratified sampling and KMV sampling, and uses RoaringBitmap structure for bottom data storage, thereby providing a unified approximate calculation service to the outside, solving the problem of offline large data set base estimation in the UV (independent visitor) scene. Compared with the precise calculation method, this method accelerates the calculation and query efficiency of the set base. At the same time, it reduces the data storage overhead of the system.

[0188] Two-stage sampling fully utilizes the advantages of stratified sampling and KMV sampling methods, reducing the amount of data storage. The bottom data storage uses the RoaringBitmap structure, so it can get a larger data compression ratio and compress the data as much as possible under the same memory overhead, thereby reducing the overall data storage overhead of the system.

[0189] By segmenting the RoaringBitmap structure, concurrent queries can be performed during queries, thereby improving the overall computing efficiency of the system.

[0190] This method can use base estimation to return analysis results in real time to obtain gains in computing efficiency and data storage. At the same time, it can theoretically guarantee that the estimated value is unbiased and the estimation variance is controllable. This technology can be widely used in data analysis, network trend analysis, database optimization and many other fields.

[0191] The application will illustrate an approximate computing solution and its engineering system implementation based on KMV and RoaringBitMap (a BIT library with high compression ratio and good performance).

[0192] In the method, the data set to be analyzed is sampled according to a certain sampling ratio, the cardinality of the sampled sample is counted, and the statistical result of the sample is used to estimate the cardinality. The estimated cardinality of the overall set = the statistical cardinality of the sampled sample / the sampling ratio. For example, the set is sampled according to a sampling ratio of 1 / 10, the cardinality of the 1 / 10 sampled sample is counted, and the estimated result of the overall set cardinality is 10*the statistical cardinality of the sampled sample. In this way, the error-controllable estimation can be realized.

[0193] The method is based on the ACS (Approximate Computing Service) system realized by the KMV algorithm idea and the RoaringBitMap data storage technology. The two-stage sampling designed in combination with the hierarchical sampling and the KMV idea meets the UV scene-based large data set cardinality estimation. The bottom layer uses 32-bit RoaringBitMap for data storage, and combines the KMV idea design calculation to support the logical operation between sets and realize the set cardinality estimation after multi-set operation.

[0194] It fully utilizes the advantages of sampling and KMV to accelerate the query, and utilizes the data compression advantage of RoaringBitMap to reduce the data storage overhead, so as to store as many mapping values as possible under a certain memory overhead to improve the obtainable K value. Finally, through the engineering system implementation, a stable approximate computing service is provided to solve the offline large data set cardinality estimation problem in the UV scene, while ensuring that the estimation error is within a controllable range.

[0195] The two-stage sampling can effectively solve the problem that the estimation error of single-set and multi-set operation is amplified due to hash conflict and the problem that the estimation error is rapidly increased after using only KMV sampling when the set cardinality exceeds a certain order of magnitude, so as to ensure that the error influence range caused by hash conflict is within 2%.

[0196] In addition, it well solves the problem that the ordinary sampling method is greatly affected by the sparsity of elements in the set. Because it cannot be ensured that the elements in the original set are uniformly distributed, the sample data after set sampling may not accurately reflect the data characteristics of the original set, thereby causing a large estimation error.

[0197] Meanwhile, the method also solves the problem that the support for the cardinality estimation after the intersection complement and nested calculation between sets is not friendly, and the intersection can only be converted into a union set for processing through the principle of inclusion-exclusion, but the principle of inclusion-exclusion may cause extreme errors, and finally the cardinality estimation error after the intersection and complement operation of multiple sets is amplified. The method also solves the problem that the sparsity of different sets is different, and it is difficult to guarantee the number of elements obtained after the logical operation between multiple sets by fixing the minimum K value, and the effective intersection cannot be obtained during the set operation, which leads to a small final K value, thereby causing a large error in the overall cardinality estimation, and the method is not suitable for offline cardinality estimation.

[0198] In the method, the int type and 32-bit roaringbitmap structure is used for storage, which avoids the problem that the data amount that can be stored by the Long type array used in the bottom layer is limited under a certain storage overhead, and a large estimation error is caused by a too small K value after the multiple set operation. The memory usage of Bitmap is irrelevant to the number of set elements, which causes storage waste. The 32-bit RoaringBitmap is selected as the bottom data storage structure of the KMV index, which has obvious advantages over the 64-bit RoaringBitmap in data sparsity and storage overhead. A larger data compression ratio is obtained by using the RoaringBitMap, and the RoaringBitmap is segmented by horizontal extension of the RoaringBitmap, so that more numerical values can be stored under a certain memory overhead to improve the K value that can be obtained by the KMV, and the problem of the cardinality estimation accuracy caused by the sparsity of set elements is compensated by using larger sample data.

[0199] The data processing apparatus of the embodiment is used to implement the corresponding deadlock processing method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein again. In addition, the functions of each module in the data processing apparatus of the embodiment can be implemented by referring to the description of the corresponding part in the foregoing method embodiments, which will not be described herein again.

[0200] Embodiment Four

[0201] Referring to Figure 4 FIG. 4 shows a structural block diagram of a data processing apparatus according to Embodiment Four of the present application.

[0202] The data processing apparatus comprises:

[0203] The first determination module 402 is configured to determine at least two sets of data to be analyzed and corresponding set logical operation operations according to the received data analysis request.

[0204] The acquisition module 404 is configured to acquire a plurality of sampling sets corresponding to the plurality of data sets to be analyzed, wherein the sampling sets are obtained by performing uniform stratified sampling and KMV sampling on mapping values of data to be analyzed in the data sets to be analyzed.

[0205] The second determination module 406 is configured to determine a target mapping value meeting a set condition from maximum mapping values of the sampling sets.

[0206] The third determination module 408 is configured to perform the set logical operation on the sampling sets, and determine an analysis result according to the set logical operation result, the target mapping value and a sampling ratio of the uniform stratified sampling.

[0207] Optionally, if the database does not store the sampling sets corresponding to the plurality of data sets to be analyzed, and the mapping values are hash values, the acquisition module 404 is configured to determine a sampling ratio and a sampling number of KMV sampling, perform hash mapping on the plurality of data sets to be analyzed by using a uniform distribution hash function, so that hash values of data to be analyzed in the plurality of data sets to be analyzed are uniformly distributed in a set interval, perform stratified sampling on the uniformly distributed hash values of the data to be analyzed according to the sampling ratio, to obtain a stratified sampling set, and perform KMV sampling on the hash values in the stratified sampling set according to the sampling number of KMV sampling, to obtain the sampling sets corresponding to the plurality of data sets to be analyzed.

[0208] Optionally, when the acquisition module 404 performs KMV sampling on the hash values in the stratified sampling set according to the sampling number of KMV sampling, to obtain the sampling sets corresponding to the plurality of data sets to be analyzed, the hash values in the stratified sampling set are sorted in ascending order, and the first k hash values in the stratified sampling set sorted in ascending order are selected to form the sampling sets corresponding to the plurality of data sets to be analyzed, where k is the sampling number.

[0209] Optionally, the apparatus further comprises:

[0210] The adjustment module 405a is configured to adjust the hash values in the sampling sets to integers, and the data length of the hash values is a length meeting a data compression condition.

[0211] The compression module 405b is configured to perform data compression on the adjusted hash values in the sampling sets, and save the sampling sets subjected to data compression to the database.

[0212] Optionally, the compression module 405b is configured to create a data compression bitmap structure for compressing the sample set, and write the hash values in the sample set into the data compression bitmap structure; split the data compression bitmap structure storing the hash values in a length direction of the hash values according to a set number of segments, to obtain m bitmap segments corresponding to the number of segments, where m is a value of the number of segments; and store the m bitmap segments into a database.

[0213] Optionally, if the database stores the sample sets corresponding to the at least two data sets to be analyzed, the obtaining module 404 is configured to read the sample sets from the database, and obtain a sampling ratio used for uniformly stratified sampling of the sample sets.

[0214] Optionally, the second determining module 406 is configured to determine maximum hash values corresponding to the sample sets, and select a minimum maximum hash value from the maximum hash values of the at least two sample sets as the target mapping value.

[0215] Optionally, the third determining module 408 is configured to, for each set logical operation, determine different sample sets involved in a current set logical operation, perform logical operations on bitmap segments corresponding to the different sample sets according to a corresponding relationship of the bitmap segments, and accumulate logical operation results corresponding to the bitmap segments, to obtain a set logical operation result corresponding to the current set logical operation; and determine an analysis result according to the set logical operation results corresponding to each set logical operation, the target mapping value, and the sampling ratio of the uniform stratified sampling.

[0216] The data processing apparatus of the embodiment is used to implement the corresponding deadlock processing method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein again. In addition, the functions of each module in the data processing apparatus of the embodiment can be implemented by referring to the descriptions of the corresponding parts in the foregoing method embodiments, which will not be described herein again.

[0217] Embodiment Five

[0218] With reference to Figure 5 , a structural schematic diagram of an electronic device according to Embodiment Five of the present application is shown, and the specific implementation of the electronic device is not limited in the embodiments of the present application.

[0219] As shown in Figure 5 , the electronic device can include a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0220] Among them:

[0221] The processor 502, the communication interface 504, and the memory 506 communicate with each other through a communication bus 508.

[0222] The communication interface 504 is configured to communicate with other electronic devices or servers.

[0223] The processor 502 is configured to execute the program 510, and specifically can execute the related steps in the foregoing data processing method embodiments.

[0224] Specifically, the program 510 can include program code including computer operation instructions.

[0225] The processor 502 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be processors of the same type, such as one or more CPUs; or can be processors of different types, such as one or more CPUs and one or more ASICs.

[0226] The memory 506 is configured to store the program 510. The memory 506 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0227] The program 510 can be specifically used to cause the processor 502 to perform the operations corresponding to the foregoing method.

[0228] The specific implementation of each step in the program 510 can refer to the corresponding description in the corresponding steps and units in the foregoing data processing method embodiments, and will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the devices and modules described above can refer to the corresponding process description in the foregoing method embodiments, and will not be described here.

[0229] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0230] The methods according to the embodiments of the present application described above can be implemented in hardware, firmware, or software, or a combination of them, and can be stored in a recording medium such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or be downloaded from a network originally stored in a remote recording medium or non-transitory machine-readable medium and stored in a local recording medium, so that the methods described herein can be processed by such software using a general-purpose computer, a special-purpose processor, or programmable or dedicated hardware such as an ASIC or FPGA. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the data processing methods described herein. Furthermore, when a general-purpose computer accesses code for implementing the data processing methods shown herein, the execution of the code will convert the general-purpose computer into a special-purpose computer for executing the data processing methods shown herein.

[0231] Those skilled in the art can realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0232] The above embodiments are only used to illustrate the present application, and not to limit the present application, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, therefore all equivalent technical solutions also belong to the scope of the present application, the patent protection scope of the present application should be defined by the claims.

Claims

1. A data processing method, comprising: Based on the data analysis requests received from independent visitors, determine at least two sets of data to be analyzed and the corresponding set logic operations; Obtain at least two sampling sets corresponding to the data sets to be analyzed. The sampling sets are obtained by uniform stratified sampling and KMV sampling of the mapping values ​​of the data to be analyzed in the data sets to be analyzed. The data to be analyzed includes at least the demographic identifier of the visitors. From the maximum mapping value of each of the sampling sets, a target mapping value that meets the set conditions is determined, so as to estimate the number of independent visitors based on the target mapping value; The set logic operation is performed on the sample set, and the analysis result of the independent visitor base is determined based on the set logic operation result, the target mapping value and the sampling ratio of the uniform stratified sampling.

2. The method according to claim 1, wherein, If the database does not contain sampling sets corresponding to the at least two sets of data to be analyzed, and the mapping value is a hash value, then obtaining the sampling sets corresponding to the at least two sets of data to be analyzed includes: Determine the sampling ratio and the sample size for KMV sampling; A uniformly distributed hash function is used to perform a hash mapping on the dataset to be analyzed, so that the hash values ​​of the data to be analyzed in the dataset are evenly distributed within a set range. Based on the sampling ratio, stratified sampling is performed on the hash values ​​of the uniformly distributed data to be analyzed to obtain a stratified sampling set. Based on the sampling quantity of the KMV sampling, KMV sampling is performed on the hash values ​​in the stratified sampling set to obtain the sampling set corresponding to the data set to be analyzed.

3. The method according to claim 2, wherein, The step of performing KMV sampling on the hash values ​​in the stratified sampling set according to the sampling quantity of the KMV sampling to obtain the sampling set corresponding to the data set to be analyzed includes: The hash values ​​in the stratified sampling set are sorted in ascending order; The first k hash values ​​are selected from the stratified sampling set in ascending order to form the sampling set corresponding to the data set to be analyzed, where k is the number of samples.

4. The method according to claim 2, wherein, The method further includes: The hash values ​​in the sampling set are adjusted to integers, and the data length of the hash values ​​is a length that satisfies the data compression conditions; The hash values ​​in the adjusted sampling set are compressed, and the compressed sampling set is saved to the database.

5. The method according to claim 4, wherein, The step of compressing the hash values ​​in the adjusted sampling set and saving the compressed sampling set to the database includes: Create a data compression bitmap structure for compressing the sampling set, and write the hash values ​​in the sampling set into the data compression bitmap structure; According to the set number of segments, the data compression bitmap structure storing the hash value is split along the length direction of the hash value to obtain m corresponding bitmap segments, where m is the number of segments; The m bitmaps are segmented and stored in the database.

6. The method according to claim 1, wherein, If the database contains sampling sets corresponding to at least two of the data sets to be analyzed, obtaining the sampling sets corresponding to at least two of the data sets to be analyzed includes: The sampling set is read from the database, and the sampling ratio used when performing uniform stratified sampling on the sampling set is obtained.

7. The method according to claim 2, wherein, Determining the target mapping value that satisfies the set conditions from the maximum mapping value of each of the sampling sets includes: Determine the maximum hash value corresponding to each sampling set; The smallest maximum hash value is selected from the maximum hash values ​​of at least two of the sampling sets as the target mapping value.

8. The method according to claim 5, wherein, The analysis result of performing the set logic operation on the sampling set and determining the independent visitor cardinality based on the set logic operation result, the target mapping value, and the sampling ratio of the uniform stratified sampling includes: For each set logical operation, the different sampling sets involved in the current set logical operation are determined, the bitmap segments corresponding to the different sampling sets are segmented according to the correspondence of the bitmap segments, and the logical operation results corresponding to each bitmap segment are accumulated to obtain the set logical operation result corresponding to the current set logical operation. The analysis results are determined based on the set logic operation results corresponding to each set logic operation, the target mapping value, and the sampling ratio of the uniform stratified sampling.

9. A data processing apparatus, comprising: The first determining module is used to determine at least two sets of data to be analyzed and corresponding set logical operation operations based on the data analysis request received in the scenario of independent visitors; The acquisition module is used to acquire at least two sampling sets corresponding to the data sets to be analyzed. The sampling sets are obtained by uniform stratified sampling and KMV sampling of the mapping values ​​of the data to be analyzed in the data sets to be analyzed. The data to be analyzed includes at least the visitor's demographic identifier. The second determining module is used to determine the target mapping value that meets the set conditions from the maximum mapping value of each of the sampling sets, so as to estimate the number of independent visitors based on the target mapping value; The third determining module is used to perform the set logic operation on the sampling set, and determine the analysis result of the independent visitor base based on the set logic operation result, the target mapping value and the sampling ratio of the uniform stratified sampling.

10. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any one of claims 1-8.

Images