De-duplication counting method, apparatus, device, and storage medium

Abstract

The present disclosure provides a deduplication counting method, device, equipment and storage medium. Each data record in a data table comprises a key field used for deduplication counting and a plurality of data fields respectively corresponding to different data dimensions. The key field is used for storing a data identifier corresponding to the data record. The data field is used for storing data content corresponding to the data record. Based on the data identifier stored in the key field, a plurality of identical data records are combined to obtain a first preprocessed data record. Then, a plurality of first preprocessed data records with the same data content stored in each data field are combined to obtain a second preprocessed data record. Since the data table is pre-processed twice, the data amount of the data table can be reduced, and the calculation speed of deduplication counting calculation based on the data table combined based on the data content is faster.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data analysis, and particularly relates to a deduplication counting method, device, equipment and storage medium. BACKGROUND

[0002] Deduplication counting is widely used; for example, in website or APP (Application) usage statistics, PV / UV is the most commonly used index, and UV (unique visitor) is the index that needs to be deduplicated, that is, the multiple access records of the same user in the statistical period are counted only once. In the field of big data analysis, the data volume is huge, and how to quickly and accurately perform deduplication counting is a technical problem to be solved in the field. SUMMARY

[0003] To overcome the problems in the related art, the present disclosure provides a deduplication counting method, device, equipment and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a deduplication counting method is provided, and the method comprises:

[0005] obtaining a data table, each data record in the data table comprising: a key field used for deduplication counting and a plurality of data fields respectively corresponding to different data dimensions; the key field is used to store a data identifier corresponding to the data record; the data field is used to store data content corresponding to the data record;

[0006] determining whether there are multiple data records with the same data identifier stored in the key field in the data table; if yes, performing data record merging on the multiple data records to obtain a first preprocessed data record; wherein the data content stored in any data field in the first preprocessed data record is the data content stored in the data field included in each data record in the multiple data records, constituting a data content set;

[0007] in response to completion of the data record merging for the data table, further determine whether there are a plurality of first pre-processing data records in the data table, the data identifiers stored in the key fields of the plurality of first pre-processing data records are different, and the data contents stored in each data field of the plurality of first pre-processing data records are all the same; if yes, further perform data content merging on the plurality of first pre-processing data records to obtain a second pre-processing data record; wherein the data content stored in any data field included in the second pre-processing data record is a data content set stored in the data field included in each of the plurality of first pre-processing data records; the second pre-processing data record further includes a newly created attribute field; the attribute field is used to store a set number corresponding to the data content set stored in each data field included in the second pre-processing data record; the set number is the number of the plurality of first pre-processing data records;

[0008] in response to completion of the data content merging for the data table, perform deduplication counting calculation based on the data table for which the data content merging is completed.

[0009] According to a second aspect of the embodiments of the present specification, a deduplication counting device is provided, and the device comprises:

[0010] an acquisition module, which acquires a data table, each data record in the data table comprising: a key field used for deduplication counting and a plurality of data fields corresponding to different data dimensions respectively; the key field is used to store a data identifier corresponding to the data record; and the data field is used to store data content corresponding to the data record;

[0011] a data record merging module, which determines whether there are a plurality of data records in the data table, the data identifiers stored in the key fields of the plurality of data records are the same; if yes, perform data record merging on the plurality of data records to obtain a first pre-processing data record; wherein the data content stored in any data field included in the first pre-processing data record is a data content set stored in the data field included in each of the plurality of data records;

[0012] The data content merging module further determines, in response to completion of the data record merging for the data table, whether there are a plurality of first pre-processing data records in the data table, the data identifiers stored in the key fields of the plurality of first pre-processing data records being different, and the data contents stored in each data field of the plurality of first pre-processing data records being same; if yes, the plurality of first pre-processing data records are further merged to obtain a second pre-processing data record; wherein the data content stored in any data field included in the second pre-processing data record is a data content set stored in the data field included in each first pre-processing data record in the plurality of first pre-processing data records; the second pre-processing data record further includes a newly created attribute field; the attribute field is used to store a set number corresponding to the data content set stored in each data field included in the second pre-processing data record; and the set number is the number of the plurality of first pre-processing data records.

[0013] The computing module performs deduplication counting calculation based on the data table for which the data content merging is completed.

[0014] According to a third aspect of the embodiments of the present specification, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the deduplication counting method according to the first aspect when executing the computer program.

[0015] According to a fourth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores a computer program, and the computer program is executable on a processor to implement the steps of the deduplication counting method according to the first aspect.

[0016] The technical solutions provided by the embodiments of the present specification can include the following beneficial effects:

[0017] In the embodiments of the present specification, the data content stored in any data field of a plurality of data records of the same user identifier is merged to form a data content set stored in a corresponding data field of a first pre-processing data record, and then a plurality of first pre-processing data records in which the data contents stored in each data field are same are merged into a second pre-processing data record, so that the number of data records is greatly reduced, and the data amount required for deduplication calculation is reduced, thereby improving the calculation speed.

[0018] In addition, the embodiment scheme can provide accurate calculation results, and there is no requirement for the type of data that needs to be counted for deduplication, and there is no need to convert the data into a numerical value using a hash algorithm like RoaringBitmap (efficient compression bitmap) and other schemes, and the generality of the embodiment scheme is better.

[0019] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the description and serve to explain the principles of the disclosure.

[0021] Figure 1A and Figure 1B are collection diagrams shown in this specification according to an exemplary embodiment, respectively.

[0022] Figure 2 is a flow chart of a deduplication counting method shown in this specification according to an exemplary embodiment.

[0023] Figure 3 is a hardware structure diagram of a computer device in which a deduplication counting apparatus is located shown in this specification according to an exemplary embodiment.

[0024] Figure 4 is a block diagram of a deduplication counting apparatus shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, unless otherwise indicated, like numbers in the drawings indicate like components. The following detailed description of the exemplary embodiments is not intended to limit the scope of the present description, but rather, for the purpose of illustration and explanation. Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0026] The terminology used in the description of the present description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present description. As used in the description of the present description and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0027] It is to be understood that although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish different sets of information from one another. For example, a first information can be termed a second information, and similarly, a second information can be termed a first information, without departing from the scope of the present description. As used herein, the word "if' can be construed to mean "when" or "in response to determining" depending on the context.

[0028] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] The deduplication count is a commonly used analysis function in data analysis, which refers to the number of different values in a column in a query data table. The function in SQL (Structured Query Language) is count(distinct col). The difference between it and the count(col) function is that there is a distinct descriptor, which means to remove duplicate values, so it is called deduplication count.

[0030] Deduplication count is widely used, for example: in website or APP (Application) usage statistics, PV / UV is the most commonly used indicator, where UV (unique visitor) is the indicator that needs to be deduplicated, that is, all access records of the same user within the statistical period are counted only once. For website or APP owners, PV (page view) represents the level of usage, and UV represents the number of users. The combination of the two numbers can accurately understand the user or usage growth of the website or APP.

[0031] Deduplication count has always been one of the important problems in the field of big data analysis. Mass data dynamic, fast and accurate deduplication calculation is a very challenging problem in the industry. The current industry solutions are difficult to balance speed, accuracy, dynamics and universality at the same time. For example, some products that include a large number of basic services, commonly used deduplication indicators not only include UV, but also include device number, template number, etc. In order to improve the operation and iteration efficiency of the product, it is necessary to support analysis of indicators in any dimension through data capabilities, but in the mass data scenario, it is difficult to achieve fast, accurate, and universal conditions for deduplication calculation using existing solutions.

[0032] For example, the following is a schematic diagram of a raw data table about users:

[0033] Number of data rows Dimension K1 Dimension K2 Dimension Ki Dimension KN User identification 1 K11 K21 Ki1 KN1 u1 2 K13 K23 Ki3 KN3 u2 …… …… …… …… …… …… 57 K1_8 K2_9 Ki_6 KN_23 ua 58 K1_8 K2_9 Ki_6 KN_23 ua 59 K1_9 K2_9 Ki_11 KN_24 ua …… …… …… …… …… …… t K1t K2t Kit KNt ut …… …… …… …… …… …… T K1T K2T KiT KNT uT

[0034] The original data table includes T original data records, each of which represents a user's access record to a website or an APP, and each original data record contains N dimensions and a user identification field, which is the key field that needs to be counted for deduplication. Each access of a user generates an access record, which corresponds to an original data record in the data table. For example, the 57th to 59th data in the above table are from the same user with the user identification ua. Although the three original data records are from the same user, the values of some dimensions in the original data records may be the same or different because the user generates each access under different circumstances.

[0035] In the field of big data analysis, the data volume of an original data table in a statistical cycle can be in the order of hundreds of millions, and it may also need to support analysis of any dimension of the index.

[0036] In existing solutions, considering the large amount of data, methods such as HyperLogLog use statistical-based inaccurate cardinality estimation, which gives an estimation result close to the true value by sampling or analyzing the statistical characteristics of the data set. Such estimation schemes can give the value of deduplication counting within a certain error range, and can complete the calculation in a short time. Although the calculation is fast, the calculation result is not accurate.

[0037] In the precise calculation scheme, a hash table or bitmap is used to record the elements that have appeared, and then the number of non-repeated elements is counted according to the recorded results. RoaringBitmap can be used to implement it, and the calculation result is accurate. However, in this scheme, bit is used to represent the presence or absence of each value in a set. If the data type of the deduplication column is a numerical value, it can be used directly, and if it is not a numerical value, the deduplication column needs to be converted into a numerical value using algorithms such as hash. For example, in the above original data table, the deduplication column is the user identification column, and the data type of the user identification is not a numerical value. Therefore, this scheme needs to map each user identification in the user identification column of the original data table to a numerical value one by one, so this scheme is only suitable for numerical type deduplication columns and requires sophisticated technical processing, and has insufficient universality. In addition, the amount of data after processing does not decrease, for example, if there are 100 million user identifications, the amount of data after processing is still 100 million.

[0038] Therefore, whether it is HyperLogLog or RoaringBitmap, the accuracy and efficiency of the calculation are solved by constructing the deduplication column, but the high cardinality phenomenon of the deduplication column itself makes it challenging to construct any data structure.

[0039] It is found through research that the deduplication counting is essentially a set operation problem. Therefore, the embodiment of the specification proposes a completely new scheme through the idea of set operation. The scheme directly bypasses the high cardinality problem in deduplication counting through set operation of the deduplication column and the query dimension, and is suitable for various scenarios requiring deduplication counting.

[0040] To simplify the problem description, as shown in Figure 1A FIG. 1 is a schematic diagram of a set according to an exemplary embodiment of the specification, assuming that there are two large sets S1 and S2 and a small set set0 i where the intersection of the sets S1 and S2 is set0, The element number calculation in the set satisfies the following additive principle (the absolute value symbol of the set in the following formula represents the element number of the set):

[0041] |S1|=|set1|+|set0|

[0042] |S2|=|set2|+|set0|

[0043] |S1∩S2|=|set0|

[0044] |S1∪S2|=|set1|+|set0|+|set2|

[0045] |S1|-|S2|=|set1|-|set2|

[0046] |S2|-|S1|=|set2|-|set1|

[0047] Assuming that set0, set1 and set2 can be constructed, the calculation between the large sets S1 and S2 can be realized through set operation, and in the deduplication counting scenario, most of the counting requirements can be obtained through addition, for example, to query |S1|, |S2|, |S1∩S2| or |S1∪S2|, all of which can be calculated by addition, and the calculation efficiency is relatively fast.

[0048] A simple example is used for illustration; taking the calculation of the number of users accessing a page as an example, assuming that there is the following original data set:

[0049] Dimension K (e.g. access channel) User identification k1 u1 k1 u2 k1 u1 k2 u3 k2 u4 k2 u1

[0050] The original data table has two columns, including: dimension K in the dimension column represents the access channel of the user accessing the page, that is, each original data record contains a key field (user identification field) and a data field for deduplication counting. The dimension K has two values: k1 and k2; The user identification column records the user identification of the access user, that is, the key column that needs to be counted for deduplication, which has four values: u1, u2, u3 and u4.

[0051] As shown in Figure 1B , the above original data table can be converted into a set for understanding: for the dimension value k1, the elements contained in the set include users u1 and u2; for the dimension value k2, the elements contained in the set include users u1, u3 and u4. The intersection of the two sets contains the element u1.

[0052] Corresponding to Figure 1A , the data record of the user containing only the dimension value k1 is set1, the data record of the user containing only the dimension value k2 is set2, and the data record of the user containing both the dimension value k1 and the dimension value k2 is set0.

[0053] In the existing accurate calculation scheme:

[0054] When querying the number of users of dimension value k1, all data containing dimension value k1 need to be found by traversing the data table, such as the first three rows of the above table, and then deduplicating the user identification to obtain the number of users, which is 2;

[0055] When querying the number of users of dimension value k2, all data containing dimension value k2 need to be found by traversing the data table, such as the last three rows of the above table, and then deduplicating the user identification to obtain the number of users, which is 3.

[0056] From the two queries, it can be seen that since the user with user identification u1 has both data records containing dimension value k1 and data records containing dimension value k2, it can be found that in the above two queries of querying the number of users of dimension value k1 and querying the number of users of dimension value k2, the data record of the user with user identification u1 (set0) is contained, and deduplication calculation based on the data record of the user with user identification u1 is required.

[0057] Based on the analysis of the principle of set operation as described above, if it can be known in advance:

[0058] The data record of the user containing only the dimension value k1 (that is, set1) is: the data record of u2;

[0059] The data record of the user containing only the dimension value k2 (that is, set2) is: the data record of u3 and the data record of u4;

[0060] The data record of the user containing both the dimension value k1 and the dimension value k2 (set0) is the data record of u1;

[0061] Therefore, when the number of users containing the dimension value k1 is queried, it is known that |set1| + |set0| is 2;

[0062] When the number of users containing the dimension value k2 is queried, it is known that |set2| + |set0| is 3.

[0063] Therefore, based on the above principle, the scheme of the embodiment is designed, and the original data records of the same access user can be merged based on the user identifier first:

[0064] Specifically, the user u1 has three original data records, two of which are the same, and the value of the dimension K is k1, so one of them can be deleted and only one can be retained. The value of the dimension K of the other is k2, so the first preprocessed data record representing the dimension value k1 and the dimension value k2 is merged.

[0065] The merged data set includes two columns, one of which is changed to the value set K_set of the dimension K, and the user identifier column remains unchanged; specifically as follows:

[0066] Set of values of dimension K K_set User identification [k1,k2] u1 k1 u2 k2 u3 k2 u4

[0067] In this way, the size of the data amount can be reduced to 4; after the first data record merging, the original data records of the same user are de-duplicated.

[0068] Next, the second data content merging is performed:

[0069] This merging is based on the value set of the dimension, and the second preprocessed data record is obtained, which contains two columns, one of which is still the value set column of the dimension K, and the other column is changed from the user identifier column to the set number column.

[0070] As an example, the data with the same value set of the dimension is merged into one, for example, there are two data with the value k2, and the set number is 2; the preprocessed data table obtained by data content merging is as follows:

[0071] Set of values of dimension K K_set Number of sets (representing number of users) [k1,k2] 1 k1 1 k2 2

[0072] The set expression corresponding to the above preprocessed data table is as follows:

[0073]

[0074] As can be seen from the above table, after merging, the preprocessed data table is actually equivalent to constructing the small set set i ;

[0075] To query the number of users accessing the site from channel k1, iterate through K_set and find the data containing k1, which is the first two data entries in the table. Add the number of records in these two data entries to get the result: |set0| and |set1|.

[0076] To query the number of users accessing the site from channel k2, iterate through K_set and find the data containing k2, which is the first and third data in the table. Add the set counts of these two data sets together, i.e., add |set0| and |set2| together. The sum of these two is the query result.

[0077] Other queries can be performed similarly, using |set0|, |set1|, and |set2| for addition and subtraction.

[0078] The above embodiments are described using one data dimension. In practical applications, the data tables shown above may involve multiple data dimensions. The embodiments described in this specification will be explained in detail below.

[0079] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a deduplication counting method according to an exemplary embodiment, which may include the following steps:

[0080] Step 202: Obtain the data table.

[0081] Each data record in the data table can contain: a key field for deduplication and several data fields corresponding to different data dimensions; the key field can be used to store the data identifier corresponding to the data record; and the data fields can be used to store the data content corresponding to the data record.

[0082] Step 204: Determine whether there are multiple data records in the data table that have the same data identifier stored in the key field; if so, merge the multiple data records to obtain the first preprocessed data record; wherein, the data content stored in any data field contained in the first preprocessed data record is the data content stored in the data field contained in each of the multiple data records, forming a set of data content.

[0083] Step 206: In response to the completion of data record merging for the data table, further determine whether there are multiple first preprocessed data records in the data table that have different data identifiers stored in the key fields and the same data content stored in each data field; if so, further merge the data content of the multiple first preprocessed data records to obtain second preprocessed data records.

[0084] The data content stored in any data field in the second pre-processed data record is a set of data contents stored in the data field of each of the first pre-processed data records; the second pre-processed data record further comprises a newly created attribute field; the attribute field is used to store a set number corresponding to the set of data contents stored in each data field in the second pre-processed data record; the set number is the number of the first pre-processed data records.

[0085] In step 208, in response to completion of the data content merging for the data table, a deduplication count calculation is performed based on the data table for which the data content merging is completed.

[0086] As an example, the data table in step 202 can be a data table related to a certain statistical indicator, for example, the statistical indicator is the number of user clicks, and the data table can include: a click data table representing clicks of a target object by a user, and the key field can be used to store a user identifier corresponding to a data record; and the deduplication count can include a deduplication count corresponding to the click UV of the target object by the user.

[0087] For example, the statistical indicator is the number of exposures of a target object to a user, and the data table can include: an exposure data table representing exposures of a target object to a user, and the key field can be used to store a user identifier corresponding to a data record; and the deduplication count can include a deduplication count corresponding to the exposure UV of the target object to the user.

[0088] There can be many different data tables according to different data analysis needs in actual applications, which are not limited in the present embodiment.

[0089] The data table can be stored in row or column, which is not limited in the present embodiment. Taking row storage as an example, each column represents a different field (or attribute), each row stores the values of these columns, and each row represents a data record in the data table.

[0090] Each original data record in the data table can include two types of fields: key fields and data fields. The key field is an important field for deduplication count, which stores a data identifier related to the original data record. This data identifier can be unique to distinguish different records. Through the key field, the data table can be deduplicated. The data field is a field that stores specific data content related to the original data record. These fields can represent different data dimensions, i.e. different aspects of data information. Of course, the data table can also include more fields according to needs in actual applications, which are not limited in the present embodiment.

[0091] For the convenience of distinguishing, the data table not processed by the merging is referred to as the original data table, and the data table finally obtained by twice merging is referred to as the preprocessed data table. Taking the original data table as an example, one representation form of the data table can be as follows:

[0092] Dimension K1 Dimension K2 Dimension Ki Dimension KN User identification K11 K21 Ki1 KN1 u1

[0093] The original data table is taken as an example of row storage, including a user identification column and N dimension columns; the user identification column is a key field; the dimension column is a column used to describe the characteristics or attributes of data, and each dimension Ki in the N dimensions is a data field. The original data table contains multiple original data records, that is, each row in the table stores the values of these columns, indicating a specific data record. Taking a click data table as an example, the key field can be a user identification field, and the N data fields are N dimensions of the user.

[0094] For the original data table, the embodiment designs twice merging processing: firstly, data record merging based on the key field, and then data content merging based on the data field.

[0095] For example, in the original data table mentioned above, three original data records in the 57th to 59th rows:

[0096] Number of data rows Dimension K1 Dimension K2 Dimension Ki Dimension KN User identification 57 K1_8 K2_9 Ki_6 KN_23 ua 58 K1_8 K2_9 Ki_6 KN_23 ua 59 K1_9 K2_9 Ki_11 KN_24 ua

[0097] Since the data in the key field is the same, that is, ua, indicating that the three original data records belong to the same user ua, the three original data records can be merged. Among them, the data contents stored in each array field of the 57th original data record and the 58th original data record are completely the same (that is, the values of each dimension are the same), and the values of part of the dimensions in the 59th original data record are different from those of the 57th and 58th original data records. The first preprocessed data record obtained by merging can be as follows:

[0098]

[0099] Therefore, the first preprocessed data obtained by merging represents the merging result of multiple original data records from the same user ua in the original data table, and the data content stored in any data field in the first preprocessed data record is the data content stored in the data field in each data record in the multiple data records, constituting a data content set.

[0100] Through the merging processing, the repeated dimension values in these records are de-duplicated, and the value set of each dimension is formed. Specifically, for the user ua, the value set of dimension K1 is [K1_8, K1_9], which indicates that the value of ua on dimension K1 is K1_8 and K1_9. Similarly, the value set of dimension K2 is [K2_9, K2_9], the value set of dimension Ki is [Ki_6, Ki_11], and the value set of dimension KN is [KN_23, KN_24].

[0101] Optionally, the result of the data record merging can be to create a new data table for storage, or to update the original data table based on the result of the data record merging. In actual applications, it can be flexibly configured according to needs, and the present embodiment does not limit this.

[0102] The specific implementation mode of the data record merging in actual applications can be configured according to needs. As an example, in some examples, the data record merging can be performed in the following manner:

[0103] G groups Group corresponding to the original data table are obtained; each group contains one original data record or contains multiple original data records with the same data identifier, the original data records in different groups have different data identifiers, and 1≤g≤G;

[0104] For each group, the following data record merging operation is performed to obtain G first pre-processed data records: it is determined that there are at least two identical original data records in the multiple original data records contained in the group Group_g; if so, at least two identical original data records in the group Group_g are deleted to one original data record to obtain an updated group Group_g; and a first pre-processed data record corresponding to the updated group Group_g is generated.

[0105] For example, after the first data record merging, the following three first pre-processed data records can be obtained:

[0106]

[0107] In the three first pre-processed data records obtained after merging, the data identifiers stored in the key fields are different, but the data contents stored in each data field are the same, that is, the N value sets SN_arr of the three data records are the same. Therefore, after the data content merging in step 206, the three first pre-processed data records can be merged into the following second pre-processed data record:

[0108]

[0109] In the above example, the second preprocessed data record contains a newly created attribute field, i.e., the column of the number of sets in the above table, which stores the value of the number of sets as 3, indicating that the number of first preprocessed data records merged to obtain the second preprocessed data record is 3. That is, the attribute field is used to store the number of sets corresponding to the set of data contents stored in each data field in the second preprocessed data record, and the number of sets is the number of the plurality of first preprocessed data records.

[0110] As an example, in actual applications, the data content merging in step 206 can be implemented in various ways, which are not limited in the embodiment. As an example, when searching for the plurality of first preprocessed data records that can be merged, the data contents stored in each data field of each first preprocessed data record can be spliced as the key of the first preprocessed data record, and then the keys of the plurality of first preprocessed data records are traversed. The plurality of first preprocessed data records with different data identifiers but the same key can be merged into a second preprocessed data record.

[0111] Suppose that there are other plurality of first preprocessed data records to be merged, as an example, the second preprocessed data record obtained by further preprocessing is as follows:

[0112]

[0113] In some examples, as shown in the above table, the second preprocessed data record can be a data record that does not contain a key field;

[0114] Further data content merging of the plurality of first preprocessed data records to obtain the second preprocessed data record can include:

[0115] Deleting the key field contained in the plurality of first preprocessed data records;

[0116] Further data content merging of the plurality of first preprocessed data records after deleting the key field to obtain the second preprocessed data record.

[0117] In the embodiment, considering that in the deduplication counting scenario, the user query requires the deduplication count value, and generally does not need to query the data identifier stored in the key field for deduplication counting, therefore, in the process of obtaining the second preprocessed data record, the embodiment can delete the key field contained in the first preprocessed data record, thereby reducing the data information contained in the second preprocessed data record and reducing the data size.

[0118] Based on the data table obtained by merging the data contents, a fast deduplication count calculation can be realized. In some examples, the deduplication count calculation based on the data table obtained by merging the data contents can include:

[0119] In response to the obtained calculation request for the deduplication count value of the target data content corresponding to the target data dimension, querying at least one second preprocessed data record containing the target data content from the data table obtained by merging the data contents;

[0120] Adding the set quantities contained in each of the at least one second preprocessed data record to obtain the deduplication count value of the target data content.

[0121] In actual applications, various deduplication count calculations can be involved, but each calculation requirement can be converted to: querying second preprocessed data records containing one or more target data contents, and obtaining a calculation result by using the set quantities in the queried data records; for querying multiple data records, an addition calculation can be used to quickly and efficiently obtain a calculation result. The scheme of the embodiment can convert deduplication count logic to addition calculation logic, and the calculation efficiency is very fast. For example:

[0122] ① Calculation of the deduplication count value of one target data content;

[0123] The calculation process can be: querying at least one second preprocessed data record containing the target data content, and adding the set quantities in the queried data records.

[0124] For example, for the calculation of the deduplication count of any target dimension value k, each data content set stored in the data dimension K_set column to which the target dimension value k belongs can be traversed in the preprocessed data table to obtain one or more second preprocessed data containing k, and the set quantities contained in these data are added. The addition result is the calculation result. As an example, the pseudo code of the above process can be as follows:

[0125] for(Ks∈K_set):

[0126]

[0127] return(vk)

[0128] The pseudo code indicates that:

[0129] For each data content set Ks stored in the data dimension K_set column, the following operations are performed:

[0130] If the target dimension value k to be queried is a subset of Ks (i.e. If the current k is a subset of Ks (i.e.

[0131] ② Computing the deduplication count value corresponding to each target data content;

[0132] The multiple target data contents can be the same target data dimension or different target data dimensions.

[0133] The computing process can be: creating a computing request corresponding to the deduplication count value of each target data content, using the computing method in ① above to obtain the deduplication count value of each target data content respectively for each target data content.

[0134] For example, if the computing request is the deduplication count calculation corresponding to multiple target dimension values respectively, the computing result of each target dimension value can be calculated respectively according to the computing process in ①.

[0135] As an example, if the computing request is to obtain the deduplication count value of each dimension value under dimension K, the computing result of each dimension value can be obtained respectively according to the computing process in ①, and each dimension value and its computing result can be returned.

[0136]

[0137] for (Ks ∈ K_set):

[0138]

[0139] return (k, vk)

[0140] The pseudo code indicates:

[0141] For each subset k in the set K, the following operations are performed:

[0142] For each data content set Ks stored under the data dimension K_set column, the following operations are performed:

[0143] If the current k is a subset of Ks (i.e. ), the value of vk is increased to Value(Ks).

[0144] The result (k, vk) is returned, where k is the current subset and vk is the accumulated value.

[0145] The code is to find the element Ks in K_set that matches k according to the subset relationship of k, and to perform an accumulation operation on the corresponding Value(Ks). Finally, return each subset k and its corresponding accumulation value vk, that is, the deduplication count value vk of each dimension value k.

[0146] ③ Calculation of the deduplication count value of the target data content containing multiple target data contents;

[0147] The multiple target data contents here can be the same target data dimension or different target data dimensions.

[0148] The calculation process is the same as ①, which can be: querying at least one second preprocessed data record containing multiple target data contents, and adding the set quantities in each data record obtained by the query.

[0149] As another example, if a calculation request for the deduplication count value of the target data content K1_8 for dimension K1 is obtained, the second preprocessed data records containing K1_8 queried from the preprocessed data table are as follows:

[0150]

[0151] Add the set quantities contained in the two second preprocessed data records containing K1_8, that is, add 3 and 4 to get the value 7, which is the deduplication count value of K1_8; In the aforementioned example of click data, it means that the number of users with dimension value K1_8 in dimension K1 is 7.

[0152] Similarly, suppose a calculation request for the deduplication count value of the target data content K1_9 for dimension K1 is obtained, and the only second preprocessed data record containing K1_9 queried is the one shown in the table above, read the set quantity 3, and get the calculation result.

[0153] The above target data content is described in one dimension. Suppose multiple calculation requests for the deduplication count values of multiple target data contents of multiple different target data dimensions are received, and the query process is the same. For example, suppose a calculation request for the deduplication count value of K1_8 and K2_9 is obtained, and the second preprocessed data containing K1_8 and K2_9 queried from the preprocessed data table is as follows:

[0154]

[0155] Similarly, add the set quantities contained in the two preprocessed data: 3+4=7, and 7 is the calculation result.

[0156] Assuming the obtained request is for calculating the deduplication count of dimension value K1_9 and K2_9, and the only data record containing K1_9 is the second preprocessed data record shown in the table above, then the set count 3 in that data record is read to obtain the calculation result.

[0157] In some examples, the data record merging and data content merging of this embodiment can be performed on the original data table to obtain a preprocessed data table, which is then stored. As analyzed above, the data fields in the preprocessed data table have changed compared to the data fields in the original data table. The data fields in the original data table correspond to data dimensions and are used to store the data content of each data dimension; while the data fields in the preprocessed data table also correspond to data dimensions, but store a set of data content for each data dimension. The deduplication count is calculated based on the preprocessed data table. Therefore, in some examples, the request to calculate the deduplication count value of the target data content corresponding to the target data dimension can be obtained in the following way:

[0158] In response to a received query request for the deduplication count of the target data content corresponding to the target data dimension in the data table, a calculation request for the deduplication count of the target data content in the data table is created based on the received query request.

[0159] As can be seen from the above embodiments, each calculation requirement can be converted into: querying data records containing one or more target data contents, and obtaining the calculation result by using the number of sets in the queried data records; for querying multiple data records, the calculation result can be obtained quickly and efficiently by using addition calculation.

[0160] Of course, other query requests may be involved in practical applications; for example, querying the difference in deduplication counts corresponding to two different target data contents. This query can also be converted into calculating the deduplication counts corresponding to these different target data contents, and then further calculations can be performed based on the difference between the two deduplication counts required in the query request. In practical applications, the specific processing can be determined according to the specific query request, and this embodiment does not limit this.

[0161] In some cases, the original data table can be the original data table for a certain statistical indicator. For example, the statistical indicator is the number of user clicks. The data table includes: a data table representing user clicks on a target object, with key fields used to store user identifiers corresponding to data records; and a deduplication count including the deduplication count corresponding to the UV (unique visitors) of user clicks on the target object. The data table obtained after merging this data table twice can then be used to calculate the deduplication count corresponding to the UV of user clicks on the target object.

[0162] Or, the statistical index is the number of exposures of the target object to the user, and the data table includes: an exposure data table representing the exposure of the target object to the user, and a key field for storing a user identifier corresponding to a data record; the deduplication count includes a deduplication count corresponding to the exposure UV of the target object to the user. After the data table is merged twice, the data table can be calculated to obtain the deduplication count corresponding to the exposure UV of the target object to the user.

[0163] In actual application, there can be one original data table for each statistical index, for example, a separate click data table or a separate exposure data table. In other examples, there can also be multiple statistical indexes in one source table, and the original data table of a certain statistical index of the embodiment is a sub-table in the source table; for example, the source table is an exposure data table, and a dimension column representing whether the user clicks is included in the exposure data table, and whether the user clicks after a certain page is exposed to the user is represented by the field. In this way, the click data table can be a sub-table in the source table. It can be understood that the original data table can be preprocessed to obtain a preprocessed data table by using the embodiment in the above different cases.

[0164] In the embodiment, since the data content stored in any data field of the multiple data records of the same user identifier is combined to form the data content set stored in the corresponding data field of the first preprocessed data, and then the multiple first preprocessed data records in which the data content stored in each data field is the same are combined into one second preprocessed data record, the number of data records is greatly reduced, and the reduction of the amount of data required for deduplication calculation improves the calculation speed.

[0165] Moreover, since the attribute field of the second preprocessed data record stores the number of the multiple first preprocessed data, the subsequent calculation process is simplified, and when calculating the deduplication count value of the data content corresponding to a certain data dimension, the set number stored in the attribute field of the second preprocessed data containing the data content can be used for addition calculation, so that the calculation result can be quickly obtained.

[0166] In addition, for a data table with multiple data dimensions, the user can freely combine the target data content to be queried according to needs when querying.

[0167] Furthermore, the embodiment scheme can provide accurate calculation results, and does not require the data type to be deduplicated to be converted into a numerical value using an algorithm such as hashing, and the universality of the embodiment scheme is better.

[0168] In practical applications, a test was conducted using an original data table containing 2 billion original data records from a real product. Using this embodiment, a preprocessed data table was obtained by merging the original data table twice, reducing the number of data records from 2 billion to 500,000. Using existing technology to query the deduplication count of a target data content in the original data table required 18 minutes and 191 cores per minute (where a core refers to a processor core). However, using the solution in this embodiment, performing the same query with the preprocessed data table required only 38 seconds and 0.27 cores per minute. Therefore, this embodiment significantly reduces the data size, reduces the time to one-hundredth of the original, and reduces the computational load to one-thousandth of the original.

[0169] Corresponding to the embodiments of the deduplication counting method described above, this specification also provides embodiments of a deduplication counting device and the computer equipment on which it is applied.

[0170] The embodiments of the deduplication counting device described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by its processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of the computer device containing the deduplication counting device described in this manual. (Except for...) Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the computer device in which the deduplication counting device 331 is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0171] like Figure 4 As shown, Figure 4 This is a block diagram illustrating a deduplication counting device according to an exemplary embodiment of this specification. The device includes:

[0172] Module 41 retrieves a data table. Each data record in the data table contains: a key field for deduplication and counting, and several data fields corresponding to different data dimensions. The key field stores the data identifier corresponding to the data record, and the data fields store the data content corresponding to the data record.

[0173] The data record merging module 42 determines whether a plurality of data records in the data table have the same data identifier stored in the key field; if so, the plurality of data records are merged to obtain a first preprocessed data record; wherein the data content stored in any data field in the first preprocessed data record is a data content set of the data content stored in the data field of each data record in the plurality of data records;

[0174] The data content merging module 43 further determines, in response to completion of the data record merging for the data table, whether a plurality of first preprocessed data records in the data table have different data identifiers stored in the key field and have the same data content stored in each data field; if so, the plurality of first preprocessed data records are further merged to obtain a second preprocessed data record; wherein the data content stored in any data field in the second preprocessed data record is a data content set of the data content stored in the data field of each first preprocessed data record in the plurality of first preprocessed data records; the second preprocessed data record further includes a newly created attribute field; the attribute field is used to store a set number corresponding to the data content set stored in each data field in the second preprocessed data record; the set number is the number of the plurality of first preprocessed data records;

[0175] The calculation module 44 performs deduplication counting calculation based on the data table for which the data content merging is completed.

[0176] In some examples, the second preprocessed data record is a data record that does not include the key field;

[0177] The data content merging module further merges the plurality of first preprocessed data records to obtain a second preprocessed data record, including:

[0178] The key field included in the plurality of first preprocessed data records is deleted;

[0179] The plurality of first preprocessed data records from which the key field is deleted are further merged to obtain a second preprocessed data record.

[0180] In some examples, the counting module performs deduplication counting calculation based on the data table for which the data content merging is completed, including:

[0181] In response to the obtained calculation request for the deduplication count value of the target data content corresponding to the target data dimension, at least one second preprocessed data record including the target data content is queried from the data table for which the data content merging is completed;

[0182] The number of sets contained in each of the at least one second pre-processed data record is added to obtain a deduplication count value of the target data content.

[0183] In some examples, the computing request for the deduplication count value of the target data content corresponding to the target data dimension is obtained in the following manner:

[0184] In response to the received query request for the deduplication count value of the target data content corresponding to the target data dimension in the data table, a computing request for the deduplication count value of the target data content in the data table is created based on the received query request.

[0185] In some examples, the data table includes: a click data table representing clicks of a target object by a user, a key field for storing a user identifier corresponding to a data record; and the deduplication count includes a deduplication count corresponding to a click UV of the target object by the user.

[0186] In some examples, the data table includes: an exposure data table representing exposure of a target object to a user, a key field for storing a user identifier corresponding to a data record; and the deduplication count includes a deduplication count corresponding to an exposure UV of the target object to the user.

[0187] The functions and effects of the modules in the deduplication count device are described in detail in the implementation process of the corresponding steps in the deduplication count method described above, and will not be repeated here.

[0188] Correspondingly, the present specification also provides a computer program product, comprising a computer program, which is executed by a processor to implement the steps of the foregoing deduplication count method embodiments.

[0189] Correspondingly, the present specification also provides a computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the deduplication count method embodiments when executing the program.

[0190] Correspondingly, the present specification also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the deduplication count method embodiments.

[0191] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiment described above is only illustrative, wherein the modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, i.e., can be located in one place or distributed on multiple network modules. Part or all of the modules can be selected to achieve the purpose of the scheme of the present specification according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0192] The above embodiments can be applied to one or more computer devices, which are devices capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0193] The computer device can be any electronic product capable of human-computer interaction with the user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive Internet Protocol Television (IPTV), a smart wearable device, etc.

[0194] The computer device can also include a network device and / or a user device. The network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0195] The network in which the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0196] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of this application as to its specific aspects, some advantages thereby are that the application can provide a method and system for providing a user with a more personalized experience.

[0197] The division of steps in the above methods is only for the sake of description, and in actual implementation, one step can be combined or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included, which is within the protection scope of the patent; adding irrelevant modifications or introducing irrelevant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the application.

[0198] The description of the word "specific example" or "some examples" or the like means that the specific features, structures, materials or characteristics described in connection with the described embodiment or example are included in at least one embodiment or example of the specification. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0199] Other embodiments of this specification will occur to those skilled in the art, having the benefit of the present specification and practice of the application claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification following the general principles thereof and including the general principles of the art or the known conventional techniques in the art to which the specification pertains. The specification and examples are to be considered exemplary only, with the true scope and spirit of the specification being indicated by the following claims.

[0200] It should be understood that the specification is not limited to the precise construction described and shown in the drawings and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the specification is limited only by the claims appended hereto.

[0201] The above only describes the preferred embodiments of the specification and does not limit the specification, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the specification shall be included in the protection scope of the specification.

Claims

1. A method for deduplicating and counting duplicates, the method comprising: Obtain a data table, wherein each data record in the data table includes: a key field for deduplication and counting, and several data fields corresponding to different data dimensions; the key field is used to store the data identifier corresponding to the data record; the data fields are used to store the data content corresponding to the data record. Determine whether there are multiple data records in the data table that have the same data identifier stored in the key field; if so, merge the multiple data records to obtain a first preprocessed data record; wherein, the data content stored in any data field contained in the first preprocessed data record is the data content stored in the data field contained in each of the multiple data records, forming a set of data content; In response to the completion of data record merging for the data table, it is further determined whether multiple first preprocessed data records exist in the data table, wherein the data identifiers stored in the key fields are different, and the data content stored in each data field is the same; if so, the data content of the multiple first preprocessed data records is further merged to obtain a second preprocessed data record; wherein the data content stored in any data field contained in the second preprocessed data record is the set of data content stored in that data field contained in each of the multiple first preprocessed data records; the second preprocessed data record also contains a newly created attribute field; the attribute field is used to store the number of sets corresponding to the set of data content stored in each data field contained in the second preprocessed data record; the number of sets is the number of the multiple first preprocessed data records; In response to the completion of data content merging for the data table, a deduplication count is performed on the data table after the data content merging is completed.

2. The method according to claim 1, wherein the second preprocessed data record is a data record that does not contain the key field; The further merging of the data content of the multiple first preprocessed data records to obtain second preprocessed data records includes: Delete the key fields contained in the multiple first preprocessed data records; The data content of the multiple first preprocessed data records with the key field deleted is further merged to obtain the second preprocessed data record.

3. The method according to claim 1, wherein the deduplication calculation based on the data table after data content merging includes: In response to the obtained request to calculate the deduplication count value of the target data content corresponding to the target data dimension, at least one second preprocessed data record containing the target data content is queried from the data table after the data content merging is completed. The deduplication count of the target data content is obtained by summing the number of sets contained in each of the at least one second preprocessed data record retrieved.

4. The method according to claim 3, wherein the request for calculating the deduplication count value of the target data content corresponding to the target data dimension is obtained in the following manner: In response to a received query request for the deduplication count of the target data content corresponding to the target data dimension in the data table, a calculation request for the deduplication count of the target data content in the data table is created based on the received query request.

5. The method according to claim 1, wherein the data table comprises: The table represents user click data for a target object, and the key field is used to store the user identifier corresponding to the data record; the deduplication count includes the deduplication count corresponding to the UV of user clicks on the target object.

6. The method according to claim 1, wherein the data table comprises: This indicates that the target object will be exposed to the user's exposure data table, and the key field is used to store the user identifier corresponding to the data record; The deduplication count includes the deduplication count corresponding to the UV exposure of the target object for the user.

7. A deduplication counting device, the device comprising: The acquisition module acquires a data table, wherein each data record in the data table includes: a key field for deduplication and counting, and several data fields corresponding to different data dimensions; the key field is used to store the data identifier corresponding to the data record; and the data fields are used to store the data content corresponding to the data record. The data record merging module determines whether there are multiple data records in the data table that have the same data identifier stored in the key field; if so, it merges the multiple data records to obtain a first preprocessed data record; wherein, the data content stored in any data field contained in the first preprocessed data record is a set of data content consisting of the data content stored in the data field contained in each of the multiple data records. The data content merging module, in response to the completion of data record merging for the data table, further determines whether multiple first preprocessed data records exist in the data table, wherein the data identifiers stored in the key fields are different, and the data content stored in each data field is the same; if so, the multiple first preprocessed data records are further merged to obtain a second preprocessed data record; wherein the data content stored in any data field of the second preprocessed data record is the set of data content stored in that data field of each of the multiple first preprocessed data records; the second preprocessed data record also includes a newly created attribute field; the attribute field is used to store the number of sets corresponding to the set of data content stored in each data field of the second preprocessed data record; the number of sets is the number of the multiple first preprocessed data records; The calculation module, in response to the completion of data content merging for the data table, performs deduplication counting calculation based on the data table with merged data content.

8. The apparatus according to claim 7, wherein the second preprocessed data record is a data record that does not contain the key field; The data content merging module further merges the data content of the multiple first preprocessed data records to obtain a second preprocessed data record, including: Delete the key fields contained in the multiple first preprocessed data records; The data content of the multiple first preprocessed data records with the key field deleted is further merged to obtain the second preprocessed data record.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images