Data processing method and device, electronic equipment and computer program product

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer program product. Relates to the field of big data, and comprises the following steps: obtaining a to-be-counted user log data set stored in a database, preprocessing the user log data set to obtain an initial processing data set, the initial processing data set comprising N initial processing data, and N being a positive integer; the initial processing data set is partitioned, M partitioned data sets are obtained, M is smaller than N, and M is a positive integer; and for the M partition data sets, performing statistical processing on each partition data set by using a cardinal number statistical algorithm to obtain M data statistical results, and performing user behavior analysis based on the M data statistical results. Through the method and the device, the technical problems of low statistical efficiency, low accuracy and high resource consumption when cardinality statistics is carried out on the to-be-counted data in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a data processing method, device, electronic device, and computer program product. Background Art

[0002] In the field of big data processing, cardinality statistics, as an important step, plays a key role in scenarios such as data analysis, search engine optimization, advertising recommendation systems, and data monitoring. The above steps can help staff deeply understand the distribution and characteristics of the data, so as to make more accurate decisions and optimization strategies.

[0003] In the related technology, sorting and deduplication method, bitmap method and database built-in method are often used for data processing. Among them, the sorting and deduplication method first sorts the data set, then traverses the data set, and counts the number of adjacent non-repeating elements to obtain the cardinality; the bitmap method uses the bitmap structure to assign a bit to each possible element to record the occurrence of the element; the database built-in method uses many relational databases to provide built-in functions to calculate the number of different values.

[0004] However, with the explosive growth of data volume, traditional precise cardinality statistics methods face severe challenges. First, when the data volume is small, sorting and deduplication can be used for operations, but when processing large data sets, that is, in real-time processing scenarios, this method has the problem of high computational complexity; the space requirement of the bitmap method is proportional to the potential types of elements. For large-scale data sets and high cardinality, the bitmap method has the problem of high memory overhead; the database built-in method is also only suitable for processing small data volumes. When processing large-scale data sets, since the identification of each unique element requires additional search or storage operations, this method also has the problem of low efficiency.

[0005] There are technical problems in related technologies such as low statistical efficiency, low accuracy and high resource consumption when performing cardinality statistics on the data to be counted, and no effective solution has been proposed yet. Summary of the invention

[0006] The main purpose of the present application is to provide a data processing method, device, electronic device and computer program product to solve the technical problems of low statistical efficiency, low accuracy and high resource consumption when performing cardinality statistics on the data to be counted in the related art.

[0007] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data processing method is provided. The method comprises: obtaining a user log data set to be counted stored in a database, preprocessing the user log data set to obtain an initial processing data set, wherein the initial processing data set comprises N initial processing data, N is a positive integer; partitioning the initial processing data set to obtain M partition data sets, wherein M is less than N, and M is a positive integer; for the M partition data sets, respectively using a cardinality statistical algorithm to perform statistical processing on each of the partition data sets to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

[0008] Furthermore, partitioning the initial processing data set to obtain M partitioned data sets includes: obtaining a log field associated with each initial processing data to obtain N log fields, wherein each log field includes at least one of the following: a time field and a numerical range field; performing field conversion processing on the N log fields using a conversion function to obtain N converted log fields; when the N converted log fields are time fields, grouping the N converted log fields according to a preset time period range to obtain M first field groups; combining the initial processing data associated with the time field in each first field group to obtain the M partitioned data sets; when the N converted log fields are numerical range fields, grouping the N converted log fields according to a preset numerical interval to obtain M second field groups; combining the initial processing data associated with the numerical range field in each second field group to obtain the M partitioned data sets.

[0009] Furthermore, partitioning the initial processed data set to obtain M partitioned data sets includes: using a hash function to calculate the hash value of each initial processed data to obtain N data hash values; grouping the N data hash values ​​using a preset hash value interval to obtain M hash value groups; and combining the initial processed data associated with the data hash values ​​in each hash value group to obtain the M partitioned data sets.

[0010] Furthermore, before using the cardinality statistics algorithm to perform statistical processing on each of the partitioned data sets to obtain M data statistical results, the method also includes: for each partitioned data set, obtaining a preset partition threshold, and comparing the preset partition threshold with the total number of initial processed data associated with the partitioned data set; when the total number of initial processed data associated with the partitioned data set is less than the preset partition threshold, determining the hash statistics algorithm as the cardinality statistics algorithm; when the number of initial processed data associated with the partitioned data set is greater than or equal to the preset partition threshold, determining the probability evaluation algorithm as the cardinality statistics algorithm.

[0011] Furthermore, in the case where the cardinality statistical algorithm is the hash statistical algorithm, each of the partitioned data sets is statistically processed using the cardinality statistical algorithm to obtain M data statistical results, including: for each partitioned data set, creating a preset array, wherein the size of the preset array is determined by the range of the initial processed data of the partitioned data set; using a hash function to map the initial processed data in the partitioned data set to the preset array to obtain a processed array; determining the types of different initial processed data in the processed array, and determining the data statistical results according to the types of the different initial processed data.

[0012] Further, in the case where the cardinality statistical algorithm is the probability evaluation algorithm, the cardinality statistical algorithm is used to perform statistical processing on each of the partitioned data sets to obtain M data statistical results, including: using a hash function to perform hash calculation on the initial processed data in the partitioned data set to obtain Y data hash values, and determining the binary representation of each data hash value to obtain Y binary data, where Y is a positive integer; obtaining L initial registers, using the preset bit in each binary data as a bitmap index, and filtering out an initial register from the L initial registers according to the bitmap index of each binary data, where L is less than Y and L is a positive integer; for a binary data, determining the prefix length of the binary data, and comparing the prefix length with the data in the initial register corresponding to the binary data, where an initial prefix length is stored in each initial register; in the case where the prefix length is greater than the initial prefix length of the initial register corresponding to the binary data, using the prefix length to update the initial prefix length in the initial register to obtain updated data in a register; performing probability statistics on the updated data in the L registers to obtain the data statistical results.

[0013] Furthermore, probability statistics are performed on the updated data in the L registers to obtain the data statistical results, including: error correction is performed on the updated data in the L registers to obtain the corrected data in the L registers, wherein the error correction method includes at least one of the following: sparse correction, long tail correction and deviation correction; the corrected data in the L registers are merged to obtain the data statistical results.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data processing device is provided. The device includes: a first acquisition unit, used to acquire a user log data set to be counted stored in a database, pre-process the user log data set, and obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, N is a positive integer; a partitioning unit, used to partition the initial processing data set to obtain M partition data sets, wherein M is less than N, and M is a positive integer; a processing unit, used to perform statistical processing on each of the M partition data sets using a cardinality statistical algorithm, respectively, to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

[0015] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned data processing methods.

[0016] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory stores an executable program, and the processor is used to run the program, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned data processing methods.

[0017] According to another aspect of an embodiment of the present invention, a computer program product is provided. The computer program product includes a computer program, wherein the computer program implements any one of the above-mentioned data processing methods when executed by a processor.

[0018] In an embodiment of the present application, a data processing method is adopted, by obtaining a user log data set to be counted stored in a database, pre-processing the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, N is a positive integer; the initial processing data set is partitioned to obtain M partition data sets, wherein M is less than N, and M is a positive integer; for the M partition data sets, a cardinality statistics algorithm is used to perform statistical processing on each of the partition data sets to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results, thereby achieving the purpose of efficiently, accurately and flexibly performing cardinality statistics on big data, thereby achieving the technical effect of improving the efficiency of cardinality statistics on big data, and being able to provide high-precision statistical results under conditions of limited resources, thereby solving the technical problems of low statistical efficiency, low accuracy and high resource consumption when performing cardinality statistics on the data to be counted. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method;

[0021] Figure 2 is a flowchart of a method for processing data provided in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a data processing device provided according to an embodiment of the present application;

[0023] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0027] It should be noted that the collected information used in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse to use it.

[0028] Example 1

[0029] According to an embodiment of the present application, a method embodiment of data processing is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 It is a hardware structure diagram of a computer terminal (or mobile device) for implementing a data processing method, such as Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 , 102n are used to illustrate) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU (Microcontroller Unit) or a programmable logic device FPGA (Field-Programmable Gate Array)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O (Input / Output Interface) interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a keyboard, a cursor control device, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) and a network interface, which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0035] Under the above operating environment, this application provides Figure 2 The data processing method shown. Figure 2 is a flow chart of a method for processing data according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0036] Step S201 , obtaining a user log data set to be counted stored in a database, and preprocessing the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, where N is a positive integer.

[0037] Data processing is widely used in many fields. In information search scenarios, data processing methods are needed to ensure that high-quality search results are provided to users. In product recommendation scenarios, data processing methods are also needed to distinguish the responses of different user groups to different products, thereby optimizing the delivery strategy and improving the relevance of recommendations and product conversion efficiency. In traffic monitoring scenarios, the use of data processing methods can help identify potential network attacks or abnormal traffic patterns, which is of great value for network security protection. In user behavior scenarios, data processing methods are used to count the base of user behavior, which is also crucial for the construction of user portraits and personalized recommendations.

[0038] When it is necessary to use the behavioral data of multiple users for user behavior analysis, in order to ensure the security of the data, the big data to be counted can be imported into the database first. Big data refers to data with large data sets, diverse types, fast generation speed and high value potential.

[0039] Furthermore, when it is necessary to use user log data for user behavior analysis, after extracting the user log data set from the above-mentioned database that can efficiently process large-scale data sets, the user log data set is subjected to data preprocessing and other operations to improve statistical accuracy, thereby obtaining an initial processed data set, wherein the data acquisition operation can be implemented through a query statement, such as a SELECT statement, and the data preprocessing operation can include filtering out records containing null values ​​or invalid formats, converting data in different formats into a unified format, and data type conversion, etc.

[0040] During data preprocessing, you can use the table engine "ReplacingMergeTree" to achieve deduplication, which can be achieved through the following code:

[0041] CREATE TABLE[IF NOT EXISTS][db.]table_name[ON CLUSTER cluster] (

[0043] name1[type1][DEFAULT|MATERIALIZED|ALIAS expr1],

[0044] name2[type2][DEFAULT|MATERIALIZED|ALIAS expr2], ...

[0046] )ENGINE=Replacing Merge Tree([ver])

[0047] [PARTITION BY expr]

[0048] [ORDER BY expr]

[0049] [SAMPLE BY expr]

[0050] [SETTINGS name=value,...];

[0051] Step S202, partitioning the initial processing data set to obtain M partitioned data sets, where M is less than N and is a positive integer.

[0052] Specifically, since the initial processing data in the initial processing data set has the characteristics of large scale, diverse types, fast generation speed and high value potential, before processing these data, they need to be sharded, that is, according to the selected partitioning strategy (such as time partitioning, range partitioning or hash partitioning), a partition is assigned to each record, and then the data set is divided into multiple smaller subsets to obtain multiple partitioned data sets. This method helps to process data in parallel, reduce resource consumption of a single calculation, and improve overall processing efficiency.

[0053] Step S203, for the M partitioned data sets, respectively use the cardinality statistical algorithm to perform statistical processing on each partitioned data set to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

[0054] Specifically, after obtaining multiple partitioned data sets, a cardinality statistics algorithm can be used to perform cardinality statistics calculations on each partitioned data set. For example, the number of non-repeating elements can be calculated using a database, or the HyperLogLog algorithm can be used for statistical techniques. Since the cardinality statistics algorithm can be performed in parallel on each partitioned data set, the computational load is reduced, thereby greatly improving the overall computational efficiency.

[0055] Furthermore, the data statistics of all partitioned data sets are summarized to obtain the final cardinality estimate, and then a series of user behavior analyses can be performed based on the cardinality estimate to extract valuable information. For example, active user groups can be identified through cardinality estimates to evaluate product usage and user engagement. Cardinality estimates can also be used to analyze user behavior patterns at different times, places or usage scenarios to customize personalized services and optimize user experience. Cardinality estimates can also be used for user grouping and market segmentation to provide a basis for personalized recommendations.

[0056] The data processing method provided in the embodiment of the present application obtains a user log data set to be counted stored in a database, pre-processes the user log data set, and obtains an initial processing data set, wherein the initial processing data set includes N initial processing data, N is a positive integer; partitions the initial processing data set to obtain M partition data sets, wherein M is less than N and M is a positive integer; for the M partition data sets, respectively uses a cardinality statistics algorithm to perform statistical processing on each partition data set to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results, and the technical problems of low statistical efficiency, low accuracy and high resource consumption when performing cardinality statistics on the data to be counted in the related art are solved, and the initial processing data set is obtained by pre-processing the obtained user log data set, and the initial processing data set is partitioned and then processed by the cardinality statistics algorithm to obtain each partition data set, so as to obtain corresponding data statistical results, and finally the data statistical results can be used to perform user behavior analysis, thereby achieving the technical effect of improving the efficiency of cardinality statistics on big data and providing high-precision statistical results under limited resources.

[0057] In order to improve computing efficiency and reduce resource consumption of a single calculation, optionally, in the data processing method provided in the embodiment of the present application, the initial processing data set is partitioned to obtain M partitioned data sets, including: obtaining a log field associated with each initial processing data to obtain N log fields, wherein each log field includes at least one of the following: a time field and a numerical range field; performing field conversion processing on the N log fields using a conversion function to obtain N converted log fields; when the N converted log fields are time fields, grouping the N converted log fields according to a preset time period range to obtain M first field groups; combining the initial processing data associated with the time field in each first field group to obtain M partitioned data sets; when the N converted log fields are numerical range fields, grouping the N converted log fields according to a preset numerical interval to obtain M second field groups; combining the initial processing data associated with the numerical range field in each second field group to obtain M partitioned data sets.

[0058] Specifically, in order to improve the efficiency of data processing, when sharding data, partitioning can be performed based on the timestamp of the data or the numerical range of the data. Time partitioning helps to quickly locate time-related data, such as analysis of user activities within a specific time range (such as peak usage hours, active user groups at night, etc.). This method is suitable for data sets that grow by time, such as log data and event data; range partitioning can distribute data evenly and avoid excessive computational burden on a single partition. For example, by dividing by user account, the behavioral differences of different user groups can be analyzed, such as the usage habits of new and old users, user preferences of specific age groups or income levels, etc. This partitioning method is suitable for numerical ranges or string ranges.

[0059] When performing partitioning operations, you can first obtain the log fields associated with each initially processed data. In order to better process the data, you can use the conversion function to convert each log field to obtain the corresponding converted log fields. These fields may include timestamps, value ranges (such as the value range of user accounts), latitude and longitude information of geographic locations, event types, etc.

[0060] Furthermore, when using time partitioning for data sharding, you can select an appropriate partitioning strategy based on business needs and data characteristics, such as partitioning by date, timestamp, year, month, day, etc., determine the partitioning expression based on the above partitioning strategy, and then use the partitioning expression to group the converted log fields. After obtaining the first field group, the initial processing data associated with the time field in the first field group is combined to obtain the corresponding partitioned data set. At this time, date partitioning can be achieved through the following code:

[0061] CREATE TABLE events (

[0063] event_date Date,

[0064] event_id UInt32,

[0065] user_id UInt32,

[0066] event_type String,

[0067] value Float32

[0068] )ENGINE=MergeTree()

[0069] PARTITION BY toYYYYMMDD(event_date)

[0070] ORDER BY(event_date,event_id);

[0071] Among them, "ENGINE = MergeTree()" is used as the table engine, which is mainly used to store and process time series data; "PARTITION BY toYYYYMMDD(event_date)" is used as the data partitioning strategy, which means using the toYYYYMMDD function to convert the date value of the event_date column into an integer value in the YYYYMMDD format, and then partition the data based on this value. Each partition represents a specific date; "ORDER BY(event_date, event_id)" indicates the sorting method, that is, sort by event_date first, and then sort by event_id. This method helps to use indexes when querying and improve query speed.

[0072] When using range partitioning for data sharding, you can determine the value interval based on the data characteristics, such as the range of user accounts from 1 to 1000, 1001 to 2000, etc., or the range of transaction amounts from 0 to 100, 101 to 200, etc. Then use the above value interval to filter out records whose values ​​fall within the range from the initial processing data set, and then obtain multiple second field groups, and then combine the initial processing data associated with the value range field in the second field group to obtain the corresponding partitioned data set. Range partitioning can be implemented through the following code:

[0073] CREATE TABLE events (

[0075] event_date Date,

[0076] event_id UInt32,

[0077] user_id UInt32,

[0078] event_type String,

[0079] value Float32

[0080] )ENGINE=MergeTree()

[0081] PARTITION BY intDiv(user_id,1000)

[0082] ORDER BY(user_id,event_date,event_id);

[0083] Among them, "event_date Date" indicates the date when the event occurred; "event_id UInt32" indicates the unique identifier of each event; "user_id UInt32" is the unique identifier of the user; "event_type String" is the type of event; ENGINE = MergeTree() indicates the use of the MergeTree table engine; "PARTITION BY intDiv(user_id,1000)" indicates the partitioning strategy of the data, that is, by using the intDiv function, the value of the user_id field is divided by 1000 and rounded, and the result is used to determine which partition the data should be stored in. This partitioning method helps to evenly distribute user-related data and avoid overloading certain partitions. Then use "ORDER BY(user_id,event_date,event_id)" to determine the sorting method, that is, first sort by user_id, then sort by event_date under each user ID, and finally sort by event_id under each date. In this way, queries based on user ID, event date and event ID are optimized to improve query performance. This embodiment uses a specific field type to perform time partitioning operations, and can also use preset conditions to perform range partitioning operations, thereby implementing partitioning operations on the initial processing data set. This can not only effectively process large-scale data sets, but also provide strong support for more in-depth user behavior analysis.

[0084] There are many ways to partition a data set. Optionally, in the data processing method provided in the embodiment of the present application, partitioning the initial processed data set to obtain M partitioned data sets includes: using a hash function to calculate the hash value of each initial processed data to obtain N data hash values; using a preset hash value interval to group the N data hash values ​​to obtain M hash value groups; and combining the initial processed data associated with the data hash values ​​in each hash value group to obtain M partitioned data sets.

[0085] Specifically, in addition to using time partitioning and range partitioning for data sharding, you can also use hash value partitioning to process data sets, where hash value partitioning is suitable for scenarios where data needs to be evenly distributed to multiple disks to improve query performance and data processing parallelism. When using hash value partitioning for processing, you can first select a hash function, where the hash function can be an intHash32() function or a lightweight hash function, and then use the selected hash function to calculate the hash value of each initially processed data to obtain the data hash value represented as an integer or binary string.

[0086] Furthermore, in order to ensure the uniform distribution of data among multiple partitions and avoid bottlenecks caused by excessive data volume in some partitions, after determining the hash value interval, the preset hash value interval can be used to group the above data hash values ​​to form multiple hash value groups, and then the data hash values ​​in each hash value group are associated with the original initial processing data to construct the corresponding partition data set. In this way, it can be ensured that the amount of data contained in each partition data set is roughly equal, which helps to maximize the efficiency of parallel processing in a distributed system. At this time, hash value partitioning can be implemented through the following code:

[0087] CREATE TABLE events (

[0089] event_date Date,

[0090] event_id UInt32,

[0091] user_id UInt32,

[0092] event_type String,

[0093] value Float32

[0094] )ENGINE=MergeTree()

[0095] PARTITION BY intHash32(user_id)%10

[0096] ORDER BY(user_id,event_date,event_id);

[0097] Among them, "event_date Date" indicates the date when the event occurred, "event_id UInt32" indicates the unique identifier of each event, "user_id UInt32" indicates the unique identifier of the user, and "event_type String" indicates the string describing the event type; "ENGINE = MergeTree()" is a table engine suitable for time series data; "PARTITION BY intHash32(user_id)%10" indicates the partitioning method of the data, that is, the value of the user_id field is converted into a hash value using the intHash32 function, and then the hash value is mapped to the range of 0 to 9 through the modulo operation "%10", so that the data is evenly distributed into 10 partitions, and finally the data is sorted using the data sorting method of "ORDER BY(user_id, event_date, event_id)", that is, first sort by user_id, then sort by event_date under each user ID, and finally sort by event_id under each date. This partitioning method ensures that the data of different users is evenly distributed to each partition, which is conducive to distributed query and parallel processing, and can avoid hot spots. This embodiment uses a hash function to calculate and group the hash values ​​of the initial processed data, and then uses a preset hash value interval to generate multiple partitioned data sets. This can not only improve the efficiency and parallelism of big data processing, but also achieve more accurate and efficient user behavior analysis and other data processing tasks while maintaining data security and privacy.

[0098] Optionally, in the data processing method provided in the embodiment of the present application, before using the cardinality statistical algorithm to perform statistical processing on each partitioned data set respectively to obtain M data statistical results, the method also includes: for each partitioned data set, obtaining a preset partition threshold, and comparing the preset partition threshold with the total number of initial processed data associated with the partitioned data set; when the total number of initial processed data associated with the partitioned data set is less than the preset partition threshold, determining the hash statistical algorithm as the cardinality statistical algorithm; when the number of initial processed data associated with the partitioned data set is greater than or equal to the preset partition threshold, determining the probability evaluation algorithm as the cardinality statistical algorithm.

[0099] Specifically, after processing using different partitioning methods, a cardinality statistics algorithm can be used to perform cardinality statistics operations, where cardinality refers to the number of non-repeating elements in a data set. For example, in a set of data {1, 2, 2, 3, 4, 4, 4}, the cardinality is 4 (i.e., non-repeating elements are 1, 2, 3, 4). When using the cardinality statistics algorithm to perform statistics on each partitioned data set, in order to ensure the accuracy and efficiency of the calculation, it is necessary to dynamically select the statistical algorithm according to the size of the partitioned data set to achieve the best balance between performance and accuracy.

[0100] Therefore, before performing cardinality statistics, we can first analyze the size of each partitioned data set, that is, determine the total number of initially processed data contained in each partitioned data set, and then compare the size of the partitioned data set with the preset partition threshold. If the size of the partitioned data set is lower than the preset partition threshold, it indicates that the amount of data in the partitioned data set is small. We can use an accurate but computationally expensive hash statistics algorithm, such as building a hash table to accurately calculate the cardinality. For small-scale data, this can provide accurate deduplication counting results.

[0101] When the size of the partitioned data set is greater than or equal to the preset partition threshold, the data set becomes huge. At this time, a probability evaluation algorithm (such as the HyperLogLog algorithm) is used for processing, that is, the cardinality is estimated by statistically analyzing the data features after hashing. It should be noted that the preset partition threshold can be dynamically adjusted according to system resources, performance requirements and data set characteristics. For example, in a scenario where the amount of data increases rapidly, the threshold may need to be increased to reduce the number of partitions using the probability evaluation algorithm, thereby reducing the overall computational complexity; conversely, in a scenario where resources are sufficient and there are higher requirements for accuracy, the threshold can be lowered and the partitions using the hash statistical algorithm can be increased. This embodiment analyzes the partitioned data set using a preset partition threshold, and then selects different cardinality statistical algorithms for statistical processing, thereby ensuring that the most suitable algorithm can be flexibly selected when processing data sets of different sizes, taking into account the efficiency of the calculation and the accuracy of the results, thereby improving the overall processing performance and resource utilization.

[0102] There are many ways to perform cardinality statistics on big data. Optionally, in the data processing method provided in the embodiment of the present application, when the cardinality statistics algorithm is a hash statistics algorithm, the cardinality statistics algorithm is used to perform statistical processing on each partitioned data set to obtain M data statistical results, including: for each partitioned data set, creating a preset array, wherein the size of the preset array is determined by the range of the initial processed data of the partitioned data set; using a hash function to map the initial processed data in the partitioned data set to the preset array to obtain a processed array; determining the types of different initial processed data in the processed array, and determining the data statistical results according to the different types of initial processed data.

[0103] Specifically, when the amount of data in the partitioned data set is small, a hash statistics algorithm can be used for cardinality statistics. Before processing the partitioned data set, the range of the data can be evaluated first, that is, the maximum and minimum values ​​that the initial processing data may take. For example, if the "user_id" field in the initial data set contains user IDs from 1 to 1000000, the size of the preset array can be 1000000, and all elements in the array are initialized to 0, and then the array is used to record the number of occurrences of each type of initial processing data.

[0104] Furthermore, a hash function is used to map the initial processing data to the index of a preset array, and the corresponding position in the array is marked to ensure that each data type is recorded in the preset array. Then the array element value of the mapping position is increased by 1, indicating that the data type appears once in this partition. For example, when "user_id" is 50000, an index position is obtained through the hash function, and then the array element value of the position is increased by 1. After processing all the data in a partitioned data set, the preset array is traversed and the number of types of hash values ​​corresponding to different initial processing data is counted, and then the number of types is used as the result of the cardinality statistics in the partitioned data set, that is, the data statistics result is obtained.

[0105] In addition, cardinality statistics can also be performed using a hash table. Similarly, an empty hash table for storing hash values ​​of data elements can be created, and then the initial processing data is mapped to a hash value using a hash algorithm, and the corresponding element or value is stored in the hash table. If the element already exists, it does not need to be stored again. Finally, the result of the cardinality statistics of the initial data set is determined based on the number of keys stored in the hash table, that is, the data statistics result is obtained. This embodiment uses a hash statistics algorithm to perform cardinality statistics, which can maximize the efficiency of data processing while maintaining the reliability of the results.

[0106] Optionally, in the data processing method provided in the embodiment of the present application, when the cardinality statistical algorithm is a probability evaluation algorithm, the cardinality statistical algorithm is used to perform statistical processing on each partitioned data set respectively to obtain M data statistical results, including: using a hash function to perform hash calculation on the initial processed data in the partitioned data set to obtain Y data hash values, and determining the binary representation of each data hash value to obtain Y binary data, where Y is a positive integer; obtaining L initial registers, using the preset bit in each binary data as a bitmap index, and screening out an initial register from the L initial registers according to the bitmap index of each binary data, where L is less than Y and L is a positive integer; for a binary data, determining the prefix length of the binary data, and comparing the prefix length with the data in the initial register corresponding to the binary data, where an initial prefix length is stored in each initial register; when the prefix length is greater than the initial prefix length of the initial register corresponding to the binary data, using the prefix length to update the initial prefix length in the initial register to obtain updated data in a register; performing probability statistics on the updated data in the L registers to obtain data statistical results.

[0107] Specifically, the probability evaluation algorithm may be the HyperLogLog algorithm, which is a probability algorithm for estimating the cardinality of a large data set. The algorithm can map data to different registers through a hash function and record the maximum prefix length to estimate the number of non-repeating elements. When the amount of data in the initial data set is large, the HyperLogLog algorithm may be used for cardinality statistics.

[0108] When performing cardinality statistics, we can first perform hash calculation on each item of initial processed data in each partitioned data set, that is, use a hash function to convert these data into a data hash value of a fixed length, and then determine the binary representation of each data hash value to obtain all binary data in each partitioned data set, wherein the length of the consecutive zeros in the prefix of each binary data can be determined, that is, the prefix length is obtained. The longer the length of the consecutive zeros, the rarer the data hash value is in the binary representation, thereby providing more information.

[0109] Since the HyperLogLog algorithm can estimate the number of different elements (i.e., cardinality) by observing the hash value of each data in the data set, registers can be used to store statistical information. For example, registers can store the prefix length of the hash value. Since the HyperLogLog algorithm can calculate the length of the prefix continuous zeros in the binary representation of the hash value conversion when encountering each new data, the register can store the length of the maximum continuous zeros in the processed hash value. In addition, the HyperLogLog algorithm can estimate the cardinality of the data set using a probabilistic method by analyzing the values ​​in the register, that is, assuming that in a uniformly distributed hash value, the probability of having a longer prefix zero is inversely proportional to the cardinality of the data set. At this time, the number of non-repeating elements in the data set can be estimated by averaging the values ​​in all registers.

[0110] Therefore, when using the HyperLogLog algorithm for statistics, multiple initial registers can be determined and their values ​​can be set to 0. These registers are used to store the maximum length of the prefix 0 in the data hash value of each initially processed data. Furthermore, a specific bit is selected from the binary representation of each hash value as a bitmap index for selecting a register. For example, if there are L initial registers, one register can be selected from these initial registers through the bitmap index to compare and update the prefix length. Then the prefix length of each binary data is compared with the initial prefix length stored in the initial register determined by the bitmap index. If the prefix length of the current binary data is greater than the initial prefix length recorded in the register, the initial prefix length in the register can be updated with the prefix length of the binary data until all binary data in the partitioned data set are compared.

[0111] Furthermore, after obtaining the prefix length of the longest consecutive zeros recorded in each register, probability statistics are performed on the data in the above registers to estimate the cardinality of the data set. At this time, the values ​​in all registers can be averaged, and then skew correction is performed based on the average value to reduce the estimation error. The corrected average value is then used to estimate the number of different elements in the data set, thereby improving the accuracy of the estimation.

[0112] It should be noted that, since the data set will be divided into multiple partitions in the big data processing, when using the HyperLogLog algorithm for cardinality statistics, it is necessary to merge the results of each partitioned data set to obtain a global estimate. This embodiment uses a probabilistic statistical algorithm to estimate the cardinality of a large data set with relatively high accuracy, and can efficiently perform deduplication statistics and maintain good performance and accuracy while processing large amounts of data.

[0113] Optionally, in the data processing method provided in the embodiment of the present application, probability statistics are performed on the updated data in the L registers to obtain data statistical results, including: error correction is performed on the updated data in the L registers to obtain corrected data in the L registers, wherein the error correction method includes at least one of the following: sparse correction, long tail correction and deviation correction; the corrected data in the L registers are merged to obtain data statistical results.

[0114] Specifically, since the HyperLogLog algorithm uses a probabilistic method when estimating the cardinality, its results have certain errors. Therefore, in order to reduce the error, a variety of error correction mechanisms can be used to improve the accuracy of the estimation. When correcting the error of the updated data row in the register, sparse correction, long tail correction or deviation correction can be used. For example, when using the sparse correction method for processing, only the registers storing data features are recorded; when using the long tail correction method for processing, the distribution of the maximum value in the register can be analyzed, and the corresponding correction factor can be applied to reduce the error in the case of high cardinality, that is, the accuracy of the estimation is improved by identifying and adjusting the abnormal distribution of the maximum value of the register; when using the deviation correction method for processing, the average deviation under different cardinality is calculated in advance, and these deviation values ​​are applied to the estimation results for correction, thereby reducing the error in the case of small cardinality and improving the accuracy of the algorithm when processing small data sets.

[0115] Furthermore, in the big data processing scenario, each partitioned data set will independently run the HyperLogLog algorithm to obtain its own data statistics. In order to obtain a global cardinality estimate, it is necessary to merge these scattered register results. At this time, all register values ​​can be compared for the maximum value at each position, so that each register position records the maximum register value at the corresponding position of all partitioned data sets, and finally obtain a more accurate cardinality estimate. This embodiment processes the updated data in the register through error correction mechanisms such as sparse correction, long tail correction, and deviation correction, which can significantly improve the estimation accuracy within different cardinality ranges, thereby improving the accuracy of data processing.

[0116] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0117] Example 2

[0118] The embodiment of the present application also provides a data processing device. It should be noted that the data processing device of the embodiment of the present application can be used to execute the data processing method provided by the embodiment of the present application. The data processing device provided by the embodiment of the present application is introduced below.

[0119] According to an embodiment of the present application, a device for implementing the above data processing method is also provided. Figure 3 is a schematic diagram of a data processing device provided according to an embodiment of the present application, such as Figure 3 As shown, the device includes: a first acquisition unit 30, a partitioning unit 31, and a processing unit 32.

[0120] The first acquisition unit 30 is used to acquire the user log data set to be counted stored in the database, and pre-process the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, where N is a positive integer;

[0121] A partitioning unit 31 is used to partition the initial processing data set to obtain M partition data sets, where M is less than N and M is a positive integer;

[0122] The processing unit 32 is used to perform statistical processing on each partition data set using a cardinality statistical algorithm for the M partition data sets to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

[0123] The data processing device provided in the embodiment of the present application obtains the user log data set to be counted stored in the database through the first acquisition unit 30, pre-processes the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, N is a positive integer; the partitioning unit 31 partitions the initial processing data set to obtain M partition data sets, wherein M is less than N and M is a positive integer; the processing unit 32 performs statistical processing on each partition data set using a cardinality statistics algorithm for the M partition data sets, respectively, to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results, thereby solving the technical problems of low statistical efficiency, low accuracy and high resource consumption when performing cardinality statistics on the data to be counted in the related art, by pre-processing the acquired user log data set to obtain an initial processing data set, partitioning the initial processing data set and processing each partition data set using a cardinality statistics algorithm, thereby obtaining corresponding data statistical results, and finally using the data statistical results to perform user behavior analysis, thereby achieving the technical effect of improving the efficiency of cardinality statistics on big data and providing high-precision statistical results under limited resources.

[0124] Optionally, in the data processing device provided in the embodiment of the present application, the partitioning unit 31 includes: a first acquisition module, used to acquire the log field associated with each initial processing data, and obtain N log fields, wherein each log field includes at least one of the following: a time field and a numerical range field; a processing module, used to perform field conversion processing on the N log fields using a conversion function, and obtain N converted log fields; a first grouping module, used to group the N converted log fields according to a preset time period range when the N converted log fields are time fields, and obtain M first field groups; a first combination module, used to combine the initial processing data associated with the time field in each first field group, and obtain M partition data sets; a second grouping module, used to group the N converted log fields according to a preset numerical interval when the N converted log fields are numerical range fields, and obtain M second field groups; a second combination module, used to combine the initial processing data associated with the numerical range field in each second field group, and obtain M partition data sets.

[0125] Optionally, in the data processing device provided in the embodiment of the present application, the partitioning unit 31 includes: a calculation module, used to use a hash function to calculate the hash value of each initial processed data to obtain N data hash values; a third grouping module, used to group the N data hash values ​​using a preset hash value interval to obtain M hash value groups; a third combination module, used to combine the initial processed data associated with the data hash value in each hash value group to obtain M partitioned data sets.

[0126] Optionally, in the data processing device provided in the embodiment of the present application, the device also includes: a second acquisition unit, used to obtain a preset partition threshold for each partition data set before using the cardinality statistical algorithm to perform statistical processing on each partition data set to obtain M data statistical results, and compare the preset partition threshold with the total number of initial processed data associated with the partition data set; a first determination unit, used to determine the hash statistical algorithm as the cardinality statistical algorithm when the total number of initial processed data associated with the partition data set is less than the preset partition threshold; and a second determination unit, used to determine the probability evaluation algorithm as the cardinality statistical algorithm when the number of initial processed data associated with the partition data set is greater than or equal to the preset partition threshold.

[0127] Optionally, in the data processing device provided in the embodiment of the present application, the processing unit 32 includes: a creation module, used to create a preset array for each partitioned data set, wherein the size of the preset array is determined by the range of the initial processed data of the partitioned data set; a mapping module, used to map the initial processed data in the partitioned data set to the preset array using a hash function to obtain a processed array; a first determination module, used to determine the types of different initial processed data in the processed array, and determine the data statistical results according to the different types of initial processed data.

[0128] Optionally, in the data processing device provided in the embodiment of the present application, the processing unit 32 includes: a calculation module, which is used to use a hash function to perform hash calculation on the initial processing data in the partitioned data set to obtain Y data hash values, and determine the binary representation of each data hash value to obtain Y binary data, where Y is a positive integer; a second acquisition module, which is used to obtain L initial registers, use the preset bit in each binary data as a bitmap index, and filter out an initial register from the L initial registers according to the bitmap index of each binary data, where L is less than Y and L is a positive integer; a second determination module, which is used to determine the prefix length of the binary data for a binary data, and compare the prefix length with the data in the initial register corresponding to the binary data, where an initial prefix length is stored in each initial register; an update module, which is used to update the initial prefix length in the initial register using the prefix length when the prefix length is greater than the initial prefix length of the initial register corresponding to the binary data, and obtain updated data in a register; a statistical module, which is used to perform probability statistics on the updated data in the L registers to obtain data statistical results.

[0129] Optionally, in the data processing device provided in the embodiment of the present application, the processing unit 32 includes: a correction module, used to perform error correction on the updated data in the L registers to obtain the corrected data in the L registers, wherein the error correction method includes at least one of the following: sparse correction, long tail correction and deviation correction; a merging module, used to merge the corrected data in the L registers to obtain data statistical results.

[0130] It should be noted that the first acquisition unit 30, the partitioning unit 31, and the processing unit 32 correspond to steps S201 to S203 in Example 1, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned units can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0131] Example 3

[0132] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal or an electronic device.

[0133] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0134] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the data processing method: obtain the user log data set to be counted stored in the database, pre-process the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, N is a positive integer; partition the initial processing data set to obtain M partition data sets, wherein M is less than N, and M is a positive integer; for the M partition data sets, respectively use the cardinality statistical algorithm to perform statistical processing on each partition data set to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

[0135] Optionally, the computer terminal can execute the program code of the following steps in the data processing method: obtaining the log field associated with each initial processing data to obtain N log fields, wherein each log field includes at least one of the following: a time field and a numerical range field; performing field conversion processing on the N log fields using a conversion function to obtain N converted log fields; when the N converted log fields are time fields, grouping the N converted log fields according to a preset time period range to obtain M first field groups; combining the initial processing data associated with the time field in each first field group to obtain M partitioned data sets; when the N converted log fields are numerical range fields, grouping the N converted log fields according to a preset numerical interval to obtain M second field groups; combining the initial processing data associated with the numerical range field in each second field group to obtain M partitioned data sets.

[0136] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the data processing method: using a hash function to calculate the hash value of each initial processed data to obtain N data hash values; using a preset hash value range to group the N data hash values ​​to obtain M hash value groups; combining the initial processed data associated with the data hash value in each hash value group to obtain M partitioned data sets.

[0137] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the data processing method: for each partitioned data set, obtain a preset partition threshold, and compare the preset partition threshold with the total number of initial processed data associated with the partitioned data set; when the total number of initial processed data associated with the partitioned data set is less than the preset partition threshold, determine the hash statistical algorithm as a cardinality statistical algorithm; when the number of initial processed data associated with the partitioned data set is greater than or equal to the preset partition threshold, determine the probability evaluation algorithm as a cardinality statistical algorithm.

[0138] Optionally, the computer terminal may execute the program code of the following steps in the data processing method: for each partitioned data set, create a preset array, wherein the size of the preset array is determined by the range of the initial processed data of the partitioned data set; map the initial processed data in the partitioned data set to the preset array using a hash function to obtain a processed array; determine the types of different initial processed data in the processed array, and determine the data statistical results according to the different types of initial processed data.

[0139] Optionally, the computer terminal can execute the program code of the following steps in the data processing method: use a hash function to perform hash calculation on the initial processed data in the partitioned data set to obtain Y data hash values, and determine the binary representation of each data hash value to obtain Y binary data, where Y is a positive integer; obtain L initial registers, use the preset bit in each binary data as a bitmap index, and filter out an initial register from the L initial registers according to the bitmap index of each binary data, where L is less than Y and L is a positive integer; for a binary data, determine the prefix length of the binary data, and compare the prefix length with the data in the initial register corresponding to the binary data, where an initial prefix length is stored in each initial register; when the prefix length is greater than the initial prefix length of the initial register corresponding to the binary data, use the prefix length to update the initial prefix length in the initial register to obtain updated data in a register; perform probability statistics on the updated data in the L registers to obtain data statistical results.

[0140] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the data processing method: perform error correction on the updated data in L registers to obtain the corrected data in the L registers, wherein the error correction method includes at least one of the following: sparse correction, long tail correction and deviation correction; merge the corrected data in the L registers to obtain data statistical results.

[0141] Optionally, Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (only one is shown) processor 402, memory 404, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0142] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method and device in the embodiment of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned data processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0143] The processor can call the information and application programs stored in the memory through the transmission device to execute the above steps in the above data processing method.

[0144] By adopting the embodiment of the present application, a data processing scheme is provided. By obtaining the user log data set to be counted stored in the database, the user log data set is preprocessed to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, N is a positive integer; the initial processing data set is partitioned to obtain M partition data sets, wherein M is less than N, and M is a positive integer; for the M partition data sets, each of the partition data sets is statistically processed using a cardinality statistics algorithm to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results, thereby achieving the purpose of efficiently, accurately and flexibly performing cardinality statistics on big data, thereby solving the technical problems of low statistical efficiency, low accuracy and high resource consumption when performing cardinality statistics on the data to be counted.

[0145] It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only, and the electronic device may also be a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), a PAD or other terminal device. Figure 4 The structure of the electronic device is not limited. Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations are shown.

[0146] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0147] Example 4

[0148] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0149] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0150] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: obtaining a user log data set to be counted stored in a database, preprocessing the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, where N is a positive integer; partitioning the initial processing data set to obtain M partitioned data sets, where M is less than N and M is a positive integer; for the M partitioned data sets, performing statistical processing on each partitioned data set using a cardinality statistical algorithm to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

[0151] Optionally, the storage medium is also configured to store program code for executing the following steps: obtaining a log field associated with each initial processing data to obtain N log fields, wherein each log field includes at least one of the following: a time field and a numerical range field; performing field conversion processing on the N log fields using a conversion function to obtain N converted log fields; when the N converted log fields are time fields, grouping the N converted log fields according to a preset time period range to obtain M first field groups; combining the initial processing data associated with the time field in each first field group to obtain M partitioned data sets; when the N converted log fields are numerical range fields, grouping the N converted log fields according to a preset numerical interval to obtain M second field groups; combining the initial processing data associated with the numerical range field in each second field group to obtain M partitioned data sets.

[0152] Optionally, the storage medium is also configured to store program code for executing the following steps: using a hash function to calculate the hash value of each initial processed data to obtain N data hash values; using a preset hash value range to group the N data hash values ​​to obtain M hash value groups; and combining the initial processed data associated with the data hash values ​​in each hash value group to obtain M partitioned data sets.

[0153] Optionally, the storage medium is also configured to store program code for performing the following steps: for each partitioned data set, obtaining a preset partition threshold, and comparing the preset partition threshold with the total number of initial processed data associated with the partitioned data set; when the total number of initial processed data associated with the partitioned data set is less than the preset partition threshold, determining the hash statistical algorithm as a cardinality statistical algorithm; when the number of initial processed data associated with the partitioned data set is greater than or equal to the preset partition threshold, determining the probability evaluation algorithm as a cardinality statistical algorithm.

[0154] Optionally, the storage medium is also configured to store program code for performing the following steps: for each partitioned data set, creating a preset array, wherein the size of the preset array is determined by the range of the initial processed data of the partitioned data set; using a hash function to map the initial processed data in the partitioned data set to the preset array to obtain a processed array; determining the types of different initial processed data in the processed array, and determining data statistical results according to the different types of initial processed data.

[0155] Optionally, the storage medium is also configured to store program code for executing the following steps: using a hash function to perform a hash calculation on the initial processed data in the partitioned data set to obtain Y data hash values, and determining the binary representation of each data hash value to obtain Y binary data, where Y is a positive integer; obtaining L initial registers, using the preset bit in each binary data as a bitmap index, and filtering out an initial register from the L initial registers according to the bitmap index of each binary data, where L is less than Y and L is a positive integer; for a binary data, determining the prefix length of the binary data, and comparing the prefix length with the data in the initial register corresponding to the binary data, where an initial prefix length is stored in each initial register; when the prefix length is greater than the initial prefix length of the initial register corresponding to the binary data, using the prefix length to update the initial prefix length in the initial register to obtain updated data in a register; performing probability statistics on the updated data in the L registers to obtain data statistical results.

[0156] Optionally, the storage medium is also configured to store program code for executing the following steps: performing error correction on the updated data in the L registers to obtain corrected data in the L registers, wherein the error correction method includes at least one of the following: sparse correction, long tail correction, and deviation correction; merging the corrected data in the L registers to obtain data statistical results.

[0157] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of the data processing method.

[0158] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0159] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0164] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a user log data set to be counted stored in a database, and preprocess the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, where N is a positive integer; Partitioning the initial processing data set to obtain M partition data sets, where M is less than N and M is a positive integer; For the M partitioned data sets, a cardinality statistical algorithm is used to perform statistical processing on each of the partitioned data sets to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

2. The method according to claim 1, characterized in that The initial processing data set is partitioned to obtain M partitioned data sets including: Obtaining a log field associated with each initial processing data to obtain N log fields, wherein each log field includes at least one of the following: a time field and a value range field; Performing field conversion processing on the N log fields using a conversion function to obtain N converted log fields; In the case where the N converted log fields are time fields, grouping the N converted log fields according to a preset time period range to obtain M first field groups; Combining the initial processed data associated with the time field in each first field group to obtain the M partition data sets; In a case where the N converted log fields are value range fields, grouping the N converted log fields according to a preset value interval to obtain M second field groups; The M partition data sets are obtained by combining the initial processed data associated with the numerical range fields in each second field group.

3. The method according to claim 1, characterized in that The initial processing data set is partitioned to obtain M partitioned data sets including: Calculate the hash value of each initially processed data using a hash function to obtain N data hash values; Grouping the N data hash values ​​using a preset hash value interval to obtain M hash value groups; The M partition data sets are obtained by combining the initial processed data associated with the data hash values ​​in each hash value group.

4. The method according to claim 1, characterized in that: Before performing statistical processing on each partition data set using a cardinality statistical algorithm to obtain M data statistical results, the method further includes: For each partition data set, obtaining a preset partition threshold, and comparing the preset partition threshold with the total number of initially processed data associated with the partition data set; In a case where the total number of initially processed data associated with the partitioned data set is less than the preset partition threshold, determining a hash statistical algorithm as the cardinality statistical algorithm; When the number of initially processed data associated with the partition data set is greater than or equal to the preset partition threshold, the probability evaluation algorithm is determined to be the cardinality statistics algorithm.

5. The method according to claim 4, characterized in that In the case where the cardinality statistical algorithm is the hash statistical algorithm, the cardinality statistical algorithm is used to perform statistical processing on each of the partitioned data sets, and the M data statistical results obtained include: For each partitioned data set, creating a preset array, wherein the size of the preset array is determined by the range of the initial processed data of the partitioned data set; Mapping the initial processed data in the partitioned data set to the preset array using a hash function to obtain a processed array; Determine the types of different initially processed data in the processed array, and determine the data statistical results according to the types of different initially processed data.

6. The method according to claim 4, characterized in that In the case where the cardinality statistical algorithm is the probability evaluation algorithm, the cardinality statistical algorithm is used to perform statistical processing on each of the partitioned data sets, and the M data statistical results obtained include: Performing hash calculation on the initial processed data in the partitioned data set using a hash function to obtain Y data hash values, and determining a binary representation of each data hash value to obtain Y binary data, where Y is a positive integer; Obtain L initial registers, use the preset bit in each binary data as a bitmap index, and filter out an initial register from the L initial registers according to the bitmap index of each binary data, wherein, L is less than Y, and L is a positive integer; For a binary data, determine the prefix length of the binary data, and compare the prefix length with the data in the initial register corresponding to the binary data, wherein each initial register stores an initial prefix length; In a case where the prefix length is greater than an initial prefix length of an initial register corresponding to the binary data, updating the initial prefix length in the initial register using the prefix length to obtain updated data in a register; Probability statistics are performed on the updated data in the L registers to obtain the data statistical results.

7. The method according to claim 6, characterized in that Probability statistics are performed on the updated data in the L registers, and the data statistics results obtained include: Performing error correction on the updated data in the L registers to obtain corrected data in the L registers, wherein the error correction method includes at least one of the following: sparse correction, long tail correction, and deviation correction; The corrected data in the L registers are combined to obtain the data statistical result.

8. A data processing device, characterized in that: include: A first acquisition unit is used to acquire a user log data set to be counted stored in a database, and pre-process the user log data set to obtain an initial processing data set, wherein the initial processing data set includes N initial processing data, where N is a positive integer; A partitioning unit, used to partition the initial processing data set to obtain M partitioned data sets, where M is less than N and M is a positive integer; The processing unit is used to perform statistical processing on each of the M partitioned data sets using a cardinality statistical algorithm to obtain M data statistical results, wherein user behavior analysis is performed based on the M data statistical results.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein the program executes the data processing method described in any one of claims 1 to 7 when running.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the data processing method described in any one of claims 1 to 7 are implemented.

Images