Data processing method and device, storage medium, and electronic device

Abstract

The present disclosure belongs to the field of computer technology, and relates to a data processing method and device, a storage medium, and an electronic device. The method includes: obtaining account information and grouping rules corresponding to the account information, and grouping the account information according to the grouping rules to obtain user data; obtaining user behavior data corresponding to the user data, and performing bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap; obtaining screening conditions for determining the target user, and performing bitmap calculation on the behavior bitmap according to the screening conditions to determine the target user and the number of users of the target user. The present disclosure reduces the storage and computing pressure caused by the rapid growth of data, and greatly improves the storage, processing, transmission and reading performance of data, meeting the requirements of high-timeliness application scenarios; further, it not only supports a variety of query operations, but also has simple query statements, which can maximize the use of computing resources and avoid the waste of computing resources.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method and a data processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] One of the important functions of the enterprise's operating platform is that advertisers and sales can select people to follow and perform operations such as analysis or advertising on the people to achieve brand business growth or other effects. In terms of technical implementation, the operating platform previously used offline computing to select people, which relied heavily on offline computing frameworks such as Hadoop and Spark. The operating platform will convert the user's input criteria into SQL (Structured Query Language) statements that can be executed on the offline computing framework and submit them so that the offline computing framework can perform calculations and output results.

[0003] However, this solution usually takes 15-30 minutes to generate a crowd, and in extreme cases when the task volume surges, it may take more than an hour. It cannot support the query of massive data, and the timeliness and user experience are too poor. In addition, when users select a crowd, they can only know the size of the crowd under the selected conditions. 90% of the selected crowd will not be reached by ads, and the purpose of advertising cannot be achieved, which greatly wastes computing resources.

[0004] In view of this, there is an urgent need in the art to develop a new data processing method and device.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data processing method, a data processing device, a computer-readable storage medium and an electronic device, thereby overcoming the technical problems of poor timeliness and waste of computing resources caused by the limitations of related technologies at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of an embodiment of the present invention, there is provided a data processing method, the method comprising: acquiring account information and a grouping rule corresponding to the account information, and grouping the account information according to the grouping rule to obtain user data;

[0009] Acquire user behavior data corresponding to the user data, and perform bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap;

[0010] A screening condition for determining a target user is obtained, and bitmap calculation is performed on the behavior bitmap according to the screening condition to determine the target user and the number of users of the target user.

[0011] In an exemplary embodiment of the present invention, the grouping the account information according to the grouping rule to obtain the user data includes:

[0012] Determining the number of groups specified in the grouping rule;

[0013] The account information is grouped according to the grouping number to obtain user data.

[0014] In an exemplary embodiment of the present invention, the grouping the account information according to the grouping number to obtain the user data includes:

[0015] Performing account calculation on the account information according to the number of groups to obtain an account calculation result;

[0016] A hash calculation is performed on the account calculation result to obtain a group identifier and user data corresponding to the group identifier.

[0017] In an exemplary embodiment of the present invention, performing bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap includes:

[0018] Performing index configuration processing on the account information to obtain account information with an index;

[0019] Based on the index, the user data is subjected to bitmap representation processing according to the user behavior data to obtain a behavior bitmap.

[0020] In an exemplary embodiment of the present invention, performing bitmap calculation on the behavior bitmap according to the screening condition to determine the target user and the number of users of the target user includes:

[0021] When there are multiple screening conditions, determining that there are multiple behavior bitmaps;

[0022] When the user data is a group, bitmap calculation is performed on a plurality of the behavior bitmaps to determine the target user and the number of users of the target user.

[0023] In an exemplary embodiment of the present invention, performing bitmap calculation on the behavior bitmap according to the screening condition to determine the target user and the number of users of the target user includes:

[0024] When there are multiple screening conditions, determining that there are multiple behavior bitmaps;

[0025] When the user data is in multiple groups, bitmap calculation is performed on the multiple behavior bitmaps according to the grouping identifiers to determine multiple single-condition users and the number of multiple single users of the multiple single-condition users;

[0026] The plurality of single condition users are connected to obtain target users, and the number of the plurality of single users is calculated to obtain the number of target users.

[0027] In an exemplary embodiment of the present invention, the method further comprises:

[0028] When the screening condition is one, determining that the behavior bitmap is one;

[0029] The behavior bitmap is characterized and meaning identified to obtain a target user and the number of the target user.

[0030] According to a second aspect of an embodiment of the present invention, there is provided a data processing device, the device comprising: a group processing module, configured to obtain account information and a grouping rule corresponding to the account information, and group the account information according to the grouping rule to obtain user data;

[0031] A bitmap representation module is configured to obtain user behavior data corresponding to the user data, and perform bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap;

[0032] The user determination module is configured to obtain screening conditions for determining the target user, and perform bitmap calculation on the behavior bitmap according to the screening conditions to determine the target user and the number of users of the target user.

[0033] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor and a memory; wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the data processing method of any of the above exemplary embodiments is implemented.

[0034] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data processing method in any of the above exemplary embodiments is implemented.

[0035] It can be seen from the above technical solutions that the data processing method, data processing device, computer storage medium and electronic device in the exemplary embodiments of the present invention have at least the following advantages and positive effects:

[0036] In the method and device provided by the exemplary embodiments of the present disclosure, on the one hand, the account information is grouped and processed to obtain user data, which solves the problem of data storage while reducing storage consumption, and at least meets the data storage needs of tens of billions in the future. On the other hand, the user data is bitmap represented and processed to obtain a behavior bitmap, which greatly reduces the storage and computing pressure caused by the rapid growth of data, and greatly improves the storage, processing, transmission and reading performance of data, meeting the requirements of high-timeliness application scenarios; further, the behavior bitmap is bitmap calculated to obtain the target user and the number of users, which not only supports a variety of query operations, but also has simple query statements and is easy to understand, and can maximize the use of computing resources and avoid the waste of computing resources.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0039] Figure 1 A flowchart schematically illustrates a data processing method in an exemplary embodiment of the present disclosure;

[0040] Figure 2 A schematic flow chart schematically illustrates a method for performing grouping processing according to grouping rules in an exemplary embodiment of the present disclosure;

[0041] Figure 3 A schematic flow chart schematically illustrates a method for performing group processing according to the number of groups in an exemplary embodiment of the present disclosure;

[0042] Figure 4 A schematic diagram schematically illustrates a flow chart of a method for performing bitmap representation processing on user data in an exemplary embodiment of the present disclosure;

[0043] Figure 5 A schematic flow chart schematically illustrates a method for bitmap calculation in an exemplary embodiment of the present disclosure;

[0044] Figure 6 A flowchart schematically illustrating another method for bitmap calculation in an exemplary embodiment of the present disclosure;

[0045] Figure 7A flowchart schematically illustrating a method for determining a target user and the number of users in an exemplary embodiment of the present disclosure;

[0046] Figure 8 A partial principle diagram schematically showing a data processing method in an application scenario in an exemplary embodiment of the present disclosure;

[0047] Fig. 9 A schematic diagram schematically illustrates a process of querying data in an application scenario in an exemplary embodiment of the present disclosure;

[0048] Fig.10 A schematic diagram schematically illustrates the statement execution of a method for determining a target user and the number of users in an application scenario in an exemplary embodiment of the present disclosure;

[0049] Fig.11 A schematic diagram schematically shows the structure of a data processing device in an exemplary embodiment of the present disclosure;

[0050] Fig.12 An electronic device for implementing a data processing method in an exemplary embodiment of the present disclosure is schematically shown;

[0051] Fig.13 A computer-readable storage medium for implementing a data processing method in an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0052] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0053] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0054] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0055] One of the important functions of the enterprise's operating platform is that advertisers and sales can select people to follow and perform operations such as analyzing or advertising on the people to achieve brand business growth or other effects. In terms of technical implementation, the operating platform previously used offline computing to select people, which relied heavily on offline computing frameworks such as Hadoop and Spark. The operating platform converts the user's input criteria into SQL statements that can be executed on the offline computing framework and submits them so that the offline computing framework can perform calculations and output results.

[0056] However, this solution usually takes 15-30 minutes to generate a crowd, and in extreme cases when the task volume surges, it may take more than an hour. It cannot support the query of massive data, and the timeliness and user experience are too poor. In addition, when users select a crowd, they can only know the size of the crowd under the selected conditions. 90% of the selected crowd will not be reached by ads, and the purpose of advertising cannot be achieved, which greatly wastes computing resources.

[0057] In view of the problems existing in the related art, the present disclosure proposes a data processing method. Figure 1 A flow chart of a data processing method is shown, Figure 1 As shown, the data processing method includes at least the following steps:

[0058] Step S110: Obtain account information and grouping rules corresponding to the account information, and group the account information according to the grouping rules to obtain user data.

[0059] Step S120: Obtain user behavior data corresponding to the user data, and perform bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap.

[0060] Step S130: Obtain screening conditions for determining target users, and perform bitmap calculation on the behavior bitmap according to the screening conditions to determine the target users and the number of target users.

[0061] In the exemplary embodiments of the present disclosure, on the one hand, the account information is grouped to obtain user data, which solves the data storage volume while reducing storage consumption, and at least meets the data storage needs of tens of billions in the future. On the other hand, the user data is bitmap represented to obtain the behavior bitmap, which greatly reduces the storage and computing pressure caused by the rapid growth of data, and greatly improves the storage, processing, transmission and reading performance of the data, meeting the requirements of high-timeliness application scenarios; further, the behavior bitmap is bitmap calculated to obtain the target user and the number of users, which not only supports a variety of query operations, but also has simple query statements and is easy to understand, and can maximize the use of computing resources and avoid the waste of computing resources.

[0062] The following is a detailed description of each step of the data processing method.

[0063] In step S110, the account information and the grouping rule corresponding to the account information are obtained, and the account information is grouped according to the grouping rule to obtain user data.

[0064] In an exemplary embodiment of the present disclosure, the account information may be account information used by a user on an application, or may be other information that uniquely represents the user on the application, and this exemplary embodiment does not specifically limit this.

[0065] The grouping rule is used to group account information so as to achieve the purpose of partitioning and processing account information with a large amount of data.

[0066] In an alternative embodiment, Figure 2 A flow chart of a method for performing grouping processing according to grouping rules is shown, such as Figure 2 As shown, the method at least includes the following steps: In step S210, the number of groups specified in the grouping rule is determined.

[0067] The grouping rule may include a grouping number, that is, the number of groups that the account information is to be divided into according to actual conditions. Generally, the grouping number may be 32, or other numbers, and this exemplary embodiment does not make any special limitation on this.

[0068] In step S220, the account information is grouped according to the grouping number to obtain user data.

[0069] In an alternative embodiment, Figure 3 A schematic flow chart of a method for performing group processing according to the number of groups is shown, such as Figure 3 As shown, the method at least includes the following steps: In step S310, account calculation is performed on the account information according to the number of groups to obtain an account calculation result.

[0070] After the number of groups is determined, account calculation can be performed on the account information according to the number of groups.

[0071] Specifically, the account calculation may be to perform a modulo calculation on the account information according to the number of groups to obtain the account calculation result.

[0072] For example, the account calculation method can refer to Table 1:

[0073]

[0074] Table 1

[0075] Among them, Pin1, Pin2, Pin3, ..., Pin10 represent 10 account information, part_id represents the result obtained by performing modulo calculation on (Pin)%3, and the account calculation result can be the result obtained by storing the account calculation result and the account information together, such as (1, pin1), (2, pin2) or (3, pin3), etc.

[0076] In step S320, a hash calculation is performed on the account calculation result to obtain a group identifier and user data corresponding to the group identifier.

[0077] After the account calculation result is obtained, part_id can be used as the grouping identifier of each group, for example, the grouping identifier can be 1, 2 or 3, etc., which is not specifically limited in this exemplary embodiment. Further, the account information in the account calculation result can also be hashed to obtain the corresponding user data.

[0078] Hash calculation can be achieved through a hash algorithm. A hash algorithm, also known as a hash algorithm, is a function that compresses a message of any length into a message digest of a fixed length. The fixed-length message digest is the hash value, that is, the user data.

[0079] It should be noted that a hash value is a short string of random letters and numbers. Hash algorithms include but are not limited to: MD4 (Message Digest-4), MD5 (Message Digest-5), SHA-1 (Secure Hash Algorithm-1), SHA-256 (Secure Hash Algorithm-256), etc.

[0080] Taking the working principle of the SHA-256 algorithm as an example, inputting any string of data into the SHA-256 algorithm will result in a 256-bit hash value. The same data input can get the same result. However, as long as the input data changes slightly, such as a 1 in the input data changes to a 0, a new result that is very different from the original result will be obtained, and the hash value of the input data calculated by the hash algorithm is unpredictable, which ensures that the process of obtaining the result of an event based on a random number is random. In addition, the hash algorithm has another feature that it is difficult to find a reverse rule, that is, it is extremely difficult to calculate the corresponding input data based on the hash value, which is considered impossible under current technological conditions. Therefore, hash calculation can ensure the security and confidentiality of user account information.

[0081] In this exemplary embodiment, the group identification of the account information and the corresponding user data can be obtained through account calculation and hash calculation, which ensures the security of the user's account information and can also realize group management of massive account information, which not only expands the storage capacity of the account information, but also improves the efficiency of subsequent determination of target users.

[0082] In step S120, user behavior data corresponding to the user data is obtained, and a bitmap representation process is performed on the user data according to the user behavior data to obtain a behavior bitmap.

[0083] In an exemplary embodiment of the present disclosure, user behavior data may be data characterizing user behaviors such as purchasing, browsing, and adding to a shopping cart, and may also include other behavior data in other application scenarios, which is not specifically limited in this exemplary embodiment.

[0084] A brand advertiser wants to run ads before the "Double 11" promotion to boost sales during the promotion. Mainly, the advertiser wants to target the following groups of people: users who have purchased the brand's products during the last two promotions (set A); users who have browsed the brand's products in the last half month (set B); users who have added the brand's products to their shopping carts but have not purchased them in the last half month (set C), and then merge these groups of people for advertising. Specifically, the user behavior data corresponding to the user data can be found in Table 2:

[0085]

[0086] Table 2

[0087] Among them, set A is obtained based on the user's purchasing behavior data, including three people {Pin1, Pin2, Pin3}; set B is obtained based on the user's browsing behavior data, including four people {Pin2, Pin3, Pin4, Pin5}; set C is actually the difference between the user behavior data of adding to the shopping cart (set C1) and the purchasing user behavior data (set C2), including three people {Pin4, Pin5, Pin6} and three people {Pin5, Pin6, Pin7} respectively.

[0088] However, after obtaining the user behavior data, there are many problems in storing it in the manner shown in Table 2.

[0089] First, the amount of data is huge. The amount of newly generated underlying detailed data is in the tens of billions every day, the internal detailed tags are in the tens of millions every day, and the user data of all users is also very large. The excessive amount of data poses a huge challenge to the performance of data storage, processing, transmission and reading.

[0090] Second, the data type is complex. For user behavior data, such as purchase behavior, it supports the selection of continuous labels such as frequency or price within a range, so it is necessary to record each user's purchase frequency and price information. This part of the data requires preprocessing of the underlying detailed data and optimization of the data volume.

[0091] Third, the user selection is complicated. Currently, users are allowed to perform intersection, union, and difference operations on any tag of their choice in any order. However, many user selection combinations can reach 20-30, and each card corresponds to multiple detailed tags at the bottom layer. This will result in the final selection of all tags to obtain thousands of detailed tags for any combination of intersection, union, and difference operations, and the logic may be very complicated.

[0092] Fourth, the query performance requirements are high. Because it is an online business, the crowd computing interface needs to return quickly, and waiting too long will lead to a worse user experience.

[0093] Fifth, high concurrency requirements. Online business means that the number of concurrent queries will not be too low. It supports simultaneous computing by multiple users and requires the stability of application services and backend storage or analysis engines to ensure the stability of external services, especially during promotion periods.

[0094] In order to improve the above problems, bitmap computing can be used to solve them. A bitmap is a very cleverly designed data structure, which consists of a series of bits and uses the bits 0 and 1 to represent two opposing states, such as whether an event occurs or whether an object exists. The operations performed on the bitmap are bitmap operations. Bit operations such as AND, OR, and NOT can be mapped to intersection, union, and difference operations on bitmap operations. In addition, mainstream NoSQL (No Structured Query Language, non-relational) databases and OLAP (On-Line Analytical Processing, online analytical processing) query engines support bitmaps.

[0095] In an alternative embodiment, Figure 4 A schematic diagram of a process for performing bitmap representation processing on user data is shown. Figure 4 As shown, the method at least includes the following steps: In step S410, index configuration processing is performed on the account information to obtain account information with an index.

[0096] Specifically, the index configuration processing of the account information can be to assign an index to each user's account information, record it in an index mapping table, and the recording method is (schema: pin, index) (structure: account information, index). In addition, there can be other ways to store indexes and account information, and this exemplary embodiment does not make special limitations on this.

[0097] When assigning an index, the index may start from 0 and may be considered as a subscript of the bitmap.

[0098] In step S420, based on the index, the user data is subjected to bitmap representation processing according to the user behavior data to obtain a behavior bitmap.

[0099] After assigning an index to the account information, the user data may be bitmap represented according to the user behavior data. In the bitmap representation process, 1 may be used to indicate that a user exists, and 0 may be used to indicate that a user does not exist.

[0100] For details, please refer to Table 3:

[0101]

[0102] Table 3

[0103] Among them, {1,1,1,0,0,0,0,0} means that the users who have purchased in the last two big promotions are users with index 0, index 1 and index 2, that is, users Pin1, Pin2 and Pin3; {0,1,1,1,1,0,0,0} means that the users who have browsed recently are users with index 1, index 2, index 3 and index 4, that is, users Pin2, Pin3, Pin4 and Pin5; {0,0,0,0,1,1,1,0} means that the users who have added items to cart recently are users with index 3, index 4 and index 5, that is, users Pin4, Pin5 and Pin6; {0,0,0,0,0,1,1,1} means that the users who have purchased recently are users with index 4, index 5 and index 6, that is, users Pin5, Pin6 and Pin7. Therefore, {1,1,1,0,0,0,0,0}, {0,1,1,1,1,0,0,0}, {0,0,0,0,1,1,1,0}, and {0,0,0,0,0,1,1,1} are behavior bitmaps obtained according to the four types of behavior data respectively.

[0104] Obviously, bitmaps can solve the problem of data volume. Specifically, a 32-bit behavior bitmap can store 2 to the power of 32 bits, which is about 2.1 billion people, while the data size is only 256MB. If the same size is stored as a string, the data volume is about 30GB, which can theoretically differ by 100 times. In actual projects, a sparse bitmap with a higher compression rate, RoaringBitmap, can also be used for storage. Greenplum can support it through plug-ins. If the behavior bitmap is not used, the amount of new user behavior data added every day exceeds 10TB. After using the behavior bitmap, the amount of new data added every day can be controlled to 500-600GB, which greatly reduces the storage and computing pressure caused by the rapid growth of data, and greatly improves the storage, processing, transmission and reading performance of data.

[0105] Among them, RoaringBitmap is an improvement based on the bitmap structure. It divides an integer into high 16 bits and low 16 bits. The high 16 bits are used as a public key and the low 16 bits are used as a value, so there will be 2 to the power of 16 partition keys. In RoaringBitmap, the key will be stored in a short[] array. If the number stored in the short[N] (N>=0andN<=65535) partition is less than 4096, RoaringBitmap uses ArrayContainer to store data by default; if the amount of data in a single partition is greater than 4096, the Container of the partition will be converted to the bitmapContainer bitmap storage method, that is, one partition key corresponds to an array of long

[1024] . In this way, each element in the long array can store 64 data (binary representation), and one partition key can store 1024*64=2^16 data. It is worth noting that when the amount of data in each partition is less than 4096 (data sparse), all data will be stored in a short array, so bitmap storage is not used in this case.

[0106] In addition, bitmaps can solve complex selection problems. In previous offline crowd selection, HiveSQL and Spark Dataframe were used. The intersection, union, and difference between different groups of people needed to be converted into SQL JOIN (connection) and UNION (merge) operations. In addition, SQL does not support difference set operations, which also need to be converted into JOIN operations. As a set, the behavior bitmap directly supports intersection, union, and difference operations in semantics. The generated query statement structure is relatively simple and easier to understand.

[0107] In addition, behavior bitmaps can also solve query performance issues. Based on the above discussion, the performance advantage of behavior bitmaps is obvious. The performance of bitmap operations is far better than that of operations between ordinary collections. In actual performance tests, the performance gap between the two is more than 10 times.

[0108] In step S130, a screening condition for determining a target user is obtained, and a bitmap calculation is performed on the behavior bitmap according to the screening condition to determine the target user and the number of target users.

[0109] In an exemplary embodiment of the present disclosure, in order to further complete the full selection of target users and determine the advertising population, the screening conditions of brand advertisers can be obtained. Continuing with the above example, the corresponding screening conditions are users who have purchased the brand's products during the last two promotions (set A); users who have browsed the brand's products in the last half month (set B); users who have added the brand's products to the shopping cart in the last half month but have not purchased them (set C), and then these groups of people are merged for advertising delivery.

[0110] After the filtering conditions are determined, bitmap calculation may be performed on the behavior bitmap according to the filtering conditions.

[0111] In an alternative embodiment, Figure 5 A schematic diagram of a method for bitmap calculation is shown, such as Figure 5 As shown, the method at least includes the following steps: In step S510, when there are multiple screening conditions, it is determined that there are multiple behavior bitmaps.

[0112] When the filtering criteria are users who have purchased products of this brand during the last two big promotions (set A); users who have browsed products of this brand in the last half month (set B); users who have added products of this brand to the shopping cart in the last half month but have not purchased them (set C), and then these groups of people are merged for advertising delivery, it can be determined that there are 3 filtering criteria, so the corresponding behavior bitmaps are {1,1,1,0,0,0,0,0}, {0,1,1,1,1,0,0,0}, {0,0,0,0,1,1,1,0} and {0,0,0,0,0,1,1,1}, which is also multiple.

[0113] It is worth noting that there is not a one-to-one correspondence between the filter conditions and the behavior bitmap, so the number of the two is not equal.

[0114] In step S520, when the user data is in multiple groups, bitmap calculation is performed on multiple behavior bitmaps according to the grouping identifiers to determine the number of multiple single-condition users and the number of multiple single users of the multiple single-condition users.

[0115] Furthermore, after the account information is grouped according to the grouping number, the 10 account information of Pin1-Pin10 are divided into three groups of user data. Among them, (1, pin1), (1, pin4), (1, pin7) and (1, pin10) with group identification 1 are the first group of user data; (2, pin2), (2, pin5) and (2, pin8) with group identification 2 are the second group of user data; (3, pin3), (3, pin6) and (3, pin9) with group identification 3 are the third group of user data. At this time, the user data is divided into multiple groups.

[0116] Then, corresponding to the user data, for the user's purchase behavior in the two most recent promotions, the behavior bitmap can also be {1,0,0,0}, {1,0,0} and {1,0,0}; for the user's most recent browsing behavior, the behavior bitmap can also be {0,1,0,0}, {1,1,0} and {1,0,0}; for the user's add-to-cart behavior and most recent purchase behavior, the method is the same and will not be repeated here.

[0117] Obviously, when the user data is in multiple groups, the behavior bitmaps must also be grouped for bitmap calculation.

[0118] Specifically, for the behavior bitmap with group identification 1, the same bitmap calculation as that of the set {A}U{B}U{{C1}-{C2}} is performed to obtain the single conditional user and the corresponding number of single users when the group identification is 1; for the behavior bitmap with group identification 2, the same bitmap calculation as that of the set {A}U{B}U{{C1}-{C2}} is performed to obtain the single conditional user and the corresponding number of single users when the group identification is 2; for the behavior bitmap with group identification 3, the same bitmap calculation as that of the set {A}U{B}U{{C1}-{C2}} is performed to obtain the single conditional user and the corresponding number of single users when the group identification is 3.

[0119] Among them, the bitmap calculation in each group can be calculated in a manner that the intersection calculation of the sets is converted into an AND operation, the union calculation of the sets is converted into an OR operation, and the difference calculation of the sets is converted into a NOT operation. Therefore, the bitmap calculation corresponding to the set {A}U{B}U{{C1}-{C2}} is A&B&(C1-C2).

[0120] In step S530, a connection process is performed on a plurality of single condition users to obtain a target user, and a quantity calculation is performed on the number of the plurality of single users to obtain the number of target users.

[0121] After obtaining multiple groups of single condition users, the multiple groups of single condition users can be connected and JOINed to obtain target users. In addition, the number of multiple groups of single users is summed up to obtain the number of target users.

[0122] It is worth noting that after the set operation is converted into the corresponding bitmap operation, the calculation results of the target users and the number of users are consistent with the calculation results of the set operation.

[0123] In this exemplary embodiment, the number of target users and circled users can be obtained through multiple sets of bitmap calculations and connection processing, the calculation results are accurate and consistent, and the data processing performance is greatly improved in many aspects.

[0124] In addition, when there is a need to determine the target users and the number of users in other situations, the purpose of circle selection and the determination of the population can also be achieved through bitmap calculation.

[0125] In an alternative embodiment, Figure 6 A flow chart of another method for bitmap calculation is shown, such as Figure 6 As shown, the method at least includes the following steps: In step S610, when there are multiple screening conditions, it is determined that there are multiple behavior bitmaps.

[0126] When the filtering criteria are users who have purchased products of this brand during the last two big promotions (set A); users who have browsed products of this brand in the last half month (set B); users who have added products of this brand to the shopping cart in the last half month but have not purchased them (set C), and then these groups of people are merged for advertising delivery, it can be determined that there are 3 filtering criteria, so the corresponding behavior bitmaps are {1,1,1,0,0,0,0,0}, {0,1,1,1,1,0,0,0}, {0,0,0,0,1,1,1,0} and {0,0,0,0,0,1,1,1}, which is also multiple.

[0127] It is worth noting that there is not a one-to-one correspondence between the filter conditions and the behavior bitmap, so the number of the two is not equal.

[0128] In step S620, when the user data is a group, bitmap calculation is performed on a plurality of behavior bitmaps to determine the target user and the number of the target user.

[0129] After the account information is grouped according to the number of groups, the 8 account information of Pin1-Pin8 can be grouped into one group, and the group identifiers are consistent. Therefore, corresponding to the user data, for the user's purchase behavior in the two most recent promotions, there is only one behavior bitmap {1,1,1,0,0,0,0,0}; for the user's most recent browsing behavior, there is only one behavior bitmap {0,1,1,1,1,0,0,0}; for the user's add-to-cart behavior, there is only one behavior bitmap {0,0,0,0,1,1,1,0}; for the user's most recent purchase behavior, there is only one behavior bitmap {0,0,0,0,0,1,1,1}.

[0130] Obviously, the bitmap calculation in this group can be calculated by converting the intersection calculation of the sets into an AND operation, the union calculation of the sets into an OR operation, and the difference calculation of the sets into a NOT operation. Therefore, the bitmap calculation corresponding to the set {A}U{B}U{{C1}-{C2}} is A&B&(C1-C2).

[0131] After performing bitmap calculation on the four behavior bitmaps according to the bitmap calculation of A&B&(C1-C2), the target users and the corresponding number of users can be directly obtained.

[0132] In this exemplary embodiment, for scenarios where the number of users is small but users need to be selected, the target users and the number of users can be determined directly through bitmap calculation, which can quickly select target users in scenarios where the number of users is small, and also simplifies the structure of the query statement, making it easy to understand and use.

[0133] In an alternative embodiment, Figure 7 A flow chart showing a method for determining target users and the number of users is shown in FIG. Figure 7 As shown, the method at least includes the following steps: In step S710, when the screening condition is one, determining that the behavior bitmap is one.

[0134] When the filtering condition is users who have purchased products of this brand during the last two promotions, it can be determined that the filtering condition is 1, so the corresponding behavior bitmap has only one {1,1,1,0,0,0,0,0}.

[0135] In step S720, the behavior bitmap is characterized and meanings are identified to obtain target users and the number of target users.

[0136] After the account information is grouped according to the number of groups, the eight account information Pin1-Pin8 can be grouped together with the same group identifier. Therefore, corresponding to the user data, for the user's purchase behavior in the last two promotions, there is only one behavior bitmap {1,1,1,0,0,0,0,0}.

[0137] Therefore, according to the principle that 1 represents the existence of a user and 0 represents the non-existence of a user in the bitmap representation processing, the representation meaning is identified, and it is determined that the target users at this time are users Pin1, Pin2 and Pin3. In addition, by counting the number of users, it can be obtained that the number of target users is 3.

[0138] Furthermore, after the account information is grouped according to the grouping number, the 10 account information of Pin1-Pin10 are divided into three groups of user data. Among them, (1, pin1), (1, pin4), (1, pin7) and (1, pin10) with group identification 1 are the first group of user data; (2, pin2), (2, pin5) and (2, pin8) with group identification 2 are the second group of user data; (3, pin3), (3, pin6) and (3, pin9) with group identification 3 are the third group of user data. At this time, the user data is divided into multiple groups.

[0139] Then, corresponding to the user data, for the user's purchase behavior in the two most recent promotions, the behavior bitmap can also be {1,0,0,0}, {1,0,0} and {1,0,0}; for the user's most recent browsing behavior, the behavior bitmap can also be {0,1,0,0}, {1,1,0} and {1,0,0}; for the user's add-to-cart behavior and most recent purchase behavior, the method is the same and will not be repeated here.

[0140] In this case, there is no need to perform bitmap calculation on the behavior bitmap. Instead, it is only necessary to identify the representation meanings of multiple behavior bitmaps separately to obtain multiple groups of target users and corresponding multiple numbers of users, and then aggregate the multiple groups of target users to obtain the final target users, and sum up the multiple numbers of users to obtain the final number of users.

[0141] In this exemplary embodiment, when there is only one screening criterion for the target users, the behavior bitmap can be directly processed for characterization and meaning recognition to obtain the corresponding target users and number of users, thereby combining the functions of user selection and target population determination into one, which has better timeliness and greater resource utilization, and avoids waste of resources.

[0142] The data processing method in the embodiment of the present disclosure is described in detail below in conjunction with an application scenario.

[0143] Figure 8 The following diagrams illustrate some of the principles of the data processing method in the application scenario. Figure 8 As shown, first, obtain the account information.

[0144] The account information may be the account information used by the user on the application, which may be Pin1, Pin2, Pin3, Pin4, Pin5, Pin6, etc. In addition, the account information may also be other information that uniquely represents the user on the application, which is not specifically limited in this exemplary embodiment.

[0145] Then, the account information is grouped according to the grouping principle to obtain user data.

[0146] The grouping rule is used to group the account information to achieve the purpose of partitioning and processing the account information with a large amount of data. The grouping rule may include a grouping number, that is, the number of groups into which the account information is to be divided according to actual conditions. Generally, the grouping number can be 3 or other numbers, and this exemplary embodiment does not specifically limit this.

[0147] In actual applications, the number of groups can be 32, which can support nearly 70 billion users and at least meet the business growth in the next 5-10 years, and the data storage capacity can also store at least two years of all user data.

[0148] After the number of groups is determined, account calculation can be performed on the account information according to the number of groups.

[0149] Specifically, the account calculation may be to perform a modulo calculation on the account information according to the number of groups to obtain the account calculation result.

[0150] Specifically, Pin1, Pin2, Pin3, ... represent N account information, part_id represents the result obtained by performing modulo calculation on (Pin)%3, and the account calculation result can be the result obtained by storing the account calculation result and the account information together, such as (1, pin1), (2, pin2) or (3, pin3), etc.

[0151] After obtaining the account calculation result, part_id can be used as the grouping identifier of each group, that is, the partition identifier in the figure. For example, the grouping identifier can be 1, 2 or 3, etc., and this exemplary embodiment does not specifically limit this. Furthermore, the account information in the account calculation result can also be hashed to obtain the corresponding user data. The hash calculation can be implemented by a hash algorithm. The hash algorithm, also known as a hash algorithm, is a function that compresses a message of any length into a message digest of a fixed length. The fixed-length message digest is the hash value, that is, the user data.

[0152] Furthermore, the account information is indexed and configured to obtain the account information with an index.

[0153] Specifically, the index configuration processing of the account information may be to assign an index to each user's account information, corresponding to the user index in the figure. In addition, the index may be recorded in an index mapping table, and the recording method is (schema: pin, index). In addition, there may be other ways to store indexes and account information, which are not specifically limited in this exemplary embodiment.

[0154] When allocating an index, the index may start from 0, which may be regarded as a subscript of the bitmap. In addition, the index may also start from 1, which is not specifically limited in this exemplary embodiment.

[0155] Then, obtain user behavior data.

[0156] Among them, the user account information corresponding to a certain user behavior data of brand 1 may be Pin1, Pin3, Pin4, Pin5, ... The user behavior data may be data representing user behaviors such as purchase, browsing, and adding to a shopping cart, and may also include other behavior data in other application scenarios, which is not specifically limited in this exemplary embodiment.

[0157] Furthermore, a bitmap representation process is performed on the user behavior data to obtain a behavior bitmap.

[0158] When an array is used to store user behavior data, brand 1 can be used as the label name, and the corresponding account information can be stored according to the index based on the group identifier.

[0159] However, there are many problems with this storage method.

[0160] First, the amount of data is huge. The amount of newly generated underlying detailed data is in the tens of billions every day, the internal detailed tags are in the tens of millions every day, and the user data of all users is also very large. The excessive amount of data poses a huge challenge to the performance of data storage, processing, transmission and reading.

[0161] Second, the data type is complex. For user behavior data, such as purchase behavior, it supports the selection of continuous labels such as frequency or price within a range, so it is necessary to record each user's purchase frequency and price information. This part of the data requires preprocessing of the underlying detailed data and optimization of the data volume.

[0162] Third, the user selection is complicated. Currently, users are allowed to perform intersection, union, and difference operations on any tag of their choice in any order. However, many user selection combinations can reach 20-30, and each card corresponds to multiple detailed tags at the bottom layer. This will result in the final selection of all tags to obtain thousands of detailed tags for any combination of intersection, union, and difference operations, and the logic may be very complicated.

[0163] Fourth, the query performance requirements are high. Because it is an online business, the crowd computing interface needs to return quickly, and waiting too long will lead to a worse user experience.

[0164] Fifth, high concurrency requirements. Online business means that the number of concurrent queries will not be too low. It supports simultaneous computing by multiple users and requires the stability of application services and backend storage or analysis engines to ensure the stability of external services, especially during promotion periods.

[0165] In order to improve the above problems, bitmap computing can be used to solve them. Bitmap is a very cleverly designed data structure, which consists of a series of bits and uses the bits 0 and 1 to represent two opposing states, such as whether an event occurs or whether an object exists. The operations performed on the bitmap are bitmap operations. Bit operations such as AND, OR, and NOT can be mapped to intersection, union, and difference operations on bitmap operations. In addition, mainstream NoSQL databases and OLAP query engines support bitmaps.

[0166] After assigning an index to the account information, the user data may be bitmap represented according to the user behavior data. In the bitmap representation process, 1 may be used to indicate that a user exists, and 0 may be used to indicate that a user does not exist.

[0167] Therefore, when stored according to the behavior bitmap, it can be in the form of {1,0,…}, {0,0,1,1,0,…}, {0,0,…,1,…}, …

[0168] For example, {1,1,1,0,0,0,0,0} means that the users who have made purchases in the last two big promotions are users with index 0, index 1, and index 2, that is, users Pin1, Pin2, and Pin3; {0,1,1,1,1,0,0,0} means that the users who have browsed recently are users with index 1, index 2, index 3, and index 4, that is, users Pin2, Pin3, Pin4, and Pin5; {0,0,0,0,1,1,1,0} means that the users who have added items to cart recently are users with index 3, index 4, and index 5, that is, users Pin4, Pin5, and Pin6; {0,0,0,0,0,1,1,1} means that the users who have made purchases recently are users with index 4, index 5, and index 6, that is, users Pin5, Pin6, and Pin7. Therefore, {1,1,1,0,0,0,0,0}, {0,1,1,1,1,0,0,0}, {0,0,0,0,1,1,1,0}, and {0,0,0,0,0,1,1,1} are behavior bitmaps obtained according to the four types of behavior data respectively.

[0169] In addition, when querying user data stored in non-bitmap format, the corresponding behavior bitmap can be dynamically generated to perform bitmap calculations of intersection, union, and difference with other bitmap populations.

[0170] Obviously, bitmaps can solve the problem of data volume. Specifically, a 32-bit behavior bitmap can store 2 to the power of 32 bits, which is about 2.1 billion people, while the data size is only 256MB. If the same size is stored as a string, the data volume is about 30GB, which can theoretically differ by 100 times. In actual projects, a sparse bitmap with a higher compression rate, RoaringBitmap, can also be used for storage. Greenplum can support it through plug-ins. If the behavior bitmap is not used, the amount of new user behavior data added every day exceeds 10TB. After using the behavior bitmap, the amount of new data added every day can be controlled to 500-600GB, which greatly reduces the storage and computing pressure caused by the rapid growth of data, and greatly improves the storage, processing, transmission and reading performance of data.

[0171] Among them, RoaringBitmap is an improvement based on the bitmap structure. It divides an integer into high 16 bits and low 16 bits. The high 16 bits are used as a public key and the low 16 bits are used as a value, so there will be 2 to the power of 16 partition keys. In RoaringBitmap, the key will be stored in a short[] array. If the number stored in the short[N] (N>=0andN<=65535) partition is less than 4096, RoaringBitmap uses ArrayContainer to store data by default; if the amount of data in a single partition is greater than 4096, the Container of the partition will be converted to the bitmapContainer bitmap storage method, that is, one partition key corresponds to an array of long

[1024] . In this way, each element in the long array can store 64 data (binary representation), and one partition key can store 1024*64=2^16 data. It is worth noting that when the amount of data in each partition is less than 4096 (data sparse), all data will be stored in a short array, so bitmap storage is not used in this case.

[0172] In addition, bitmaps can solve complex selection problems. In previous offline crowd selection, HiveSQL and Spark Dataframe were used. The intersection, union, and difference between different groups of people needed to be converted into SQL JOIN and UNION operations. In addition, SQL does not support difference set operations, which also need to be converted into JOIN operations. As a set, the behavior bitmap directly supports intersection, union, and difference operations in semantics, and the generated query statement structure is relatively simple and easier to understand.

[0173] In addition, behavior bitmaps can also solve query performance issues. Based on the above discussion, the performance advantage of behavior bitmaps is obvious. The performance of bitmap operations is far better than that of operations between ordinary collections. In actual performance tests, the performance gap between the two is more than 10 times.

[0174] Fig. 9 The schematic diagram of the process of querying data in the application scenario is shown as follows: Fig. 9 As shown, the brand initiates a query request for purchasing users and browsing users of Brand 1. Moreover, the query request is within the promotion period from November 1, 2019 to November 11, 2019.

[0175] The query service then queries the Greenplum architecture based on the query request.

[0176] Greenplum is mainly composed of three parts: Master node, Segment node, and interconnect. Greenplum master is the entrance to the Greenplum database system, accepting client connections and submitted SQL statements, and distributing the workload to other database instances (segment instances), which store and process data. Greenpluminterconnect is responsible for communication between different PostgreSQL instances. Greenplum segment is an independent PostgreSQL database, and each segment stores a part of the data. Most query processing is done by the segment.

[0177] The Master node does not store any user data, but only controls access to clients and stores metadata of table distribution logic. The Segment node is responsible for data storage and can optimize the distribution key to fully utilize the io performance of the Segment node to expand the io performance of the entire cluster.

[0178] Different storage methods can be used according to data popularity or access patterns. Different data in a table can use different physical storage methods, which can be row storage, column storage, or external table.

[0179] After the query, several groups of people including {b-1, purchase, 1, audience, 11-01}, {b-1, purchase, 10, audience, 11-02}, {b-1, purchase, 20, audience, 11-03}, ..., {b-1, purchase, 32, audience, 11-11} are located in tag 1 (tag1).

[0180] The targeting conditions for these groups can be the purchase behavior of the last two promotions tag="b-1"andbusiness_type="purchase"and((date>='2019-11-01'and date<='2019-11-11')or(date>='2020-06-01'and date<='2020-06-18')); recent browsing behavior tag="b-1"andbusiness_type="browse"and(date>='2020-09-15'and date<='2020-09-30'); recent purchase behavior tag="b-1"andbusiness_type="cart"and(date>='2020-09-15'and date<='2020-09-30'); recent purchase behavior tag="b-1"andbusiness_type="purchase"and(date>='2020-09-15'and date<='2020-09-30'); date<='2020-09-30'). In addition, other positioning conditions may also be used, and this exemplary embodiment does not specifically limit this.

[0181] At this point, it can be determined that there are multiple filtering conditions, so there are also multiple corresponding behavior bitmaps.

[0182] Furthermore, since the group identifiers are 1, 10, 20, ..., 32, etc., it can be determined that the user data also consists of multiple groups.

[0183] Further, bitmap calculation is performed on multiple behavior bitmaps according to the grouping identifier to determine multiple single condition users and multiple single user numbers of multiple single condition users. Then, multiple single condition users are connected to obtain target users, and the number of multiple single users is calculated to obtain the number of target users.

[0184] Fig.10 A statement execution diagram of a method for determining target users and the number of users in an application scenario is shown, such as Fig.10 As shown, for example, after the account information is grouped according to the grouping number, the 10 account information of Pin1-Pin10 are divided into three groups of user data. Among them, (1, pin1), (1, pin4), (1, pin7) and (1, pin10) with group identification 1 are the first group of user data; (2, pin2), (2, pin5) and (2, pin8) with group identification 2 are the second group of user data; (3, pin3), (3, pin6) and (3, pin9) with group identification 3 are the third group of user data. At this time, the user data is multiple groups.

[0185] Then, corresponding to the user data, for the user's purchase behavior in the two most recent promotions, the behavior bitmap can also be {1,0,0,0}, {1,0,0} and {1,0,0}; for the user's most recent browsing behavior, the behavior bitmap can also be {0,1,0,0}, {1,1,0} and {1,0,0}; for the user's add-to-cart behavior and most recent purchase behavior, the method is the same and will not be repeated here.

[0186] Obviously, when the user data is in multiple groups, the behavior bitmaps must also be grouped for bitmap calculation.

[0187] Specifically, for the behavior bitmap with group identification 1, the same bitmap calculation as that of the set {A}U{B}U{{C1}-{C2}} is performed to obtain the single conditional user and the corresponding number of single users when the group identification is 1; for the behavior bitmap with group identification 2, the same bitmap calculation as that of the set {A}U{B}U{{C1}-{C2}} is performed to obtain the single conditional user and the corresponding number of single users when the group identification is 2; for the behavior bitmap with group identification 3, the same bitmap calculation as that of the set {A}U{B}U{{C1}-{C2}} is performed to obtain the single conditional user and the corresponding number of single users when the group identification is 3.

[0188] The counting of the number of single users can be implemented by a cardinality function. In addition, it can also be implemented by other functions, which is not specifically limited in this exemplary embodiment.

[0189] Among them, the bitmap calculation in each group can be calculated in a manner that the intersection calculation of the sets is converted into an AND operation, the union calculation of the sets is converted into an OR operation, and the difference calculation of the sets is converted into a NOT operation. Therefore, the bitmap calculation corresponding to the set {A}U{B}U{{C1}-{C2}} is A&B&(C1-C2).

[0190] After obtaining multiple groups of single condition users, the multiple groups of single condition users can be connected and JOINed to obtain target users. In addition, the number of multiple groups of single users is summed up to obtain the number of target users.

[0191] The sum calculation of the number of single users under multiple groups of single conditions may be implemented by a sum function. In addition, it may also be implemented by other methods, which are not specifically limited in this exemplary embodiment.

[0192] It is worth noting that after the set operation is converted into the corresponding bitmap operation, the calculation results of the target users and the number of users are consistent with the calculation results of the set operation.

[0193] In addition, when the filtering criteria are users who have purchased products of this brand during the last two major promotions (set A); users who have browsed products of this brand in the past half month (set B); users who have added products of this brand to the shopping cart in the past half month but have not purchased them (set C), and then these groups of people are merged for advertising delivery, it can be determined that there are 3 filtering criteria, so the corresponding behavior bitmaps are {1,1,1,0,0,0,0,0}, {0,1,1,1,1,0,0,0}, {0,0,0,0,1,1,1,0} and {0,0,0,0,0,1,1,1}, which is also multiple.

[0194] Furthermore, after the account information is grouped according to the number of groups, the eight account information of Pin1-Pin8 can be grouped together, and the group identifiers are consistent. Therefore, corresponding to the user data, for the user's purchase behavior in the two most recent promotions, there is only one behavior bitmap {1,1,1,0,0,0,0,0}; for the user's most recent browsing behavior, there is only one behavior bitmap {0,1,1,1,1,0,0,0}; for the user's add-to-cart behavior, there is only one behavior bitmap {0,0,0,0,1,1,1,0}; for the user's most recent purchase behavior, there is only one behavior bitmap {0,0,0,0,0,1,1,1}.

[0195] Obviously, the bitmap calculation in this group can be calculated by converting the intersection calculation of the sets into an AND operation, the union calculation of the sets into an OR operation, and the difference calculation of the sets into a NOT operation. Therefore, the bitmap calculation corresponding to the set {A}U{B}U{{C1}-{C2}} is A&B&(C1-C2).

[0196] After performing bitmap calculation on the four behavior bitmaps according to the bitmap calculation of A&B&(C1-C2), the target users and the corresponding number of users can be directly obtained.

[0197] In the application scenarios disclosed in the present invention, on the one hand, the account information is grouped to obtain user data, which solves the problem of data storage while reducing storage consumption, and at least meets the data storage needs of tens of billions in the future. On the other hand, the user data is bitmap represented to obtain the behavior bitmap, which greatly reduces the storage and computing pressure caused by the rapid growth of data, and greatly improves the storage, processing, transmission and reading performance of data, meeting the requirements of high-timeliness application scenarios; further, the behavior bitmap is bitmap calculated to obtain the target user and the number of users, which not only supports a variety of query operations, but also has simple query statements and is easy to understand, and can maximize the use of computing resources and avoid the waste of computing resources.

[0198] Furthermore, in an exemplary embodiment of the present disclosure, a data processing device is also provided. Fig.11 A schematic diagram of the structure of a data processing device is shown in FIG. Fig.11 As shown, the data processing device 1100 may include: a group processing module 1110, a bitmap representation module 1120 and a user determination module 1130. Among them:

[0199] The grouping processing module 1110 is configured to obtain account information and grouping rules corresponding to the account information, and to group the account information according to the grouping rules to obtain user data; the bitmap representation module 1120 is configured to obtain user behavior data corresponding to the user data, and to perform bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap; the user determination module 1130 is configured to obtain filtering conditions for determining the target user, and to perform bitmap calculation on the behavior bitmap according to the filtering conditions to determine the target user and the number of users of the target user.

[0200] In an exemplary embodiment of the present invention, the grouping the account information according to the grouping rule to obtain the user data includes:

[0201] Determining the number of groups specified in the grouping rule;

[0202] The account information is grouped according to the grouping number to obtain user data.

[0203] In an exemplary embodiment of the present invention, the grouping the account information according to the grouping number to obtain the user data includes:

[0204] Performing account calculation on the account information according to the number of groups to obtain an account calculation result;

[0205] A hash calculation is performed on the account calculation result to obtain a group identifier and user data corresponding to the group identifier.

[0206] In an exemplary embodiment of the present invention, performing bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap includes:

[0207] Performing index configuration processing on the account information to obtain account information with an index;

[0208] Based on the index, the user data is subjected to bitmap representation processing according to the user behavior data to obtain a behavior bitmap.

[0209] In an exemplary embodiment of the present invention, performing bitmap calculation on the behavior bitmap according to the screening condition to determine the target user and the number of users of the target user includes:

[0210] When there are multiple screening conditions, determining that there are multiple behavior bitmaps;

[0211] When the user data is a group, bitmap calculation is performed on a plurality of the behavior bitmaps to determine the target user and the number of users of the target user.

[0212] In an exemplary embodiment of the present invention, performing bitmap calculation on the behavior bitmap according to the screening condition to determine the target user and the number of users of the target user includes:

[0213] When there are multiple screening conditions, determining that there are multiple behavior bitmaps;

[0214] When the user data is in multiple groups, bitmap calculation is performed on the multiple behavior bitmaps according to the grouping identifiers to determine multiple single-condition users and the number of multiple single users of the multiple single-condition users;

[0215] The plurality of single condition users are connected to obtain target users, and the number of the plurality of single users is calculated to obtain the number of target users.

[0216] In an exemplary embodiment of the present invention, the method further comprises:

[0217] When the screening condition is one, determining that the behavior bitmap is one;

[0218] The behavior bitmap is characterized and meaning identified to obtain a target user and the number of the target user.

[0219] The specific details of the above-mentioned data processing device 1100 have been described in detail in the corresponding data processing method, so they will not be repeated here.

[0220] It should be noted that, although several modules or units of the data processing device 1100 are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0221] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0222] Refer to the following Fig.12 1200 according to this embodiment of the present invention will be described. Fig.12The electronic device 1200 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0223] like Fig.12 As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include, but are not limited to: the at least one processing unit 1210, the at least one storage unit 1220, a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210), and a display unit 1240.

[0224] The storage unit stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 executes the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0225] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1221 and / or a cache memory unit 1222 , and may further include a read-only memory unit (ROM) 1223 .

[0226] The storage unit 1220 may also include a program / utility 1224 having a set (at least one) of program modules 1225, such program modules 1225 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0227] Bus 1230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0228] The electronic device 1200 may also communicate with one or more external devices 1400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0229] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0230] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0231] refer to Fig.13 As shown, a program product 1300 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0232] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0233] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0234] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0235] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0236] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire account information and a grouping rule corresponding to the account information, and group the account information according to the grouping rule to obtain user data; Acquire user behavior data corresponding to the user data, and perform bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap; Obtain filtering conditions for determining the target user, and perform bitmap calculation on the behavior bitmap according to the filtering conditions to determine the target user and the number of users of the target user, wherein performing bitmap calculation on the behavior bitmap according to the filtering conditions to determine the target user and the number of users of the target user includes: when the filtering conditions are multiple, determining that the behavior bitmap is multiple; when the user data is multiple groups, performing bitmap calculation on the multiple behavior bitmaps according to the grouping identifier to determine multiple single-condition users and the number of multiple single users of the multiple single-condition users; performing connection processing on the multiple single-condition users to obtain the target user, and performing quantity calculation on the number of the multiple single users to obtain the number of users of the target user.

2. The data processing method according to claim 1, characterized in that: The grouping the account information according to the grouping rule to obtain the user data includes: Determining the number of groups specified in the grouping rule; The account information is grouped according to the grouping number to obtain user data.

3. The data processing method according to claim 2, characterized in that: The grouping the account information according to the grouping number to obtain the user data includes: Performing account calculation on the account information according to the number of groups to obtain an account calculation result; A hash calculation is performed on the account calculation result to obtain a group identifier and user data corresponding to the group identifier.

4. The data processing method according to claim 3, characterized in that: The performing bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap includes: Performing index configuration processing on the account information to obtain account information with an index; Based on the index, the user data is subjected to bitmap representation processing according to the user behavior data to obtain a behavior bitmap.

5. The data processing method according to claim 3, characterized in that: The performing bitmap calculation on the behavior bitmap according to the screening condition to determine the target user and the number of users of the target user includes: When there are multiple screening conditions, determining that there are multiple behavior bitmaps; When the user data is a group, bitmap calculation is performed on a plurality of the behavior bitmaps to determine the target user and the number of users of the target user.

6. The data processing method according to claim 1, characterized in that: The method further comprises: When the screening condition is one, determining that the behavior bitmap is one; The behavior bitmap is characterized and meaning identified to obtain a target user and the number of the target user.

7. A data processing device, characterized in that: include: a grouping processing module, configured to obtain account information and a grouping rule corresponding to the account information, and perform grouping processing on the account information according to the grouping rule to obtain user data; A bitmap representation module is configured to obtain user behavior data corresponding to the user data, and perform bitmap representation processing on the user data according to the user behavior data to obtain a behavior bitmap; The user determination module is configured to obtain filtering conditions for determining the target user, and perform bitmap calculation on the behavior bitmap according to the filtering conditions to determine the target user and the number of users of the target user, wherein performing bitmap calculation on the behavior bitmap according to the filtering conditions to determine the target user and the number of users of the target user includes: when the filtering conditions are multiple, determining that the behavior bitmap is multiple; when the user data is multiple groups, performing bitmap calculation on the multiple behavior bitmaps according to the grouping identifier to determine multiple single-condition users and the number of multiple single users of the multiple single-condition users; performing connection processing on the multiple single-condition users to obtain the target user, and performing quantity calculation on the number of the multiple single users to obtain the number of users of the target user.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1 to 6 by executing the executable instructions.

Images