Data processing method and apparatus, storage medium, and device

Abstract

The application discloses a data processing method and device, a storage medium and equipment. The method comprises the following steps: acquiring data shards corresponding to each label crowd input by a front-end application from a pre-constructed database; generating each query statement based on metadata in each data shard, and running each query statement in the database to obtain each bitmap set; performing calculation on each bitmap set according to a calculation mode input by the front-end application to obtain a target bitmap set; determining a behavior path analysis result based on the target bitmap set, and sending the behavior path analysis result to the front-end application. The method uses the calculation between bitmap sets to replace network IO operations, reduces the consumption of computing resources, and the efficiency of the calculation between bitmap sets is much higher than that of network IO operations, so that the processing efficiency of label crowd data can be improved, the time required for the behavior path analysis process is reduced, and the data processing efficiency of the behavior path analysis is effectively improved.

Description

Technical Field

[0001] This application relates to the field of big data, and more particularly to a data processing method, apparatus, storage medium, and device. Background Technology

[0002] Traditional user behavior path analysis, when filtering out the corresponding groups of people based on different scenarios, usually requires correlation calculation between the behavioral data of users in the group and the labeled groups in order to obtain the user behavior path analysis results.

[0003] Currently, in the process of performing user behavior path analysis, network I / O operations are required on user behavior data and tag audience data. Due to the large amount of tag audience data, the number of network I / O operations increases, and a lot of time is spent processing the tag audience data (i.e. audience selection), which greatly reduces the data processing efficiency of behavior path analysis. Summary of the Invention

[0004] This application provides a data processing method, apparatus, storage medium, and device for improving the data processing efficiency of the behavior path analysis process.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A data processing method, comprising:

[0007] Retrieve data shards corresponding to each tagged user group input by the front-end application from a pre-built database; the data shards are used to store tag data and metadata corresponding to the tagged user group; the tag data includes a bitmap corresponding to each user ID shown in the tag group; the bitmap is generated based on the hash value corresponding to the user ID; the hash value is generated by calling a preset hash function;

[0008] Based on the metadata in each of the data shards, each query statement is generated and executed in the database to obtain each bitmap set; the query statement is an SQL statement used to query the bitmaps contained in the tag data;

[0009] Based on the calculation method input by the front-end application, calculations are performed on each of the bitmap sets to obtain the target bitmap set;

[0010] Based on the target bitmap set, the behavior path analysis results are determined and sent to the front-end application.

[0011] Optionally, the database construction process includes:

[0012] Obtain the metadata of each of the aforementioned tagged user groups in the business system; the tagged user group includes a set of user IDs; the set of user IDs includes multiple user IDs that meet the filtering rules of the tagged user group; the metadata includes the definition of the tagged user group;

[0013] A preset hash function is invoked to generate a hash value corresponding to each user ID;

[0014] A bitmap corresponding to each user ID is generated based on the hash value corresponding to each user ID;

[0015] Based on each of the tagged groups and the bitmap corresponding to each user ID in the user ID set shown for each tagged group, generate tag data corresponding to each of the tagged groups;

[0016] Each of the aforementioned tag data and the metadata corresponding to each of the aforementioned tag data are saved to each data segment;

[0017] The database is constructed based on each of the aforementioned data shards.

[0018] Optional, also includes:

[0019] Acquire various behavioral events of the business system; each behavioral event includes at least a user ID.

[0020] A preset hash function is invoked to generate a hash value corresponding to the user ID shown in each of the aforementioned behavioral events;

[0021] Based on each behavioral event and the hash value corresponding to the user ID shown by each behavioral event, behavioral data corresponding to each behavioral event is generated;

[0022] The behavioral data of the user IDs in the user ID set shown by the tag data is saved to the data fragment that stores the tag data.

[0023] Optionally, each tagged audience input by the front-end application includes a first tagged audience and a second tagged audience;

[0024] The step of retrieving data shards corresponding to each tagged user group input by the front-end application from a pre-built database includes:

[0025] Obtain a first data shard and a second data shard from a pre-built database; the first data shard is a data shard containing first tag data; the first tag data is tag data corresponding to the first tag population; the second data shard is a data shard containing second tag data; the second tag data is tag data corresponding to the second tag population.

[0026] Optionally, the step of generating various query statements based on the metadata in each of the data shards, and running each query statement in the database to obtain various bitmap sets includes:

[0027] Based on the metadata in the first data shard, a first query statement is generated and executed to obtain a first bitmap set; the first query statement is an SQL statement used to query the bitmaps contained in the first tag data; the first bitmap set includes multiple bitmaps contained in the first tag data.

[0028] Based on the metadata in the second data shard, a second query statement is generated and executed to obtain a second bitmap set; the second query statement is an SQL statement used to query the bitmaps contained in the second tag data; the second bitmap set includes multiple bitmaps contained in the second tag data.

[0029] Optionally, the step of calculating each bitmap set according to the calculation method input by the front-end application to obtain the target bitmap set includes:

[0030] Based on the calculation method, the first bitmap set and the second bitmap set are calculated to obtain the target bitmap set.

[0031] Optionally, the data shards are also used to store behavioral data; the behavioral data includes user IDs;

[0032] The step of determining the behavior path analysis result based on the target bitmap set and sending the behavior path analysis result to the front-end application includes:

[0033] Multiple target data shards are obtained from the database; each target data shard contains target behavior data; each target behavior data contains target user IDs; and each target user ID corresponds to a user ID shown in the target bitmap set.

[0034] Behavioral path analysis is performed on the label data and behavioral data in each target data segment to obtain the behavioral path analysis results for each target data segment.

[0035] A preset proxy service is invoked to summarize and process the analysis results of each behavior path to obtain the target behavior path analysis result, and then the target behavior path analysis result is sent to the front-end application.

[0036] A data processing apparatus, comprising:

[0037] The data shard acquisition unit is used to acquire data shards corresponding to each tagged user group input by the front-end application from a pre-built database; the data shards are used to store tag data corresponding to the tagged user group and metadata of the tagged user group; the tag data includes a bitmap corresponding to each user ID shown in the tagged user group; the bitmap is generated based on the hash value corresponding to the user ID; the hash value is generated by calling a preset hash function;

[0038] The bitmap acquisition unit is used to generate various query statements based on the metadata in each of the data fragments, and to run each query statement in the database to obtain various bitmap sets; the query statement is an SQL statement used to query the bitmaps contained in the tag data;

[0039] The set calculation unit is used to perform calculations on each of the bitmap sets according to the calculation method input by the front-end application to obtain the target bitmap set;

[0040] The path analysis unit is used to send the user ID corresponding to each bitmap in the target bitmap set to the front-end application.

[0041] A computer-readable storage medium includes a stored program, wherein the program executes the data processing method described above.

[0042] A data processing device includes: a processor, a memory, and a bus; the processor and the memory are connected via the bus.

[0043] The memory is used to store a program, and the processor is used to run the program, wherein the program executes the data processing method when it runs.

[0044] The technical solution provided in this application retrieves data fragments corresponding to each tagged user group input by the front-end application from a pre-built database. Based on the metadata in each data fragment, various query statements are generated and executed in the database to obtain various bitmap sets. According to the calculation method input by the front-end application, calculations are performed on each bitmap set to obtain a target bitmap set. Based on the target bitmap set, the behavior path analysis results are determined and sent to the front-end application. This application utilizes calculations between bitmap sets to replace network I / O operations, reducing the consumption of computing resources. Furthermore, the efficiency of calculations between bitmap sets is far higher than that of network I / O operations, which can improve the processing efficiency of tagged user group data and reduce the time required for the behavior path analysis process, thereby effectively improving the data processing efficiency of behavior path analysis. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1a A flowchart illustrating a data processing method provided in an embodiment of this application;

[0047] Figure 1b A flowchart illustrating a data processing method provided in an embodiment of this application;

[0048] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of the architecture of a data processing device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] like Figure 1a and Figure 1b The diagram shown is a flowchart illustrating a data processing method provided in an embodiment of this application, including the following steps:

[0052] S101: Obtain the metadata of each behavioral event, each tagged user group, and each tagged user group from the business system.

[0053] Among them, behavioral events include, but are not limited to: event code, user ID, and occurrence time. The event code represents the order of the behavioral events, the user ID represents the user who performed the behavioral event, and the occurrence time represents when the behavioral event occurred.

[0054] Specifically, assuming the event code of the behavior event is 100, the user ID is 001, and the occurrence time is 12 o'clock, then the behavior event represents the 100th behavior event that occurred in the business system within a preset time period, and the executor of the behavior event is the user with user ID 001, and the behavior event specifically occurred at 12 o'clock.

[0055] A tagged audience includes a tag value, audience code, audience name, audience type, snapshot date, and a set of user IDs. The tag value represents user attributes (such as age, gender, education level, etc.), the audience code represents the threshold corresponding to the user attribute, the audience name represents the naming of the audience, the audience type represents the name of the filtering rules for the tagged audience (the filtering rules are composed of tag values, audience codes, and time ranges), the snapshot date represents the time range, and the set of user IDs includes multiple user IDs that meet the filtering rules for the tagged audience.

[0056] Specifically, assuming the tag value of the tagged audience is age, the audience code is 18-30, the audience name is young people, the audience type is teenagers, and the snapshot date is xxxx year xx month xx day, then the screening rule for the teenagers is: users who registered before xxxx year xx month xx day and whose age is 18-30.

[0057] It should be noted that the metadata of the tagged audience includes the definition of the tagged audience.

[0058] S102: Call the preset hash function to generate a hash value corresponding to the user ID shown in each behavior event.

[0059] The preset hash function includes, but is not limited to, the ciytHash64 function. The hash value generated by calling the ciytHash64 function can be a Uint64 value (i.e., a 64-bit integer hash value).

[0060] S103: Generate behavioral data corresponding to each behavioral event based on each behavioral event and the hash value corresponding to the user ID shown in each behavioral event.

[0061] S104: Generate a bitmap corresponding to each user ID based on the hash value corresponding to each user ID.

[0062] The specific principles of bitmap usage are common knowledge familiar to those skilled in the art. Specifically, in this application embodiment, the Roaringbitmap algorithm can be used to generate a bitmap corresponding to each user ID.

[0063] S105: Generate tag data corresponding to each tag group based on each tag group and the bitmap corresponding to each user ID in the user ID set shown for each tag group.

[0064] S106: Save each tag data and the metadata corresponding to each tag data to each data shard, and build a database based on each data shard.

[0065] Each data shard stores only one tag data and the corresponding metadata.

[0066] It should be noted that the database can specifically be a ClickHouse database, and the data shards that make up the ClickHouse database can specifically be shards. Generally speaking, the physical storage of the ClickHouse database is carried by replicas, and shards are mainly used to implement horizontal partitioning of tag data.

[0067] S107: Save the behavioral data of the user IDs in the user ID set containing the tag data to the data fragment containing the tag data.

[0068] S108: Parse the audience selection command sent by the front-end application to obtain the set of tagged audiences and the calculation method corresponding to the set of tagged audiences.

[0069] The tagged audience set includes the first-tag audience and the second-tag audience. The calculation methods include, but are not limited to, intersection calculation, union calculation, and difference calculation.

[0070] S109: Retrieve the first data shard and the second data shard from the database.

[0071] The first data shard contains data with a first tag, which is the tag data corresponding to the first-tag population. The second data shard contains data with a second tag, which is the tag data corresponding to the second-tag population.

[0072] S110: Based on the metadata in the first data shard, generate the first query statement and run the first query statement to obtain the first graph set.

[0073] The first query statement is an SQL statement used to query the bitmaps contained in the first tag data. The first bitmap set includes multiple bitmaps contained in the first tag data.

[0074] S111: Based on the metadata in the second data shard, generate a second query statement and run the second query statement to obtain the second bitmap set.

[0075] The second query statement is an SQL statement used to query the bitmaps contained in the second tag data. The second bitmap set includes multiple bitmaps contained in the second tag data.

[0076] It should be noted that S110 and S111 can be executed concurrently.

[0077] S112: Based on the calculation method, calculate the first bitmap set and the second bitmap set to obtain the target bitmap set.

[0078] S113: Send the user ID corresponding to each bitmap in the target bitmap set to the front-end application.

[0079] S114: After receiving the behavior path analysis command sent by the front-end application, retrieve multiple target data shards from the database.

[0080] Among them, the target data shard is a data shard containing target behavior data, the target behavior data is behavior data containing target user ID, and the target user ID is the user ID corresponding to the bitmap shown in the target bitmap set.

[0081] S115: Perform behavioral path analysis on the label data and behavioral data in each target data segment to obtain the behavioral path analysis results for each target data segment.

[0082] S116: Call the preset proxy service to summarize and process the analysis results of each behavior path, obtain the target behavior path analysis results, and send the target behavior path analysis results to the front-end application.

[0083] In particular, calling the proxy service to summarize and process the analysis results of each behavior path can avoid consuming too many computing resources for behavior path analysis, thus effectively improving the efficiency of behavior path analysis.

[0084] Based on the processes shown in S101-S116 above, this embodiment can achieve the following beneficial effects:

[0085] 1. Compared with existing user ID storage methods, this embodiment uses bitmap storage to store the user IDs contained in the tagged population, which greatly saves storage costs.

[0086] 2. Compared with existing crowd selection methods, this embodiment calculates the target bitmap set by calculating the first bitmap set and the second bitmap set. Based on the operation characteristics of bitmaps, it can be seen that the crowd selection process will not consume too much computing resources, and the processing speed of crowd selection will not decrease due to the growth of tag data, and will still maintain high processing efficiency, thereby ensuring that the performance of the cluster will not be consumed.

[0087] 3. Compared with existing data storage methods, this embodiment uses data fragmentation to store tag data and behavior data, which can greatly save memory space and speed up the processing efficiency of behavior path analysis.

[0088] 4. One-click selection and processing of tagged audiences: When different scenarios require tagged data of different dates, the corresponding snapshot date can be selected arbitrarily under the tagged audience without the need for SQL reassembly and development, greatly enhancing scalability.

[0089] In summary, this embodiment utilizes computation between bitmap sets to replace network I / O operations, reducing the consumption of computing resources. Furthermore, the efficiency of computation between bitmap sets is far higher than that of network I / O operations, which can improve the processing efficiency of tagged audience data and reduce the time required for behavioral path analysis, thereby effectively improving the data processing efficiency of behavioral path analysis.

[0090] It should be noted that S101 mentioned in the above embodiments is an optional implementation of the data processing method described in this application. Furthermore, S116 mentioned in the above embodiments is also an optional implementation of the data processing method described in this application. Therefore, the process shown in the above embodiments can be summarized as follows: Figure 2 The method described.

[0091] like Figure 2 The diagram shown is a flowchart illustrating another data processing method provided in this application, including the following steps:

[0092] S201: Retrieve data shards from a pre-built database that correspond to each tagged user group input by the front-end application.

[0093] The data shards are used to store tag data and metadata corresponding to the tagged population. The tag data includes a bitmap corresponding to each user ID shown in the tag population. The bitmap is generated based on the hash value corresponding to the user ID. The hash value is generated by calling a preset hash function.

[0094] S202: Based on the metadata in each data shard, generate each query statement and run each query statement in the database to obtain each bitmap set.

[0095] The query statement is an SQL statement used to query the bitmap contained in the tag data.

[0096] S203: Based on the calculation method input by the front-end application, perform calculations on each bitmap set to obtain the target bitmap set.

[0097] S204: Based on the target bitmap set, determine the behavior path analysis results and send the behavior path analysis results to the front-end application.

[0098] In summary, this embodiment utilizes computation between bitmap sets to replace network I / O operations, reducing the consumption of computing resources. Furthermore, the efficiency of computation between bitmap sets is far higher than that of network I / O operations, which can improve the processing efficiency of tagged audience data and reduce the time required for behavioral path analysis, thereby effectively improving the data processing efficiency of behavioral path analysis.

[0099] Corresponding to the data processing method provided in the above embodiments of this application, the embodiments of this application also provide a data processing device.

[0100] like Figure 3 The diagram shown is a schematic representation of the architecture of a data processing device provided in an embodiment of this application, comprising:

[0101] The shard acquisition unit 100 is used to acquire data shards corresponding to each tagged group of people input by the front-end application from a pre-built database. The data shards are used to store the tag data and metadata of the tag group. The tag data includes a bitmap corresponding to each user ID shown in the tag group. The bitmap is generated based on the hash value corresponding to the user ID. The hash value is generated by calling a preset hash function.

[0102] The database construction process includes: acquiring the metadata of each tagged user group from the business system; the tagged user group includes a set of user IDs; the user ID set includes multiple user IDs that meet the filtering rules of the tagged user group; the metadata includes the definition of the tagged user group; calling a preset hash function to generate a hash value corresponding to each user ID; generating a bitmap corresponding to each user ID based on the hash value corresponding to each user ID; generating tag data corresponding to each tagged user group based on each tagged user group and the bitmap corresponding to each user ID in the set of user IDs shown for each tagged user group; saving each tag data and the metadata corresponding to each tag data to each data shard; and constructing the database based on each data shard.

[0103] The database construction process also includes: obtaining various behavioral events of the business system; behavioral events include at least user IDs; calling a preset hash function to generate a hash value corresponding to the user ID shown in each behavioral event; generating behavioral data corresponding to each behavioral event based on each behavioral event and the hash value corresponding to the user ID shown in each behavioral event; and saving the behavioral data containing user IDs from the set of user IDs shown in the tag data to the data shards that store the tag data.

[0104] In addition, each tag audience input by the front-end application includes a first tag audience and a second tag audience.

[0105] The shard acquisition unit 100 is specifically used to: acquire a first data shard and a second data shard from a pre-built database; the first data shard is a data shard containing first tag data; the first tag data is tag data corresponding to the first tag population; the second data shard is a data shard containing second tag data; the second tag data is tag data corresponding to the second tag population.

[0106] Bitmap acquisition unit 200 is used to generate various query statements based on the metadata in each data shard, and run each query statement in the database to obtain various bitmap sets; the query statement is an SQL statement used to query the bitmaps contained in the tag data.

[0107] The bitmap acquisition unit 200 is specifically used for: generating a first query statement based on the metadata in the first data segment, and running the first query statement to obtain a first bitmap set; the first query statement is an SQL statement used to query the bitmaps contained in the first tag data; the first bitmap set includes multiple bitmaps contained in the first tag data; generating a second query statement based on the metadata in the second data segment, and running the second query statement to obtain a second bitmap set; the second query statement is an SQL statement used to query the bitmaps contained in the second tag data; the second bitmap set includes multiple bitmaps contained in the second tag data.

[0108] The set calculation unit 300 is used to perform calculations on each bitmap set according to the calculation method input by the front-end application to obtain the target bitmap set.

[0109] The set calculation unit 300 is specifically used to: calculate the first bitmap set and the second bitmap set according to the calculation method input by the front-end application to obtain the target bitmap set.

[0110] The path analysis unit 400 is used to determine the behavior path analysis results based on the target bitmap set and send the behavior path analysis results to the front-end application.

[0111] The path analysis unit 400 is specifically used for: retrieving multiple target data shards from the database; the target data shards are data shards containing target behavioral data; the target behavioral data are behavioral data containing target user IDs; the target user IDs are user IDs corresponding to the bitmaps shown in the target bitmap set; performing behavioral path analysis on the tag data and behavioral data in each target data shard to obtain the behavioral path analysis results for each target data shard; calling a preset proxy service to aggregate and process the behavioral path analysis results of each shard to obtain the target behavioral path analysis results, and sending the target behavioral path analysis results to the front-end application.

[0112] In summary, this embodiment utilizes computation between bitmap sets to replace network I / O operations, reducing the consumption of computing resources. Furthermore, the efficiency of computation between bitmap sets is far higher than that of network I / O operations, which can improve the processing efficiency of tagged audience data and reduce the time required for behavioral path analysis, thereby effectively improving the data processing efficiency of behavioral path analysis.

[0113] This application also provides a computer-readable storage medium including a stored program, wherein the program executes the data processing method provided in this application.

[0114] This application also provides a data processing device, including a processor, a memory, and a bus. The processor and the memory are connected via the bus. The memory is used to store a program, and the processor is used to run the program. When the program runs, it executes the data processing method provided in this application, including the following steps:

[0115] Retrieve data shards corresponding to each tagged user group input by the front-end application from a pre-built database; the data shards are used to store tag data and metadata corresponding to the tagged user group; the tag data includes a bitmap corresponding to each user ID shown in the tag group; the bitmap is generated based on the hash value corresponding to the user ID; the hash value is generated by calling a preset hash function;

[0116] Based on the metadata in each of the data shards, each query statement is generated and executed in the database to obtain each bitmap set; the query statement is an SQL statement used to query the bitmaps contained in the tag data;

[0117] Based on the calculation method input by the front-end application, calculations are performed on each of the bitmap sets to obtain the target bitmap set;

[0118] Based on the target bitmap set, the behavior path analysis results are determined and sent to the front-end application.

[0119] Specifically, based on the above embodiments, the database construction process includes:

[0120] Obtain the metadata of each of the aforementioned tagged user groups in the business system; the tagged user group includes a set of user IDs; the set of user IDs includes multiple user IDs that meet the filtering rules of the tagged user group; the metadata includes the definition of the tagged user group;

[0121] A preset hash function is invoked to generate a hash value corresponding to each user ID;

[0122] A bitmap corresponding to each user ID is generated based on the hash value corresponding to each user ID;

[0123] Based on each of the tagged groups and the bitmap corresponding to each user ID in the user ID set shown for each tagged group, generate tag data corresponding to each of the tagged groups;

[0124] Each of the aforementioned tag data and the metadata corresponding to each of the aforementioned tag data are saved to each data segment;

[0125] The database is constructed based on each of the aforementioned data shards.

[0126] Specifically, based on the above embodiments, it also includes:

[0127] Acquire various behavioral events of the business system; each behavioral event includes at least a user ID.

[0128] A preset hash function is invoked to generate a hash value corresponding to the user ID shown in each of the aforementioned behavioral events;

[0129] Based on each behavioral event and the hash value corresponding to the user ID shown by each behavioral event, behavioral data corresponding to each behavioral event is generated;

[0130] The behavioral data of the user IDs in the user ID set shown by the tag data is saved to the data fragment that stores the tag data.

[0131] Specifically, based on the above embodiments, each tagged group input by the front-end application includes a first tagged group and a second tagged group;

[0132] The step of retrieving data shards corresponding to each tagged user group input by the front-end application from a pre-built database includes:

[0133] Obtain a first data shard and a second data shard from a pre-built database; the first data shard is a data shard containing first tag data; the first tag data is tag data corresponding to the first tag population; the second data shard is a data shard containing second tag data; the second tag data is tag data corresponding to the second tag population.

[0134] Specifically, based on the above embodiments, the step of generating various query statements based on the metadata in each data shard, and running each query statement in the database to obtain various bitmap sets includes:

[0135] Based on the metadata in the first data shard, a first query statement is generated and executed to obtain a first bitmap set; the first query statement is an SQL statement used to query the bitmaps contained in the first tag data; the first bitmap set includes multiple bitmaps contained in the first tag data.

[0136] Based on the metadata in the second data shard, a second query statement is generated and executed to obtain a second bitmap set; the second query statement is an SQL statement used to query the bitmaps contained in the second tag data; the second bitmap set includes multiple bitmaps contained in the second tag data.

[0137] Specifically, based on the above embodiments, the step of calculating each bitmap set according to the calculation method input by the front-end application to obtain the target bitmap set includes:

[0138] Based on the calculation method, the first bitmap set and the second bitmap set are calculated to obtain the target bitmap set.

[0139] Specifically, based on the above embodiments, the data sharding is also used to store behavioral data; the behavioral data includes user IDs;

[0140] The step of determining the behavior path analysis result based on the target bitmap set and sending the behavior path analysis result to the front-end application includes:

[0141] Multiple target data shards are obtained from the database; each target data shard contains target behavior data; each target behavior data contains target user IDs; and each target user ID corresponds to a user ID shown in the target bitmap set.

[0142] Behavioral path analysis is performed on the label data and behavioral data in each target data segment to obtain the behavioral path analysis results for each target data segment.

[0143] A preset proxy service is invoked to summarize and process the analysis results of each behavior path to obtain the target behavior path analysis result, and then the target behavior path analysis result is sent to the front-end application.

[0144] If the functions described in the methods of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computing device readable storage medium. Based on this understanding, the parts of this application that contribute to the prior art or the technical solutions can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, mobile computing device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0145] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining, from a pre-constructed database, data shards corresponding to each tag population input by a front-end application; the data shards are used to save tag data corresponding to the tag populations and metadata of the tag populations; the tag data comprises a bitmap corresponding to each user ID of the tag population; the bitmap is generated based on a hash value corresponding to the user ID; the hash value is generated by calling a preset hash function; generating each query statement based on the metadata in each data shard and running each query statement in the database to obtain each bitmap set; the query statement is an SQL statement used to query the bitmap contained in the tag data; performing calculation on each bitmap set in the database according to a calculation method input by the front-end application to obtain a target bitmap set, so as to replace IO operation with bitmap calculation and reduce consumption of computing resources; determining a behavior path analysis result based on the target bitmap set and sending the behavior path analysis result to the front-end application.

2. The method of claim 1, wherein, The construction process of the database comprises the following steps: obtaining each tag population of a business system and metadata of each tag population; the tag population comprises a user ID set; the user ID set comprises a plurality of user IDs meeting a filtering rule of the tag population; the metadata comprises a definition of the tag population; calling a preset hash function to generate a hash value corresponding to each user ID; generating a bitmap corresponding to each user ID based on the hash value corresponding to each user ID; generating tag data corresponding to each tag population based on each tag population and the bitmap corresponding to each user ID in the user ID set of each tag population; saving each tag data and metadata corresponding to each tag data in each data shard; constructing the database based on each data shard.

3. The method of claim 2, wherein, The method further comprises the following steps: obtaining each behavior event of the business system; the behavior event at least comprises a user ID; calling a preset hash function to generate a hash value corresponding to the user ID of each behavior event; generating behavior data corresponding to each behavior event based on each behavior event and the hash value corresponding to the user ID of each behavior event; saving behavior data containing a user ID in the user ID set of the tag data in the data shard storing the tag data.

4. The method of claim 2, wherein, Each tag population input by the front-end application comprises a first tag population and a second tag population; the step of obtaining, from a pre-constructed database, data shards corresponding to each tag population input by a front-end application comprises the following steps: obtaining a first data shard and a second data shard from a pre-constructed database; the first data shard is a data shard containing first tag data; the first tag data is tag data corresponding to the first tag population; the second data shard is a data shard containing second tag data; the second tag data is tag data corresponding to the second tag population.

5. The method of claim 4, wherein, The generating each query statement based on the metadata in each of the data shards and running each of the query statements in the database to obtain each bitmap set comprises: generating a first query statement based on the metadata in the first data shard and running the first query statement to obtain a first bitmap set; the first query statement is an SQL statement for querying bitmaps contained in the first label data; the first bitmap set comprises a plurality of bitmaps contained in the first label data; generating a second query statement based on the metadata in the second data shard and running the second query statement to obtain a second bitmap set; the second query statement is an SQL statement for querying bitmaps contained in the second label data; the second bitmap set comprises a plurality of bitmaps contained in the second label data.

6. The method of claim 5, wherein, The calculating each of the bitmap sets according to the calculation manner input by the front-end application to obtain a target bitmap set comprises: calculating the first bitmap set and the second bitmap set according to the calculation manner to obtain a target bitmap set.

7. The method of claim 1, wherein, The data shard is also used to save behavior data; the behavior data comprises a user ID; The determining a behavior path analysis result based on the target bitmap set and sending the behavior path analysis result to the front-end application comprises: obtaining a plurality of target data shards from the database; the target data shard is a data shard containing target behavior data; the target behavior data is behavior data containing a target user ID; the target user ID is a user ID corresponding to a bitmap in the target bitmap set; performing behavior path analysis on label data and behavior data in each of the target data shards to obtain a behavior path analysis result of each of the target data shards; calling a preset proxy service to aggregate each of the behavior path analysis results to obtain a target behavior path analysis result and sending the target behavior path analysis result to the front-end application.

8. A data processing apparatus, characterized by, Comprise: a shard obtaining unit, configured to obtain, from a pre-constructed database, a data shard corresponding to each label crowd input by a front-end application; The data shard is used to save label data corresponding to the label crowd and metadata of the label crowd; the label data comprises a bitmap corresponding to each user ID indicated by the label crowd; The bitmap is generated based on a hash value corresponding to the user ID; the hash value is generated by calling a preset hash function; a bitmap obtaining unit, configured to generate each query statement based on the metadata in each of the data shards and run each of the query statements in the database to obtain each bitmap set; The query statement is an SQL statement for querying bitmaps contained in the label data; a set calculating unit, configured to calculate each of the bitmap sets according to a calculation manner input by the front-end application in the database to obtain a target bitmap set, so as to replace IO operation by bitmap calculation and reduce consumption of computing resources. A path analysis unit is configured to determine a behavior path analysis result based on the target bitmap set and send the behavior path analysis result to the front-end application.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program performs the data processing method of any one of claims 1-7.

10. A data processing device, characterized by Comprise: a processor, a memory and a bus; the processor is connected with the memory through the bus; the memory is used for storing a program, and the processor is used for running the program, wherein the program performs the data processing method of any one of claims 1-7 when running.

Images