User data processing method, device and server

Abstract

The present application provides a method, device and server for processing user data, which relates to the field of data processing. The method includes: in response to the current user log data collected by the log collection module of the client, extracting information from the current user log data to determine the current user data corresponding to each user; wherein the current user data includes user identification, log generation time and attention information of a specified dimension; determining the new user tags and retained user tags of the current user data based on pre-stored historical user data; determining the new user information and retained user information corresponding to the current user log data based on the current user data containing the user type tag. The present application does not require external triggering, and can obtain the new user information and retained user information in real time, thereby improving the real-time performance of data result processing; at the same time, when there is a lot of log data, there is no need to store massive log data, thereby reducing resource consumption.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device and server for processing user data. Background Art

[0002] With the development of internet technology, analyzing user data for internet products has become increasingly important for attracting new users and adjusting marketing strategies. Related technologies often analyze user log data by extracting it at specified intervals and performing offline batch processing, or by directly writing it to a database and performing calculations during queries. However, this approach doesn't provide timely analysis results, and processing massive amounts of data can also lead to excessive resource consumption due to the limited data-carrying capacity of databases. Summary of the Invention

[0003] The purpose of the present invention is to provide a user data processing method, device and server, which can obtain new user information and retained user information in real time without external triggering, thereby improving the real-time performance of data result processing; at the same time, when there is a lot of log data, there is no need to store massive log data, thereby reducing resource consumption.

[0004] In a first aspect, the present invention provides a method for processing user data, providing application services to a client through a server, and a log collection module is provided in the client; the log collection module is used to collect user log data in real time; the method includes: in response to the current user log data collected by the log collection module of the client, extracting information from the current user log data, and determining the current user data corresponding to each user; wherein the current user data includes a user identifier, a log generation time, and attention information of a specified dimension; determining a user type label of the current user data based on pre-stored historical user data; wherein the user type label includes a new user label and a retained user label; determining the new user information and retained user information corresponding to the current user log data based on the current user data containing the user type label; wherein both the new user information and the retained user information contain attention information of the specified dimension.

[0005] In an optional embodiment, historical user data is stored in the first data table in the form of key-value pairs, and the key-value pair corresponding to the historical user data of each user includes: a key number determined based on the user identifier of the user and the attention information of a specified dimension, and a key value determined based on the log generation time of the historical user data; the step of determining the user type label of the current user data based on the pre-stored historical user data includes: determining a first key-value pair corresponding to the current user data; wherein the first key-value pair includes a first key number determined based on the user identifier of the current user and the attention information of a specified dimension, and a first key value determined based on the log generation time corresponding to the current user data; determining the user type label of the current user data based on the first key-value pair and the key-value pairs of the historical user data in the first data table.

[0006] In an optional embodiment, the step of determining the user type label of the current user data based on the first key-value pair and the key-value pair of historical user data in the first data table includes: storing the current user data in the first data table; searching for historical user data corresponding to the first key number in the first data table, and determining whether there is historical user data corresponding to other key values different from the first key value; if not, determining that the user type label of the current user data is a new user label; if yes, determining that the user type label of the current user data is a retained user label.

[0007] In an optional embodiment, the steps of determining the new user information and retained user information corresponding to the current user log data based on the current user data containing the user type tag include: copying the current user data containing the user type tag multiple times to generate a first data stream and a second data stream respectively; based on the user type tag, diverting the new user information corresponding to the current user log data from the first data stream; and determining whether there is user data with the same user identifier and different log generation time in the second data stream, and if so, determining the current user log data corresponding to the user identifier in the second data stream as the retained user information.

[0008] In an optional embodiment, the second data stream is stored in a second data table in the form of a second key-value pair, and the second data table also includes pre-stored historical user data; the second key-value pair includes a second key-value pair for a newly added user tag and a second key-value pair for a retained user tag; the step of judging whether there is user data with the same user identifier and a different log generation time in the second data stream, and if so, determining the current user log data corresponding to the user identifier in the second data stream as the retained user information, includes: unifying the format of the second key-value pairs corresponding to different user type tags to generate a second data table corresponding to the second key-value pairs in the same format; searching in the second data table based on the second key-value pair whether there is user data with the same user identifier and a different log generation time as in the second data stream, and if so, determining the current user log data corresponding to the user identifier in the second data stream as the retained user information.

[0009] In an optional embodiment, the specified dimension includes multiple different dimensions; the method also includes: aggregating and counting the new user information and / or retained user information according to the attention information of the specified dimension to obtain the number of users corresponding to the attention information of different dimensions.

[0010] In an optional implementation, the method further includes: storing the newly added user information and the retained user information in an external database, and the external database is used for the user to view the user information of the client.

[0011] In a second aspect, the present invention provides a user data processing device, which provides application services to a client through a server, and a log collection module is provided in the client; the log collection module is used to collect user log data in real time; the device includes: an information extraction module, which is used to respond to the current user log data collected by the log collection module of the client, extract information from the current user log data, and determine the current user data corresponding to each user; wherein the current user data includes user identification, log generation time and attention information of a specified dimension; a type determination module, which is used to determine the user type label of the current user data based on pre-stored historical user data; wherein the user type label includes a new user label and a retained user label; a data processing module, which is used to determine the new user information and retained user information corresponding to the current user log data based on the current user data containing the user type label; wherein both the new user information and the retained user information contain attention information of a specified dimension.

[0012] In a third aspect, the present invention provides a server comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods in the aforementioned embodiments.

[0013] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods of the aforementioned implementation methods.

[0014] The user data processing method, device and server provided by the embodiment of the present invention provide application services to a client provided with a log collection module through the server, wherein the log collection module provided by the client is used to collect user log data in real time. The method extracts information from the current user log data in response to the current user log data collected by the log collection module of the client, determines the current user data corresponding to each user including the user identifier, log generation time and information of interest in a specified dimension, then determines the new user tag and retained user tag corresponding to the current user data based on pre-stored historical user data, and finally determines the new user information and retained user information corresponding to the current user log data, both of which contain information of interest in the specified dimension, based on the current user data containing the user type tag. In this way, when the server receives the user log data collected in real time by the client, it will automatically perform information extraction and type information classification processing, thereby obtaining the new user information and retained user information in real time without external triggering, thereby improving the real-time performance of data result processing; at the same time, when the log data is large, there is no need to store massive log data, reducing resource consumption.

[0015] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a method for processing user data provided by an embodiment of the present invention;

[0019] Figure 2A flowchart of determining new user information and retained user information provided by an embodiment of the present invention;

[0020] Figure 3 A flowchart of a specific method for processing user data provided by an embodiment of the present invention;

[0021] Figure 4 A structural diagram of a user data processing device provided by an embodiment of the present invention;

[0022] Figure 5 A structural diagram of a specific user data processing device provided by an embodiment of the present invention;

[0023] Figure 6 A structural diagram of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0025] With the widespread adoption of internet technology, numerous internet products with similar functions have sprung up like mushrooms after rain. Among similar internet products, those with the largest user base enjoy a significant advantage. Therefore, user base is crucial to a product. A product's user base consists of two components: new users and retained users. For new users, product and operations personnel require data to analyze and evaluate user acquisition across various channels to reduce acquisition costs and improve acquisition efficiency. For retained users, product and operations personnel require data to analyze, evaluate, and monitor the changes in retention rates of different types of users, providing appropriate user operations strategies to improve user engagement. Therefore, given the massive volume of internet data, it is crucial to quickly, accurately, and in real time calculate new user acquisition and retention metrics to help product operators adjust user acquisition and marketing strategies in a timely manner.

[0026] To calculate user acquisition and retention metrics, we first need to collect user product usage data. This is usually done by embedding user behavior logs in the product, which are automatically triggered and reported when the user uses the product. By processing and calculating the reported log data, we can obtain the corresponding data metrics. In related technologies, the calculation schemes for data metrics generally adopt the following two methods:

[0027] Method 1: Offline batch computing, based on data warehouse technology. The scheduling system pulls user behavior log data from the previous day every day. Using data warehouse technology, the data is extracted, converted, and loaded multiple times to obtain data indicator calculation results, which are then stored in an external database.

[0028] Method 2: MPP database calculation: Detailed log data is written directly to the database, and calculations are triggered and results are retrieved when queries are made. Detailed user behavior log data is written to the MPP database using a data import tool. When users query data metrics, the database's computing power is leveraged to directly calculate metrics from the detailed data. This is a typical approach that reduces metric calculation time by increasing computer hardware resources.

[0029] However, the above-mentioned offline batch processing solution has the problem of untimely data acquisition, and usually requires calculation to be triggered once a day; while the MPP database calculation solution has the problems of excessive computer resource consumption and limited data carrying capacity, and usually the amount of data supported cannot be too large. Based on this, the embodiments of the present invention provide a user data processing method, device and server, which can obtain new user information and retained user information in real time without external triggering, thereby improving the real-time performance of data result processing; at the same time, when there is a lot of log data, there is no need to store massive log data, which reduces resource consumption.

[0030] To facilitate understanding of this embodiment, we first introduce in detail a method for processing user data disclosed in an embodiment of the present invention. This method provides application services to the client through a server, and a log collection module is provided in the client, wherein the log collection module is used to collect user log data in real time. In one embodiment, a log embedding point can be pre-placed in an application service (such as an application (APP)), which is usually pre-placed in an Internet product, and automatically triggers and reports user log data when the user uses the product.

[0031] See also Figure 1 As shown, the method includes the following steps:

[0032] Step S102 : In response to the current user log data collected by the log collection module of the client, information is extracted from the current user log data to determine the current user data corresponding to each user.

[0033] The above-mentioned current user log data is log data generated in real time based on user usage, such as user registration events, user ID, user name, time when the user logged into the app, user's frequently used addresses, user's frequently used IP addresses, method of downloading the app, etc. In one embodiment, a data format can be pre-set to store the collected current user log data.

[0034] The above-mentioned current user data includes user identification, log generation time and attention information of specified dimensions. Among them, the attention information of the specified dimension can be selectively set according to user needs. Optionally, if you want to analyze which region has a large number of new users, the attention information of the specified dimension can be the region where the user is located; if you want to analyze which registration method has a higher user retention rate, the attention information of the specified dimension can be selected as the user's download method, etc. In actual applications, the attention information of the specified dimension can include one or more, which can be extracted accordingly from the current user data according to actual needs.

[0035] Step S104: determining a user type label of current user data based on pre-stored historical user data.

[0036] Historical user data is extracted from user log data received before the current time. This historical user data also includes the user ID, log generation time, and information about a specific dimension. Because the dimensions of interest may vary at different times, the information about a specific dimension in historical user data may not be exactly the same as that in the current user data.

[0037] The above user type tags include new user tags and retained user tags. By determining the corresponding user tags for the current user data, it is easier to distinguish between new users and retained users.

[0038] Step S106 : determining the newly added user information and the retained user information corresponding to the current user log data based on the current user data including the user type tag.

[0039] The above-mentioned new user information and retained user information both contain attention information of specified dimensions. In one embodiment, the new user information may include the number of new users, the source of new users (such as the method of addition), etc., and the retained user information may include the number of retained users, retention time, retention rate, etc.

[0040] An embodiment of the present invention provides a method for processing user data. The method extracts information from the current user log data collected by a log collection module of a client in response to the current user log data, determines the current user data corresponding to each user including the user identification, log generation time and attention information of a specified dimension, and then determines the new user tag and retained user tag corresponding to the current user data based on pre-stored historical user data. Finally, the new user information and retained user information corresponding to the current user log data, both including attention information of a specified dimension, are determined based on the current user data including the user type tag. Therefore, the new user information and retained user information can be obtained in real time without external triggering, thereby improving the real-time performance of data result processing. At the same time, when there is a lot of log data, there is no need to store massive log data, thereby reducing resource consumption.

[0041] In an optional embodiment, the above-mentioned historical user data is stored in the first data table in the form of a key-value pair, and the key-value pair corresponding to each user's historical user data includes: a key number determined according to the user's user ID and the attention information of the specified dimension, and a key value determined according to the log generation time of the historical user data. In one embodiment, the above-mentioned first data table can be a data table in a Key-Value database, then correspondingly, the key number is the Rowkey composed of the user ID user_id and the attention information dim_info under the specified dimension, and the log generation time log_time is the value of the Rowkey. For example, the key-value pair can be Rowkey = 10001#user region; value = 2021-10-01, that is, the user ID is 10001, the attention information of the specified dimension is the user region, and the log generation time is October 1, 2021.

[0042] Accordingly, when determining the user type label of the current user data based on the pre-stored historical user data, the first key-value pair corresponding to the current user data can be determined first, wherein the first key-value pair includes a first key number determined based on the user identifier of the current user and the attention information of the specified dimension, and a first key value determined based on the log generation time corresponding to the current user data, and then the user type label of the current user data is determined based on the first key-value pair and the key-value pairs of the historical user data in the first data table.

[0043] The above-mentioned first key value is the first key number Rowkey determined by the user identifier corresponding to the current user data and the attention information of the specified dimension, and the first key value value determined by the log generation time of the current user data. Since Rowkey is a key number related to the user identifier, the user identifier in the first key value corresponding to the same user is the same. Therefore, when the current user data corresponding to the same user identifier is received, the current user data can be stored in the storage location for the same user in the data table, that is, multiple versions of user data are stored through Rowkey, which avoids the resource consumption problem caused by excessive data volume and realizes real-time deduplication of data.

[0044] Since the first key number in the first key-value pair includes the user ID, and the first key value is determined by the log generation time of the user data, the time when the same user used the product can be obtained through the first key value. If the data table includes the key value corresponding to the historical user data of the same user before the current moment (that is, the log generation time), then the user type label of the current user data can be determined to be a retained user label; if the data table does not include the key value corresponding to the historical user data before the current moment (that is, the log generation time), then the user type label corresponding to the current user data is a new user label. Determining the user type label corresponding to the current user data through the first key-value pair can improve the efficiency of determining new users and retained users.

[0045] In one embodiment, when determining the user type label of the current user data based on the first key-value pair and the key-value pair of historical user data in the first data table, the current user data can be stored in the first data table and then compared with the historical user data, or the current user data can be directly compared with the historical user data in the first data table.

[0046] (1) Store first and then check: store the current user data in the first data table that has stored historical user data, and search the first data table for historical user data corresponding to the first key number, and then determine whether there is historical user data corresponding to other key values different from the first key value. If not, determine that the user type label of the current user data is a new user label. If so, determine that the user type label of the current user data is a retained user label.

[0047] (2) Direct viewing without storage: Check whether the first key number exists in the first data table storing historical user data. If not, determine that the user type label of the current user data is a new user label. If so, determine that the user type label of the current user data is a retained user label. In specific implementation, the first key number to be searched can be the user ID corresponding to the first key number. If the first key number does not exist in the first data table storing historical user data, it means that the user corresponding to the current user data has not used the Internet product, that is, the user corresponding to the current user data is a new user, and is marked with a new user label. Correspondingly, if the first key number already exists in the first data table storing historical user data, it means that the user corresponding to the current user data has used the Internet product, that is, the user corresponding to the current user data is a retained user, and is marked with a retained user label.

[0048] This method can improve the accuracy of label determination and the flexibility of label determination, providing users with a flexible processing method.

[0049] Furthermore, in order to facilitate the separate processing of newly added user data and retained user data when the data is continuously increasing, and to avoid affecting the calculation accuracy due to delays in the data link, when determining the newly added user information and retained user information corresponding to the current user log data based on the current user data containing the user type tag, refer to Figure 2 As shown, the following steps S202 to S206 may be included:

[0050] Step S202: The current user data including the user type tag is copied multiple times to generate a first data stream and a second data stream, respectively, wherein the data contents included in the first data stream and the second data stream are identical.

[0051] Step S204: based on the user type tag, extract the newly added user information corresponding to the current user log data from the first data stream.

[0052] Step S206 , determining whether there is user data with the same user ID and different log generation time in the second data stream; if so, determining the current user log data corresponding to the user ID in the second data stream as the retained user information.

[0053] For the above step S204, when the newly added user information corresponding to the current user log data is diverted from the first data stream, the newly added user information corresponding to the current user log data corresponding to the newly added user tag can be extracted from the first data stream based on the newly added type tag, or the retained user information corresponding to the current user log data corresponding to the retained user tag can be removed from the first data stream based on the retained type tag to obtain the newly added user information.

[0054] With respect to step S206 above, the second data stream is stored in the second data table in the form of second key-value pairs, which also includes pre-stored historical user data. The second key-value pairs include a second key-value pair for newly added user tags and a second key-value pair for retained user tags. In one example, when writing the second data stream to the second data table, the second data stream can be written in different formats based on the user type tags in the data stream, that is, the second key-value pairs for newly added user tags and the second key-value pairs for retained user tags are written in different formats.

[0055] In one embodiment, the second key-value pair for the newly added user tag may include: a second key number determined by the user ID of the user and the log generation time and a second key value determined by a preset value, wherein the preset value may be a fixed value such as 0, 1, etc., or may be left blank; the second key-value pair for the retained user tag includes: a random number in a specified range (such as 1-1000) as the second key number and the user ID of the user and the second key value determined by the log generation time. From then on, the newly added user tag and the retained user tag have been consumed when the second data stream is written to the second data table, and the newly added user data and the retained user data are represented in the second data table respectively by different second key-value pair formats.

[0056] Furthermore, in the second data table, the retained data is written under random numbers 1-1000, that is, different data for the same user ID can be stored in different versions under the same random number, thereby reducing the resource usage in the data table when there is massive data.

[0057] Since the second key-value pairs in different formats can be used to store new users and retained users, when determining whether there is user data with the same user identifier and different log generation times in the second data stream, if so, the current user log data corresponding to the user identifier in the second data stream is determined as retained user information. In order to facilitate the acquisition of user retention information based on different log generation times, the following steps 3.1 and 3.2 may be further included:

[0058] Step 3.1: Unify the format of the second key-value pairs corresponding to different user type tags to generate a second data table corresponding to the second key-value pairs in the same format. For example, if two data (user_id = 10001, 10002) are received respectively: Data 1: rowkey = 855 value = 10001 # 2021-10-01, Data 2: rowkey = 10002 # 2021-10-02 value = 0, after formatting Data 1 and Data 2, two data with the same format are obtained: Data 1: {user_id = 10001, log_time = 2021-10-01}, Data 2: {user_id = 10002, log_time = 2021-10-02}, so that subsequent data judgment can be performed more quickly and intuitively. Among them, data 1: rowkey=855value=10001#2021-10-01, it can be determined that the user ID is 10001 and the log generation time is 2021-10-01, which is the data of the retained user, that is, it is not the data generated by the user's first use of the product, and data 2: rowkey=10002#2021-10-02value=0, it can be determined that the data with user ID 10002 is the data of the new user, that is, the data generated by the user's first use of the product.

[0059] Step 3.2: Based on the second key-value pair, search in the second data table whether there is user data with the same user ID as the second data stream and a different log generation time. If so, the current user log data corresponding to the user ID in the second data stream is determined as the retained user information. For example, two pieces of data are received, data 1: {user_id=10001, log_time=2021-10-05}, data 2: {user_id=10002, log_time=2021-10-06}. When judging whether data 1 and data 2 are retained user data, you can search in the second data table based on the second key-value pair whether there is user data with the same user ID as the second data stream and different log generation time. If Rowkey=10001#2021-10-02 exists in the second data table, then user 10001 of data 1 is a 3-day retained user. If not, then it is not. If Rowkey=10002#2021-10-03 exists in the second data table, then data 2 is a 3-day retained user. If not, then it is not.

[0060] Furthermore, the retention time of the retained user information can be determined based on the multiple log generation times corresponding to the retained users in the second data table. For example, for the second data stream {user_id=10001, log_time=2021-10-01} and Rowkey=10001#2021-09-27, it can be determined that the retention time of the user with user ID 10001 is 4 days.

[0061] The embodiment of the present invention provides a specific method for processing user data applied to a Key-Value database. Figure 3 As shown in the figure, boxes represent products or data entities, circles represent code implementation, and arrows represent the direction of data flow. In specific implementation, the server needs to use a data collection SDK, a message queue system, a stream computing system, a key-value database system, and a relational database system. In this example, Apache Flink is used as the stream computing system. Specifically, the following steps (1) to (6) are included:

[0062] Step 1: The data collection SDK collects the current user log data corresponding to the embedded logs in the Internet product, and delivers the current user log data obtained in real time to the message queue 1 of the message queue system.

[0063] Step 2: Generate Flink task 1 in the stream computing system, read the current user log data from message queue 1, and extract the user_id, log time, and dim_info information of interest in a specific dimension (such as city, channel, etc.) from the embedded log. For example, the information of interest in a specific dimension could be the number of new users. Depending on the demand scenario, the national number of new users and the number of new users by province could be included. If the same user used the product in Zhejiang and Jiangsu provinces, a new user would be recorded in each province dimension. The user_id and dim_info are encoded and used as the rowkey of the KV database system, with the log time log_time as the value of the rowkey. This data is then written to Hbase table 1 (the first data table mentioned above) of the KV database system. Hbase is a key-value database. In this Hbase table 1, the same rowkey can store multiple versions of data (that is, the same k can have multiple v).

[0064] Step 3: Write Hbase postPut Hook 1 and attach it to Hbasetable 1. Hbase postPut Hook 1 receives the rowkey and value parameters generated in step 2. It searches for all historical versions of data in Hbase table 1 based on the rowkey. If the data and value match, the user is considered a new user. If not, the user is considered an existing user. The new data message is composed of the rowkey and value parameters, along with the new user tag flag1 or the existing user tag flag2, and is sent to message queue 2 of the message queue system.

[0065] Step 4: Generate Flink Task 2 in the stream computing system, extract data from Message Queue 2, and split the data into two data streams (i.e., the first data stream and the second data stream). Filter out old users from the first data stream based on the new and old user flags. A simple summation is performed on the data to calculate the number of new users, and the result is written to Result Data Table 1 in the relational database system. The other data stream (i.e., the second data stream) is written to Hbase Table 2 (i.e., the second data table mentioned above). When the user type tag flag in the data is flag1, the RowKey is set to the encoding of user_id and log_time, and the Value is set to 0. When the user type tag flag in the data is flag2, the RowKey is set to a random integer between 1 and 1000, and the Value is the encoding of user_id and log_time.

[0066] Step 5: Write HBase postPut Hook 2 and attach it to HBase table 2. HBase postPut Hook 2 receives the rowkey and value parameters generated in step 4. If the rowkey is an integer between 1 and 1000, it formats the value and sends it to message queue 3. If the rowkey is not an integer between 1 and 1000, it formats the rowkey and sends it to message queue 3.

[0067] Step 6: Generate Flink task 3 in the stream computing system, read data from message queue 3, map Hbasetable2 into a dimension table, associate the data stream read from message queue 3 with the dimension table, and then count to obtain real-time retained user information (such as the number of retained users and retention time). The calculation results are written to result data table 2 in the relational database. The Hbase table serves as the stream computing data source, and mapping it into a dimension table during calculation is a function provided by the stream computing framework.

[0068] By storing new user information and retained user information in an external database (such as the relational database system in the above example), the readability of the data processing structure can be improved, making it easier for users to view user information on the client.

[0069] Optionally, the designated dimension includes multiple different dimensions. When multiple dimensions are selected, a threshold ratio can be pre-set for each dimension to determine which dimension's user information has a greater impact on user addition or retention. In one embodiment, the method further includes: aggregating and counting new user information and / or retained user information based on the attention information of the designated dimension to obtain the number of users corresponding to the attention information of the different dimensions, and storing the obtained user counts (new user counts or retained user counts) in an external database for easy review and statistics by statisticians.

[0070] In summary, the user data processing method provided by the embodiments of the present invention connects data processing units in series through a stream computing system. The computing process is driven by streaming data messages, eliminating the need for external triggers, and can reduce data latency from days to seconds. Real-time user deduplication is achieved using a KV database and dimensional encoding. At the same data scale, the computer hardware resources required by a KV database are far lower than those of an MPP database, reducing resource consumption.

[0071] In view of the above-mentioned method for processing user data, an embodiment of the present invention provides a user data processing device, which provides application services to the client through a server, and a log collection module is provided in the client; the log collection module is used to collect user log data in real time; see Figure 4 As shown, the device includes the following parts:

[0072] An information extraction module 402 is configured to extract information from the current user log data collected by the client's log collection module to determine the current user data corresponding to each user; the current user data includes a user identifier, log generation time, and information of interest in a specified dimension;

[0073] Type determination module 404, configured to determine a user type tag of current user data based on pre-stored historical user data; wherein the user type tag includes a new user tag and a retained user tag;

[0074] The data processing module 406 is used to determine the new user information and the retained user information corresponding to the current user log data based on the current user data containing the user type tag; wherein both the new user information and the retained user information contain attention information of a specified dimension.

[0075] The user data processing device provided by the embodiment of the present invention automatically performs information extraction and type information classification when the server receives user log data collected in real time by the client, thereby obtaining new user information and retained user information in real time without the need for external triggering, thereby improving the real-time performance of data result processing; at the same time, when there is a large amount of log data, there is no need to store massive log data, thereby reducing resource consumption.

[0076] In some embodiments, the historical user data is stored in the first data table in the form of key-value pairs, where the key-value pair corresponding to each user's historical user data includes: a key number determined based on the user identifier of the user and the attention information of a specified dimension, and a key value determined based on the log generation time of the historical user data;

[0077] The type determination module 404 is further configured to:

[0078] Determine a first key-value pair corresponding to the current user data; wherein the first key-value pair includes a first key number determined based on the user identifier of the current user and the attention information of a specified dimension, and a first key value determined based on the log generation time corresponding to the current user data; determine a user type label for the current user data based on the first key-value pair and the key-value pairs of historical user data in the first data table.

[0079] In some implementations, the type determination module 404 is further configured to:

[0080] The current user data is stored in a first data table; historical user data corresponding to the first key number is searched in the first data table, and it is determined whether there is historical user data corresponding to other key values different from the first key value; if not, the user type label of the current user data is determined to be a new user label; if so, the user type label of the current user data is determined to be a retained user label.

[0081] In some implementations, the data processing module 406 is further configured to:

[0082] The current user data containing the user type tag is copied multiple times to generate a first data stream and a second data stream respectively; based on the user type tag, the newly added user information corresponding to the current user log data is separated from the first data stream; it is determined whether there is user data with the same user identifier and different log generation time in the second data stream. If so, the current user log data corresponding to the user identifier in the second data stream is determined as the retained user information.

[0083] In some embodiments, the second data stream is stored in a second data table in the form of a second key-value pair, and the second data table also includes pre-stored historical user data; the second key-value pair includes a second key-value pair for a newly added user tag and a second key-value pair for a retained user tag;

[0084] The data processing module 406 is further configured to:

[0085] The second key-value pairs corresponding to different user type tags are formatted in a unified manner to generate a second data table corresponding to the second key-value pairs in the same format; based on the second key-value pairs, a search is performed in the second data table to determine whether there is user data with the same user identifier as the second data stream and a different log generation time; if so, the current user log data corresponding to the user identifier in the second data stream is determined as retained user information.

[0086] In some embodiments, the specified dimension includes a plurality of different dimensions;

[0087] The above device also includes:

[0088] The counting module 408 is used to aggregate and count the newly added user information and / or the retained user information according to the attention information of the specified dimension, and obtain the number of users corresponding to the attention information of different dimensions.

[0089] In some embodiments, the above apparatus further comprises:

[0090] The storage module 410 is used to store the newly added user information and the retained user information in an external database. The external database is used for users to view the user information of the client.

[0091] In summary, the embodiment of the present invention provides a specific structural diagram of a user data processing device, see Figure 5 As shown, the apparatus includes an information extraction module 402 , a type determination module 404 , a data processing module 406 , a counting module 408 and a storage module 410 .

[0092] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0093] An embodiment of the present invention provides a server. Specifically, the server includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0094] Figure 6A structural diagram of a server provided in an embodiment of the present invention, wherein the server 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, wherein the processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0095] The memory 61 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 63 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0096] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0097] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0098] The processor 60 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method may be performed by hardware integrated logic circuits or software instructions within the processor 60. The processor 60 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention may be directly executed by a hardware decoding processor or by a combination of hardware and software modules within the decoding processor. The software modules may be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 61 , and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0099] The computer program products of the user data processing methods, devices, and servers provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0100] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In addition, in the description of the embodiments of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0102] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0103] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0104] Finally, it should be noted that the above embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for processing user data, characterized in that: Provide application services to the client through the server, and the client is provided with a log collection module; The log collection module is used to collect user log data in real time; the method includes: In response to the current user log data collected by the log collection module of the client, information is extracted from the current user log data to determine the current user data corresponding to each user; wherein the current user data includes a user identifier, a log generation time, and attention information of a specified dimension; if the current user is an internet user, the attention information of the specified dimension includes a region; Determine a user type label for the current user data based on pre-stored historical user data; wherein the user type label includes a new user label and a retained user label; Determining, based on current user data containing the user type tag, newly added user information containing the attention information of the specified dimension and retained user information containing the attention information of the specified dimension, so as to perform statistics on users corresponding to the attention information of the specified dimension; The step of determining the user type label of the current user data based on pre-stored historical user data includes: Determine a first key number according to the user identifier of the current user and the attention information of the specified dimension, and determine a first key value according to the log generation time corresponding to the current user data; Determining a first key-value pair corresponding to the current user data based on the first key number and the first key value; The first key-value pair is compared with the key-value pair of the historical user data to determine the user type label of the current user data.

2. The method according to claim 1, characterized in that The historical user data is stored in the first data table in the form of key-value pairs. The key-value pairs corresponding to each user's historical user data include: a key number determined based on the user ID of the user and the attention information of the specified dimension, and a key value determined based on the log generation time of the historical user data.

3. The method according to claim 2, characterized in that The step of comparing the first key-value pair with the key-value pairs in the historical user data to determine the user type label of the current user data includes: storing the current user data in a first data table; Searching the first data table for historical user data corresponding to the first key number, and determining whether there is historical user data corresponding to other key values different from the first key value; If not, determining that the user type tag of the current user data is a new user tag; If yes, it is determined that the user type label of the current user data is a retained user label.

4. The method according to claim 1, wherein The step of determining the newly added user information and the retained user information corresponding to the current user log data according to the current user data containing the user type tag includes: Copying the current user data including the user type tag multiple times to generate a first data stream and a second data stream respectively; Based on the user type tag, separating the newly added user information corresponding to the current user log data from the first data stream; It is determined whether there is user data with the same user identifier and different log generation time in the second data stream. If so, the current user log data corresponding to the user identifier in the second data stream is determined as the retained user information.

5. The method according to claim 4, characterized in that The second data stream is stored in a second data table in the form of a second key-value pair, and the second data table also includes pre-stored historical user data; the second key-value pair includes a second key-value pair for a newly added user tag and a second key-value pair for a retained user tag; The step of determining whether there is user data in the second data stream having the same user identifier and attention information of the specified dimension but different log generation times, and if so, determining the current user log data corresponding to the user identifier in the second data stream as retained user information includes: Unifying the formats of the second key-value pairs corresponding to different user type tags to generate a second data table corresponding to the second key-value pairs in the same format; Based on the second key-value pair, the second data table is searched to see whether there is user data with the same user ID as the second data stream and a different log generation time. If so, the current user log data corresponding to the user ID in the second data stream is determined as retained user information.

6. The method according to claim 1 or 4, characterized in that The specified dimension includes a plurality of different dimensions; The method further comprises: The newly added user information and / or the retained user information is aggregated and counted according to the attention information of the specified dimension to obtain the number of users corresponding to the attention information of different dimensions.

7. The method according to claim 1, characterized in that The method further comprises: The newly added user information and the retained user information are stored in an external database, and the external database is used for users to view user information of the client.

8. A user data processing device, characterized in that: Provide application services to the client through the server, and the client is provided with a log collection module; The log collection module is used to collect user log data in real time; the device includes: an information extraction module, configured to extract information from the current user log data collected by the log collection module of the client, and determine current user data corresponding to each user; wherein the current user data includes a user identifier, a log generation time, and information of interest in a specified dimension; if the current user is an internet user, the information of interest in the specified dimension includes region; A type determination module, configured to determine a user type label of the current user data based on pre-stored historical user data; wherein the user type label includes a new user label and a retained user label; a data processing module, configured to determine, based on current user data containing the user type tag, newly added user information containing the attention information of the specified dimension and retained user information containing the attention information of the specified dimension, so as to collect statistics on users corresponding to the attention information of the specified dimension; The type determination module is specifically used to: Determine a first key number according to the user identifier of the current user and the attention information of the specified dimension, and determine a first key value according to the log generation time corresponding to the current user data; Determining a first key-value pair corresponding to the current user data based on the first key number and the first key value; The first key-value pair is compared with the key-value pair of the historical user data to determine the user type label of the current user data.

9. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images