User traffic data retention prediction method and system based on distributed big data computing platform

Abstract

The present application belongs to the technical field of big data processing, and particularly relates to a user traffic data retention prediction method and system based on a distributed big data computing platform. The correlation between daily incremental latest active user data and past N-day active user data is pre-calculated to determine the active users in the past N days and the users retained up to the present, and the active and retained user detail data in the past N days is stored in the daily partition to predict the retained user data information in the future N days. Compared with the traditional user retention calculation method, the present application retains the daily active and retained user details, can support N-day retention index calculation under different dimensions, and significantly improves the scalability and reusability of the retention model. Furthermore, the distributed big data computing platform is used as a data processing platform to solve the bottleneck problem of the traditional transactional database in big data calculation, such as user retention calculation of daily traffic data of at least 10 GB or more.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of big data processing, and particularly relates to a user traffic data retention prediction method and system based on a distributed big data computing platform. BACKGROUND

[0002] With the continuous development of Internet technology, the generation of data has grown in an explosive manner. It is inevitable for enterprises to use data to determine the development state of the business and the user stickiness, especially for game platforms, live broadcast platforms, short video platforms, e-commerce platforms, social network platforms and the like. The owners of the applications are more concerned about user retention and user retention rate. The retention index can reflect the operating state of the application and the degree of user dependence on the application to a certain extent. For the currently popular short video applications and e-commerce applications, the daily access user volume of these applications can reach the level of hundreds of millions. User retention refers to the use of an application by a user for a period of time, and after a period of time, the user continues to use the application. Such a user is called a retained user. The proportion of retained users in the users of the previous time period is called the user retention rate. Commonly used time periods include the next day, seven days, and thirty days. These time periods are often selected to reflect the ability of the application to retain users within a certain period of time.

[0003] The traffic data of the application belongs to the behavior log type data. One of the characteristics of this type of data is that the data volume is large. The user access data volume of a target platform on a single day is at least 10 GB. The performance of the current technology using a traditional OLTP online transaction processing database cannot meet the needs of user retention data analysis. In addition, the traditional data model for calculating user retention can only fixedly calculate the retention rate and the number of retained users for a certain two time periods. This data model has the problems of low expandability, repeated calculation, and waste of computing resources. SUMMARY

[0004] Therefore, the present application provides a user traffic data retention prediction method and system based on a distributed big data computing platform to solve the problems of the traditional transactional database being unable to complete large data volume user retention calculation and having low expandability, repeated calculation, and waste of computing resources in the calculation method.

[0005] According to the design scheme provided by the present application, a user traffic data retention prediction method based on a distributed big data computing platform is provided, which includes the following contents:

[0006] User access data information of a target platform within a preset time period is obtained, the user access data information is uploaded to a distributed big data computing platform, and a preset partition field is used for partition storage in the distributed big data computing platform;

[0007] Updating the data stored in the distributed big data platform partition at a preset frequency within an observation time period;

[0008] According to the specified time period query instruction, the first partition data table of the user accessing the target platform in the specified historical time period and the second partition data table of the user accessing the target platform in the specified recent time period are obtained respectively, and the first partition data table and the second partition data table are associated according to the specified retention business index.

[0009] The data of the second partition data table and the first partition data table are subtracted, the partition field date difference is used as the retention day dimension partition field, and the retention detail data table of the specified time period is generated, and the life cycle of the retention detail data table is set as the specified time period.

[0010] The retention detail data table is used to obtain the user active number at any time node in the specified time period and the user retention number and user retention rate in the corresponding future time period.

[0011] As the user flow data retention prediction method based on the distributed big data computing platform in the application, further, in the user access data information of the target platform in the preset time period, the distributed log collection tool or the data synchronization tool is used to collect the user access data information in the target platform.

[0012] As the user flow data retention prediction method based on the distributed big data computing platform in the application, further, the user access data information obtained includes but is not limited to: user id or device id for identifying business entity field, user access time information, and user access channel.

[0013] As the user flow data retention prediction method based on the distributed big data computing platform in the application, further, in the distributed big data computing platform, a preset partition field is used for partition storage, including: first, for the obtained user access data information, a preset partition field is used for deduplication under the specified business attribute, then the deduplicated user access data information is stored in the distributed big data computing platform by the preset partition field.

[0014] As the user flow data retention prediction method based on the distributed big data computing platform in the application, further, the preset partition field is to use the preset date and / or clock time as the partition field.

[0015] As the user flow data retention prediction method based on the distributed big data computing platform of the present application, further, in the observation time period, the data stored in the distributed big data platform partition is updated at a preset frequency, and the target platform user access data information in the daily and / or hourly time interval is uploaded to the latest partition of the distributed big data platform for storage in an incremental extraction manner.

[0016] As the user flow data retention prediction method based on the distributed big data computing platform of the present application, further, the first partition data table and the second partition data table are associated with the specified retention business indicators, the first partition data table is used as the left table, the second partition data table is used as the right table, and the left association method is used for association processing under the preset specified retention business indicators.

[0017] Further, the present application also provides a user flow data retention prediction system based on a distributed big data computing platform, comprising: a data uploading module, a data updating module, a data association module, a retention processing module and a prediction output module, wherein,

[0018] The data uploading module is used for obtaining the user access data information of the target platform in the preset time period, uploading the user access data information to the distributed big data computing platform, and storing the data in the distributed big data computing platform by using the preset partition field for partition storage;

[0019] The data updating module is used for updating the data stored in the distributed big data platform partition at a preset frequency in the observation time period;

[0020] The data association module is used for obtaining the first partition data table of the user accessing the target platform in the specified historical time period and the second partition data table of the user accessing the target platform in the specified recent time period according to the specified time period query instruction respectively, and associating the first partition data table and the second partition data table with the specified retention business indicators;

[0021] The retention processing module is used for subtracting the data of the second partition data table and the first partition data table, using the partition field date difference as the retention day dimension partition field, and generating the retention detail data table of the specified time period, and setting the life cycle of the retention detail data table as the specified time period;

[0022] The prediction output module is used for obtaining the user active number at any time node in the specified time period and the user retention number and user retention rate in the corresponding future time period by using the retention detail data table.

[0023] The beneficial effects of the present application are:

[0024] The application obtains access data information of a target platform in a preset time period, uses a data processing mode of a big data computing platform, and obtains user retention of the target platform, thereby solving the bottleneck that a traditional transaction database cannot calculate user retention of a large amount of data; since the retention user details data is saved, the data model can be greatly improved in expandability and reusability, the waste of computing resources is effectively reduced, and the application prospect is good. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A user traffic data retention prediction process schematic diagram based on a distributed big data computing platform in the embodiment;

[0026] Figure 2 A user traffic data retention calculation principle schematic diagram based on a distributed big data computing platform in the embodiment. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the application more clear, specific and obvious, the application will be further described in detail below with reference to the drawings and technical scheme.

[0028] The embodiment of the application provides a user traffic data retention prediction method based on a distributed big data computing platform, referring to Figure 1 As shown in the figure, the method comprises the following contents:

[0029] S101, obtaining user access data information of a target platform in a preset time period, uploading the user access data information to a distributed big data computing platform, and storing the user access data information in the distributed big data computing platform by using a preset partition field;

[0030] S102, updating the data stored in the distributed big data platform by partition in a preset frequency in an observation time period;

[0031] S103, obtaining a first partition data table of a user accessing the target platform in a specified historical time period and a second partition data table of a user accessing the target platform in a specified recent time period according to a specified time period query instruction respectively, and associating the first partition data table and the second partition data table according to a specified retention business index;

[0032] S104, subtracting the data of the second partition data table and the first partition data table, using a partition field date difference as a retention day dimension partition field, and generating a retention detail data table of the specified time period, and setting a life cycle of the retention detail data table as the specified time period;

[0033] S105, using the retention detail data table to obtain a user active number at any time node in the specified time period and a user retention number and a user retention rate in a corresponding future time period.

[0034] By acquiring the access data information of the target platform in a preset time period, the data processing mode of the big data computing platform is used to acquire the user retention of the target platform, and the bottleneck that the traditional transaction database cannot perform large data volume calculation of user retention is solved. By saving the detailed data of the retained users, the expandability and reusability of the data model can be greatly improved, and the waste of computing resources can be effectively reduced.

[0035] In the embodiment, the data information of the user accessing the target platform in a preset time period is acquired, such as the data information of the user accessing the target platform in the past month. The access data information at least includes user id (user_id) or device id (device_id), access time (create_time), access terminal (platform), access page (page), etc.

[0036] The target platform described herein can be any Internet service platform, such as a short video platform such as Kuaishou, Douyin, Bilibili, etc., an e-commerce service platform such as Taobao, Jingdong, Pinduoduo, etc., a communication chat application such as Weibo, WeChat, QQ, etc.

[0037] For large data volume log data, a log collection tool can be used, such as a distributed log collection tool, including but not limited to Flume; for log data with low data volume and existing in a transaction database, a data synchronization tool can be used, including but not limited to Datax and Sqoop, to upload data to a distributed big data computing platform.

[0038] In the preferred embodiment, further, a preset partition field is used for partition storage in the distributed big data computing platform, including: first, for the acquired user access data information, a preset partition field is used for deduplication under a specified business attribute, and then the deduplicated user access data information is stored in the distributed big data computing platform in a partitioned manner according to the preset partition field.

[0039] A partition table is created in the distributed big data computing platform, and a date partition is used as the partition field to store daily incremental user access data information; if the data volume is too large, an hour partition field can be added. For example: dt=20221018, hour=01, dt=20221018, hour=02.

[0040] As a preferred embodiment, further, the first partition data table and the second partition data table are associated according to a specified retention business index, the first partition data table is used as the left table, the second partition data table is used as the right table, and a left association method is used to perform association processing under a preset specified retention business index.

[0041] For example Figure 2As shown, a time period to be observed is selected, such as the last 30 days, and this interval determines that the active users and the retained users of any day in this time period and the user retention rate can be observed and used as a target table A, i.e., a first partition data table, in the logical processing; the partition data of the active users of the target platform in the last day is selected as a target table B, i.e., a second partition data table.

[0042] The daily incremental synchronization of the active user data to the partition table of the big data platform is adopted, the N-day active data (A table) and the latest daily active user (B table) are associated in a left-association manner, the association and processing are performed according to a specific retention definition (the retention definition may be various, such as the retention of accessing the same page in two days or the retention of accessing the target platform through the same channel), the difference between the latest active user partition and the N-day active user partition is obtained, the difference is used as a partition field as the dimension of the retention days, and the final detailed data is obtained as a target table three (a retention detailed wide table C table). The active user number in any day in N days and the retained user number in the future N days can be obtained through the retention detailed data wide table (C table). The target table C generated after the processing is used to form a daily scheduling task for data processing according to the observation time period; the partition data is refreshed daily in the observation time period, and the dynamic update of the retention data is realized through this ingenious data preprocessing method. Compared with the traditional user retention calculation method, the daily active and retained user details are retained, the N-day retention index calculation under different dimensions can be supported, the scalability and reusability of the retention model are obviously improved, and the distributed big data computing platform is used as the data processing platform, so that the bottleneck problem of the user retention calculation of the traditional transactional database cannot be solved, such as the daily traffic data of at least 10 GB.

[0043] Further, based on the above method, the embodiment of the present application further provides a user traffic data retention prediction system based on a distributed big data computing platform, comprising: a data uploading module, a data updating module, a data association module, a retention processing module and a prediction output module, wherein,

[0044] The data uploading module is used to obtain the user access data information of the target platform in a preset time period, upload the user access data information to the distributed big data computing platform, and store the data in the distributed big data computing platform in a preset partition field.

[0045] The data updating module is used to update the data stored in the distributed big data platform in a preset time period in a preset frequency.

[0046] The data association module is configured to acquire a first partition data table of the user accessing the target platform in a specified historical time period and a second partition data table of the user accessing the target platform in a specified recent time period according to a specified time period query instruction, and associate the first partition data table and the second partition data table according to a specified retention business index.

[0047] The retention processing module is configured to subtract the data of the second partition data table and the first partition data table, use a partition field date difference value as a retention day dimension partition field, and generate a retention detail data table of the specified time period, and set a life cycle of the retention detail data table as the specified time period.

[0048] The prediction output module is configured to acquire a user active number at any time node in the specified time period and a user retention number and a user retention rate in a corresponding future time period by using the retention detail data table.

[0049] Unless specifically stated otherwise, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0050] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0051] The units and method steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example are generally described in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation does not exceed the scope of the present application.

[0052] The distributed big data computing platform in the embodiment can adopt Maxcompute or an open source Hadoop data platform. The memory includes a physical device for storing information, and is usually a memory that stores information in a digital form on a medium by using an electric, magnetic or optical method. For example, various memories that store information by using an electric method, such as RAM and ROM; various memories that store information by using a magnetic method, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a bubble memory and a U disk; and various memories that store information by using an optical method, such as a CD or a DVD. Of course, there are other memories, such as a quantum memory and a graphene memory. The device composed of the memory, the processor and the computer program is implemented by executing corresponding program instructions in the computer by the processor, and the processor can be loaded with various operating systems, such as a windows operating system, a Linux system, an android system and an iOS system.

[0053] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, and are used to illustrate the technical solutions of the present application, rather than limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features, within the technical scope disclosed by the present application. Such modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for user traffic data retention prediction based on a distributed big data computing platform, characterized in that, The application comprises the following contents: Obtain user access data information of a target platform in a preset time period, upload the user access data information to a distributed big data computing platform, and store the data in the distributed big data computing platform by using a preset partition field; Update the data stored in the distributed big data platform in a preset frequency in an observation time period; Obtain a first partition data table of user access to the target platform in a specified historical time period and a second partition data table of user access to the target platform in a specified recent time period according to a specified time period query instruction, and associate the first partition data table and the second partition data table by using a specified retention business index; Subtract the data of the second partition data table from the data of the first partition data table, use the date difference of the partition field as a retention day dimension partition field, and generate a retention detail data table of the specified time period, set the life cycle of the retention detail data table as the specified time period, and dynamically update the retention data in the retention detail data table by refreshing the partition data of the distributed big data platform every day in the observation time period; Use the retention detail data table to obtain the number of active users at any time node in the specified time period, the number of user retention in the future time period corresponding to any time node, and the user retention rate. 2.The user traffic data retention prediction method based on distributed big data computing platform of claim 1, wherein, Obtain user access data information of a target platform in a preset time period, and collect the user access data information in the target platform by using a distributed log collection tool or a data synchronization tool. 3.The user traffic data retention prediction method based on distributed big data computing platform of claim 1, wherein, The obtained user access data information includes but is not limited to user id or device id for identifying business entity field, user access time information, and user access channel.

4. The method for user traffic data retention prediction based on distributed big data computing platform of claim 1, wherein, Store the obtained user access data information in the distributed big data computing platform by using a preset partition field, which comprises the following steps: first, remove the duplicate data of the obtained user access data information in a specified business attribute by using a preset partition field, and then store the de-duplicated user access data information in the distributed big data computing platform by using a preset partition field.

5. The method for user traffic data retention prediction based on distributed big data computing platform according to claim 1 or 4, characterized in that, The preset partition field uses a preset date and / or clock time as a partition field.

6. The method for user traffic data retention prediction based on distributed big data computing platform of claim 1, wherein, In the observation time period, update the data stored in the distributed big data platform in a preset frequency, use daily and / or hourly as a time interval, and use an incremental extraction method to upload the user access data information of the target platform in the time interval to the latest partition of the distributed big data platform for storage.

7. The method for user traffic data retention prediction based on distributed big data computing platform of claim 1, wherein, In the association of the first partition data table and the second partition data table by using a specified retention business index, the first partition data table is used as a left table, the second partition data table is used as a right table, a left association method is used, and the association is processed under a preset specified retention business index. 8.A user traffic data retention prediction system based on a distributed big data computing platform, characterized in that, The application comprises a data uploading module, a data updating module, a data association module, a retention processing module, and a prediction output module, wherein, The data uploading module is used to obtain user access data information of a target platform in a preset time period, upload the user access data information to a distributed big data computing platform, and store the data in the distributed big data computing platform by using a preset partition field. The data updating module is configured to update the data stored in the distributed big data platform partition at a preset frequency within an observation time period. The data association module is configured to obtain a first partition data table of a user accessing a target platform within a specified historical time period and a second partition data table of the user accessing the target platform within a specified recent time period according to a specified time period query instruction, and associate the first partition data table and the second partition data table according to a specified retention business index. The retention processing module is configured to subtract the data of the first partition data table and the second partition data table, use a partition field date difference as a retention day dimension partition field, and generate a retention detail data table of the specified time period, set a life cycle of the retention detail data table as the specified time period, and dynamically update the retention data in the retention detail data table by refreshing the distributed big data platform partition data every day within the observation time period. The prediction output module is configured to obtain a user active number at any time node within the specified time period and a user retention number and a user retention rate in a future time period corresponding to any time node by using the retention detail data table.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps of any one of claims 1-7. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps of any one of claims 1-7. ​ 10. A computer-readable storage medium, characterized in that, ​

Images