A behavior data time processing and saving method and device and related medium

Abstract

The application discloses a kind of behavior data time processing and saving method, device and related medium, the method includes: through log server collection user log information and carry out disk landing;Receive the user behavior data on the log server, and carry out abnormal time processing;Receive the user behavior data in the data warehouse, carry out data drift processing to the user behavior data;The event header file corresponding to the user behavior data is parsed, corresponding user behavior data generation time is obtained, and data storage is carried out based on the user behavior data generation time according to preset time period.The application solves the data deviation caused by time zone difference and time anomaly of user behavior data by unifying data time zone, removing abnormal time, solving cross-time drift, so as to reduce the error of user behavior analysis caused by abnormal data, improve user behavior analysis accuracy.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a method, apparatus and related medium for processing and storing behavioral data over time. Background Technology

[0002] Currently, with globalization, the number of users targeting software products has exploded, and user distribution has become increasingly diversified. Most software companies collect legitimate user behavior data to enhance their products. However, considering that users are distributed across different time zones, time discrepancies can cause data interference. Therefore, eliminating this interference to improve the accuracy of collecting legitimate user behavior data is a problem that those skilled in the art need to solve. Summary of the Invention

[0003] This invention provides a method, apparatus, computer device, and storage medium for processing and storing behavioral data over time, aiming to reduce errors in user behavior analysis caused by abnormal data and improve the accuracy of user behavior analysis.

[0004] In a first aspect, embodiments of the present invention provide a method for processing and storing behavioral data over time, including:

[0005] User log information is collected through a log server and written to disk; wherein, the user log information includes multiple user behavior data and the corresponding user behavior data generation time, and the user behavior data generation time is a timestamp identifier in UTC time zone;

[0006] The system receives user behavior data from the log server and performs abnormal time processing. The abnormal time processing process includes: determining whether the time when the corresponding user behavior data was generated is within the start and end time points of a preset time period; if so, storing the user behavior data in a data warehouse; if not, discarding the corresponding user behavior data directly.

[0007] Receive user behavior data from the data warehouse and perform data drift processing on the user behavior data; wherein, the data drift processing process includes: adding the generation time of the user behavior data to the corresponding event header file;

[0008] The event header file corresponding to the user behavior data is parsed to obtain the corresponding user behavior data generation time, and the data is stored according to a preset time period based on the user behavior data generation time.

[0009] Secondly, an embodiment of the present invention provides a behavioral data time processing and storage apparatus, comprising:

[0010] The collection and disk writing unit is used to collect user log information through the log server and write it to disk; wherein, the user log information includes multiple user behavior data and the corresponding user behavior data generation time, and the user behavior data generation time is a timestamp identifier in UTC time zone;

[0011] An exception handling unit is used to receive user behavior data from the log server and perform exception time processing; wherein, the exception time processing process includes: determining whether the generation time of the corresponding user behavior data is within the start and end time points of a preset time period; if yes, storing the user behavior data in a data warehouse; if no, directly discarding the corresponding user behavior data.

[0012] A drift processing unit is used to receive user behavior data from the data warehouse and perform data drift processing on the user behavior data; wherein, the data drift processing process includes: adding the generation time of the user behavior data to the corresponding event header file;

[0013] The file parsing unit is used to parse the event header file corresponding to the user behavior data, obtain the corresponding user behavior data generation time, and store the data according to a preset time period based on the user behavior data generation time.

[0014] Thirdly, according to an embodiment of the present invention, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the behavioral data time processing and storage method as described in the first aspect.

[0015] Fourthly, according to an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the behavioral data time processing and storage method as described in the first aspect.

[0016] This invention provides a method, apparatus, computer device, and storage medium for processing and storing behavioral data based on time. The method includes: collecting user log information through a log server and writing it to disk; wherein the user log information includes multiple pieces of user behavior data and corresponding user behavior data generation times, the user behavior data generation times being timestamps in UTC time zones; receiving user behavior data from the log server and performing abnormal time processing; wherein the abnormal time processing process includes: determining whether the corresponding user behavior data generation time is within the start and end time points of a preset time period; if yes, storing the user behavior data in a data warehouse; if no, directly discarding the corresponding user behavior data; receiving user behavior data from the data warehouse and performing data drift processing on the user behavior data; wherein the data drift processing process includes: adding the user behavior data generation time to a corresponding event header file; parsing the event header file corresponding to the user behavior data to obtain the corresponding user behavior data generation time, and storing the data based on the user behavior data generation time according to a preset time period. This invention addresses data deviations caused by time zone differences and time anomalies in user behavior data by unifying data time zones, removing abnormal times, and resolving cross-time drift. This reduces errors in user behavior analysis caused by abnormal data and improves the accuracy of user behavior analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for processing and storing behavioral data over time, provided in an embodiment of the present invention;

[0019] Figure 2 A network flowchart of a behavioral data time processing and storage method provided in an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a sub-process of a behavioral data time processing and storage method provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic block diagram of a behavioral data time processing and storage device provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0024] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] Please see below. Figure 1 , Figure 1 The flowchart of a behavior data time processing and storage method provided in an embodiment of the present invention specifically includes steps S101 to S104.

[0027] S101. Collect user log information through a log server and write it to disk; wherein, the user log information includes multiple user behavior data and the corresponding user behavior data generation time, and the user behavior data generation time is a timestamp identifier in UTC time zone;

[0028] S102. Receive user behavior data from the log server and perform abnormal time processing; wherein, the abnormal time processing process includes: determining whether the corresponding user behavior data generation time is within the start and end time points of a preset time period; if yes, store the user behavior data in a data warehouse; if no, discard the corresponding user behavior data directly.

[0029] S103. Receive user behavior data from the data warehouse and perform data drift processing on the user behavior data; wherein, the data drift processing process includes: adding the generation time of the user behavior data to the corresponding event header file;

[0030] S104. Parse the event header file corresponding to the user behavior data, obtain the corresponding user behavior data generation time, and store the data according to a preset time period based on the user behavior data generation time.

[0031] In this embodiment, combined with Figure 2 First, user log information, including user behavior data and its corresponding generation time, is collected through a log server. Then, based on the relationship between the generation time of the user behavior data and the start and end times of a preset time period, anomaly processing is performed on the user log information to remove outliers. Next, the user behavior data with outliers removed undergoes data shifting to add it to an event header file. Finally, the event header file is parsed to obtain the generation time of the user behavior data, and the data is saved based on this generation time.

[0032] This embodiment addresses data deviations caused by different time zones and time anomalies in user behavior data by unifying data time zones, removing abnormal times, and resolving cross-time drift. This reduces errors in user behavior analysis caused by abnormal data and improves the accuracy of user behavior analysis.

[0033] In one embodiment, the step of collecting user log information through a log server and writing it to disk includes:

[0034] The system receives user behavior data and the corresponding user behavior data generation time through a written log receiving interface, and performs IO operations accordingly.

[0035] The user behavior data is generated and the time is stored in the time field to obtain user log information, which is then written to the log server.

[0036] In this embodiment, at the source of user behavior data generation, the user behavior data is identified by a timestamp in the UTC time zone, thereby solving the problem of collecting user behavior data in different time zones. When different users use the product, the collection time is recorded based on the device's uptime. It should be noted that in practical applications, some users collect their personal behavior data through cracked or outdated software. Although the reporting time of the collected data is unified in the time zone, the actual behavior time can still have significant deviations. User behavior data is actually a series of log entries. Therefore, this embodiment uses a log server to write a log receiving interface to obtain user log information and performs I / O operations after receiving the user log information to write it to disk. Here, to prevent network latency from causing data time errors, this embodiment uses the occurrence time of user behavior data as the data time and defines it as a ts field (i.e., the time field). That is, the log entry representing user behavior data contains two parts: the specific behavior and the ts. Preferably, the log entry format is uniformly JSON, which facilitates subsequent data processing.

[0037] In one embodiment, receiving user behavior data from the log server in step S102 includes:

[0038] Receive user behavior data from the log server via Flume;

[0039] The event body of Flume is parsed to obtain the time field, and the time when the user behavior data was generated is obtained based on the time field.

[0040] The abnormal time handling process in step S102 further includes:

[0041] Determine whether the time of generation of the corresponding user behavior data meets the following conditions:

[0042] systemTime > todayTime and eventTime is less than todayTime, where systemTime is the current system time, todayTime is the starting time of the preset time period, and eventTime is the time when the user behavior data was generated.

[0043] If the conditions are not met, the user behavior data is retained; if the conditions are met, the corresponding user behavior data is discarded.

[0044] Specifically, the start and end time points of the current preset time period include a start time point and an end time point. Among them, the start time point of the current preset time period is earlier than the starting calculation time point of the current preset time period. The time period between the start time point and the starting calculation time point is a buffer time period, and the end time point of the current preset time period is the starting calculation time point of the next preset time period.

[0045] When processing the abnormal time of the received user behavior data in this embodiment, first, methods such as TatilDirSouce can be used to receive the user behavior data stored on disk, and the standard for judging abnormal values can be set. For example, abnormal value processing is performed on a daily basis, and the abnormal time processing is implemented by writing custom Java code for the Interceptor (interceptor). For example Figure 3 As shown, define the time stamp of 23:50 of the previous day as beforeTime, leaving a buffer of 10 minutes to prevent data loss caused by network fluctuations; define the time stamp of 00:00 of the next day as afterTime. Since cross-day data will be transmitted to the next day, no buffer is required here; obtain the current system time as systemTime, which can be used for subsequent data drift processing; define the time stamp of 00:00 of the current day as todayTime, and make judgments in combination with the system time.

[0046] Then, by parsing the Event of Flume, the time when the corresponding user behavior data of the user behavior data is generated is obtained as eventTime. Specifically, the Event of Flume consists of a Body and a Header. Among them, the Header stores some attributes of the Event, and the Body is the specific data. At this time, by parsing the Body and obtaining the ts field, the time when the user behavior data is generated can be obtained.

[0047] Next, it is judged whether the time eventTime when the user behavior data is generated meets the standard, and a direct discard operation is taken when it is determined to be an abnormal value. Specifically as follows:

[0048] If the time eventTime when the user behavior data is generated is less than beforeTime, it is directly discarded;

[0049] If the time eventTime when the user behavior data is generated is greater than afterTime, it is directly discarded;

[0050] After the above two-step operations, the time range of the time when the user behavior data is generated that remains is:

[0051] beforeTime (23:50 of the previous day) < eventTime < afterTime (00:00 of the next day)

[0052] After testing, it was found that some data in the data meeting the conditions still had the time of the previous day at 23:00 as of 23:00 on the current day. This caused the HDFS (Hadoop Distributed File System) files to fail to close properly after the data was saved. Therefore, to solve this problem, this embodiment adds the following judgment condition:

[0053] systemTime > todayTime and eventTime is less than todayTime;

[0054] User behavior data that meets the above criteria will be discarded based on its generation time; otherwise, it will be retained.

[0055] After the above steps, abnormal time data can be removed, and the remaining user behavior data can be transmitted in real time according to a unified time zone, thus avoiding the problem of inaccurate behavior data analysis results due to time abnormalities.

[0056] In one embodiment, storing the user behavior data in a data warehouse includes:

[0057] Data is transferred to Kafka via Flume's Kafka channel.

[0058] In this embodiment, as Figure 2 The S3 architecture addresses high concurrency issues. To prevent server crashes due to excessive data volume, Kafka is used as a buffer for data transfer, with Flume's KafkaChannel handling the data transmission. Furthermore, for building a real-time data warehouse to analyze user behavior data, frameworks like Flink / SparkStrimming can be used to directly read data from Kafka for processing. This allows user behavior data to be directly stored in HDFS for offline data warehouse analysis.

[0059] In one embodiment, receiving user behavior data from the data warehouse includes:

[0060] Data is received from Kafka via KafkaSource and written to the offline data warehouse HDFS via HdfsSink.

[0061] In this embodiment, data is received from Kafka via KafkaSource, and then written to the offline data warehouse HDFS via HdfsSink. It should be noted that after time-washing of the data, some cross-day data due to network fluctuations may still exist. Therefore, this embodiment uses the TimestampInterceptor example from the Flume official documentation, writing a custom TimestampInterceptor in Java to use the user behavior data generation time ts and add it to the Event Header. Flume then automatically parses the Event Header, thus solving the data drift problem. Specifically, the user behavior data generation time is first obtained by retrieving the ts field from the Event Body, then the Event Header is obtained, and ts is added to the Header using a put operation for subsequent data writing operations.

[0062] Write the Flume runtime application (conf) and use a custom interceptor. Flume will automatically parse the data timestamps in the Event Header and use the custom path format through the Flume HDFS Sink. The Flume HDFS Sink will store data on a daily basis according to the HDFS path information.

[0063] It should be noted that this embodiment uses Flume for streaming behavioral data collection and performs outlier handling in real time. Its core idea is to standardize data time, remove outlier data timestamps, and then save the data based on the data time. Of course, in other embodiments, mainstream frameworks such as Flink and Spark Strimming can be used to replace Flume for streaming data processing.

[0064] Figure 4 This is a schematic block diagram of a behavioral data time processing and storage device 400 provided in an embodiment of the present invention. The device 400 includes:

[0065] The collection and disk writing unit 401 is used to collect user log information through the log server and write it to disk; wherein, the user log information includes multiple user behavior data and the corresponding user behavior data generation time, and the user behavior data generation time is a timestamp identifier in UTC time zone;

[0066] An exception handling unit 402 is used to receive user behavior data from the log server and perform exception time processing; wherein, the exception time processing process includes: determining whether the generation time of the corresponding user behavior data is within the start and end time points of a preset time period; if yes, storing the user behavior data in a data warehouse; if no, directly discarding the corresponding user behavior data.

[0067] The drift processing unit 403 is used to receive user behavior data in the data warehouse and perform data drift processing on the user behavior data; wherein, the data drift processing process includes: adding the generation time of the user behavior data to the corresponding event header file;

[0068] The file parsing unit 404 is used to parse the event header file corresponding to the user behavior data, obtain the corresponding user behavior data generation time, and store the data according to a preset time period based on the user behavior data generation time.

[0069] In one embodiment, the exception handling unit 402 further includes:

[0070] The condition judgment unit is used to determine whether the generation time of the corresponding user behavior data meets the following conditions:

[0071] systemTime > todayTime and eventTime is less than todayTime, where systemTime is the current system time, todayTime is the starting time of the preset time period, and eventTime is the time when the user behavior data was generated.

[0072] The retain or discard unit is used to retain the user behavior data if the condition is not met, and to discard the corresponding user behavior data directly if the condition is met.

[0073] In one embodiment, the collection and tray dropping unit 401 includes:

[0074] The IO operation unit is used to receive the user behavior data and the corresponding user behavior data generation time through the written log receiving interface, and to perform IO operations.

[0075] The storage and disk persistence unit is used to store the generation time of the user behavior data in a time field to obtain user log information, and to persist the user log information to the log server.

[0076] In one embodiment, the start and end times of the current preset time period include a start time and an end time. The start time of the current preset time period is earlier than the starting time of the current preset time period. The time period between the start time and the starting time is a buffer period. The end time of the current preset time period is the starting time of the next preset time period.

[0077] In one embodiment, the exception handling unit 402 includes:

[0078] The data receiving unit is used to receive user behavior data from the log server via Flume;

[0079] The event parsing unit is used to parse the event body of Flume, obtain the time field, and obtain the time when the user behavior data was generated based on the time field.

[0080] In one embodiment, the exception handling unit 402 includes:

[0081] The data relay unit is used to relay data to Kafka via Flume's Kafka channel.

[0082] In one embodiment, the drift processing unit 403 includes:

[0083] The data writing unit is used to receive data from the Kafka via KafkaSource and write the data to the offline data warehouse HDFS via HdfsSink.

[0084] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0085] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0086] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0087] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0088] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for processing and storing behavioral data over time, characterized in that, include: User log information is collected through a log server and written to disk; wherein, the user log information includes multiple user behavior data and the corresponding user behavior data generation time, and the user behavior data generation time is a timestamp identifier in UTC time zone; The system receives user behavior data from the log server and performs abnormal time processing. The abnormal time processing includes: determining whether the generation time of the corresponding user behavior data falls within the start and end times of a preset time period; if so, storing the user behavior data in a data warehouse; if not, discarding the corresponding user behavior data. The abnormal time processing also includes: determining whether the generation time of the corresponding user behavior data satisfies the following conditions: systemTime > todayTime and eventTime is less than todayTime, where systemTime is the current system time, todayTime is the start time of the preset time period, and eventTime is the generation time of the user behavior data; if the conditions are not met, the user behavior data is retained; if the conditions are met, the corresponding user behavior data is discarded. Receive user behavior data from the data warehouse and perform data drift processing on the user behavior data; wherein, the data drift processing process includes: adding the generation time of the user behavior data to the corresponding event header file; The event header file corresponding to the user behavior data is parsed to obtain the corresponding user behavior data generation time, and the data is stored according to a preset time period based on the user behavior data generation time.

2. The method for processing and storing behavioral data according to claim 1, characterized in that, The process of collecting user log information through a log server and writing it to disk includes: The system receives user behavior data and the corresponding user behavior data generation time through a written log receiving interface, and performs IO operations accordingly. The user behavior data is generated and the time is stored in the time field to obtain user log information, which is then written to the log server.

3. The method for processing and storing behavioral data according to claim 1, characterized in that, The start and end times of the current preset time period include a start time and an end time. The start time of the current preset time period is earlier than the starting time of the current preset time period. The time period between the start time and the starting time is a buffer period. The end time of the current preset time period is the starting time of the next preset time period.

4. The method for processing and storing behavioral data according to claim 2, characterized in that, Receiving user behavior data from the log server includes: Receive user behavior data from the log server via Flume; The event body of Flume is parsed to obtain the time field, and the time when the user behavior data was generated is obtained based on the time field.

5. The method for processing and storing behavioral data according to claim 1, characterized in that, The step of storing the user behavior data in a data warehouse includes: Data is transferred to Kafka via Flume's Kafka channel.

6. The method for processing and storing behavioral data according to claim 5, characterized in that, Receiving user behavior data from the data warehouse includes: Data is received from Kafka via KafkaSource and written to the offline data warehouse HDFS via HdfsSink.

7. A device for processing and storing behavioral data over time, characterized in that, include: The collection and disk writing unit is used to collect user log information through the log server and write it to disk; wherein, the user log information includes multiple user behavior data and the corresponding user behavior data generation time, and the user behavior data generation time is a timestamp identifier in UTC time zone; An exception handling unit is used to receive user behavior data from the log server and perform exception time processing. The exception time processing process includes: determining whether the generation time of the corresponding user behavior data falls within the start and end time points of a preset time period; if yes, storing the user behavior data in a data warehouse; if no, discarding the corresponding user behavior data directly. The exception time processing process also includes: determining whether the generation time of the corresponding user behavior data satisfies the following condition: systemTime > todayTime and eventTime is less than todayTime, where systemTime is the current system time, todayTime is the start time point of the preset time period, and eventTime is the generation time of the user behavior data; if the condition is not met, the user behavior data is retained; if the condition is met, the corresponding user behavior data is discarded directly. A drift processing unit is used to receive user behavior data from the data warehouse and perform data drift processing on the user behavior data; wherein, the data drift processing process includes: adding the generation time of the user behavior data to the corresponding event header file; The file parsing unit is used to parse the event header file corresponding to the user behavior data, obtain the corresponding user behavior data generation time, and store the data according to a preset time period based on the user behavior data generation time.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the behavioral data time processing and storage method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the behavioral data time processing and storage method as described in any one of claims 1 to 6.

Images