A method and system for processing logs of an iptv device

Abstract

The application discloses an IPTV equipment log processing method and system, and mainly relates to the technical field of IPTV equipment log processing. The method comprises the following steps: storing the operation log reported by the received IPTV equipment to a distributed file system according to a preset time unit; querying the attribute information of full-amount users, and pre-storing the associated information to a cache database; performing batch processing on the log data stored in a previous time unit of the preset time unit, calculating the continuous online time period of each user and recording to a first database; and performing batch processing on the log data stored in the current preset time unit, calculating the media content playing time period of each user and recording to the first database. The application has the beneficial effect that it systematically solves the problems of data disorder, calculation accuracy, resource utilization rate and state consistency in the processing of the log of ten million IPTV equipment.

Description

Technical Field

[0001] This invention relates to the field of IPTV device log processing technology, specifically to an IPTV device log processing method and system based on group local sorting and state splitting. Background Technology

[0002] Existing IPTV equipment log processing systems often face the following technical bottlenecks:

[0003] 1. Low efficiency in processing out-of-order data: Device-reported logs are partially out of order due to network transmission delays. If a global sorting strategy is used, the time and space complexity is as high as O(n logs). It is difficult to handle data processing scales of millions per minute;

[0004] 2. Complex calculation logic: The calculation of user playback time periods is directly coupled based on raw logs (such as heartbeat, power on / off, and on-demand records), resulting in intertwined business logic and high system complexity and maintenance costs.

[0005] 3. Difficulty in ensuring consistency of stream processing status: Under the stream processing architecture, differences in the processing speed of subtasks can easily lead to misjudgment of the user's online status, affecting the accuracy of the calculation results;

[0006] 4. Low resource utilization: Traditional stream processing requires long-term use of computing resources, making it difficult to support the needs of historical data backtracking and repair.

[0007] Furthermore, in real-world environments with tens of millions of devices, the inherent problem of device clock asynchrony is prevalent. Some devices (based on empirical data, this accounts for less than 1% of the total) exhibit significant errors in system time compared to standard time. This severe deviation in device-side clocks poses a serious challenge to log timestamp-based processing systems. Existing technologies typically rely simply on device-reported timestamps or can only handle minute errors on the order of seconds, lacking effective fault-tolerance mechanisms for clock skew problems involving both magnitude and scale.

[0008] Therefore, there is an urgent need for an IPTV device log processing method and system based on grouped local sorting and state splitting to solve the above problems. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for processing IPTV device logs based on grouped local sorting and state splitting. It systematically solves the problems of data disorder, calculation accuracy, resource utilization and state consistency faced in the processing of logs from tens of millions of IPTV devices.

[0010] To achieve the above objectives, the present invention employs the following technical solution:

[0011] On one hand, the present invention provides a method for processing IPTV device logs, comprising the following steps:

[0012] Step S1: Store the received operation logs reported by the IPTV device to the distributed file system according to a preset time unit;

[0013] Step S2: Query the attribute information of all users and pre-store the associated information in the cache database;

[0014] Step S3: Batch process the log data stored in the previous time unit of the preset time unit, calculate the continuous online time period of each user and record it in the first database;

[0015] Step S4: Batch process the log data stored within the current preset time unit, calculate the media content playback time period for each user, and record it in the first database.

[0016] Preferably, step S3 includes:

[0017] Data ingestion and parsing steps: Read the raw log file of the previous preset time unit from the distributed file system and parse it into a structured log data stream;

[0018] Fault-tolerant waterline generation steps: Configure a sliding window for the structured log data stream. The sliding window has a capacity of N and is used to store the event timestamps of the most recent N logs. Whenever a new log arrives, insert its event timestamp into the window and remove the oldest timestamp. Calculate the median of the timestamps in the window and use this median as the current waterline for emission.

[0019] Grouping and local sorting steps: The data stream is partitioned according to user identifiers, and a fixed-capacity ordered buffer is allocated to each user; when a new log belonging to one of the users arrives, it is inserted into the corresponding buffer according to the event time and sorted. If the buffer is full, the element with the earliest event time is removed and output to the downstream processing logic; a first event time timer is registered. When the current water level exceeds the batch processing window end time defined for the previous preset time unit, the first event time timer is triggered to output all remaining log data in the corresponding buffer in event time order.

[0020] State initialization and online calculation steps: Load the intermediate states of the online period generated by the previous batch of processing from the first database, filter them according to the preset distributed filtering rules, load the filtered states into memory to build a local state mapping; calculate the user's continuous online time period based on the heartbeat log and power on / off log, combined with the local state mapping.

[0021] Result output and state persistence steps: Associate the completed online time period records with the association information obtained from the cache database and write them into the final result table of the first database; When the batch processing job ends, store the unfinished online time periods as new intermediate states in the intermediate state table of the first database.

[0022] In the fault-tolerant water level generation step, the capacity N is an odd number.

[0023] The median is calculated by sorting the N timestamps stored in the sliding window and taking the value at the middle position of the sorted sequence.

[0024] Preferably, in the grouping and local sorting steps:

[0025] The capacity of the fixed-capacity ordered buffer is set to M;

[0026] The sorting operation is performed immediately after each new log entry is inserted;

[0027] Register a first event timer. When the current water level exceeds the predefined batch processing window end time, the first event timer will be triggered to output all remaining log data in the corresponding buffer in the order of event time.

[0028] Preferably, in the state initialization and online calculation steps:

[0029] The filtering based on preset distributed filtering rules specifically refers to:

[0030] For each intermediate state record to be loaded, a target subtask index is calculated based on the user identifier, the maximum parallelism preset by the job, the total number of current subtasks, and the index of the current subtask in the record.

[0031] The intermediate state record is loaded into the current task instance only if the calculated target subtask index is equal to the actual index of the current subtask.

[0032] Preferably, the state initialization and online calculation steps further include: a second event timer registration step.

[0033] When processing the first log data corresponding to a certain user identifier, register a second event timer with a trigger time set to a specific future time.

[0034] Preferably, in the result output and state persistence steps, the unfinished online period is treated as a new intermediate state and persistently stored in the intermediate state table of the first database, specifically including:

[0035] Collect the online time status data of all users in memory for any unfinished tasks;

[0036] The status data is sent via the side output stream;

[0037] In the intermediate state table of the first database, a temporary partition is created to write the state data and the batch identifier in the state data is updated.

[0038] Use the temporary partition replacement function of the first database to update the data in the temporary partition with the latest data in the intermediate state table;

[0039] Specifically, when the batch processing job ends, the second event timer is called back.

[0040] Preferably, step S4 includes:

[0041] Data ingestion and parsing steps: Read the raw log file of the current preset time unit from the distributed file system and parse it into a structured log data stream;

[0042] Water level line generation and grouping sorting steps: Execute the fault-tolerant water level line generation step and the grouping and local sorting steps in step S3 in sequence;

[0043] The steps for initializing the associated status and calculating playback are as follows: Load the user online time period status data generated in this batch processing and the intermediate status data of the user playback time period generated in the previous batch processing from the first database, and filter and load them according to the distributed filtering rules; Calculate the user's media content playback time period based on the user's on-demand playback log and the effective online time range defined by the loaded user online time period status data as a constraint.

[0044] Playback result output and status persistence steps: After the calculated playback period record is associated with the related information, it is written into the playback result table of the first database; when the batch processing job ends, the unfinished playback period is used as a new intermediate state and persistently stored in the playback intermediate state table of the first database.

[0045] Preferably, the calculation of the user's media content playback time period in the association state initialization and playback calculation step specifically involves:

[0046] Set a current playback record for each user;

[0047] When a new playback log for a user arrives, determine whether its event time falls within any of the user's online time periods that have been loaded;

[0048] If it is determined that the new playback log falls within any online time period of the user who has already loaded the data, then it is further determined whether the new playback log and the current playback record belong to the same media content identifier;

[0049] If the determination is yes, then merge and update the end time of the current playback record to the event time of the new log;

[0050] If the determination is negative, the current playback record will be output as a playback period, and a new current playback record will be created with a new playback log.

[0051] On the other hand, this embodiment also provides an IPTV device log processing system for implementing the above-mentioned IPTV device log processing method, including:

[0052] The log storage module is used to store the operation logs reported by the IPTV device to the distributed file system according to a preset time unit;

[0053] The associated information pre-storage module is used to query the attribute information of all users and pre-store the associated information in the high-speed cache database;

[0054] A batch processing engine for scheduling and executing batch processing jobs;

[0055] The online time period analysis module runs on the batch processing computing engine and is used to process the log data stored in the previous preset time unit, calculate the continuous online time period of each user, and write the final result and intermediate status into the first database.

[0056] The playback behavior analysis module runs on the batch processing computing engine and is used to process the log data stored within the current preset time unit, calculate the media content playback time period for each user, and write the final result and intermediate state into the first database.

[0057] The online time period analysis module includes:

[0058] The fault-tolerant water level generation unit is used to configure a sliding window for the log data stream and generate a water level based on the median of the event timestamps within the window;

[0059] Grouped local sorting units are used to partition the data stream by user identifier and set an ordered buffer for each user;

[0060] The state management unit is used to load and filter previous intermediate states from the first database, and to persist the current intermediate state at the end of batch processing.

[0061] The time period calculation unit is used to calculate the continuous online time period based on the heartbeat log and power on / off log, combined with the loading status.

[0062] Preferably, the batch processing computing engine is implemented using the Apache Flink framework; the distributed file system is the Hadoop Distributed File System; the high-speed cache database is a Redis database; and the first database is a Doris database.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] 1. This invention partitions data by user identifier and sets up a small, ordered queue for each user within each partition for local sorting. Combined with a fault-tolerant watermark generated based on the midpoint of a sliding window, which has strong anti-interference capabilities, it provides a stable event time advancement benchmark for the system while tolerating significant clock errors in some devices. This combination systematically solves the challenge of ordering massive, locally disordered log data with clock skew in batch processing scenarios, laying a solid foundation for subsequent accurate time-period calculations.

[0065] 2. This invention abandons traditional methods that directly rely on raw timestamps or only handle minor errors. It innovatively introduces a water level generation mechanism based on the median of a sliding window onto the global data stream. This mechanism utilizes the statistical stability of the median to ensure that even with a certain number of extreme abnormal timestamps, the generated water level still represents the reasonable event timeline for most normal devices. This effectively avoids the problem of severe water level oscillations and stagnation caused by seriously inaccurate clocks of individual devices, which could lead to subsequent timer erroneous triggering or time window chaos. This significantly improves the system's robustness in real-world, complex environments.

[0066] 3. This invention employs a "state decomposition" design concept to break down complex user behavior analysis (such as online time periods and playback time periods) into independent computational tasks with clear dependencies and the ability to be executed in batches. By designing a dedicated intermediate state table and utilizing a temporary partition switching mechanism with atomicity guarantees for persistence, consistency, traceability, and repairability of states across batches are ensured. Furthermore, the introduced distributed filtering mechanism ensures that each parallel computing instance loads only the state data within its scope of responsibility, avoiding the memory and network overhead of full loading. This design not only simplifies the business logic of individual computational tasks and reduces system complexity and maintenance costs, but also provides a reliable state foundation for achieving accurate correlation calculations (such as playback behavior must occur within an online time period). Attached Figure Description

[0067] Figure 1 This is a flowchart of the method of the present invention;

[0068] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0069] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0070] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0071] Example:

[0072] like Figure 1 As shown, this embodiment provides a method for processing IPTV device logs, including the following steps:

[0073] Step S1: Store the received operation logs reported by the IPTV device to the distributed file system according to a preset time unit;

[0074] Step S2: Query the attribute information of all users and pre-store the associated information in the cache database;

[0075] Step S3: Batch process the log data stored in the previous time unit of the preset time unit, calculate the continuous online time period of each user and record it in the first database;

[0076] Step S4: Batch process the log data stored within the current preset time unit, calculate the media content playback time period for each user, and record it in the first database.

[0077] Specifically:

[0078] 1. Operation log reporting:

[0079] IPTV terminal devices collect user actions in real time, such as powering on, powering off, heartbeat, video-on-demand, and page navigation, and generate structured operation log data. This log data is then reported to the server's data receiving module via the network. An example of the operation log data format is as follows:

[0080] {

[0081] "user_id": "1234567890",

[0082] "device_mac": "a1:b2:c3:d4:e5:f6",

[0083] "event_time": "20240101103015", / / Event time, in YYYYMMDDHHMMSS format

[0084] "log_time": "20240101103020", / / Log arrival time on the server

[0085] "event_type": "HEARTBEAT", / / Event type: BOOT, SHUTDOWN, HEARTBEAT, VOD_PLAY, etc.

[0086] "asset_id": "YHB00199512190654", / / Media asset ID, which exists if it is a video-on-demand event.

[0087] "play_progress": 400, / / Current playback progress (seconds)

[0088] "video_duration": 1440 / / Total video duration (seconds)

[0089] }

[0090] 2. Operation log storage:

[0091] After receiving the logs, the server persists them in batches to a specified directory in the HDFS distributed file system according to a preset time unit (1 hour in this example). The file naming rules are as follows:

[0092] iptv_log_YYYYMMDDHH.log.

[0093] 3. User information association:

[0094] Before starting the batch processing task, the basic attribute information (such as user ID, subscription package, registration region, etc.) of all users is queried from the business database (Doris in this embodiment), and this related information is stored in the high-speed cache database Redis with a reasonable expiration time set for quick querying during subsequent data enrichment processing.

[0095] 4. Online time period analysis:

[0096] The online time period analysis task is implemented by an Apache Flink batch job. Its core objective is to calculate the continuous online time period for each user within an hour based on heartbeat and power-on / off logs, specifically including:

[0097] S41. Data Ingestion: The Flink job reads raw device log files from HDFS for a specified time window (i.e., T hours, such as iptv_log_2024010110.log);

[0098] S42. Log parsing: Use Flink's MapFunction to parse each line of log text in string format into a structured log data object (such as DeviceLogPO).

[0099] S43. Fault-Tolerant Watermark Generation: To address the issue of device clock skew, a custom WatermarkGenerator operator is inserted immediately after the data stream source. This operator sets up a global sliding window (capacity N=9) to store the event timestamps of the 9 most recent logs (converted to long integer millisecond values). Whenever a new log arrives, its timestamp is inserted into the window and the oldest timestamp is discarded. Then, the median of the timestamps in the window is calculated and this median is emitted as the current watermark. This method can effectively tolerate the impact of a few (no more than 4) extreme time skew data points.

[0100] S44. Data Partitioning and Local Sorting: The parsed data stream is partitioned by user_id using KeyBy. In each KeyedProcessFunction, a fixed-capacity ordered queue of 10 is set up for each unique user_id, sorted by event time. When a new log belonging to a user arrives, it is inserted into the queue and immediately sorted. Due to the small queue capacity, the sorting overhead is negligible. After sorting, if the queue size exceeds 10, the element with the earliest event time is removed and sent to the downstream time period calculation logic. Simultaneously, an event time timer with a trigger time set to the far future (e.g., Long.MAX_VALUE) is registered and triggered at the end of batch processing to ensure that any remaining data in the buffer is cleared and sent out.

[0101] S45. Online Time Period Calculation: This step is executed in KeyedProcessFunction and includes:

[0102] S451, State Initialization: In the operator's open() method, the previous batch identifier (previous_batch_id=2024010109) is determined based on the current batch identifier (e.g., batch_id=2024010110). A query is initiated to the intermediate state table of the Doris database to obtain the intermediate state data of all users online during the previous period with batch_id=previous_batch_id. To avoid loading the full data for each parallel subtask instance, a distributed filtering mechanism is introduced: For each queried record, a hash value is calculated based on its user_id and mapped to the index range of the current subtask. Only records that should be processed by this instance are loaded, and two mappings are built in memory:

[0103] Map <String, List <deviceonlineperiod>>finishedPeriodsMap,

[0104] Used to store historical time periods that have been taken offline;

[0105] Map<String, DeviceOnlinePeriod> currentOnlinePeriodMap

[0106] Used to store the current time period that has not yet been taken offline;

[0107] S452, Timer Registration: When processing the first data element of a user, check the flag bit. If no timer has been registered, register an event time timer with a trigger time set to the far future (such as Long.MAX_VALUE) and set the flag bit to true to ensure that it is registered only once in the entire batch processing process.

[0108] S453, Time Period Calculation: For each ordered log data:

[0109] If it is a BOOT event, the online time period of the user in currentOnlinePeriodMap will be output and recorded in finishedPeriodsMap. A new current online time period will be created based on this event time (both the start and end times will be set to the event time) and stored in currentOnlinePeriodMap.

[0110] If it is a non-BOOT event, check the current online period status, calculate the difference between the current event time and the last valid time (i.e., the offline time) recorded in the current online period. If the difference does not exceed the preset session timeout threshold (e.g., 30 minutes), update the end time of the current online period to the current event time. If it exceeds the threshold, it is considered that the previous online session has ended, the current online period is marked as completed and output and recorded in finishedPeriodsMap, and then a new current online period is created based on the new event and stored in currentOnlinePeriodMap.

[0111] S46. Result Output and State Persistence:

[0112] Output results: The calculated and determined user online time period records (i.e. offline time periods) are associated with the user static information queried from Redis, assembled into a complete business object, and then written in batches to the final result table of the Doris database (e.g., user_online_period).

[0113] Intermediate state persistence: Override the operator's onTimer method. When the batch job ends, the Flink framework will call back this method. In this callback, the online records in currentOnlinePeriodMap are traversed, and the current batch identifier (e.g., batch_id=2024010110) is used to determine whether the user's no-message time has exceeded the session timeout threshold. If it has, it is output and recorded in finishedPeriodsMap. All finishedPeriodsMap and currentOnlinePeriodMap are sent through the side output stream. In the Sink operator associated with the side output stream, this batch of intermediate state data is written to a temporary partition of the Doris intermediate state table, and the Doris temporary partition replacement function is used to atomically update the intermediate state table and update the state flag bits. This mechanism ensures the consistency of the state.

[0114] 5. Playback Behavior Analysis:

[0115] The playback behavior analysis task is also implemented using Flink batch jobs. Its process is similar to that of online time-of-day analysis, with the core difference being that the computation logic depends on the results of online time-of-day analysis, including:

[0116] S51, Data Ingestion and S52, Log Parsing: S41 and S42, which are analyzed during the same online period, read and parse log data within the same time window (T hours);

[0117] S53, fault-tolerant water level generation and S54, data partitioning and local sorting: same as S43 and S44 in online time period analysis;

[0118] S55. Playback Period Calculation: This step is executed in KeyedProcessFunction and includes:

[0119] S551, State Initialization: In the open() method, in addition to loading the intermediate state of the user playback period from the previous batch (same method as S451), the key is to associate and load the user online period state generated by this online period analysis task (i.e., the online period result and intermediate state of batch_id=2024010110). This also uses a distributed filtering mechanism, built in memory:

[0120] Map <String, List <deviceonlineperiod>>onlinePeriodsMap (stores user online time periods) and Map<String, TVStreamPO> currentPlayPeriodMap (stores the current playback period that the user has not yet finished);

[0121] S552, Timer Registration: Same as S452;

[0122] S553, Association Calculation: For each ordered playback-related log (e.g., VOD_PLAY):

[0123] First, based on the user_id and the event time, query the onlinePeriodsMap to determine whether the playback event occurred within a user's online time period. Only if the event time falls within a certain online time period is the playback behavior considered valid.

[0124] For valid playback events, check if the user has a current playback record in currentPlayPeriodMap;

[0125] If it does not exist, or if it exists but the media asset ID of the current record is different from the new event, then the existing current playback record (if it exists) will be output as a playback period (it is necessary to determine whether it has ended), and a new current playback record will be created based on the new event;

[0126] If they exist and have the same media asset ID, then merge and update the end time of the current playback record to the time of the new event;

[0127] At the same time, it is necessary to determine whether the playback behavior ends naturally due to the user going offline, based on the end time of the online period;

[0128] S56. Result Output and State Persistence:

[0129] Output results: After associating the calculated and determined user playback period records (such as when the user has gone offline or the playback content has been switched) with the user static information, write them in batches into the final result table of Doris (such as user_play_period).

[0130] Intermediate state persistence: Similar to the intermediate state persistence mechanism of S46, the intermediate state of the unfinished playback period is persisted to the intermediate state table of Doris.

[0131] like Figure 2 As shown, this embodiment also provides an IPTV device log processing system, including:

[0132] The log storage module is used to store the operation logs reported by the IPTV device to the distributed file system according to a preset time unit;

[0133] The associated information pre-storage module is used to query the attribute information of all users and pre-store the associated information in the high-speed cache database;

[0134] A batch processing engine for scheduling and executing batch processing jobs;

[0135] The online time period analysis module runs on the batch processing computing engine and is used to process the log data stored in the previous preset time unit, calculate the continuous online time period of each user, and write the final result and intermediate status into the first database.

[0136] The playback behavior analysis module runs on the batch processing computing engine and is used to process the log data stored within the current preset time unit, calculate the media content playback time period for each user, and write the final result and intermediate state into the first database.

[0137] The online time-period analysis module includes:

[0138] The fault-tolerant water level generation unit is used to configure a sliding window for the log data stream and generate a water level based on the median of the event timestamps within the window;

[0139] Grouped local sorting units are used to partition the data stream by user identifier and set an ordered buffer for each user;

[0140] The state management unit is used to load and filter previous intermediate states from the first database, and to persist the current intermediate state at the end of batch processing.

[0141] The time period calculation unit is used to calculate the continuous online time period based on the heartbeat log and power on / off log, combined with the loading status;

[0142] In this embodiment, the batch processing computing engine is implemented using the Apache Flink framework; the distributed file system is the Hadoop Distributed File System; the high-speed cache database is the Redis database; and the first database is the Doris database.

[0143] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.< / deviceonlineperiod> < / deviceonlineperiod>

Claims

1. A method for IPTV device log processing, the method comprising: The method comprises the following steps: Step S1: storing the operation log reported by the IPTV device in a distributed file system according to a preset time unit; Step S2: querying the attribute information of a full-amount user and pre-storing the attribute information of the user in a cache database; Step S3: performing batch processing on the log data stored in a previous time unit of the preset time unit, calculating the continuous online time period of each user and recording the same in a first database; Step S4: performing batch processing on the log data stored in the current preset time unit, calculating the media content playing time period of each user and recording the same in the first database; Step S3 comprises: a data ingestion and parsing step: reading the original log file of the previous preset time unit from the distributed file system and parsing and converting the same into a structured log data stream; a fault-tolerant water level generation step: configuring a sliding window for the structured log data stream, the capacity of the sliding window being N, for storing the event time stamps of the latest N logs; whenever a new log arrives, inserting the event time stamp of the new log into the window and removing the oldest time stamp, calculating the median of the time stamps in the window and emitting the median as the current water level; a grouping and local sorting step: partitioning the data stream according to the user identifier and allocating an ordered buffer with a fixed capacity to each user; when a new log belonging to one of the users arrives, inserting the new log into the corresponding buffer according to the event time and sorting the same, and if the buffer is full, removing the element with the earliest event time and outputting the same to the downstream processing logic; registering a first event time timer, which triggers the output of all the remaining log data in the corresponding buffer in the order of event time when the current water level exceeds the batch processing window end time defined for the previous preset time unit; a state initialization and online calculation step: loading the intermediate state of the online time period generated by the previous batch processing from the first database, screening the same according to a preset distributed filtering rule, loading the screened state into the memory to construct a local state map, and calculating the continuous online time period of the user according to the heartbeat log and the startup / shutdown log and in combination with the local state map; a result output and state persistence step: associating the calculated online time period record with the attribute information of the user obtained from the cache database and writing the same into the final result table of the first database; at the end of the batch processing job, storing the unfinished online time period as a new intermediate state in the intermediate state table of the first database; In the fault-tolerant water level generation step, the value of the capacity N is an odd number; The median is calculated by sorting the N time stamps stored in the sliding window and taking the value at the middle position of the sorted sequence; In the state initialization and online calculation step: The screening according to the preset distributed filtering rule specifically comprises: For each intermediate state record to be loaded, a target subtask index is calculated according to the user identifier in the record, the maximum parallelism degree preset for the job, the current total number of subtasks and the current subtask index. Load the intermediate state record in the row to the current task instance only when the calculated target subtask index is equal to the actual index of the current subtask; Step S4 comprises: A data ingestion and parsing step: reading the original log files of the current preset time unit from the distributed file system and parsing and converting into a structured log data stream; A waterline generation and grouping and sorting step: sequentially performing the fault-tolerant waterline generation step and the grouping and local sorting step in step S3; An associated state initialization and play calculation step: loading the user online session state data generated by the current batch processing and the user play session intermediate state data generated by the previous batch processing from the first database, and performing screening and loading according to the distributed filtering rule; based on the user's play log and taking the valid online time range defined by the loaded user online session state data as a constraint, calculating the user's media content play session; A play result output and state persistence step: writing the calculated play session record and the user's attribute information into the play result table of the first database; at the end of the batch processing job, persisting the unfinished play session as new intermediate state to the play intermediate state table of the first database.

2. The IPTV device log processing method of claim 1, wherein, In the grouping and local sorting step: The capacity of the fixed-capacity ordered buffer is set to M; The sorting operation is performed immediately after each new log is inserted; A first event time timer is registered, which triggers the first event time timer to output all remaining log data in the corresponding buffer in event time order when the current waterline exceeds the predefined end time of the batch processing window.

3. The IPTV device log processing method of claim 1, wherein, The state initialization and online calculation step further comprises a second event time timer registration step: In the batch processing job, when the first log data corresponding to a certain user identifier is processed, a second event time timer is registered, which is triggered at a specific time in the future.

4. The IPTV device log processing method of claim 3, wherein, In the result output and state persistence step, persisting the unfinished online session as new intermediate state to the intermediate state table of the first database specifically comprises: Collecting all unfinished online session state data of users in memory; Sending the state data through a side output stream; In the intermediate state table of the first database, creating a temporary partition to write the state data and updating the batch identifier in the state data; Using the temporary partition replacement function of the first database to update the data in the temporary partition to the latest data of the intermediate state table; Wherein, at the end of the batch processing job, the second event time timer is called back.

5. The IPTV device log processing method of claim 1, wherein, In the associated state initialization and play calculation step, the calculation of the user's media content play session specifically comprises: Setting a current play record for each user; When the new play log of the user arrives, it is determined whether the event time falls within any loaded online session of the user; If it is determined to fall within any loaded online session of the user, it is further determined whether the new play log and the current play record belong to the same media content identifier; If it is determined to be, the end time of the current play record is updated to the event time of the new log. If the judgment is no, the current play record is output as a play period, and a new current play record is created with a new play log.

6. An IPTV device log processing system for implementing the IPTV device log processing method of any one of claims 1 to 5, characterized by, The method comprises the following steps: A log storage module is configured to store the operation logs reported by the IPTV device according to a preset time unit in a distributed file system. A user attribute information pre-storage module is configured to query the attribute information of all users and pre-store the attribute information of the users in a cache database. A batch processing computing engine is configured to schedule and execute batch processing jobs. An online period analysis module is configured to run on the batch processing computing engine, process the log data stored in a previous preset time unit, calculate the continuous online time period of each user, and write the final result and intermediate state into a first database. A play behavior analysis module is configured to run on the batch processing computing engine, process the log data stored in a current preset time unit, calculate the media content play time period of each user, and write the final result and intermediate state into the first database. The online period analysis module comprises: A fault-tolerant water level line generation unit is configured to configure a sliding window for log data flow and generate a water level line based on the median of event timestamps in the window. A grouping local sorting unit is configured to partition the data flow according to user identifiers and set an ordered buffer for each user. A state management unit is configured to load and filter the previous intermediate state from the first database, and persist the current intermediate state at the end of batch processing. A period calculation unit is configured to calculate the continuous online time period according to heartbeat logs and power-on / off logs, combined with the loaded state.

7. The IPTV device log processing system of claim 6, wherein, The batch processing computing engine is implemented by using an Apache Flink framework; the distributed file system is a Hadoop distributed file system; the cache database is a Redis database; and the first database is a Doris database.

Images