An iptv program recommendation method and system based on multi-source data

Abstract

The application discloses an IPTV program recommendation method and system based on multi-source data, and mainly relates to the technical field of television program recommendation. The method comprises the following steps: collecting multi-source heterogeneous data, managing the multi-source heterogeneous data, and constructing a data base; based on a batch processing engine, extracting historical data from the data base and persistently storing the historical data into a feature vector database; accessing behavior logs reported by a set top box in real time, and writing real-time features extracted into a cache; based on full training data in the data base and feature vectors in the feature vector database and the cache, outputting a personalized program recommendation prediction result for a target user; and dynamically adjusting parameters of the recommendation model according to feedback behavior data of the target user for the recommended content, so as to realize continuous self-optimization of the recommendation effect. The application has the beneficial effect that it combines a deep learning recommendation algorithm, realizes real-time modeling of user interest, and accurately recommends programs.

Description

Technical Field

[0001] This invention relates to the field of television program recommendation technology, specifically to an IPTV program recommendation method and system based on multi-source data. Background Technology

[0002] IPTV has become an important gateway to home entertainment. However, with the explosive growth of content supply, users are facing the common problem of "information overload"—the live, on-demand, and replay content resources provided by the platform often exceed 100,000 hours, but users watch for less than 3 hours on average per day. The low efficiency of program selection has become a key bottleneck restricting the improvement of user experience.

[0003] Current IPTV platforms generally possess a foundation for collecting user behavior data, including structured data such as viewing history, dwell time, and on-demand searches, generating over 2TB of user behavior logs daily. This data provides the necessary fuel for building recommendation systems, but existing systems lack multi-source data fusion and effective algorithmic mining capabilities. By combining program content data and introducing deep learning-based recommendation algorithms, it is possible to extract multi-dimensional features of program content (such as genre, actors, and style), model user interests in real time, and ultimately create a personalized intelligent program recommendation service.

[0004] Current solutions typically rely solely on single viewing records or on-demand logs, making it difficult to effectively integrate multi-dimensional user data such as live viewership ratings, time-shifted playback patterns, page access paths, search and collection intentions, and subscription conversions with rich program content data. This results in a one-sided user profile and an inability to fully reflect the true interest structure of users in a home setting. To address this, this invention proposes an IPTV program recommendation method and system based on multi-source data. By constructing a data governance system that includes multi-dimensional user data and program content data, it calculates user characteristics in real time for model inference, solving the problems of insufficient data utilization and poor real-time recommendation performance in existing technologies, and providing a more efficient and accurate viewing experience.

[0005] Therefore, there is an urgent need for an IPTV program recommendation method and system based on multi-source data to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide an IPTV program recommendation method and system based on multi-source data, which combines deep learning recommendation algorithms to achieve real-time modeling of user interests and accurate program recommendation.

[0007] To achieve the above objectives, the present invention employs the following technical solution: On the one hand, the present invention provides an IPTV program recommendation method based on multi-source data, comprising the following steps: Step S1: Collect multi-source heterogeneous data and process the multi-source heterogeneous data. The data includes, but is not limited to, user behavior logs, program metadata, user basic information, and public network supplementary data. Perform unified cleaning, standardization processing, and distributed storage on the multi-source heterogeneous data to build a data foundation. Step S2: Based on the batch processing engine, extract historical data from the data base, perform feature engineering and vectorization on the user's basic attributes, long-term behavior aggregation indicators and preference distribution, as well as the program's static attributes, content tags and statistical indicators, to construct a full user profile and program profile, and persist them to the feature vector database. Step S3: Access the behavior logs reported in real time by the set-top box, extract real-time features from the behavior logs through the stream processing engine, and write the extracted real-time features into the cache; Step S4: Based on the full training data in the data base and the feature vectors in the feature vector database and cache, use the recommendation model for training and inference, and output personalized program recommendation prediction results for the target user; Step S5: Based on the feedback behavior data of the target user regarding the recommended content, dynamically adjust the parameters of the recommendation model to achieve continuous self-optimization of the recommendation effect.

[0008] Preferably, step S1 specifically includes: The multi-source heterogeneous data is deduplicated, filtered for null values, and formatted. A legal time window threshold is set to remove log records with abnormal event times. To address the differences in data structures from different sources, field mapping and semantic normalization are performed to form a unified standard data model; The standardized data is used to build detailed tables and aggregate tables, which are then written into a distributed data warehouse.

[0009] Preferably, step S2 specifically includes: Cleaned historical data is extracted from a distributed data warehouse to form a feature dataset; The word vector model is used to train user behavior sequences to generate user feature vectors, and the text embedding model is used to train program description text to generate program feature vectors. The two vectors are then unified to the same dimension and spliced ​​or weighted and fused. The completed user feature vectors and program feature vectors are written in batches to a high-performance vector database and loaded into the memory cache.

[0010] Preferably, step S3 specifically includes: The message middleware is used to access the set-top box to report logs in real time, complete the format parsing and event recognition, and handle the log out-of-order problem based on the event time and water level mechanism; Data streams are split according to user identifiers, and a sliding window is used to count the viewing records and durations within the most recent preset time period to update the user's short-term preference vector. The calculated real-time features are serialized and written to the cache with the user identifier as the key, and a time-to-live is set.

[0011] Preferably, step S4 specifically includes: The model training dataset is generated by extracting real user behavior as positive samples from a distributed data warehouse and constructing negative samples through exposure without clicks or random negative sampling. A recommendation model is built based on a deep learning recommendation framework, distributed training is initiated, and the intermediate model state is periodically saved through a checkpoint mechanism to support resuming training from breakpoints. The trained model is evaluated offline using a test set. Once the model's performance is confirmed to meet the standards, the service ratio of the new model is gradually increased to complete the release. Historical versions of the model are retained to support rollback operations.

[0012] Preferably, step S5 specifically includes: After the recommendation results are uploaded in real time, explicit feedback behaviors such as program collection and subscription, as well as implicit feedback behaviors such as viewing time and page visits are collected from users. The feedback behaviors are accurately attributed to the corresponding recommendation strategies and model versions, and an incremental training dataset is constructed based on the feedback behaviors. Using the incremental training interface of the deep learning recommendation framework, the incremental training dataset is input to the online training node. Only the fully connected layer or part of the embedding layer of the ranking model is locally updated. The online recommendation request is not interrupted during the parameter update process. The updated model parameters are directly loaded into the inference service online to achieve hot model update. After the model parameters are updated, the system continuously monitors the core recommendation business metrics. If an abnormal downward trend is detected in the core recommendation business metrics, the system will automatically trigger a rollback mechanism to restore the current recommendation model version or strategy weight to the previous stable version, ensuring the robustness and continuous self-optimization of the online recommendation service.

[0013] On the other hand, the present invention provides an IPTV program recommendation system based on multi-source data, for implementing the IPTV program recommendation method based on multi-source data as described above, including: The multi-source data governance module collects heterogeneous data from multiple sources and completes unified cleaning, standardized processing, and distributed storage to build a data foundation. The offline profile building module extracts historical data from the data base based on the batch processing engine, performs feature engineering and vectorization on users and programs, builds full user and program profiles, and persists them to the feature vector database. The near-line real-time computing module accesses the behavior logs reported in real time by the set-top box, performs real-time feature extraction through the stream processing engine, and writes real-time user features into the high-speed cache; The model inference service module reads the full training data in the data base and combines it with the feature vector database and the feature vector in the cache. It then uses the recommendation model for training and inference to output a personalized preference prediction for the user's program. The feedback and optimization module collects explicit and implicit user feedback on recommended content and dynamically adjusts the parameters of the recommendation model to achieve self-optimization of recommendation performance.

[0014] Preferably, the distributed storage in the multi-source data governance module uses the Doris database; The offline profile building module uses Milvus vector database as its feature vector database; the near-line real-time computing module uses Kafka as its message middleware, Flink as its stream processing engine, and Redis as its cache. The model inference service module is based on the EasyRec deep learning recommendation framework, employing a dual-tower model to perform the recall phase and a Wide&Deep model to perform the ranking phase.

[0015] Preferably, the feedback and optimization module is specifically configured as follows: The collected feedback data is used to construct an incremental training dataset. The incremental training interface of the EasyRec framework is called to update the local parameters of the ranking model, and the updated model parameters are loaded into the inference service online. If a decline in core recommendation metrics is detected, the model version or strategy weights will be automatically rolled back to the previous stable version.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention provides an IPTV program recommendation method and system based on multi-source data, which constructs a high-quality full-domain data foundation by uniformly accessing set-top box behavior logs, media asset libraries and public network data and performing strict filtering, cleaning and standardized mapping. 2. This invention innovatively integrates Flink offline batch processing and stream processing technologies to construct a full user / program profile based on behavior sequences and program metadata and store it in Milvus, and to calculate and write real-time features based on sliding windows to Redis, thereby realizing real-time updates of short-term interest features. 3. This invention employs a hierarchical architecture combining a dual-tower model and a Wide&Deep model for recall and ranking, coupled with a Checkpoint breakpoint training mechanism, which significantly improves model training efficiency and recommendation accuracy. 4. This invention drives incremental training and local parameter updates by collecting user feedback data in real time, forming a complete closed loop from data collection, feature calculation, model inference to online optimization. It effectively solves industry pain points such as inconsistent quality of multi-source data, strong timeliness of user interests, and lagging recommendation effects, and greatly improves the intelligence level and user experience of IPTV recommendation systems. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0018] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0019] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0020] Example: like Figure 1 As shown, this embodiment provides an IPTV program recommendation method based on multi-source data, including the following steps: Step S1: Collect multi-source heterogeneous data and process the multi-source heterogeneous data; perform unified cleaning, standardization processing and distributed storage on the multi-source heterogeneous data to build a data foundation; Step S2: Based on the batch processing engine, extract historical data from the data base, perform feature engineering and vectorization on the user's basic attributes, long-term behavior aggregation indicators and preference distribution, as well as the program's static attributes, content tags and statistical indicators, to build a full user profile and program profile, and persist them to the feature vector database. Step S3: Access the behavior logs reported in real time by the set-top box, extract real-time features from the behavior logs through the stream processing engine, and write the extracted real-time features into the cache; Step S4: Based on the full training data in the data base and the feature vectors in the feature vector database and cache, use the recommendation model for training and inference, and output personalized program recommendation prediction results for the target user; Step S5: Based on the feedback behavior data of the target users regarding the recommended content, dynamically adjust the parameters of the recommendation model to achieve continuous self-optimization of the recommendation effect.

[0021] The following embodiment will provide a detailed explanation of the above steps.

[0022] Step S1, Multi-source data integration and governance: In the specific implementation process of this embodiment, data integration and governance are the foundation for subsequent recommendations. The specific implementation steps are as follows: 1. Multi-source data access and dirty data removal: The data sources accessed by the system include: user behavior logs reported by IPTV set-top boxes, program content data in the media asset library, supplementary data (such as Douban ratings, movie reviews, etc.) crawled from the public network by web crawling tools, and basic data in the user information database and subscription database. The system performs a strict cleaning operation on the user behavior logs reported by the set-top box: (1) Time anomaly filtering: Set a legal time window threshold, specifically, the current system time is backdated by 2 days to the next hour. If the event_time in the log is 2070 or 1980, which is obviously outside this time window, it is judged as an illegal time and the log is directly discarded. Key field validation: Perform null value checks on key fields such as user_id, program_code, and event_type; if any are missing, discard the record. Garbled characters and format filtering: Regular expressions are used to filter garbled characters and invisible control characters in non-critical fields; field length is constrained and validated, for example, the length of user_id is limited to 8 to 32 characters, and data exceeding or falling short of this limit is considered abnormal and is removed; in addition, to address the issue of duplicate data reporting caused by repeated user operations, the system deduplicates data based on serial number or timestamp. 2. Standardization of heterogeneous data and unification of models: Because the data formats of different business types (such as live streaming and video-on-demand) differ, the system uses "|" as the field separator. The first field represents the business type code: 01 represents live streaming business and 02 represents video-on-demand business. The system first classifies and distributes the data stream according to the business type code. To address the issue of inconsistent field naming across different business scenarios, the system configures unified field mapping rules. For example, the user ID in the live streaming source and the uid in the video-on-demand source are uniformly mapped to the standardized field user_id; the programID in the live streaming source and the content_code in the video-on-demand source are uniformly mapped to the standardized field program_code; and the playback duration field is mapped to watch_duration. In addition, the system parses and standardizes business parameters to ensure that downstream models can consume them uniformly. 3. Distributed storage: The system uses the Doris distributed database for data storage. For historical full data, Doris's Stream Load method is used in conjunction with offline computing tasks for batch import to ensure data integrity. During writing, table structures are constructed according to data characteristics. Duplicate Key detail table: used for storing raw behavior logs, facilitating historical record tracing, such as creating user_collect (user favorites table), user_search (user search table), etc.; Aggregate Key tables are used to store pre-aggregated metrics by user, program, time period, etc. For example, user_live (user live broadcast table) and user_video (user on-demand table) can be created, with pre-aggregated metrics such as playback count and viewing duration, to balance detailed traceability and high-performance analysis.

[0023] Step S2, Offline Profile Construction: Offline profile building is mainly used to characterize users' long-term interests and the characteristic attributes of programs: 1. Offline data source extraction and preprocessing: The system connects to the Doris database through the Flink batch processing engine, calls the JDBC interface or the Doris native Connector, and extracts historical data from the above detail tables and aggregation tables. The extraction strategy adopts time partitioning, extracting the full data of the previous day at midnight every day or extracting historical window data by week / month to reduce I / O overhead. Extract basic user attributes from the user information table, such as gender, age group, and region code; collect long-term behavioral aggregation metrics from the live streaming and on-demand aggregation table, such as cumulative viewing time, active days, and average daily views over the past 30 and 90 days; calculate preference distribution, such as the viewing percentage of each type of program; extract static program attributes from program metadata, such as type, cast list, director, production company, and release date; obtain content tags (theme, style, etc.); and collect statistical metrics, such as cumulative play count, average completion rate, and rating. 2. Multi-dimensional feature engineering and vectorized representation: The system uses the Flink batch processing engine to perform deep feature calculations on the above indicators, transforming discrete and sparse business fields into dense and computable numerical features or vectors. (1) User feature vector generation: The user behavior log is sorted by event_time using user_id as the grouping key to generate a user behavior sequence in the form of [play, pause, fast forward, ..., exit]. The Gensim natural language processing library is used to train the behavior sequence based on the Skip-gram algorithm or CBOW algorithm to learn the relationship between user operation behavior and program, and finally generate user feature vector (User Embedding). (2) Program feature vector generation: The text fields such as program introduction, tags, actor and director information are segmented and cleaned, and the statistical indicators (such as number of plays and completion rate) are normalized by Min-Max. The input is fed into the pre-trained text embedding model (such as BERT, Sentence-BERT) and the program feature vector (Item Embedding) is output. (3) Unified Dimension Fusion: All user feature vectors and program feature vectors are processed in a unified dimension, for example, uniformly mapped to 128 dimensions or 256 dimensions, and L2 normalization is applied. 3. Persistence of feature vector database: The system triggers a Flink batch processing task every morning at midnight via the DolphinScheduler scheduling platform. This task automatically scans all valid user and program data from the previous day, performs the aforementioned feature calculation and vectorization process, and writes the generated feature vectors in batches into the high-performance vector database Milvus. Subsequently, the feature vectors in Milvus are loaded into the memory cache for real-time use by the online recommendation system, effectively reducing recommendation response latency.

[0024] Step S3, near-line real-time calculation: Nearline real-time computation is primarily used to capture short-term shifts in user interests, ensuring the timeliness of recommendations. 1. Real-time data stream access for user behavior logs: After the behavior logs of set-top boxes from various operators are uniformly formatted, they are written to the Kafka message queue by the log collection service. To address the differences in logs from different operators, the system is configured with independent Kafka Topics (e.g., China Unicom iptv-log-lt, China Mobile iptv-log-yd, China Telecom iptv-log-dx) to achieve business isolation and parallel consumption. Flink consumes Kafka data in real time, and uses the event_type business type field in the log to split the data stream for processing; at the same time, the system sets a maximum out-of-order time of 30 seconds, and sets a watermark based on the event_time field in the log as the event time to ensure that the data remains ordered under a certain network latency. 2. Real-time feature calculation: To capture recent changes in interests, the system uses a sliding window to aggregate and statistically analyze behavioral data. The window length is set to 30 minutes, and the current user's behavioral characteristics are calculated every 30 minutes to achieve near real-time updates. Using user_id and program_code as grouping keys (KeyBy), the system sorts the user's behavioral sequences within 30 minutes by event_time and extracts features. It then uses a Skip-gram or CBOW algorithm, similar to the offline stage, to generate short-term user preference vectors in real time. 3. Feature caching persistence: Whenever Flink's sliding window triggers computation and generates a new feature vector, the system overwrites the old state value with the new value and serializes the calculated real-time features into the Redis cache with user_id or user_id:program_code as the key. At the same time, TTL (Time to Live) is set to ensure the effectiveness of cache management and ensure that the recommendation service reads the user's latest real-time interest state.

[0025] Step S4, Model Inference Service: 1. Construction of training data samples: The system extracts high-quality user behavior samples from the Doris data warehouse and constructs positive and negative samples and partitions the dataset. Positive sample construction: PLAY behavior where the user's playback time exceeds the threshold (e.g., ≥60 seconds) and COLLECT behavior are defined as valid positive behaviors; Negative sample construction: User playback time of less than 3 seconds is negatively sampled; at the same time, random negative sampling is added to alleviate the bias. The ratio of positive to negative samples is set to 1:5 to alleviate the training bias caused by the imbalance of positive and negative samples. Finally, the positive and negative samples are aligned and concatenated for feature fields. 2. Model Training: The system relies on the open-source deep learning recommendation framework EasyRec, and writes configuration files to define the data processing pipeline, feature engineering module, model structure and training parameters. In the recall phase, a two-tower model is used, with the user tower and program tower modeling the representation of users and programs respectively, and outputting a 128-dimensional dense vector for matching and recall; in the ranking phase, a wide & deep model is used, with the wide part memorizing high-frequency co-occurrence features and the deep part learning high-order nonlinear feature combinations. During training, the system initiates distributed training tasks, automatically synchronizing gradients and updating parameters. A checkpoint mechanism is implemented, saving model parameters and optimizer state to a distributed file system every 10 minutes or after processing 1 million samples. When a training node fails and training is interrupted, the system automatically resumes training from the most recent checkpoint, significantly improving the reliability and fault tolerance of large-scale training. 3. Model evaluation, release, and inference: The system uses a test set to evaluate the trained model offline, calculates recommendation performance metrics including AUC (Area Under Curve), GAUC (Group AUC), and Top-K hit rate (Hit Rate@K), selects the recall and ranking model version with the best performance metrics, and loads it into the online recommendation service. During the inference process, the online recommendation service reads the user profile and program profile feature vectors cached in Milvus and Redis as the initial input to the model. When the system is turned on for the first time each day, it provides users with a cold start recommended program list based on historical features. After the near-line computing module completes the real-time feature update every 30 minutes, the online service directly reads the latest real-time feature vector in Redis, merges it with the long-term profile and program features, and inputs it into the model for inference. Finally, it generates a real-time personalized program recommendation list and pushes it to the IPTV terminal.

[0026] Step S5, Feedback and Optimization Mechanism: After the recommendation results are launched, closed-loop optimization is initiated, and its core implementation process is as follows: 1. Feedback Data Collection and Attribution: Continuously extract feedback behaviors from the latest logs reported by the set-top box. Explicit feedback includes users’ clear intention behaviors such as program collection and subscription; implicit feedback includes users’ viewing time (such as exceeding a certain time as valid viewing) and page access path. The above feedback behaviors are accurately attributed to the recommendation strategy and model version of this recommendation, and incrementally constructed into an online training dataset. 2. Incremental training and hot updating of model parameters: By utilizing the incremental training interface provided by the EasyRec framework, incremental samples are input to the online training node. In order to balance training efficiency and recommendation performance, only the fully connected layers or part of the embedding layers of the ranking model are locally updated, while the large-scale embedding layers at the bottom are frozen to avoid the huge computational consumption caused by full retraining. The parameter update process does not interrupt the online recommendation request, and the updated model parameters are directly loaded online to the inference service node, realizing the model "hot loading" and achieving model self-optimization without affecting the user experience. 3. Online indicator monitoring and automatic rollback: After the parameters are updated, core business metrics such as CTR (click-through rate) and average viewing time per user are continuously monitored. If an abnormal downward trend is detected in the metrics, a rollback mechanism is automatically triggered to restore the model version or strategy weights used by the current online service to the previous stable version, ensuring the robustness and continuity of the online recommendation service. Through the complete closed loop from data collection, feature calculation, model inference to online optimization, this invention achieves continuous self-optimization of IPTV program recommendation effects.

[0027] like Figure 2 As shown, this embodiment also provides an IPTV program recommendation system based on multi-source data, including: The multi-source data governance module collects heterogeneous data from multiple sources and completes unified cleaning, standardized processing, and distributed storage to build a data foundation. The offline profile building module extracts historical data from the data base based on the batch processing engine, performs feature engineering and vectorization on users and programs, builds full user and program profiles, and persists them to the feature vector database. The near-line real-time computing module accesses the behavior logs reported in real time by the set-top box, performs real-time feature extraction through the stream processing engine, and writes real-time user features into the high-speed cache; The model inference service module reads the full training data from the data base and combines it with the feature vector database and the feature vector in the cache. It then uses the recommendation model for training and inference to output a personalized preference prediction for the user's program. The feedback and optimization module collects explicit and implicit user feedback on recommended content and dynamically adjusts the parameters of the recommendation model to achieve self-optimization of recommendation performance.

[0028] The distributed storage in the multi-source data governance module uses the Doris database; The feature vector database used in the offline profile building module is the Milvus vector database; In the near-line real-time computing module, the message middleware is Kafka, the stream processing engine is Flink, and the high-speed cache is Redis; The model inference service module is based on the EasyRec deep learning recommendation framework, employing a dual-tower model for the recall phase and a Wide&Deep model for the ranking phase.

[0029] The feedback and optimization module is specifically configured as follows: The collected feedback data is used to construct an incremental training dataset. The incremental training interface of the EasyRec framework is called to update the local parameters of the ranking model, and the updated model parameters are loaded into the inference service online. If a decline in core recommendation metrics is detected, the model version or strategy weights will be automatically rolled back to the previous stable version.

[0030] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention.

Claims

1. A method for IPTV program recommendation based on multi-source data, characterized in that, Includes the following steps: Step S1: Collect multi-source heterogeneous data and process the multi-source heterogeneous data. The data includes, but is not limited to, user behavior logs, program metadata, user basic information, and public network supplementary data. Perform unified cleaning, standardization processing, and distributed storage on the multi-source heterogeneous data to build a data foundation. Step S2: Based on the batch processing engine, extract historical data from the data base, perform feature engineering and vectorization on the user's basic attributes, long-term behavior aggregation indicators and preference distribution, as well as the program's static attributes, content tags and statistical indicators, to construct a full user profile and program profile, and persist them to the feature vector database. Step S3: Access the behavior logs reported in real time by the set-top box, extract real-time features from the behavior logs through the stream processing engine, and write the extracted real-time features into the cache; Step S4: Based on the full training data in the data base and the feature vectors in the feature vector database and cache, use the recommendation model for training and inference, and output personalized program recommendation prediction results for the target user; Step S5: Based on the feedback behavior data of the target user regarding the recommended content, dynamically adjust the parameters of the recommendation model to achieve continuous self-optimization of the recommendation effect.

2. The IPTV program recommendation method based on multi-source data according to claim 1, characterized in that, Step S1 specifically includes: The multi-source heterogeneous data is deduplicated, filtered for null values, and formatted. A legal time window threshold is set to remove log records with abnormal event times. To address the differences in data structures from different sources, field mapping and semantic normalization are performed to form a unified standard data model; The standardized data is used to build detailed tables and aggregate tables, which are then written into a distributed data warehouse.

3. The IPTV program recommendation method based on multi-source data according to claim 1, characterized in that, Step S2 specifically includes: Cleaned historical data is extracted from a distributed data warehouse to form a feature dataset; The word vector model is used to train user behavior sequences to generate user feature vectors, and the text embedding model is used to train program description text to generate program feature vectors. The two vectors are then unified to the same dimension and spliced ​​or weighted and fused. The completed user feature vectors and program feature vectors are written in batches to a high-performance vector database and loaded into the memory cache.

4. The IPTV program recommendation method based on multi-source data according to claim 1, characterized in that, Step S3 specifically includes: The message middleware is used to access the set-top box to report logs in real time, complete the format parsing and event recognition, and handle the log out-of-order problem based on the event time and water level mechanism; Data streams are split according to user identifiers, and a sliding window is used to count the viewing records and durations within the most recent preset time period to update the user's short-term preference vector. The calculated real-time features are serialized and written to the cache with the user identifier as the key, and a time-to-live is set.

5. The IPTV program recommendation method based on multi-source data according to claim 1, characterized in that, Step S4 specifically includes: The model training dataset is generated by extracting real user behavior as positive samples from a distributed data warehouse and constructing negative samples through exposure without clicks or random negative sampling. A recommendation model is built based on a deep learning recommendation framework, distributed training is initiated, and the intermediate model state is periodically saved through a checkpoint mechanism to support resuming training from breakpoints. The trained model is evaluated offline using a test set. Once the model's performance is confirmed to meet the standards, the service ratio of the new model is gradually increased to complete the release. Historical versions of the model are retained to support rollback operations.

6. The IPTV program recommendation method based on multi-source data according to claim 1, characterized in that, Step S5 specifically includes: After the recommendation results are uploaded in real time, explicit feedback behaviors such as program collection and subscription generated by users, as well as implicit feedback behaviors such as viewing time and page visits, are collected. The feedback behaviors are accurately attributed to the corresponding recommendation strategies and model versions, and an incremental training dataset is constructed based on the feedback behaviors. Using the incremental training interface of the deep learning recommendation framework, the incremental training dataset is input to the online training node. Only the fully connected layer or part of the embedding layer of the ranking model is locally updated. The online recommendation request is not interrupted during the parameter update process. The updated model parameters are directly loaded into the inference service online to achieve hot model update. After the model parameters are updated, the system continuously monitors the core recommendation business metrics. If an abnormal downward trend is detected in the core recommendation business metrics, the system will automatically trigger a rollback mechanism to restore the current recommendation model version or strategy weight to the previous stable version, ensuring the robustness and continuous self-optimization of the online recommendation service.

7. A multi-source data based IPTV program recommendation system for implementing the multi-source data based IPTV program recommendation method according to any one of claims 1-6, characterized in that, include: The multi-source data governance module collects heterogeneous data from multiple sources and completes unified cleaning, standardized processing, and distributed storage to build a data foundation. The offline profile building module extracts historical data from the data base based on the batch processing engine, performs feature engineering and vectorization on users and programs, builds full user and program profiles, and persists them to the feature vector database. The near-line real-time computing module accesses the behavior logs reported in real time by the set-top box, performs real-time feature extraction through the stream processing engine, and writes real-time user features into the high-speed cache; The model inference service module reads the full training data in the data base and combines it with the feature vector database and the feature vector in the cache. It then uses the recommendation model for training and inference to output a personalized preference prediction for the user's program. The feedback and optimization module collects explicit and implicit user feedback on recommended content and dynamically adjusts the parameters of the recommendation model to achieve self-optimization of recommendation performance.

8. The IPTV program recommendation system based on multi-source data according to claim 7, characterized in that, The distributed storage in the multi-source data governance module uses the Doris database. The offline profile building module uses Milvus vector database as its feature vector database; the near-line real-time computing module uses Kafka as its message middleware, Flink as its stream processing engine, and Redis as its cache. The model inference service module is based on the EasyRec deep learning recommendation framework, employing a dual-tower model to perform the recall phase and a Wide&Deep model to perform the ranking phase.

9. The IPTV program recommendation system based on multi-source data according to claim 7, characterized in that, The feedback and optimization module is specifically configured as follows: The collected feedback data is used to construct an incremental training dataset. The incremental training interface of the EasyRec framework is called to update the local parameters of the ranking model, and the updated model parameters are loaded into the inference service online. If a decline in core recommendation metrics is detected, the model version or strategy weights will be automatically rolled back to the previous stable version.

Images