Recommendation system data processing method and device based on data lake and storage medium
By using data lake technology to classify and store feature data in the recommendation system, and using the synergy between Flink, Lindorm and Paimon, the efficiency and timeliness of traditional systems when processing large-scale feature data is solved, and efficient storage and Deep utilization provides a solid technical foundation and provides high-quality training samples for machine learning.
Patent Information
- Application Number
- CN202411551835.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When traditional recommendation systems process large-scale and high-dimensional feature data, there are problems such as low storage and computing efficiency, untimely updates and poor scalability. Especially in scenarios with high real-time requirements, it is difficult to effectively store and quickly retrieve feature data, and it is impossible to Ensure the timeliness and accuracy of sample data.
Using the data lake-based recommended system data processing method, real-time processing is achieved by classifying feature data into short-term effective and long-term effective categories, using permanent storage and timed expired column clusters, and combining the synergy of Apache Flink, Lindorm, and Paimon and in-depth analysis to build a complete data processing and analysis process to ensure the timeliness and in-depth utilization of data.
It realizes efficient storage and access of feature data of the recommendation system, ensures the timeliness and deep utilization of data, provides high-quality training samples for machine learning, alleviates the problem of feature time travel, and provides a basis for continuous optimization of the recommendation system.
Smart Images

Figure CN119065613A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a data lake-based recommendation system data processing method, device and storage medium. Background Art
[0002] Recommendation systems have become the core components of many Internet applications. Their role is to accurately match user needs with massive amounts of information, improve user satisfaction and platform operational efficiency. However, with the surge in data volume, traditional recommendation systems have exposed problems such as low storage and computing efficiency, untimely updates, and poor scalability when processing large-scale, high-dimensional feature data. Especially in scenarios with high real-time requirements, it is difficult to effectively store and quickly retrieve feature data, and the timeliness and accuracy of sample data cannot be guaranteed.
[0003] In recent years, the rise of data lake technology has provided a new idea for solving the above problems. With its open and flexible data storage architecture, data lake supports the storage and analysis of massive data, especially in processing unstructured and semi-structured data. However, there is no related data processing method for recommendation system based on data lake technology in the existing technology. Summary of the invention
[0004] The present invention provides a data lake-based recommendation system data processing method, device and storage medium, which can effectively solve the above problems.
[0005] The present invention is achieved in that: The present invention provides a recommendation system data processing method based on a data lake, comprising the following steps: S1, classify the feature data of different modules, and then subdivide the feature data into short-term effective categories and long-term effective categories according to the real-time requirements of the feature data, and then use the permanent storage column cluster for the long-term effective features to ensure data integrity, and the short-term effective features are allocated to the timed expiration column cluster to achieve classified storage of the feature data of each user ID in the database; wherein, the short-term effective features include two categories: hourly update and daily update; S2, obtain the operation behavior generated by the user in the client app, and push the operation behavior to the Kafka message queue in real time, and use the stream processing capability of Apache Flink to obtain the operation behavior data stream from the Kafka message queue; S3, processing the data stream in real time, querying the corresponding user behavior and feature data in the database cache according to the user ID and operation behavior data stream, and constructing sample data including user ID, product ID, and operation behavior; S4, writing the sample data in batches into a Paimon partition table through Apache Flink, where the Paimon partition strategy is multi-dimensional partitioning based on time, user ID or operation type; S5, regularly perform offline data processing, conduct in-depth analysis on the sample data in the Paimon partition table to aggregate user exposure data, generate label information, and use the label information and feature data at the exposure moment as user training features, wherein the user training features are used to provide training samples for machine learning.
[0006] Preferably, short-term valid features are divided into high-frequency updated short-term valid features and low-frequency updated short-term valid features through a dynamic label mechanism, and the high-frequency updated short-term valid features are stored in the Redis cache, and the frequently updated short-term valid features are stored in Lindorm, so as to realize hierarchical storage and fast access of feature data of user ID.
[0007] Preferably, the classification of the short-term effective features and the long-term effective features is based on a machine learning model predicting the update frequency of different features to dynamically adjust the category of the features.
[0008] Preferably, the feature data of the long-term effective category includes gender, city, and mobile phone number location, which are updated in real time based on user input and directly allocated to the permanent storage column cluster for storage.
[0009] Preferably, the tag information includes real-time tags and offline tags, the real-time tags are used for real-time recommendation scenarios for users, and the offline tags provide long-term training data for machine learning models.
[0010] Preferably, after obtaining the operation behavior data stream from the Kafka message queue using the stream processing capability of Apache Flink, the method further includes: Detect abnormal data in the operation behavior data stream, wherein the abnormal data includes repeated behaviors and invalid data.
[0011] A second embodiment of the present invention provides a data lake-based recommendation system data processing device, including: processor; A memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute any of the methods described above.
[0012] A third embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement any of the methods described above.
[0013] The beneficial effects of the present invention are as follows: a data processing method, device and storage medium for a recommendation system based on a data lake, firstly, classifies the feature data of different modules, and then subdivides the feature data into short-term valid categories and long-term valid categories according to the real-time requirements of the feature data, and then uses a permanent storage column cluster for the long-term valid features to ensure data integrity, while the short-term valid features are allocated to a timed expiration column cluster, so as to realize the classified storage of the feature data of each user ID in Lindorm, thereby solving the efficient storage and access of the feature data of the recommendation system; secondly, combining the synergy of Flink, Lindorm and Paimon, constructing a complete set of data processing and analysis processes, realizing the timeliness of features and the deep utilization of data, and providing a solid technical foundation for the continuous optimization of the recommendation system; finally, using the features when the exposure log is generated as the training features of the user, so as to provide high-quality training samples for the machine learning model, and further, can greatly alleviate the problem of feature time travel. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0015] Figure 1 is a flow chart of a data processing method for a recommendation system based on a data lake provided by an embodiment of the present invention; Figure 2 It is a storage table of the feature data provided by the present invention in Lindorm; Figure 3 It is a user behavior and feature data table for real-time query of Lindorm provided by the present invention; Figure 4 It is a calculation result table of the user behavior and feature data table provided by the present invention. DETAILED DESCRIPTION
[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0017] Reference Figure 1 As shown, an embodiment of the present invention provides a data processing method for a recommendation system based on a data lake, comprising the following steps: S1, classify the feature data of different modules, and then subdivide the feature data into short-term effective categories and long-term effective categories according to the real-time requirements of the feature data. The long-term effective features are stored in permanent column clusters to ensure data integrity, while the short-term effective features are allocated to timed expiration column clusters to achieve classified storage of feature data of each user ID in Lindorm. The short-term effective features include two categories: hourly update and daily update. S2, obtain the operation behavior generated by the user in the client app, and push the operation behavior to the Kafka message queue in real time, and use the stream processing capability of Apache Flink to obtain the operation behavior data stream from the Kafka message queue; S3, processing the data stream in real time, querying the corresponding user behavior and feature data in the database cache according to the user ID and operation behavior data stream, and constructing sample data including user ID, product ID, and operation behavior; S4, writing the sample data in batches into a Paimon partition table through Apache Flink, where the Paimon partition strategy is multi-dimensional partitioning based on time, user ID or operation type; S5, regularly perform offline data processing, conduct in-depth analysis on the sample data in the Paimon partition table to aggregate user exposure data, generate label information, and use the label information and feature data at the exposure moment as user training features, wherein the user training features are used to provide training samples for machine learning.
[0018] In step S1, long-term effective category feature data is a real-time update feature, which is directly allocated to the permanent storage column cluster for storage to ensure the latest status; while the hourly updated and daily updated feature data are aggregated and then allocated to the timed expiration column cluster for storage to reduce storage space and improve query efficiency.
[0019] See also Figure 2 Specifically, the long-term effective feature data may include gender (sex), city (city), and mobile phone number location (ump), which are updated in real time based on user input and directly assigned to a permanent storage column cluster (such as column cluster 1 in the table) for storage. In addition, the most recent recharge time (ilot) in column cluster 1 is also updated in real time based on user behavior records.
[0020] The effective features updated daily include data such as "click rate in the past day (frm1r)", "click rate in the past 3 days (frm3r)" and "user consumption related data (d:consume_fes)", which can be added and adjusted according to actual needs and are not limited here. Among them, the "click rate in the past day (frm1r)" and "click rate in the past 3 days (frm3r)" are aggregated through the user behavior record table, and then allocated to the timed expiration column cluster for storage (for example, as shown in column cluster d in the table); the "user consumption related data (d:consume_fes)" is formed by aggregating and merging "wealth level (g7)", "past day consumption amount (sas)" and "past day recharge amount (plras)" into one field, and then allocated to the timed expiration column cluster (for example, as shown in column cluster d in the table) for storage. If the information updated daily is not updated the next day, it means that the feature information is expired and will be cleaned up regularly.
[0021] Furthermore, the hourly updated effective features include "list of content clicked in the past hour", etc., which can be added and adjusted according to actual needs, and are not limited here. The "list of content clicked in the past hour (h:ms_fes)" is aggregated through the user behavior record table, and then allocated to the timed expiration column cluster (column cluster h) for storage. If the user does not operate for an hour and the information is not updated, it means that the feature information is expired and will be cleaned up regularly.
[0022] Furthermore, in this embodiment, the dynamic tag mechanism is subdivided into two categories: high-frequency update and low-frequency update. The feature data of high-frequency update is stored in the Redis cache for fast access and to meet real-time requirements, while the feature data of low-frequency update is stored in Lindorm for persistent storage and management of data. It should be noted that different storage locations and strategies can be used according to the update frequency of the feature, so as to achieve hierarchical management of the feature data of the user ID, making data access faster and storage more efficient.
[0023] Of course, in a possible implementation of the present invention, a machine learning model can also be used to predict the update frequency of each feature, dynamically dividing the features into short-term and long-term categories, and through dynamic machine learning classification and hierarchical storage strategies, the system can adapt to different update requirements of feature data. In addition, the combination of Redis and Lindorm not only achieves real-time response, but also meets long-term data management needs.
[0024] In step S2, the client pushes the user's operation behavior in the app to the embedded point kafka in real time, and uses the stream processing capability of Apache Flink to consume and process the data stream of the kafka queue to obtain the user_ID, product ID, and operation behavior, thereby ensuring the timeliness and continuity of the data and laying the foundation for the real-time generation of samples.
[0025] In step S3, based on the user's operation behavior on the client, a data stream containing the user ID, product ID, and operation behavior is obtained in real time, and the data stream is transferred to the database cache. The user ID is then used to quickly query the corresponding user behavior data and feature data in the cache, and this information is integrated with the current operation behavior to construct sample data containing the user ID, product ID, and operation behavior. In this process, real-time data update and feature splicing are achieved through stream processing technology, so that the sample data can accurately reflect the user's latest operations and feature changes. See Figure 3 : Figure 3 To query Lindorm's user behavior and feature data in real time, we can query the corresponding fields in the l-column cluster and the d-column cluster in Lindorm by using the user ID. Similarly, the product ID also has corresponding feature information, which is not listed here.
[0026] In step S4, the step of efficiently writing the sample data into the partition table of Paimon through Apache Flink specifically includes: Through real-time processing of data streams, key information such as user ID, product ID, and operation behavior are extracted and integrated to generate sample data containing user behavior characteristics. Subsequently, the generated sample data is written in batches into Paimon's partition table using the efficient writing function of Apache Flink. To ensure efficient data access and management, the partitioning strategy of Paimon's partition table is based on a multi-dimensional design, which may include but is not limited to time, user ID, and operation type. Based on the multi-dimensional partitioning method, data can be quickly located according to different time periods, user characteristics, or behavior types, facilitating subsequent query and analysis.
[0027] In step S5, the user's exposure records are aggregated to form a basic data set containing information such as user ID, exposure content, and exposure times. Then, the user's training features are generated in combination with the user's exposure time and related feature data. These training features, together with the aggregated label data, constitute high-quality machine learning training samples. Label information is divided into real-time labels and offline labels according to the application scenario. Real-time labels are suitable for scenarios where users are recommended in real time, and can help the model respond to users' immediate needs in a timely manner. Offline labels are based on longer-term data accumulation and serve as long-term training data for machine learning models, providing in-depth information for the continuous optimization of the model. The system can not only adapt to the dynamic recommendation needs of real-time scenarios, but also continuously train models on long-term accumulated data, thereby maintaining the accuracy of predictions and recommendations over a longer time scale.
[0028] Furthermore, the problem of feature time travel can be greatly alleviated. For example, the time of the last recharge of user 2 changed between exposure and click. This may be caused by the exposure of the current product 20100. If the feature data at the time of click is used, the problem of feature travel will occur. The standardized training data generated by parsing and flattening the field is as follows, see Figure 4 shown.
[0029] Figure 4 for Figure 3 The result of calculating the features after label aggregation and feature splitting of the data. For example, user 1 only has exposure data, so the label is 0, and user 2 has click behavior, so the label is 1. At the same time, some features are standardized, such as the latest recharge time is uniformly converted to the number of hours from now, and the recharge amount in the past day is bucketed according to the interval.
[0030] Paimon tables not only support efficient writing and querying, but can also be used as DataWorks external tables and integrated into DataWorks' data development process for more complex data analysis and monitoring. Through DataWorks' ETL operations, we can further explore data value, optimize recommendation algorithms, and ensure sample stability and quality.
[0031] The embodiment of the present invention further provides a data lake-based recommendation system data processing device, including: processor; A memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the data lake-based recommendation system data processing method.
[0032] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor of a device where the computer-readable storage medium is located to implement the data lake-based recommendation system data processing method.
[0033] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0034] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0035] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0036] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0037] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A data processing method for a recommendation system based on a data lake, characterized in that: The following steps are involved: S1, classify the feature data of different modules, and then subdivide the feature data into short-term effective categories and long-term effective categories according to the real-time requirements of the feature data, and then use the permanent storage column cluster for the long-term effective features to ensure data integrity, while the short-term effective features are allocated to the timed expiration column cluster to achieve classified storage of the feature data of each user ID in the database; wherein the short-term effective features include two categories: hourly update and daily update; S2, obtain the operation behavior generated by the user in the client app, and push the operation behavior to the Kafka message queue in real time, and use the stream processing capability of Apache Flink to obtain the operation behavior data stream from the Kafka message queue; S3, processing the data stream in real time, querying the corresponding user behavior and feature data in the database cache according to the user ID and operation behavior data stream, and constructing sample data including user ID, product ID, and operation behavior; S4, writing the sample data in batches into a Paimon partition table through Apache Flink, where the Paimon partition strategy is multi-dimensional partitioning based on time, user ID or operation type; S5, regularly perform offline data processing, conduct in-depth analysis on the sample data in the Paimon partition table to aggregate user exposure data, generate label information, and use the label information and feature data at the exposure moment as user training features, wherein the user training features are used to provide training samples for machine learning.
2. The data processing method for a recommendation system based on a data lake according to claim 1, characterized in that: In step S1, short-term valid features are divided into high-frequency updated short-term valid features and low-frequency updated short-term valid features through a dynamic label mechanism, and the high-frequency updated short-term valid features are stored in the Redis cache, and the frequently updated short-term valid features are stored in Lindorm, so as to realize hierarchical storage and fast access of feature data of user ID.
3. The data processing method for a recommendation system based on a data lake according to claim 1, characterized in that: The classification of the short-term effective features and the long-term effective features is based on a machine learning model predicting the update frequency of different features to dynamically adjust the category of the features.
4. The data processing method for a recommendation system based on a data lake according to claim 1, characterized in that: The feature data of the long-term effective category includes gender, city, and mobile phone number location, which are updated in real time based on user input and directly allocated to the permanent storage column cluster for storage.
5. The data processing method for a recommendation system based on a data lake according to claim 1, characterized in that: The tag information includes real-time tags and offline tags. The real-time tags are used for real-time recommendation scenarios for users, and the offline tags provide long-term training data for machine learning models.
6. The data processing method for a recommendation system based on a data lake according to claim 1, characterized in that: After obtaining the operation behavior data stream from the Kafka message queue using the stream processing capability of Apache Flink, the method further includes: Detect abnormal data in the operation behavior data stream, wherein the abnormal data includes repeated behaviors and invalid data.
7. A data processing device for a recommendation system based on a data lake, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Flink streaming processing engine method and device for real-time recommendation and computer equipment
CA3150183A1
User behavior statistical analysis method based on Flink streaming processing
CN112000636A
Flink streaming processing engine method and device for real-time recommendation and computer equipment
CN112948450A
Real-time intelligent marketing recommendation system and method based on Flink framework and application of real-time intelligent marketing recommendation system and method
CN115936787A
