Real-time data recommendation method and device, storage medium and electronic equipment

Abstract

The invention relates to a real-time data recommendation method and device, a storage medium and electronic equipment. The method comprises the steps of sending real-time data to a real-time queue under the condition that the real-time data is received; sending all total data in the database table to a real-time queue; adding each piece of data in the real-time queue into a tense table corresponding to the respective identification number, and recording the respective update time at the same time; according to the data with the latest updating time in each tense table, a data lake table is updated, and the data lake table is used for recommending an algorithm to use all data in the data lake table to generate a real-time recommendation result. The technical problem that the system development and maintenance cost is too high due to the fact that two different big data computing frameworks are used in the content pool production process is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a real-time data recommendation method, device, storage medium and electronic device. Background Art

[0002] The data production architecture of the recommended content pool of most video software in the prior art adopts the Lambda architecture widely used in the industry. This architecture divides data production into two main links: the offline data production link and the real-time data production link. The offline data production link processes the full amount of data. Specifically, the offline link is responsible for calculating all the recommended video content pools in all recommendation scenarios. Due to the huge amount of data processed, offline data production is only performed once a day. In the offline data production link, the Spark framework is used for data calculation. Spark is a distributed data processing engine that is good at processing large-scale data sets and can effectively perform complex analysis tasks. The results of offline calculations are finally stored in Hive tables. Hive is a data warehouse system based on Hadoop that can process and query large-scale data sets. The daily calculation results store the content pool data in the Hive table for use by the algorithms of various recommendation scenarios. Unlike the offline link, the real-time data production link processes incremental data. Only when the data changes will it enter the real-time data production link to recalculate the content pool to which the video belongs. The real-time data production link uses the Flink framework for data processing. Flink is a real-time stream processing engine that can process infinite data streams and provide low-latency and high-throughput data processing capabilities. The results of real-time processing are ultimately stored in the Kafka real-time queue. Kafka is a distributed stream processing platform that is good at processing real-time data streams and provides message publishing and subscription functions. The downstream system obtains real-time information of the video content pool by listening to the real-time messages in the Kafka queue.

[0003] One feature of the Lambda architecture is that data processing is divided into two parts: offline and real-time, which process full and incremental data respectively. This means that two different sets of code need to be prepared to process data during implementation. Although some business logic can be reused, the frameworks used for offline and real-time processing are different, and the specific implementation methods are also different. First, the Spark framework is used for offline data production, while the Flink framework is used for real-time data production. The use of this dual framework ensures that the system can efficiently process different data volumes and data types. Spark is suitable for processing large-scale batch data, while Flink is good at processing real-time streaming data. The data produced by offline tasks is stored in Hive tables, while the data produced by real-time tasks is stored in Kafka real-time queues. This separation of data storage methods makes the system more flexible and efficient when processing and storing different types of data. This architecture brings certain challenges to the use of downstream systems. If the algorithm wants to build a real-time recommendation and recall system, it needs to connect to two heterogeneous data sources at the same time: offline Hive tables and real-time Kafka queues. Connecting and integrating these two different data sources puts higher requirements and costs on the development and maintenance of downstream systems. The downstream system not only needs to process the full amount of data stored offline, but also needs to monitor the incremental data in the Kafka queue in real time. This complex data processing and docking puts higher requirements on the stability and performance of the system. Therefore, in the existing technology, two different big data computing frameworks are used in the content pool production process. Maintaining two sets of code will result in high maintenance costs and high system complexity. Summary of the invention

[0004] The present application provides a real-time data recommendation method, device, storage medium and electronic device to solve the technical problem that the cost of system development and maintenance is too high due to the use of two different big data computing frameworks in the content pool production process.

[0005] In a first aspect, the present application provides a real-time recommendation method for data, comprising: when real-time data is received, sending the real-time data to a real-time queue; sending all the full data in a database table to the real-time queue; adding each data in the real-time queue to a temporal table corresponding to its respective identification number, and recording its respective update time; updating a data lake table according to the data with the latest update time in each of the temporal tables, wherein the data lake table is used for a recommendation algorithm to generate real-time recommendation results using all the data in the data lake table.

[0006] In the second aspect, the present application provides a real-time recommendation device for data, including: a first sending module, used to send the real-time data to a real-time queue when the real-time data is received; a second sending module, used to send all the full data in the database table to the real-time queue; an adding module, used to add each data in the real-time queue to a temporal table corresponding to the respective identification number, and record the respective update time; an updating module, used to update the data lake table according to the data with the latest update time in each of the above temporal tables, wherein the data lake table is used for the recommendation algorithm to generate real-time recommendation results using all the data in the data lake table.

[0007] As an optional example, the above-mentioned device also includes: a saving module, which is used to send the above-mentioned real-time data to the real-time queue and save the above-mentioned real-time data to the above-mentioned database table when receiving the real-time data.

[0008] As an optional example, the above-mentioned adding module includes: a first processing unit, used to take each data in the above-mentioned real-time queue as current data, and perform the following operations on the above-mentioned current data: obtain the identification number of the above-mentioned current data; search for the target temporal table corresponding to the identification number of the above-mentioned current data in all the above-mentioned temporal tables; if the above-mentioned target temporal table exists in all the above-mentioned temporal tables, add the above-mentioned current data and its update time to the above-mentioned target temporal table.

[0009] As an optional example, the first processing unit is further used to search for a target temporal table corresponding to the identification number of the current data in all the temporal tables, and if the target temporal table does not exist in all the temporal tables, create the target temporal table; and add the current data and its update time to the target temporal table.

[0010] As an optional example, the above-mentioned device also includes: a processing module, which is used to, after adding each data in the above-mentioned real-time queue to the temporal table corresponding to the respective identification number, use each of the above-mentioned temporal tables as the current temporal table, and perform the following operations on the above-mentioned current temporal table: sort all the data in the above-mentioned current temporal table from late to early according to the update time.

[0011] As an optional example, the update module includes: an acquisition unit, used to acquire the first row of data in each of the above-mentioned temporal tables and determine it as the first data; a second processing unit, used to take each of all the above-mentioned first data as current data, and perform the following operations on the above-mentioned current data: search for the target identification number of the above-mentioned current data in the above-mentioned data lake table; if the above-mentioned target identification number exists in the above-mentioned data lake table, modify the second data corresponding to the above-mentioned target identification number in the above-mentioned data lake table to the above-mentioned first data.

[0012] As an optional example, the second processing unit is also used to save the second data to the data lake table after searching for the target identification number of the current data in the data lake table if the target identification number does not exist in the data lake table.

[0013] In a third aspect, the present application provides a storage medium storing a computer program, wherein the computer program executes the above-mentioned real-time recommendation method for data when executed by a processor.

[0014] In a fourth aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the real-time recommendation method for the data through the computer program.

[0015] In an embodiment of the present application, when real-time data is received, the real-time data is sent to the real-time queue; all the full data in the database table is sent to the real-time queue; each data in the real-time queue is added to the temporal table corresponding to the respective identification number, and the respective update time is recorded; according to the data with the latest update time in each of the temporal tables, the data lake table is updated, wherein the data lake table is used for the recommendation algorithm to generate real-time recommendation results using all the data in the data lake table. Since in the above method, both the full data and the real-time data are sent to the source of the real-time data production link, that is, the real-time queue, Then, the temporal table and its deduplication query characteristics are used to deduplicate offline data and real-time data, retaining only the latest data and finally saving it in the data lake table, so that the output process of the recommendation content pool is merged from the original two data links into one, and the final data processing endpoint is also only saved in a unique and near-real-time updated data lake table. The downstream only needs to connect to the data in the data lake to obtain near-real-time content pool information, thereby significantly improving the content distribution efficiency and reducing the maintenance cost of the recommendation system, and thus solving the technical problem that the use of two different big data computing frameworks in the content pool production process leads to too high a cost of system development and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0019] Figure 1 is a flowchart of an optional real-time data recommendation method according to an embodiment of the present application;

[0020] Figure 2 is a schematic structural diagram of an optional real-time data recommendation device according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0024] According to a first aspect of an embodiment of the present application, a real-time data recommendation method is provided. Optionally, as follows: Figure 1 As shown, the above method includes:

[0025] S102, when real-time data is received, sending the real-time data to a real-time queue;

[0026] S104, sending all the full amount of data in the database table to the real-time queue;

[0027] S106, adding each data in the real-time queue to the temporal table corresponding to the respective identification number, and recording the respective update time;

[0028] S108, updating the data lake table according to the data with the latest update time in each temporal table, wherein the data lake table is used by the recommendation algorithm to generate real-time recommendation results using all the data in the data lake table.

[0029] Optionally, the amount of video data in the entire site of each video software in the prior art is huge. As a part of content distribution, the recommendation system plays an important role in the distribution of video software. The most important initial step of the recommendation system is to build a recommendation scenario content pool. There are a large number of recommendation scenarios in video software, such as when a user just enters multiple channels on the homepage, these are personalized recommendation scenarios. Different recommendation scenarios have different requirements for content pools. For example, the homepage hopes to push newly launched hot dramas, while the movie channel may push classic movies with high ratings in film history, which may be works from a long time ago. Therefore, it is necessary to circle and define different recommendation content pools for different recommendation scenarios. All video metadata of video software is stored in a unified data platform within the company. The recommended content pool data production system originally adopted the Lambda architecture, which includes offline data production links and real-time data production links. Offline data production link: According to certain rules, content pool data that meets the requirements of each recommendation scenario is screened from the unified data platform every day and stored in the Hive offline data warehouse. The algorithm team of each recommendation scenario will extract the video information of the content pool from Hive, perform subsequent algorithm model training and other steps, and select the content that users are most interested in and recommend it to users. Since all data is calculated and filtered, the calculation of the offline data link takes several hours. Real-time data production link: only processes video content with changed data. For example, when a new drama is launched, its metadata is updated. The content management platform listens to the metadata message changes, recalculates the current content pool for each changed video in the real-time data production link, and sends the calculation results to the Kafka real-time queue. The downstream algorithm team can obtain the latest online video by connecting to the real-time queue to build real-time recommendations. Therefore, the content management platform needs to maintain two data links, and due to the different amounts of data processed, different big data processing frameworks (such as Spark and Flink) need to be used, which means that two sets of codes need to be maintained, which increases the complexity and maintenance cost of the system. In addition, the full content pool data is provided to the recommendation algorithm team through two storage media, one is the Hive offline data warehouse, and the other is the real-time Kafka queue. To achieve real-time recommendations, the algorithm team needs to connect to two different types of data sources and integrate two copies of data, which increases a lot of development and maintenance work.

[0030] Therefore, in this embodiment, a recommendation content pool production batch stream integrated architecture based on Flink (Apache Flink, a distributed stream processing and batch processing framework) and IceBerg (Apache Iceberg, an open table format for large-scale data lake management) is proposed. Apache Flink is an open source stream processing framework, mainly used for distributed data processing, especially real-time and batch processing of large-scale data. It has high performance, low latency and flexible processing capabilities, and can process and analyze large-scale data streams and data sets. Apache Iceberg is a high-performance open source table format library designed for large-scale data lake storage. Its main purpose is to solve some performance and manageability problems encountered by traditional big data table formats such as Apache Hive when processing large-scale data. This architecture abandons the original offline data production link, sends the full amount of offline data to the real-time production link every day, and deduplicates offline and real-time tasks through the temporal table and deduplication query capabilities of the Flink framework, retaining only the latest data of each video. Among them, the real-time queue is the Kafka real-time queue, the temporal table is the temporal table generated by Flink, and the data lake table is the IceBerg table. Apache Kafka is an open source distributed stream processing platform that is mainly used for building real-time data pipelines and streaming applications. It is used for high-throughput, low-latency data stream processing while being highly scalable and fault-tolerant.

[0031] The specific implementation is as follows: The Kafka source queue will receive both real-time data that changes in real time and full data once a day. Therefore, due to the simultaneous processing of two types of data, duplicate data IDs (Identifiers) will appear in the Kafka source queue. The Flink computing task generates a temporal table for each data ID. A temporal table is a table that can store historical states and record the state of data at different time points. In each temporal table, Flink sorts the same data IDs according to the "last update time" to ensure that the latest data is always in the first row. In this way, data deduplication can be achieved by extracting the first row of sorted data from each temporal table and retaining the latest record. When a piece of data is written to the IceBerg table, IceBerg will check whether the data ID already exists in the table. If the data ID already exists, IceBerg will overwrite it to ensure that the latest data is always retained in the table. If the data ID does not exist, IceBerg will add a new record to save the data. Through this update mechanism, the IceBerg table can keep the latest data at any time. Therefore, the recommendation algorithm team only needs to connect to the IceBerg data lake to obtain the latest online data and build real-time recommendation results. This method not only simplifies the data processing process, improves the real-time and accuracy of the data, but also significantly reduces the maintenance cost of the system. The recommendation algorithm team can obtain the latest video data more efficiently and build more accurate real-time recommendation results, thereby improving the user experience.

[0032] Optionally, in this embodiment, both the full data and the real-time data are sent to the source of the real-time data production link, that is, the real-time queue, and then the temporal table and its deduplication query characteristics are used to deduplicate the offline data and the real-time data, and only the latest data is retained and finally saved in the data lake table, so that the output process of the recommendation content pool is merged from the original two data links into one, and the final data processing end point is also only saved in a unique and near real-time updated data lake table. The downstream only needs to connect to the data in the data lake to obtain near-real-time content pool information, thereby achieving the purpose of significantly improving the content distribution efficiency and reducing the maintenance cost of the recommendation system, and further solving the technical problem that the use of two different big data computing frameworks in the content pool production process leads to too high a cost of system development and maintenance.

[0033] As an optional example, when real-time data is received, while sending the real-time data to the real-time queue, the method further includes:

[0034] Save real-time data to database tables.

[0035] Optionally, in this embodiment, when real-time data is received, in order to ensure the integrity and consistency of the data, in addition to sending the real-time data to the real-time queue, the real-time data also needs to be saved in a database table.

[0036] As an optional example, each data in the real-time queue is added to a temporal table corresponding to the respective identification number, and the respective update time is recorded, including:

[0037] Take each data in the real-time queue as the current data and perform the following operations on the current data:

[0038] Get the identification number of the current data;

[0039] Find the target temporal table corresponding to the identification number of the current data in all temporal tables;

[0040] If the target temporal table exists in all temporal tables, the current data and its update time are added to the target temporal table.

[0041] As an optional example, after searching all temporal tables for a target temporal table corresponding to the identification number of the current data, the method further includes:

[0042] If the target temporal table does not exist in all temporal tables, create the target temporal table;

[0043] Add the current data and the time it was updated to the target temporal table.

[0044] Optionally, in this embodiment, each piece of data usually contains a unique identification number and other relevant information, including update time. The Kafka source queue will simultaneously receive real-time data that changes in real time and full data once a day. Therefore, due to the simultaneous processing of two types of data, duplicate data identification numbers will appear in the Kafka source queue. The Flink computing task generates a temporal table for each data identification number. The temporal table is a table that can store historical states and can record the state of data at different time points. When adding each data in the real-time queue to the temporal table corresponding to the respective identification number, taking a certain data as an example, a unique identification number is extracted from the data, and the target temporal table corresponding to the identification number of the data is searched in all available temporal tables. If the target temporal table exists, the data and its update time are added to the target temporal table. If the target temporal table does not exist, a new temporal table is created according to the data identification number. The new temporal table will be used to store all data corresponding to the identification number, and then the data and its update time are added to the newly created target temporal table. By analogy, all data in the Kafka source queue are added to the corresponding temporal table.

[0045] As an optional example, after adding each data in the real-time queue to the temporal table corresponding to the respective identification number, the method further includes:

[0046] Take each temporal table as the current temporal table and perform the following operations on the current temporal table:

[0047] Sort all data in the current temporal table according to the update time from latest to earliest.

[0048] Optionally, in this embodiment, after each data is added to the corresponding temporal table, the data in the temporal table is sorted, and all the data in the temporal table are sorted from latest to earliest according to the update time, which means that the most recently updated data will be at the front, ensuring that the latest data is always in the first row. In this way, only the first row of sorted data needs to be extracted from each temporal table to achieve data deduplication and retain the latest record. The temporal table can store the state of data at different time points, support historical tracking of data and query of the latest state. Through the temporal table, the system can record the version and changes of the data to ensure that the latest or specific time point data is used during processing and analysis.

[0049] As an optional example, updating the data lake table according to the latest updated data in each temporal table includes:

[0050] Obtain the first row of data in each temporal table and determine it as the first data;

[0051] Take each data in all the first data as the current data, and perform the following operations on the current data:

[0052] Find the target identification number of the current data in the data lake table;

[0053] When the target identification number exists in the data lake table, the second data corresponding to the target identification number in the data lake table is modified to the first data.

[0054] As an optional example, after searching the data lake table for the target identification number of the current data, the method further includes:

[0055] If the target identification number does not exist in the data lake table, the second data is saved in the data lake table.

[0056] Optionally, in this embodiment, IceBerg has the ability to update records compared to the traditional offline data warehouse Hive. Hive can only add new records, while IceBerg can flexibly handle data updates and keep the data up to date. When a piece of data is written to the IceBerg table, IceBerg will check whether the identification number of the data already exists in the table. If the identification number of the data already exists, IceBerg will overwrite it to ensure that the latest data is always retained in the table. If the identification number of the data does not exist, IceBerg will add a new record to save the data. Through this update mechanism, the IceBerg table can keep the latest data at any time. Therefore, the recommendation algorithm team only needs to connect to the IceBerg data lake to obtain the latest online data and build real-time recommendation results. Specifically, the first row of data is extracted from each temporal table and determined as the first data. Each first data is used as the current data and processed in sequence. The target identification number corresponding to the identification number of the current data is searched in the IceBerg table. If the target identification number exists, the second data corresponding to the target identification number in the IceBerg table is modified to the first data. If the target identification number does not exist, the first data is saved to the IceBerg table.

[0057] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0058] According to another aspect of the embodiment of the present application, a real-time data recommendation device is also provided. Figure 2 As shown, including:

[0059] A first sending module 202 is used to send the real-time data to the real-time queue when receiving the real-time data;

[0060] The second sending module 204 is used to send all the full amount of data in the database table to the real-time queue;

[0061] An adding module 206, for adding each data in the real-time queue to a temporal table corresponding to the respective identification number, and recording the respective update time;

[0062] The updating module 208 is used to update the data lake table according to the data with the latest update time in each temporal table, wherein the data lake table is used for the recommendation algorithm to generate real-time recommendation results using all the data in the data lake table.

[0063] Optionally, the amount of video data in the entire site of each video software in the prior art is huge. As a part of content distribution, the recommendation system plays an important role in the distribution of video software. The most important initial step of the recommendation system is to build a recommendation scenario content pool. There are a large number of recommendation scenarios in video software, such as when a user just enters multiple channels on the homepage, these are personalized recommendation scenarios. Different recommendation scenarios have different requirements for content pools. For example, the homepage hopes to push newly launched hot dramas, while the movie channel may push classic movies with high ratings in film history, which may be works from a long time ago. Therefore, it is necessary to circle and define different recommendation content pools for different recommendation scenarios. All video metadata of video software is stored in a unified data platform within the company. The recommended content pool data production system originally adopted the Lambda architecture, which includes offline data production links and real-time data production links. Offline data production link: According to certain rules, content pool data that meets the requirements of each recommendation scenario is screened from the unified data platform every day and stored in the Hive offline data warehouse. The algorithm team of each recommendation scenario will extract the video information of the content pool from Hive, perform subsequent algorithm model training and other steps, and select the content that users are most interested in and recommend it to users. Since all data is calculated and filtered, the calculation of the offline data link takes several hours. Real-time data production link: only processes video content with changed data. For example, when a new drama is launched, its metadata is updated. The content management platform listens to the metadata message changes, recalculates the current content pool for each changed video in the real-time data production link, and sends the calculation results to the Kafka real-time queue. The downstream algorithm team can obtain the latest online video by connecting to the real-time queue to build real-time recommendations. Therefore, the content management platform needs to maintain two data links, and due to the different amounts of data processed, different big data processing frameworks (such as Spark and Flink) need to be used, which means that two sets of codes need to be maintained, which increases the complexity and maintenance cost of the system. In addition, the full content pool data is provided to the recommendation algorithm team through two storage media, one is the Hive offline data warehouse, and the other is the real-time Kafka queue. To achieve real-time recommendations, the algorithm team needs to connect to two different types of data sources and integrate two copies of data, which increases a lot of development and maintenance work.

[0064] Therefore, in this embodiment, a recommendation content pool production batch stream integrated architecture based on Flink (Apache Flink, a distributed stream processing and batch processing framework) and IceBerg (Apache Iceberg, an open table format for large-scale data lake management) is proposed. Apache Flink is an open source stream processing framework, mainly used for distributed data processing, especially real-time and batch processing of large-scale data. It has high performance, low latency and flexible processing capabilities, and can process and analyze large-scale data streams and data sets. Apache Iceberg is a high-performance open source table format library designed for large-scale data lake storage. Its main purpose is to solve some performance and manageability problems encountered by traditional big data table formats such as Apache Hive when processing large-scale data. This architecture abandons the original offline data production link, sends the full amount of offline data to the real-time production link every day, and deduplicates offline and real-time tasks through the temporal table and deduplication query capabilities of the Flink framework, retaining only the latest data of each video. Among them, the real-time queue is the Kafka real-time queue, the temporal table is the temporal table generated by Flink, and the data lake table is the IceBerg table. Apache Kafka is an open source distributed stream processing platform that is mainly used for building real-time data pipelines and streaming applications. It is used for high-throughput, low-latency data stream processing while being highly scalable and fault-tolerant.

[0065] The specific implementation is as follows: The Kafka source queue will receive both real-time data that changes in real time and full data once a day. Therefore, due to the simultaneous processing of two types of data, duplicate data IDs (Identifiers) will appear in the Kafka source queue. The Flink computing task generates a temporal table for each data ID. A temporal table is a table that can store historical states and record the state of data at different time points. In each temporal table, Flink sorts the same data IDs according to the "last update time" to ensure that the latest data is always in the first row. In this way, data deduplication can be achieved by extracting the first row of sorted data from each temporal table and retaining the latest record. When a piece of data is written to the IceBerg table, IceBerg will check whether the data ID already exists in the table. If the data ID already exists, IceBerg will overwrite it to ensure that the latest data is always retained in the table. If the data ID does not exist, IceBerg will add a new record to save the data. Through this update mechanism, the IceBerg table can keep the latest data at any time. Therefore, the recommendation algorithm team only needs to connect to the IceBerg data lake to obtain the latest online data and build real-time recommendation results. This method not only simplifies the data processing process, improves the real-time and accuracy of the data, but also significantly reduces the maintenance cost of the system. The recommendation algorithm team can obtain the latest video data more efficiently and build more accurate real-time recommendation results, thereby improving the user experience.

[0066] Optionally, in this embodiment, both the full data and the real-time data are sent to the source of the real-time data production link, that is, the real-time queue, and then the temporal table and its deduplication query characteristics are used to deduplicate the offline data and the real-time data, and only the latest data is retained and finally saved in the data lake table, so that the output process of the recommendation content pool is merged from the original two data links into one, and the final data processing end point is also only saved in a unique and near real-time updated data lake table. The downstream only needs to connect to the data in the data lake to obtain near-real-time content pool information, thereby achieving the purpose of significantly improving the content distribution efficiency and reducing the maintenance cost of the recommendation system, and further solving the technical problem that the use of two different big data computing frameworks in the content pool production process leads to too high a cost of system development and maintenance.

[0067] As an optional example, the above device further includes:

[0068] The saving module is used to send the real-time data to the real-time queue and save the real-time data in the database table when receiving the real-time data.

[0069] Optionally, in this embodiment, when real-time data is received, in order to ensure the integrity and consistency of the data, in addition to sending the real-time data to the real-time queue, the real-time data also needs to be saved in a database table.

[0070] As an optional example, adding modules includes:

[0071] The first processing unit is used to take each data in the real-time queue as current data and perform the following operations on the current data:

[0072] Get the identification number of the current data;

[0073] Find the target temporal table corresponding to the identification number of the current data in all temporal tables;

[0074] If the target temporal table exists in all temporal tables, the current data and its update time are added to the target temporal table.

[0075] As an optional example, the first processing unit is further configured to create the target temporal table if the target temporal table does not exist in all the temporal tables after searching for the target temporal table corresponding to the identification number of the current data in all the temporal tables;

[0076] Add the current data and the time it was updated to the target temporal table.

[0077] Optionally, in this embodiment, each piece of data usually contains a unique identification number and other relevant information, including update time. The Kafka source queue will simultaneously receive real-time data that changes in real time and full data once a day. Therefore, due to the simultaneous processing of two types of data, duplicate data identification numbers will appear in the Kafka source queue. The Flink computing task generates a temporal table for each data identification number. The temporal table is a table that can store historical states and can record the state of data at different time points. When adding each data in the real-time queue to the temporal table corresponding to the respective identification number, taking a certain data as an example, a unique identification number is extracted from the data, and the target temporal table corresponding to the identification number of the data is searched in all available temporal tables. If the target temporal table exists, the data and its update time are added to the target temporal table. If the target temporal table does not exist, a new temporal table is created according to the data identification number. The new temporal table will be used to store all data corresponding to the identification number, and then the data and its update time are added to the newly created target temporal table. By analogy, all data in the Kafka source queue are added to the corresponding temporal table.

[0078] As an optional example, the above device further includes:

[0079] The processing module is used to add each data in the real-time queue to the temporal table corresponding to the respective identification number, take each temporal table as the current temporal table, and perform the following operations on the current temporal table:

[0080] Sort all data in the current temporal table according to the update time from latest to earliest.

[0081] Optionally, in this embodiment, after each data is added to the corresponding temporal table, the data in the temporal table is sorted, and all the data in the temporal table are sorted from latest to earliest according to the update time, which means that the most recently updated data will be at the front, ensuring that the latest data is always in the first row. In this way, only the first row of sorted data needs to be extracted from each temporal table to achieve data deduplication and retain the latest record. The temporal table can store the state of data at different time points, support historical tracking of data and query of the latest state. Through the temporal table, the system can record the version and changes of the data to ensure that the latest or specific time point data is used during processing and analysis.

[0082] As an optional example, the update module includes:

[0083] An acquisition unit, used to acquire the first row of data in each temporal table and determine it as the first data;

[0084] The second processing unit is configured to take each data in all the first data as current data and perform the following operations on the current data:

[0085] Find the target identification number of the current data in the data lake table;

[0086] When the target identification number exists in the data lake table, the second data corresponding to the target identification number in the data lake table is modified to the first data.

[0087] As an optional example, the second processing unit is further configured to save the second data to the data lake table after searching the data lake table for the target identification number of the current data, if the target identification number does not exist in the data lake table.

[0088] Optionally, in this embodiment, IceBerg has the ability to update records compared to the traditional offline data warehouse Hive. Hive can only add new records, while IceBerg can flexibly handle data updates and keep the data up to date. When a piece of data is written to the IceBerg table, IceBerg will check whether the identification number of the data already exists in the table. If the identification number of the data already exists, IceBerg will overwrite it to ensure that the latest data is always retained in the table. If the identification number of the data does not exist, IceBerg will add a new record to save the data. Through this update mechanism, the IceBerg table can keep the latest data at any time. Therefore, the recommendation algorithm team only needs to connect to the IceBerg data lake to obtain the latest online data and build real-time recommendation results. Specifically, the first row of data is extracted from each temporal table and determined as the first data. Each first data is used as the current data and processed in sequence. The target identification number corresponding to the identification number of the current data is searched in the IceBerg table. If the target identification number exists, the second data corresponding to the target identification number in the IceBerg table is modified to the first data. If the target identification number does not exist, the first data is saved to the IceBerg table.

[0089] For other examples of this embodiment, please refer to the above examples and will not be repeated here.

[0090] Figure 3 is a schematic diagram of an optional electronic device according to an embodiment of the present application, such as Figure 3 As shown, it includes a processor 302, a communication interface 304, a memory 306 and a communication bus 308, wherein the processor 302, the communication interface 304 and the memory 306 communicate with each other through the communication bus 308, wherein,

[0091] Memory 306, used for storing computer programs;

[0092] The processor 302 is used to implement the following steps when executing the computer program stored in the memory 306:

[0093] When receiving real-time data, the real-time data is sent to the real-time queue;

[0094] Send all the data in the database table to the real-time queue;

[0095] Add each data in the real-time queue to the temporal table corresponding to the respective identification number, and record the respective update time;

[0096] The data lake table is updated according to the latest updated data in each temporal table, where the data lake table is used by the recommendation algorithm to generate real-time recommendation results using all the data in the data lake table.

[0097] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The communication interface is used for communication between the above electronic device and other devices.

[0098] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0099] As an example, the memory 306 may include, but is not limited to, the first sending module 202, the second sending module 204, the adding module 206, and the updating module 208 in the real-time recommendation device for data. In addition, it may also include, but is not limited to, other module units in the real-time recommendation device for data, which will not be described in detail in this example.

[0100] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0101] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0102] It can be understood by those skilled in the art that Figure 3The structure shown is for illustration only. The device for implementing the real-time recommendation method for the above data may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 3 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 3 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 3 Different configurations shown.

[0103] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0104] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program executes the steps in the above-mentioned real-time data recommendation method when executed by a processor.

[0105] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0106] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0107] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0108] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A real-time data recommendation method, characterized in that: include: When receiving the real-time data, sending the real-time data to the real-time queue; Send all the full amount of data in the database table to the real-time queue; Add each data in the real-time queue to the temporal table corresponding to the respective identification number, and record the respective update time; The data lake table is updated according to the data with the latest update time in each of the temporal tables, wherein the data lake table is used for the recommendation algorithm to generate real-time recommendation results using all the data in the data lake table.

2. The method according to claim 1, characterized in that When receiving the real-time data, while sending the real-time data to the real-time queue, the method further includes: The real-time data is saved in the database table.

3. The method according to claim 1, characterized in that The adding each data in the real-time queue to the temporal table corresponding to the respective identification number and recording the respective update time comprises: Take each data in the real-time queue as current data and perform the following operations on the current data: Obtaining the identification number of the current data; Searching for a target temporal table corresponding to the identification number of the current data in all the temporal tables; In the case that the target temporal table exists in all the temporal tables, the current data and its update time are added to the target temporal table.

4. The method according to claim 3, characterized in that After searching all the temporal tables for a target temporal table corresponding to the identification number of the current data, the method further includes: When the target temporal table does not exist in all the temporal tables, creating the target temporal table; The current data and its update time are added to the target temporal table.

5. The method according to claim 1, characterized in that After adding each data in the real-time queue to the temporal table corresponding to the respective identification number, the method further includes: Take each of the temporal tables as a current temporal table, and perform the following operations on the current temporal table: All data in the current temporal table are sorted from latest to earliest according to update time.

6. The method according to claim 1, characterized in that The updating of the data lake table according to the latest updated data in each of the temporal tables includes: Obtain the first row of data in each of the temporal tables and determine it as the first data; Each data in all the first data is taken as current data, and the following operations are performed on the current data: Searching the target identification number of the current data in the data lake table; In a case where the target identification number exists in the data lake table, the second data corresponding to the target identification number in the data lake table is modified to the first data.

7. The method according to claim 6, characterized in that After searching the data lake table for the target identification number of the current data, the method further includes: If the target identification number does not exist in the data lake table, the second data is saved in the data lake table.

8. A real-time data recommendation device, characterized in that: include: A first sending module, configured to send the real-time data to a real-time queue when receiving the real-time data; A second sending module is used to send all the full amount of data in the database table to the real-time queue; An adding module, used for adding each data in the real-time queue to a temporal table corresponding to the respective identification number, and recording the respective update time; An updating module is used to update the data lake table according to the data with the latest update time in each of the temporal tables, wherein the data lake table is used by the recommendation algorithm to generate real-time recommendation results using all the data in the data lake table.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images