Content recommendation method, device, apparatus, medium, and computer program product

Abstract

The application discloses a content recommendation method and device, equipment, medium and computer program product, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring a first feature representation corresponding to first type feature data of candidate content; acquiring a second feature representation corresponding to second type feature data of the candidate content; performing recommendation probability prediction on at least two account types based on the first feature representation and the second feature representation, to obtain recommendation probabilities corresponding to the at least two account types respectively; and in response to the recommendation probability of the candidate content under a target account type meeting a recommendation condition, pushing the candidate content to a target account under the target account type. The content is recommended to the account by the recommendation probability of the candidate content under different account types, so that the recommendation accuracy of a new account in a cold start stage is ensured.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, medium, and computer program product for recommending content. Background Technology

[0002] With the rapid development of the Internet and the lowering of the threshold for content production, more and more content is entering users' field of vision through the Internet. The above content is then pushed to users' terminals after being filtered by recommendation systems.

[0003] In related technologies, recommendation systems generally implement content recommendation through an expert-driven approach. That is, by inputting the user's browsing history and the content to be recommended into the recommendation system, the recommendation algorithm in the system determines whether the content to be recommended meets the user's interests, and thus determines the recommended content to be sent to the user's terminal.

[0004] However, since the aforementioned recommendation system requires users to have already browsed content during the recommendation process, and requires a large amount of historical browsing records to support the accuracy of the recommendation system, this leads to the problem of low content recommendation accuracy during the cold start phase for new users due to the lack of historical browsing records. Summary of the Invention

[0005] This application provides a content recommendation method, apparatus, device, medium, and computer program product, which can improve the accuracy of content recommendation during the cold start phase. The technical solution is as follows:

[0006] On the one hand, a content recommendation method is provided, the method comprising:

[0007] Obtain a first feature representation corresponding to the first type of feature data of the candidate content, wherein the first type of feature data is used to indicate the publication attribute information of the candidate content;

[0008] Obtain a second feature representation corresponding to the second type of feature data of the candidate content, wherein the second type of feature data is used to indicate the content information of the candidate content;

[0009] Based on the first feature representation and the second feature representation, recommendation probability prediction is performed for at least two account types to obtain the recommendation probability corresponding to each of the at least two account types. The recommendation probability is used to indicate the interest of the account under the account type in the candidate content. The at least two account types include the target account type.

[0010] In response to the fact that the recommendation probability of the candidate content under the target account type meets the recommendation criteria, the candidate content is pushed to the target account under the target account type.

[0011] On the other hand, a content recommendation device is provided, the device comprising:

[0012] The acquisition module is used to acquire a first feature representation corresponding to the first type of feature data of the candidate content, wherein the first type of feature data is used to indicate the publication attribute information of the candidate content;

[0013] The acquisition module is further configured to acquire a second feature representation corresponding to the second type of feature data of the candidate content, wherein the second type of feature data is used to indicate the content information of the candidate content;

[0014] The prediction module is used to predict the recommendation probability of at least two account types based on the first feature representation and the second feature representation, and to obtain the recommendation probability corresponding to the at least two account types respectively. The recommendation probability is used to indicate the interest of the account under the account type in the candidate content. The at least two account types include the target account type.

[0015] The sending module is used to push the candidate content to the target account under the target account type in response to the candidate content's recommendation probability meeting the recommendation conditions.

[0016] On the other hand, a computer device is provided, the terminal including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the recommended method described in any of the embodiments of this application.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one piece of program code is stored therein, the program code being loaded and executed by a processor to implement the recommended method described in any of the embodiments of this application.

[0018] On the other hand, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the recommended method described in any of the above embodiments.

[0019] The technical solution provided in this application includes at least the following beneficial effects:

[0020] When content recommendation is needed, the recommendation probability of candidate content under different account types is predicted based on its characteristics at the publishing attribute level and content information level. This yields the recommendation probability corresponding to each candidate content and account type, and determines whether to push the candidate content to the target account based on the target account's account type. In other words, when recommending content, only the account type of the account to be recommended needs to be obtained, and content recommendation can be made based on the recommendation probability of the candidate content under that account type, thus ensuring the accuracy of recommendations for new accounts during the cold start phase. Furthermore, since the prediction is based on the content's publishing attribute information and content information, it is also suitable for newly added candidate content, thereby ensuring the recommendation efficiency of new content during the cold start phase. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application;

[0023] Figure 2 This is a schematic diagram of a server architecture provided in an exemplary embodiment of this application;

[0024] Figure 3 This is a flowchart illustrating a recommended method for providing content in an exemplary embodiment of this application;

[0025] Figure 4 This is a flowchart of a recommendation probability prediction method provided in an exemplary embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating the processing of the first type of feature data by the Wide portion according to an exemplary embodiment of this application;

[0027] Figure 6 This is a schematic diagram illustrating the processing of the second type of feature data by the Deep part according to an exemplary embodiment of this application;

[0028] Figure 7 This is a schematic diagram of a deep-shallow fusion multimodal model provided in an exemplary embodiment of this application;

[0029] Figure 8 This is a flowchart of a recommendation probability prediction method provided in another exemplary embodiment of this application;

[0030] Figure 9 This is a generation process of the second feature representation in a visual modality provided by an exemplary embodiment of this application;

[0031] Figure 10 This is a flowchart of a method for recommending content provided in an exemplary embodiment of this application;

[0032] Figure 11 This is a recommended device structure block diagram of the content provided in an exemplary embodiment of this application;

[0033] Figure 12 This is a recommended device structure block diagram of the content provided in another exemplary embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] First, a brief introduction to the terms used in the embodiments of this application:

[0037] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0038] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0039] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0040] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, and map building.

[0041] Key technologies in speech technology include Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech emerging as one of the most promising methods.

[0042] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0043] In related technologies, content recommendation systems typically employ an expert-driven approach. This involves inputting the user's browsing history and the content to be recommended into the system. The recommendation algorithm then determines whether the content matches the user's interests, thus determining the recommended content to send to the user's device. This recommendation algorithm is a matching algorithm built upon both user understanding and content understanding; it requires both the target account's browsing history and tags indicating the user's interests within the recommended content.

[0044] Therefore, the above recommendation system has at least the following problems:

[0045] (1) During the training phase, it is necessary to determine the interest labels corresponding to the sample content. Since the interest labels require a certain understanding of the content, the interest labels of the training samples are usually expert-driven. That is, the interest labels corresponding to the training samples are manually labeled. In order to ensure the accuracy of the model, a large number of training samples are needed to train the model. This results in a large consumption of human resources when acquiring sample data during the training phase.

[0046] (2) There are significant problems with new users. Specifically, during the cold start phase of a newly created account, the recommendation system cannot accurately obtain the user's interests due to the lack of or limited browsing history, resulting in low accuracy in content recommendations.

[0047] (3) The cold start efficiency of new content is also low. That is, for newly added content, interest tags need to be marked. If the marking of interest tags is completed by the back-end reviewers, the delivery process of new content will be lengthened. If the marking of interest tags is completed by the content uploader, there may be incorrect marking or no marking, which may lead to incorrect content push or low exposure rate of content in the cold start stage due to the lack of interest tags.

[0048] The content recommendation method provided in this application reduces the manual cost of generating sample data by extracting the aforementioned recommendation probabilities of sample content relative to various account types from the feedback logs recorded by the platform during content recommendation. Furthermore, for new accounts, with authorization, the account type can be labeled by obtaining basic account information, and content recommendations can be made based on that account type. This reduces the information cost requirements during the account's cold start phase and improves the accuracy of content recommendations. Moreover, for new content, recommendations can be made based on the publication attributes and content information of candidate content, further reducing the information cost requirements during the cold start phase and improving the accuracy of recommendations when pushing new content to accounts.

[0049] The solutions provided in this application involve technologies such as natural language processing and machine learning in artificial intelligence. The recommended methods provided in this application can be applied to the following scenarios:

[0050] This method can be applied to video recommendation scenarios in the form of a feed: Feed products, in suitable scenarios, can provide appropriate content, possessing the following characteristics: suitable for mobile screens, allowing for continuous, fluid scrolling; and a sufficiently large data volume to constantly refresh new content. The application value of feed products is mainly reflected in the following aspects: convenient reading; compared to traditional books, magazines, and videos, feed products offer a convenient reading / browsing method; massive amounts of information; and freshness and timeliness. Applying feeds to video scenarios creates feed-style video pushes, specifically, for example, short video platforms.

[0051] In this video recommendation scenario, the server retrieves candidate videos to be recommended from the video library, inputs the relevant data of the candidate videos into the content recognition model of the recommendation system, the content recognition model identifies them, determines the recommendation probability of the candidate video under various account types, and then transmits the candidate videos marked with recommendation probabilities to the scheduling center, which then distributes the candidate content to the appropriate account terminals.

[0052] The content recommendation process provided by this method can help newly published video content be quickly pushed to accounts that meet the push criteria. At the same time, for newly registered accounts, accounts can be marked based on their basic information, improving the accuracy of push notifications for new accounts during the cold start phase.

[0053] Optionally, this method can also be applied to recommendation systems in various scenarios such as article reading platforms, social platforms, and music playback platforms, without specific limitations.

[0054] Based on the above introduction of terms and application scenarios, the implementation environment of the embodiments of this application will be described. For example... Figure 1 As shown, the computer system of this implementation environment includes: terminal device 110, server 120 and communication network 130.

[0055] Terminal device 110 includes various forms of devices such as mobile phones, tablets, desktop computers, laptops, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Terminal device 110 runs a target application, which includes a service module for recommending content to the logged-in target account. This service module recommends content when the user uses the target application. Illustratively, the target application can be traditional application software, cloud application software, a mini-program or application module within a host application, or a web platform; no specific limitations are imposed here. Optionally, the aforementioned target application can be an e-commerce application, a short video application, an audio application, a novel application, etc., without specific restrictions.

[0056] Server 120 provides backend services to the target application in terminal device 110, including content recommendation services. Illustratively, upon obtaining authorization from the target account, server 120 retrieves the target account information from terminal device 110, including the target account type. Server 120 retrieves candidate content from a content library, inputs the candidate content into a pre-trained content recognition model, and outputs the recommendation probabilities of the candidate content under at least two account types. If the recommendation probability of the candidate content under the target account type meets the recommendation criteria, the candidate content is sent as recommended content to terminal device 110. After receiving the recommended content, terminal device 110 displays the recommended content through the target application.

[0057] It is worth noting that the aforementioned server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0058] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0059] In some embodiments, the server 120 can also be implemented as a node in a blockchain system. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0060] Indicatively, terminal device 110 and server 120 are connected via communication network 130, which can be a wired network or a wireless network, and is not limited here.

[0061] The recommended methods provided in this application are applied to a server. The specific architecture of the server is illustrated herein, such as... Figure 2 As shown, it illustrates a schematic diagram of a server architecture 200 provided in an exemplary embodiment of this application. The server architecture 200 includes an uplink / downlink content interface service 210, a content database 220, a scheduling center service 230, a manual review service 240, a deduplication service 250, a content recognition service 260, an account information database 270, a browsing log database 280, and a content distribution exit service 290.

[0062] The following functions can be achieved by individual functional services or combinations of services:

[0063] The aforementioned uplink / downlink content interface service 210 connects to the content producer 201, which provides content to be published via a mobile terminal or an Application Programming Interface (API) system. Specifically, the content producer 201 obtains the upload server interface address through communication with the uplink / downlink content interface service, and then uploads the local file.

[0064] The aforementioned uplink and downlink content interface service 210 is also connected to the content consumer 202. When the content consumer 202 needs to obtain recommended content, it communicates with the uplink and downlink content interface service 210 to obtain the index information (usually the address of the content) for accessing the recommended content, and then downloads the corresponding file for display. Taking video as an example, the content consumer 202 downloads the video streaming media file and plays it through a local player.

[0065] After receiving the content to be published uploaded by the content producer 201, the uplink / downlink content interface service 210 writes the metadata of the content to be published into the content database 220. This metadata includes file size, cover image link, bitrate, file format, title, publication time, author information, video file size, video format, whether it is marked as original or first release, and the content's classification during manual review. Before writing the metadata into the content database 220, the uplink / downlink content interface service 210 first performs a standard transcoding operation on the content to be published, and then asynchronously returns the metadata after transcoding is complete.

[0066] Meanwhile, the uplink and downlink content interface service 210 sends the content to be published to the scheduling center service 230. The scheduling center service 230 performs subsequent content processing and circulation of the content to be published, which mainly includes machine processing and manual review. The machine processing involves calling the deduplication service 250 to identify completely duplicate and similar content. The deduplication results are written to the content database 220. Completely duplicate content will not be subject to repeated secondary processing by humans, thereby reducing the amount of content that needs to be processed in the manual review process and reducing the consumption of human resources.

[0067] Specifically, the deduplication service 250 vectorizes the content, then creates indexes for the vectors, and determines the degree of repetition by comparing the distances between the vectors. To reduce the computational load of subsequent processing, the deduplication service 250 can first deduplicate the short text of the content's title.

[0068] The dispatch center service 230 then sends the deduplicated content to the manual review service 240. The manual review service 240 needs to read the original information of the content itself from the content database. It is usually a complex web database-based system. The manual reviewer performs a preliminary filtering of the content to determine if it violates any rules. Based on the preliminary review, the manual reviewer performs a second review of the content, which mainly involves classifying and labeling or confirming the content. In this process, the accuracy and efficiency of the content labeling, especially the tags, are improved through human-machine collaboration.

[0069] The dispatch center service 230 also communicates with the content recognition service 260, which includes a data reporting interface 261, a data processing unit 262, a model training unit 263, and a recognition service unit 264.

[0070] During the process of displaying or playing recommended content on the content consumption terminal 202, browsing data related to the recommended content during the download process will be reported to the data reporting interface 261 in the content recognition service 260. The aforementioned browsing data may include one or more of the following: stuttering information during the loading phase of the recommended content, loading time, playback clicks, interaction information (such as likes, forwards, favorites, sharing, etc.), and content browsing time.

[0071] The data reporting interface 261 stores the above browsing information in the browsing log database 280. When it is necessary to train the content recognition model, the data processing unit 262 obtains browsing information from the browsing log database 280 and account information from the account information database 270. Based on the browsing information and account information, the data is aggregated and cleaned to obtain sample data.

[0072] After obtaining the sample data from the data processing unit 262, the model training unit 263 trains a content recognition model based on the sample data and applies the content recognition model to the recognition service unit 264.

[0073] The identification service unit 264 is connected to the dispatch center service 230. The identification service unit 264 predicts the recommendation probability of each account type based on the content to be published provided by the dispatch center service 230, and uses the recommendation probability to mark the content. The marking result is returned to the dispatch center service 230. The dispatch center service 230 sends the above marking result to the content distribution exit service 290. The content distribution exit service 290 matches the content consumer end 202 that receives the content based on the marking result.

[0074] Please refer to Figure 3 This illustrates a recommended method for the content shown in one embodiment of this application, in which the method is applied to, for example, Figure 1 In the server shown, the method includes:

[0075] Step 301: Obtain the first feature representation corresponding to the first type of feature data of the candidate content.

[0076] Optionally, the content format of the above candidate content includes one or more combinations of video, image, text, and audio formats.

[0077] Optionally, the candidate content can be professionally generated content (PGC), that is, content provided by professional content production agencies or organizations; it can also be user-generated content (UGC), that is, content created by users and displayed or provided to other users through an internet platform; or it can be professionally generated content (PUGC), that is, content produced in the form of UGC but relatively close to PGC, obtained by combining UGC and PGC.

[0078] For illustrative purposes, the first type of feature data mentioned above is used to indicate the publication attribute information of candidate content. For illustrative purposes, the publication attribute information mentioned above is used to indicate the attribute information that distinguishes the content from the content itself when it is published to the Internet, and the publication attribute information is not directly related to the characteristics expressed by the content itself.

[0079] Optionally, the first type of feature data includes at least one of the following: the classification level of the candidate content, the account information of the publishing account, and the length of the candidate content.

[0080] The classification level of the candidate content indicates the classification level determined during the publishing stage based on the content's domain. This level is determined by the different degrees of generalization of the contextual relationships expressed internally. Different classification levels encompass different ranges, and each candidate content includes at least one classification level. In an example, if the candidate content is an article explaining a certain brand of mobile phone in 2022, then the article's first-level classification is "technology," the second-level classification is "smartphones," and the third-level classification is "a certain brand." Illustratively, the classification level of the candidate content can be customized by the publishing account during content upload, or it can be determined by a classification model provided by the server. This classification model can be a supervised learning model that classifies content using a classification tree structure.

[0081] A publishing account is used to indicate the account that uploads and publishes candidate content. Optionally, the account information of the publishing account may include at least one of the following: account level, number of followers on the content platform, account creation time, number of content published, and historical views of the published content. The account level can be the level initialized during the registration or authentication phase; that is, the account level does not change with account usage and indicates the account's authority on the content publishing platform. In one example, account levels are divided into 1-5, with a positive correlation between account level and the level of authority represented by the account. For example, an official media account might be set to level 5. The number of followers mentioned above can reflect the popularity of the content published by the account to a certain extent. In some embodiments, different numbers of followers are divided into different tiers. For example, less than 1,000 followers is tier 0, 1,000-10,000 followers is tier 1, 10,000-50,000 followers is tier 2, 50,000-100,000 followers is tier 3, 100,000-500,000 followers is tier 4, 500,000-1,000,000 followers is tier 5, 1,000,000-2,000,000 followers is tier 6, 2,000,000-5,000,000 followers is tier 7, 5,000,000-10,000,000 followers is tier 8, 10,000,000-30,000,000 followers is tier 9, and more than 30,000,000 followers is tier 10. The higher the tier, the higher the popularity.

[0082] The length of candidate content is used to measure the time required for the candidate content to be fully viewed. For example, when the candidate content is in video format, the video duration is used as the length of the candidate content; when the candidate content is in image, text, or a combination of both, the number of screens used to display the candidate content during the viewing process is used as the length of the candidate content—that is, the more screens, the longer the candidate content.

[0083] In some embodiments, the recommendation system includes a first processing module for processing first-type feature data, the first processing module including a transformation network for extracting features from the first-type feature data. Illustratively, the first-type feature data is input into the transformation network, and a first feature representation is output.

[0084] Optionally, the aforementioned transformation network can be at least one of the following networks that can be used for feature extraction: Recurrent Neural Network (RNN), Convolutional Neural Networks (CNN), Transformer, Autoencoder (AE).

[0085] Step 302: Obtain the second feature representation corresponding to the second type of feature data of the candidate content.

[0086] Schematic illustration: The second type of feature data described above is used to indicate the content information of the candidate content. Optionally, the second type of feature data includes at least one of the candidate content's title, description, tags, and content.

[0087] The title or description of the candidate content can be set by the publishing account when publishing the candidate content. When the publishing account does not set a title or description for the candidate content, the server can also automatically extract the title or description based on the content of the candidate content.

[0088] The tags for the aforementioned candidate content can be tags selected or entered by the publishing account when publishing the candidate content, or tags automatically added by the server based on the content of the candidate content. For illustrative purposes, these tags are those displayed at a specified location during the display of the candidate content, such as at the bottom of the video playback. In one example, if the candidate content is an article explaining a certain brand of mobile phone in 2022, and the article states that both model A and model B of the phone use the XX chip, then the tags could be "#ModelA, #ModelB, #XXChip".

[0089] The content of the aforementioned candidate content refers to the specific data displayed during the presentation of the candidate content. In some embodiments, when the aforementioned candidate content is in video format, the content includes at least one of the video frames, audio data, and subtitle data of the candidate content; in other embodiments, when the aforementioned candidate content is in image format, the content includes at least one of the image data of the candidate content and the text data in the image; in other embodiments, when the aforementioned candidate content is in text format, the content includes the text data of the candidate content.

[0090] In some embodiments, the recommendation system includes a second processing module for processing second-type feature data, the second processing module including a feature extraction network for extracting features from the second-type feature data.

[0091] Schematic illustration: The second-type feature data is input into different feature extraction networks based on its data modality. The data modality refers to the presentation format of the candidate content. The data modality of the second-type feature data can include a visual modality (i.e., video or image data), an audio modality (i.e., audio data), or a text modality (i.e., text data).

[0092] Responding to the fact that the data modality of the second type of feature data is visual, the second type of feature data is input into the image feature extraction network to obtain the second feature representation. The image feature extraction network is used to extract features based on the image data of the candidate content. Optionally, the image feature extraction network can be at least one of the following networks: CNN, Visual Geometry Group Network (VGG), Feature Pyramid Network (FPN), 3D Convolutional Network (C3DNetwork), Swin-Transformer, etc.

[0093] Responding to the fact that the data mode of the second type of feature data is audio mode, the second type of feature data is input into the audio feature extraction network to obtain the second feature representation. The audio feature extraction network is used to extract features based on the audio data of the candidate content. Optionally, the audio feature extraction network can be a neural network such as CNN or VGGish network, or it can be implemented using traditional audio feature extraction methods, such as Mel Frequency Cepstrum Coefficient (MFCC) feature extraction, zero-crossing rate, and spectral centroid feature extraction.

[0094] Responding to the fact that the data modality of the second type of feature data is text modality, the second type of feature data is input into the text feature extraction network to obtain the second feature representation. The text feature extraction network is used to extract features based on the text data of the candidate content. Optionally, the text feature extraction network can be at least one of the following networks: Graph Neural Network (GNN), CNN, RNN, Residual Network (ResNet), etc.

[0095] Step 303: Based on the first feature representation and the second feature representation, predict the recommendation probability for at least two account types to obtain the recommendation probability corresponding to each of the at least two account types.

[0096] The recommendation probability is used to indicate the degree of interest of accounts under an account type in candidate content. That is, it is a prediction of the degree of interest that users corresponding to an account type may have when browsing candidate content.

[0097] For illustrative purposes, the account types mentioned above are account tags used within the target application to categorize different accounts. Optionally, the account types mentioned above can be obtained by categorizing accounts based on their basic information with authorization.

[0098] Optionally, accounts can be categorized based on at least one of the following basic information: geographic location, gender, and age. In one example, accounts on the platform are divided into A account types. Specifically, the categorization is done by dividing the accounts into two main categories based on the gender of the user. Each main category is then further subdivided based on age, creating account types for each age group within each gender. For example, account type A might include "Female 0-17 years old; Female 18-22 years old; Female 23-30 years old; ...; Female 61 years and older; Male 0-17 years old; Male 18-22 years old; Male 23-30 years old; ...," where A is a positive integer. This illustrative explanation of account categorization based on gender and age is not limited to this method; accounts can also be categorized based on other information.

[0099] In some embodiments, the recommender system includes a prediction module for predicting recommendation probabilities based on a first feature representation and a second feature representation. Illustratively, this prediction module includes a fusion unit for fusing the first and second feature representations, and a prediction unit for classifying the fused feature representation to obtain recommendation probabilities. Specifically, the first and second feature representations are encoded and fused to obtain a fused feature representation; based on the fused feature representation, the recommendation probabilities corresponding to at least two account types are predicted to obtain recommendation probabilities corresponding to at least two account types respectively.

[0100] Step 304: In response to the fact that the recommendation probability of the candidate content under the target account type meets the recommendation conditions, the candidate content is pushed to the target account under the target account type.

[0101] The target account type mentioned above refers to the account type of the target account.

[0102] As an illustration, when content recommendation is required, the terminal device displays a prompt interface asking whether to authorize the application to obtain the account's basic information for content recommendation. In response to receiving a confirmation signal on the prompt interface, the server determines the target account type by obtaining the target account's basic information and recommends content to the target account based on that target account type.

[0103] Optionally, the recommendation probability corresponding to the candidate content can be identified and marked when the candidate content is entered into the database. When it is necessary to recommend content to the target account, it can be determined directly based on the already marked identification results, thereby improving the recommendation efficiency when recommending content to the target account, while reducing the multiple identification of the same candidate content and reducing the waste of processing resources.

[0104] Schematic illustration: The above recommendation criteria are used to filter candidate content based on the recommendation probability of candidate content under the target account type. In some embodiments, the above recommendation criteria may indicate that in response to the recommendation probability of candidate content under the target account type reaching a recommendation threshold, the candidate content is determined to meet the recommendation criteria; in other embodiments, the above recommendation criteria may indicate that the Z obtained candidate content are sorted according to the recommendation probability of each candidate content under the target account type to obtain a candidate content queue, and the K candidate content with the highest recommendation probability are obtained from the candidate content queue as candidate content that meets the recommendation criteria, where J is a positive integer and K is a positive integer less than J.

[0105] In some embodiments, when the recommendation probability of candidate content under the target account type meets the recommendation criteria, the server can push the candidate content to the terminal device in real time in the form of streaming media; in other embodiments, provided that sufficient authorization is obtained, when the recommendation probability of candidate content under the target account type meets the recommendation criteria, the server can pre-send the candidate content to the terminal device, that is, the terminal device pre-stores the candidate content, and when the target application is opened, the terminal device can quickly recommend the candidate content.

[0106] In summary, the content recommendation method provided in this application, when content recommendation is required, predicts the recommendation probability of candidate content under different account types based on the characteristics of the candidate content at the publishing attribute information level and the characteristics at the content information level. This yields the recommendation probability corresponding to each candidate content and each account type, and determines whether to push the candidate content to the target account based on the account type of the candidate content. That is, when recommending content, only the account type of the account to be recommended needs to be obtained, and content recommendation can be performed based on the recommendation probability of the candidate content under this account type, thus ensuring the recommendation accuracy of new accounts during the cold start phase. Furthermore, since the prediction is based on the publishing attribute information and content information of the content itself, it is also suitable for newly added candidate content, thereby ensuring the recommendation efficiency of new content during the cold start phase.

[0107] Please refer to Figure 4 This document illustrates a method for predicting recommendation probabilities in the content recommendations presented in one embodiment of this application. In this embodiment, a deep & wide multimodal model is employed to predict the recommendation probabilities. The method includes:

[0108] Step 401: Obtain M types of first-class feature data and N types of second-class feature data of the candidate data.

[0109] Where M and N are positive integers.

[0110] In some embodiments, the candidate content includes at least two types of first-class feature data, i.e., M≥2 and M is a positive integer.

[0111] In some embodiments, the candidate content includes at least two types of second-class feature data, i.e., N≥2 and N is a positive integer. The at least two types of second-class feature data are data from different data modalities; that is, the candidate content is obtained from at least two data modalities, including at least two of visual, audio, and text modalities.

[0112] In some embodiments, when the candidate content is in text form, the preferred content mentioned above includes only the second type of feature data of the text modality.

[0113] Step 402: Input the i-th type of first-class feature data into the i-th transformation network for feature transformation to obtain the i-th first feature representation.

[0114] The i mentioned above is a positive integer.

[0115] Optionally, the aforementioned transformation network can be at least one of RNN, CNN, Transformer, AE, etc., which can be used for feature extraction.

[0116] To illustrate, let's take the Transformer as an example. The Transformer is used to encode and decode the first type of feature data according to the attention mechanism, so as to obtain the first feature representation.

[0117] Schematic illustration: the first processing module in the recommendation system is the Wide part of the Deep&Wide multimodal model, which includes M Transformer units. Optionally, these Transformer units may include single-layer Transformers or multi-layer Transformers. Each Transformer has the same structure, but the parameters of the Transformers corresponding to different first-class feature data are different.

[0118] Schematic illustration: The Transformer described above is a model that includes encoders and decoders, which can be single or multiple. An encoder includes a self-attention layer and a fully connected feedforward neural network. When a Transformer includes multiple encoders, the parameters of the feedforward neural network for each encoder are identical, but their functions are independent. The decoder part also has a similar hierarchical structure to the encoder part, except that an Encoder-Decoder-Attention layer is added between the self-attention layer and the fully connected layer to focus on the first type of feature data of the input itself.

[0119] Step 403: Perform feature concatenation on the M first feature representations to obtain the third feature representation.

[0120] After obtaining the first feature representations corresponding to each type of feature data through a transformation network, the M first feature representations are linked together through a linking layer to obtain the third feature representation. This third feature representation is used to provide the ability to memorize the semantic features of candidate content. That is, during the recommendation probability prediction process, the model can locate the recommendation scope of candidate content to account types with high interest in the recommended content based on the third feature representation of the candidate content.

[0121] Indicative, such as Figure 5 As shown, this illustration depicts a schematic diagram of the processing of the first type of feature data by the Wide section in an exemplary embodiment of this application. The four types of first-type feature data 501 are respectively input to the corresponding single-layer Transformer 510, and the output is a first feature representation a. i 502, the four first feature representations a are processed through the connection layer 520. i Connect 502 to obtain the third feature representation x503.

[0122] Step 404: Input the j-th type of second-class feature data into the feature extraction network corresponding to the j-th data modality to obtain the j-th second feature representation.

[0123] Schematic, the second processing module in the recommendation system is the Deep part of the Deep&Wide multimodal model, which includes feature processing units corresponding to various modalities. In some embodiments, the second type of feature data is input into the feature extraction network corresponding to the data modality to obtain a second feature representation. The second feature representation is used to provide the ability to generalize the semantic features of candidate content. That is, in the process of predicting recommendation probability, the model can generalize the recommendation range of candidate content to account types with high interest in the same or similar content based on the second feature representation of the candidate content.

[0124] In some embodiments, when the candidate content includes content in the form of video or images, it is necessary to obtain second type feature data in the visual modality.

[0125] Optionally, when the candidate content is in video format, the video data of the candidate content can be used as the second type of feature data. That is, in this case, in response to the data modality of the second type of feature data being visual, the second type of feature data is input into a video feature extraction network to obtain a second feature representation. The video feature extraction network is used to extract features based on the video data of the candidate content, for example, C3D Net. Optionally, when the candidate content is in video format, the video data of the candidate content can also be segmented into frames to obtain image data as the second feature data. That is, in this case, in response to the data modality of the second type of feature data being visual, the second type of feature data is input into an image feature extraction network to obtain a second feature representation. The image feature extraction network is used to extract features based on the image data of the candidate content, for example, Swin-Transformer.

[0126] When the candidate content is in the form of an image, the image data of the candidate content can be used as the second type of feature data. That is, in this case, the data modality of the second type of feature data is the visual modality. The second type of feature data is input into the image feature extraction network to obtain the second feature representation.

[0127] In some embodiments, when the candidate content includes video or audio content, it is necessary to obtain the second type of feature data in the audio modality.

[0128] In response to the data modality of the second type of feature data being audio modality, the second type of feature data is input into the audio feature extraction network to obtain the second feature representation. The audio feature extraction network is used to extract features based on the audio data of the candidate content, such as the VGGish network.

[0129] As an illustration, when the candidate content is in video format, the aforementioned audio data can be extracted from the video data of the candidate content.

[0130] In some embodiments, when the candidate content includes video, audio, graphic, or text content, it is necessary to obtain the second type of feature data in the text modality.

[0131] In response to the fact that the data modality of the second type of feature data is text modality, the second type of feature data is input into the text feature extraction network to obtain the second feature representation. The text feature extraction network is used to extract features based on the text data of the candidate content, for example, CNN.

[0132] For illustrative purposes, the aforementioned text data includes the title, description, and tag data of the candidate content. For illustrative purposes, when the candidate content is in video format, the aforementioned text data also includes text data obtained from the video data of the candidate content through speech-to-text or subtitle recognition; when the candidate content is in audio format, the aforementioned text data also includes text data obtained from the audio data of the candidate content through speech-to-text recognition; when the candidate content is in image-text format, the aforementioned text data also includes text data obtained from the image data of the candidate content through OCR recognition, as well as text data included in the candidate content itself; when the aforementioned candidate content is in text format, the aforementioned text data also includes text data included in the candidate content itself.

[0133] Indicative, such as Figure 6 As shown, this illustration depicts a schematic diagram of the processing of the second type of feature data by the Deep section according to an exemplary embodiment of this application. The five types of second-type feature data 601 are first input to the embedding layer 610 to generate an embedding vector b. i 602, Embedding vector b i The inputs 602 are fed into the corresponding feature extraction network i620, and the output is the second feature representation y. i 603.

[0134] Step 405: Based on the second feature representation and the third feature representation, predict the recommendation probability for at least two account types to obtain the recommendation probability corresponding to each of the at least two account types.

[0135] Indicatively, the shallow expression of candidate content is indicated by the third feature representation, and the deep expression of candidate content is indicated by the second feature representation. Combining the shallow and deep representations improves the accuracy of predicting the recommendation probability of candidate content.

[0136] In some embodiments, the Deep&Wide multimodal model includes a prediction module that processes the second feature representation and the third feature representation and predicts the recommendation probability. The prediction module includes a connection unit, a fusion unit, and a prediction unit. The connection unit connects the third feature representation and N second feature representations to obtain an intermediate feature representation. The intermediate feature representation is then input to the fusion unit for feature fusion to obtain a fused feature representation. The fused feature representation is then output by the prediction unit to provide the recommendation probability for each account type.

[0137] In one example, the connection unit could be a feedforward neural network or any other network structure that can connect feature representations.

[0138] In one example, the fusion unit can be a CrossModel-TransformersEncoder fusion network, where the CrossModel-TransformersEncoder fusion network is a feature fusion network based on the Transformer structure and implemented through cross-modal coding.

[0139] Schematic, the cross-modal layer implementing cross-modal encoding in the aforementioned fusion network consists of at least two self-attention sublayers, at least one bidirectional cross-attention sublayer, and two feedforward sublayers. In a specific implementation, the fusion network may include multiple cross-modal layers, in which case the cross-modal layers need to be stacked, i.e., the output of the k-th layer is used as the input of the (k+1)-th layer. Inside the k-th layer, a multi-directional cross-attention sublayer is first applied. This sublayer contains multiple unidirectional cross-attention sublayers. Taking an intermediate feature representation including two data modalities as an example, the unidirectional cross-attention sublayer includes attention from intermediate feature representation a to intermediate feature representation b, and attention from intermediate feature representation b to intermediate feature representation a. The cross-attention sublayer is used to exchange information and align entities between the two modes to learn a joint cross-modal representation, thereby obtaining the fused feature representation of the input prediction unit.

[0140] In one example, the prediction unit could be a softmax layer, which is used to predict the recommendation probability under different account types based on the fused feature representation, thereby obtaining the recommendation probability used to indicate the recommendation situation.

[0141] Indicative, such as Figure 7The diagram illustrates a schematic of a shallow-deep fusion multimodal model 700 provided in an exemplary embodiment of this application. The shallow layer 710 of the shallow-deep fusion multimodal model 700 includes M transformation networks 711 and a first connection layer 712. The deep layer 720 includes an embedding layer 721 and N feature extraction networks 722. The third feature representation output by the shallow layer 710 and the N second feature representations output by the N feature extraction networks 722 are input to the second connection layer 731. The second connection layer 731 inputs the intermediate feature representations to the fusion network 732. The fusion network 732 inputs the output fused feature representations to the prediction layer 733, and outputs the recommendation probability of candidate content relative to each account type.

[0142] In summary, the recommendation probability prediction method for content recommendation provided in this application, when content recommendation is required, predicts the recommendation probability of candidate content under different account types based on the characteristics of the candidate content at the publishing attribute information level and the characteristics at the content information level, thereby obtaining the recommendation probability corresponding to each candidate content and each account type. That is, since the prediction is based on the publishing attribute information and content information of the content itself, it is also suitable for newly added candidate content, thus ensuring the recommendation efficiency of new content during the cold start phase.

[0143] In this embodiment, candidate content is processed by Deep & Wide and multimodal fusion. The Wide part is suitable for extracting relatively sparse information in the candidate content, namely the publication attribute information of the candidate content, while the Deep part is suitable for extracting similar features generalized by embedding in the candidate content, that is, processing the content information. The combination of the two improves the accuracy of indicating the features of the candidate content, thereby improving the accuracy of the prediction results.

[0144] Please refer to Figure 8 This illustrates a recommendation probability prediction method according to an embodiment of this application. In this embodiment, for... Figure 3 The Deep section (i.e., step 304) provides an illustrative explanation of the processing procedures under different data modalities. Taking video content as an example, the second type of feature data corresponding to the candidate content includes second type feature data under visual modality, audio modality, and text modality. The method includes:

[0145] Step 4041: In response to the data modality of the second type of feature data being visual modality, the second type of feature data is input into the image feature extraction network to obtain the second feature representation.

[0146] Indicatively, when the data modality of the second type of feature data is the visual modality, the second type of feature data includes an image sequence of candidate content. That is, the above-mentioned image sequence is obtained by dividing the video data of the candidate content into frames, and the second feature representation in the video modality is obtained through the image sequence.

[0147] In some embodiments, by converting the image sequence into an embedding vector, and by inputting the embedding vector into an image feature extraction network, a second feature representation is obtained. The embedding vector of the image sequence is a low-dimensional vector used to represent the video, and is also an "implicit" feature based on the video content. The second feature representation includes two meanings: (1) a low-dimensional dense feature under representation learning, the specific form of which can be a set of arrays; (2) a similarity measurement vector under metric learning, wherein the distance between two vectors in the feature space can represent the similarity between them.

[0148] In a schematic manner, a main frame image is obtained from an image sequence. The main frame image is used to represent images with similar scenes in the image sequence. By using the sliding window and the original window provided by the image feature extraction network, the relationship between each pixel in the main frame image is established, and the global attention features corresponding to the main frame image are obtained. In response to the image sequence including at least two main frame images, the global attention features of at least two main frame images are aggregated to obtain a second feature representation.

[0149] Specifically, the process involves inputting an image sequence from a video, extracting frames using a Temporal Segment Network (TSN) to obtain main frame images, and then using a sliding window and an original window provided by SwinT to establish relationships between pixels in the main frame images, obtaining global attention features for each main frame image. Finally, image embedding vectors obtained from the intermediate layers of the multimodal video classification model NeXtVLad are used, and the second feature representation is obtained by summing and averaging these image embedding vectors. NeXtVLad, as described above, is a model structure for aggregating frame-level features into video-level features in large-scale video classification tasks.

[0150] Indicative, such as Figure 9As shown, this illustrates the generation process of the second feature representation in a visual modality provided by an exemplary embodiment of this application. The image sequence 910 corresponding to the video content is processed by frame extraction to obtain the main frame image 920 (including main frame image A, main frame image B, and main frame image C). The main frame image 920 is input to the SwinT 930 to obtain global attention features 940 (including features A, B, and C). The global attention features 940 are input to the NeXtVLad 950 to output the second feature representation.

[0151] In other embodiments, when the candidate content is a combination of text and images, for the second type of feature data in the visual modality, the cover image can be directly input into the Transformer to represent the cover image features, thus obtaining the second feature representation in the visual modality.

[0152] Step 4042: In response to the data modality of the second type of feature data being audio modality, the second type of feature data is input into the audio feature extraction network to obtain the second feature representation.

[0153] For illustrative purposes, when the data modality of the second type of feature data is audio, the second type of feature data includes the audio data of the candidate content. Optionally, the aforementioned audio data may be data extracted from video content.

[0154] In some embodiments, the audio feature extraction network first generates a spectrogram corresponding to the audio data, then maps the spectrogram to a preset filter bank to generate spectral features corresponding to the audio data. Audio frames are combined based on the spectral features to obtain audio frame features, and sound events within these features are extracted to obtain a second feature representation.

[0155] In one example, the audio is preprocessed to obtain a Mel spectrogram, then VGGish is used to extract features from the Mel spectrogram, and finally, a NeXtVlad network is used to obtain video-level audio features. Illustratively, the above preprocessing process can involve selecting audio from a specified time period within the video content and located at a fixed frequency point, then performing a short-time Fourier transform on the audio using a Hamming window to obtain the spectrogram. For example, selecting 16kHz audio from the first 10 minutes of the video, using a 25ms Hamming window and a 10ms frame shift to perform a short-time Fourier transform on the audio to obtain the spectrogram. After obtaining the transformed spectrogram, it is mapped onto a 64th-order Mel filter bank to calculate the Mel spectrum. Then, the Mel spectrum features are framed in 960ms intervals with no overlap between frames, each frame being 10ms long and containing 64 Mel frequency bands. VGGish is used as the feature extractor; VGGish is a feature extractor with strong expressive capabilities for scene-based sound events. The specific values ​​mentioned above are for illustrative purposes only and may be implemented with other values ​​as needed. No restrictions are imposed here.

[0156] As an illustration, adding an audio modality can significantly improve the accuracy of semantic modeling for content such as video and music.

[0157] Step 4043: In response to the data modality of the second type of feature data being text modality, the second type of feature data is input into the text feature extraction network to obtain the second feature representation.

[0158] Schematic illustration: When the data modality of the second type of feature data is text, the second type of feature data includes the text data of the candidate content. In some embodiments, when the candidate content is video content, the aforementioned text data can be at least one of the following: the title corresponding to the video content, tags, text information obtained by ASR, and text information recognized by OCR. The title is generally a subjective description of the content by the publisher, typically encompassing the high-level semantics the video intends to convey. Tags, on the other hand, encompass the semantics extracted from the video content in a lower dimension. Since the information expressed by the title and tags may be insufficient, text information from the video content can be obtained through ASR or OCR to enrich the text data.

[0159] In some embodiments, taking OCR as an example, after obtaining text information through OCR, the text information can be denoised to improve the accuracy of the text information. The denoising process can include at least one of the following operations: filtering single characters / pure numbers / pure letters, filtering characters with small position offsets and high text repetition rates between adjacent frames, and filtering characters with small heights at the bottom of the screen.

[0160] Once the text data is acquired, it is input into a text feature extraction network. Based on the semantic expression corresponding to the text data, features are extracted to obtain a second feature representation. In one example, the aforementioned text feature extraction network can be a Bidirectional Encoder Representation from Transformers (BERT) model pre-trained on a large-scale text corpus, or it can be any other network model capable of extracting speech features from text expressions; no limitation is imposed here.

[0161] In summary, the recommendation probability prediction method provided in this application extracts features of candidate content under multiple data modalities in the Deep part and integrates the features of candidate content under multiple modalities, thereby enhancing the accuracy of the model in representing the features of candidate content and improving the recommendation accuracy of the model when applied to the recommendation system.

[0162] In some embodiments, the Deep&Wide multimodal model used to predict recommendation probabilities in the above-described recommendation system acquires sample content and label data for the sample content during the training phase. The label data indicates whether the sample content is a positive or negative sample under at least two account types. Illustratively, by inputting the sample content into the model to be trained for prediction, prediction scores corresponding to the sample content and at least two account types are obtained. The difference between the predicted scores and the label data is used to determine the loss value of the prediction result. The model parameters of the model to be trained are adjusted according to the loss value to iteratively train the model. During training, the model automatically learns the association between different content and different account types, thus enabling it to determine the recommendation probability relative to different account types based on the content itself during the model application phase. Upon convergence of the loss value during training, the target model is determined to be the model used in the actual recommendation system to predict recommendation probabilities.

[0163] For illustrative purposes, the loss value described above is obtained by substituting the predicted score and label data into the loss function. Optionally, the loss function may be at least one of the following: squared loss function, absolute value loss function, and cross-entropy loss function, without specific limitations.

[0164] Indicatively, during the training phase, the aforementioned sample content, its data collection, and the construction of its label data can be achieved through recommendation logs recorded by a server, corresponding to content that has undergone a complete recommendation process. These recommendation logs record data obtained from different accounts browsing the content during the information flow content distribution process. In some embodiments, label data corresponding to the sample content is obtained by matching the pre-defined, finely granular account types with the aforementioned recommendation logs.

[0165] In one example, taking video content as the sample content, the recommendation log recorded on the server can have a data format of (t, uid). i vid v ,exp i,v clk i,v playdur i,v Complete i,v ), where t is the data reporting timestamp, that is, the timestamp at which the recommendation log records the browsing behavior of the account, and uid i Used to indicate account i, vid v Used to indicate video v, exp i,v Used to indicate the number of times video v is viewed for account i, clk i,v Used to indicate the number of clicks received by video v during the browsing process of account i, playdur i,v Used to indicate the duration of video v viewed by account i, Complete i,v Used to indicate the completion rate of video v for account i, and to determine the level of engagement during playback.

[0166] Based on the defined account types and their corresponding needs, a set of accounts is extracted from the recommendation logs (taking the target account type as an example). This allows for the filtering of the recommendation logs to obtain records corresponding to the target account type. In the data aggregation and cleaning stages, recommendation logs within a specific recommendation probability range are selected based on the vid. v The data is grouped according to preset rules, and browsing data such as exposure, clicks, and duration are aggregated to obtain the aggregated data structure as (vid). v ,exp v clk v playdur v Complete v ), where the total number of impressions of video v under the target account type is exp v It is calculated from Formula 1, where exp i,v This is used to indicate the number of times video v is displayed for account i, where U is the set of accounts of the target account type mentioned above.

[0167] Formula 1: exp v =sum i∈U (exp i,v )

[0168] Total clicks clk v Calculated using Formula 2, where clk i,v This is used to indicate the number of clicks received by video v during the browsing process of account i, where U is the set of accounts of the aforementioned target account type.

[0169] Formula 2: clk v =sum i∈U (clk i,v )

[0170] Average click-through rate (CTR) v Calculated by formula 3, where exp v For the total number of exposures mentioned above, clk v This represents the total number of clicks mentioned above.

[0171] Formula 3: CTR v =clk v / exp v

[0172] Average playback duration v Calculated using Formula 4, where playdur i,v This is used to indicate the viewing duration of account i for video v, where U is the set of accounts of the target account type mentioned above.

[0173] Formula 4: playdur v =avg i∈U (playdur i,v )

[0174] Average playback completion rate v Calculated by Formula 5, where Complete i,v This is used to indicate the completion rate of video v for account i, where U is the set of accounts of the target account type mentioned above.

[0175] Formula 5: Complete v =avg i∈U (Complete i,v )

[0176] Furthermore, through average click-through rate (CTR) v Formula 6 can be used to obtain the average click-through rate of the target account type for all content. Where V is the set of videos extracted as all of the above content.

[0177] Formula Six:

[0178] By average playback duration v Formula 7 can be used to obtain the average duration of all content for the target account type. Where V is the set of videos extracted as all of the above content.

[0179] Formula 7:

[0180] By average playback completion rate Complete v Formula 8 can be used to obtain the average completion rate of all content for the target account type. Where V is the set of videos extracted as all of the above content.

[0181] Formula 8:

[0182] In some embodiments, it can be achieved through or One type of data is labeled positively or negatively; that is, the overall business optimization goal is determined, and one of the three types of data is selected as the judgment criterion based on the optimization goal. In one example, taking click-through rate as an example, if video v... The video v is then marked as a positive sample under the target account type; otherwise, it is compared as a negative sample under the target account type.

[0183] In other embodiments, a multi-task optimization objective can also be set, that is, a judgment can be made by combining at least two of the above-mentioned data. Illustratively, a weighted average can be taken after normalizing at least two types of data to make the judgment.

[0184] This method can be applied to recommendation systems in various scenarios. Here, we take the information flow scenario as an example to illustrate the recommendation system:

[0185] To illustrate, in order to continuously improve the accuracy of the content recognition model and capture changes in account preferences, a closed loop of the recommendation system is formed by combining offline mining with online prediction.

[0186] In one example, such as Figure 10The diagram illustrates a flowchart of a content recommendation method provided in an exemplary embodiment of this application, including an offline mining section 1010 and an online prediction section 1020. The offline mining section 1010 obtains recommendation logs and account information from a recommendation log database 1001 and an account information database 1002, and inputs them into a data aggregation module 1011 for data aggregation. This data aggregation module 1011 is configured with a scheduled task. After data aggregation is completed, the data aggregation module 1011 inputs the processed data into a sample construction and cleaning module 1012, outputting sample content and corresponding label data. The sample content and label data are then input into an offline model training module 1013 for model training, outputting a trained candidate content recognition model. The candidate content recognition model is input into the model evaluation module 1014. The model evaluation module 1014 uses offline A / B testing to screen the candidate content recognition model and the content recognition model currently used in the online prediction part 1020, and determines the better model as the content recognition model used in the recognition model module 1021 in the online prediction part 1020. If the newly trained candidate content recognition model is better, the candidate content recognition model replaces the model in the recognition model module 1021; otherwise, the content recognition model in the recognition model module 1021 continues to be used. The recognition model module 1021 obtains candidate content from the content database 1022, identifies and predicts it, marks the recommendation probability on the candidate content, and inputs the candidate content into the recommendation system 1023. The recommendation system 1023 obtains the account type from the account information database 1002 and recommends the candidate content to the terminal device 1030 based on the recognition results of the candidate content and its marking. The terminal device 1030 uploads the browsing history to the recommendation log database 1001 for recording, thereby realizing the closed loop of the entire process.

[0187] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the account information and account types involved in this application were obtained with full authorization.

[0188] Please refer to Figure 11 This illustrates a recommended device structure block diagram of an exemplary embodiment of the content provided in this application, the device comprising the following modules:

[0189] The acquisition module 1110 is used to acquire a first feature representation corresponding to the first type of feature data of the candidate content, wherein the first type of feature data is used to indicate the publication attribute information of the candidate content;

[0190] The acquisition module 1110 is further configured to acquire a second feature representation corresponding to the second type of feature data of the candidate content, wherein the second type of feature data is used to indicate the content information of the candidate content;

[0191] Prediction module 1120 is used to predict the recommendation probability of at least two account types based on the first feature representation and the second feature representation. The recommendation probability is used to indicate the interest of the account under the account type in the candidate content, and to obtain the recommendation probability corresponding to the at least two account types respectively. The at least two account types include the target account type.

[0192] The sending module 1130 is used to push the candidate content to the target account under the target account type in response to the candidate content's recommendation probability meeting the recommendation conditions.

[0193] In some alternative embodiments, such as Figure 12 As shown, the prediction module 1120 includes a fusion unit 1121 and a prediction unit 1122;

[0194] The fusion unit 1121 is used to encode and fuse the first feature representation and the second feature representation to obtain a fused feature representation;

[0195] The prediction unit 1122 is used to predict the recommendation probability of the at least two account types based on the fusion feature representation, so as to obtain the recommendation probability corresponding to the at least two account types respectively.

[0196] In some optional embodiments, the candidate content includes at least two types of first-class feature data;

[0197] The fusion unit 1121 is used to perform feature concatenation on at least two first feature representations to obtain a third feature representation;

[0198] The prediction unit 1122 is used to predict the recommendation probability of the at least two account types based on the second feature representation and the third feature representation, so as to obtain the recommendation probability corresponding to the at least two account types respectively.

[0199] In some optional embodiments, the acquisition module 1110 includes a first acquisition unit 1111;

[0200] The first acquisition unit 1111 is used to input the i-th type of first-class feature data into the i-th transformation network for feature transformation to obtain the i-th first feature representation. The transformation network is used to encode and decode the first-class feature data according to the attention mechanism, where i is a positive integer.

[0201] In some optional embodiments, the acquisition module 1110 includes a second acquisition unit 1112;

[0202] The second acquisition unit 1112 is used to acquire the second type of feature data under at least two data modalities corresponding to the candidate content, wherein the data modalities are used to indicate the display format of the candidate content; and input the second type of feature data into the feature extraction network corresponding to the data modalities to obtain the second feature representation.

[0203] In some optional embodiments, the second acquisition unit 1112 is further configured to, in response to the data modality of the second type of feature data being a visual modality, input the second type of feature data into an image feature extraction network to obtain the second feature representation, wherein the image feature extraction network is used to extract features based on the image data of the candidate content; in response to the data modality of the second type of feature data being an audio modality, input the second type of feature data into an audio feature extraction network to obtain the second feature representation, wherein the audio feature extraction network is used to extract features based on the audio data of the candidate content; in response to the data modality of the second type of feature data being a text modality, input the second type of feature data into a text feature extraction network to obtain the second feature representation, wherein the text feature extraction network is used to extract features based on the text data of the candidate content.

[0204] In some optional embodiments, when the data modality of the second type of feature data is a visual modality, the second type of feature data includes an image sequence of the candidate content;

[0205] The second acquisition unit 1112 is further configured to acquire a main frame image from the image sequence, the main frame image representing images with similar scenes in the image sequence; establish the connection between each pixel in the main frame image through the sliding window and the original window provided by the image feature extraction network to obtain the global attention feature corresponding to the main frame image; and, in response to the image sequence including at least two main frame images, aggregate the global attention features of the at least two main frame images to obtain the second feature representation.

[0206] In some optional embodiments, when the data modality of the second type of feature data is an audio modality, the second type of feature data includes the audio data of the candidate content;

[0207] The second acquisition unit 1112 is further configured to generate a spectrogram corresponding to the audio data; map the spectrogram to a preset filter group to generate a spectral feature corresponding to the audio data; combine audio frames based on the spectral feature to obtain an audio frame feature; and extract features from the sound events in the audio frame feature to obtain the second feature representation.

[0208] In some optional embodiments, when the data modality of the second type of feature data is text modality, the second type of feature data includes the text data of the candidate content;

[0209] The second acquisition unit 1112 is further configured to extract features from the text data based on the semantic expression of the text data to obtain the second feature representation.

[0210] In summary, the content recommendation device provided in this application, when content recommendation is required, predicts the recommendation probability of candidate content under different account types based on the characteristics of the candidate content at the publishing attribute information level and the characteristics at the content information level. This yields the recommendation probability corresponding to each candidate content and each account type, and determines whether to push the candidate content to the target account based on the account type of the candidate content. That is, when recommending content, only the account type of the account to be recommended needs to be obtained, and content recommendation can be performed based on the recommendation probability of the candidate content under that account type, thus ensuring the recommendation accuracy of new accounts during the cold start phase. Furthermore, since the prediction is based on the publishing attribute information and content information of the content itself, it is also suitable for newly added candidate content, thereby ensuring the recommendation efficiency of new content during the cold start phase.

[0211] It should be noted that the content recommendation device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the content recommendation device and the content recommendation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0212] Figure 13 This illustration shows a schematic diagram of the structure of a server provided in an exemplary embodiment of this application. Specifically, it includes the following structure.

[0213] Server 1300 includes a Central Processing Unit (CPU) 1301, a system memory 1304 including Random Access Memory (RAM) 1302 and Read Only Memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. Server 1300 also includes a mass storage device 1306 for storing the operating system 1313, application programs 1314, and other program modules 1315.

[0214] Mass storage device 1306 is connected to central processing unit 1301 via a mass storage controller (not shown) connected to system bus 1305. Mass storage device 1306 and its associated computer-readable media provide non-volatile storage for server 1300. That is, mass storage device 1306 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0215] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1304 and mass storage device 1306 described above can be collectively referred to as memory.

[0216] According to various embodiments of this application, server 1300 can also be connected to a remote computer on a network, such as the Internet. That is, server 1300 can be connected to network 1312 via network interface unit 1311 connected to system bus 1305, or it can also use network interface unit 1311 to connect to other types of networks or remote computer systems (not shown).

[0217] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0218] Embodiments of this application also provide a computer device including a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the recommended methods provided in the above-described method embodiments. Optionally, the computer device may be a terminal or a server.

[0219] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the recommended methods provided in the above-described method embodiments.

[0220] Embodiments of this application also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the recommended method described in any of the above embodiments.

[0221] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0222] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0223] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of recommending content, characterized by, The method includes: Acquire at least two types of first-class feature data and at least two types of second-class feature data in at least two data modalities for candidate content. The first-class feature data is used to indicate the publication attribute information of the candidate content and includes at least one of the following: the category level of the candidate content, the account information of the publishing account, and the length of the candidate content. The second-class feature data is used to indicate the content information of the candidate content. The data modalities are used to indicate the display format of the candidate content and include at least one of the following: the title, description, tags, and content data of the candidate content. By using the shallow part of the deep-shallow fusion multimodal model, the i-th type of first-class feature data is input into the i-th transformation network for feature transformation to obtain the i-th first feature representation. The shallow part includes at least two transformation networks, which are used to encode and decode the first-class feature data according to the attention mechanism, where i is a positive integer. At least two of the first feature representations are concatenated to obtain a third feature representation; Through the deep part of the deep-shallow fusion multimodal model, the second type of feature data under the at least two data modalities are respectively input into the feature extraction network corresponding to the data modalities to obtain the second feature representations corresponding to the at least two data modalities respectively. The deep part includes the feature extraction network corresponding to the at least two data modalities respectively. Based on the third feature representation and the second feature representation, the deep and shallow fusion multimodal model predicts the recommendation probability for at least two account types, and obtains the recommendation probability corresponding to the at least two account types respectively. The recommendation probability is used to indicate the interest of the account under the account type in the candidate content. The account type is an account tag that classifies different accounts according to the basic information of the account within the target application. The at least two account types include the target account type. In response to the fact that the recommendation probability of the candidate content under the target account type meets the recommendation criteria, the candidate content is pushed to the target account under the target account type.

2. The method according to claim 1, characterized in that, The step of predicting recommendation probabilities for at least two account types based on the third feature representation and the second feature representation, to obtain the recommendation probabilities corresponding to the at least two account types respectively, includes: The third feature representation and the second feature representation are encoded and fused to obtain a fused feature representation; Based on the fusion feature representation, the recommendation probability is predicted for the at least two account types to obtain the recommendation probability corresponding to each of the at least two account types.

3. The method according to claim 1 or 2, characterized in that, The step of inputting the second type of feature data under the at least two data modalities into the feature extraction network corresponding to the data modalities respectively to obtain the second feature representations corresponding to the at least two data modalities includes: In response to the fact that the data mode of the second type of feature data is a visual mode, the second type of feature data is input into an image feature extraction network to obtain the second feature representation. The image feature extraction network is used to extract features based on the image data of the candidate content. In response to the fact that the data mode of the second type of feature data is audio mode, the second type of feature data is input into an audio feature extraction network to obtain the second feature representation. The audio feature extraction network is used to extract features based on the audio data of the candidate content. In response to the fact that the data modality of the second type of feature data is text modality, the second type of feature data is input into a text feature extraction network to obtain the second feature representation. The text feature extraction network is used to extract features based on the text data of the candidate content.

4. The method according to claim 3, characterized in that, When the data modality of the second type of feature data is a visual modality, the second type of feature data includes the image sequence of the candidate content; The step of inputting the second type of feature data into the image feature extraction network to obtain the second feature representation includes: Obtain a main frame image from the image sequence, the main frame image being used to represent images with similar scenes in the image sequence; By using the sliding window and the original window provided by the image feature extraction network, the relationship between each pixel in the main frame image is established, and the global attention feature corresponding to the main frame image is obtained. In response to the image sequence comprising at least two main frame images, the global attention features of the at least two main frame images are aggregated to obtain the second feature representation.

5. The method according to claim 3, characterized in that, When the data modality of the second type of feature data is an audio modality, the second type of feature data includes the audio data of the candidate content; The step of inputting the second type of feature data into the audio feature extraction network to obtain the second feature representation includes: Generate the spectrogram corresponding to the audio data; The spectrogram is mapped to a preset filter group to generate the acoustic spectrum features corresponding to the audio data; Based on the aforementioned acoustic spectrum features, audio frames are combined to obtain audio frame features; The sound events in the audio frame features are extracted to obtain the second feature representation.

6. The method according to claim 3, characterized in that, When the data modality of the second type of feature data is text modality, the second type of feature data includes the text data of the candidate content; The step of inputting the second type of feature data into the text feature extraction network to obtain the second feature representation includes: Based on the semantic expression of the text data, features are extracted from the text data to obtain the second feature representation.

7. A content recommendation device, characterized in that, The device includes: The acquisition module is used to acquire at least two types of first-class feature data and at least two types of second-class feature data under at least two data modalities. The first-class feature data is used to indicate the publication attribute information of the candidate content and includes at least one of the following: the classification level of the candidate content, the account information of the publishing account, and the length of the candidate content. The second-class feature data is used to indicate the content information of the candidate content. The data modalities are used to indicate the display format of the candidate content and include at least one of the following: the title, description, tags, and content data of the candidate content. The first acquisition unit is used to input the i-th type of first-class feature data into the i-th transformation network for feature transformation through the shallow part of the deep-shallow fusion multimodal model to obtain the i-th first feature representation. The shallow part includes at least two transformation networks. The transformation networks are used to encode and decode the first-class feature data according to the attention mechanism, where i is a positive integer. A fusion unit is used to perform feature concatenation on at least two of the first feature representations to obtain a third feature representation; The second acquisition unit is used to input the second type of feature data under the at least two data modes into the feature extraction network corresponding to the data mode through the deep part of the deep-shallow fusion multimodal model, so as to obtain the second feature representation corresponding to the at least two data modes respectively. The deep part includes the feature extraction network corresponding to the at least two data modes respectively. The prediction module is used to predict the recommendation probability of at least two account types based on the third feature representation and the second feature representation using the deep and shallow fusion multimodal model, and to obtain the recommendation probability corresponding to the at least two account types respectively. The recommendation probability is used to indicate the interest of the account under the account type in the candidate content. The account type is an account tag in the target application that classifies different accounts according to the basic information of the account. The at least two account types include the target account type. The sending module is used to push the candidate content to the target account under the target account type in response to the candidate content's recommendation probability meeting the recommendation conditions.

8. The apparatus according to claim 7, characterized in that, The prediction module includes a fusion unit and a prediction unit; The fusion unit is used to encode and fuse the third feature representation and the second feature representation to obtain a fused feature representation; The prediction unit is used to predict the recommendation probability of the at least two account types based on the fused feature representation, so as to obtain the recommendation probability corresponding to the at least two account types respectively.

9. The apparatus according to claim 7 or 8, characterized in that, The second acquisition unit is further configured to, in response to the data modality of the second type of feature data being visual, input the second type of feature data into an image feature extraction network to obtain the second feature representation, wherein the image feature extraction network is used to extract features based on the image data of the candidate content; in response to the data modality of the second type of feature data being audio, input the second type of feature data into an audio feature extraction network to obtain the second feature representation, wherein the audio feature extraction network is used to extract features based on the audio data of the candidate content; and in response to the data modality of the second type of feature data being text, input the second type of feature data into a text feature extraction network to obtain the second feature representation, wherein the text feature extraction network is used to extract features based on the text data of the candidate content.

10. The apparatus according to claim 9, characterized in that, When the data modality of the second type of feature data is a visual modality, the second type of feature data includes the image sequence of the candidate content; The second acquisition unit is further configured to acquire a main frame image from the image sequence, the main frame image being used to represent images with similar scenes in the image sequence; and to establish the connection between each pixel in the main frame image through the sliding window and the original window provided by the image feature extraction network, thereby obtaining the global attention features corresponding to the main frame image. In response to the image sequence comprising at least two main frame images, the global attention features of the at least two main frame images are aggregated to obtain the second feature representation.

11. The apparatus according to claim 9, characterized in that, When the data modality of the second type of feature data is an audio modality, the second type of feature data includes the audio data of the candidate content; The second acquisition unit is further configured to generate a spectrogram corresponding to the audio data; map the spectrogram to a preset filter group to generate a spectral feature corresponding to the audio data; and combine audio frames based on the spectral feature to obtain audio frame features; The sound events in the audio frame features are extracted to obtain the second feature representation.

12. The apparatus according to claim 9, characterized in that, When the data modality of the second type of feature data is text modality, the second type of feature data includes the text data of the candidate content; The second acquisition unit is further configured to extract features from the text data based on the semantic expression corresponding to the text data, and obtain the second feature representation.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the recommended method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the recommended method as described in any one of claims 1 to 6.

15. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the recommended method as described in any one of claims 1 to 6.

Images