A content recommendation method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of computers, in particular to the field of artificial intelligence, and provides a content recommendation method and device, an electronic device and a storage medium, to improve the recommendation accuracy of multimedia content. The method comprises the following steps: acquiring content feature information of each multimedia content in a multimedia content set to be recommended; determining the recommendation sequence of each multimedia content in a cyclic iteration manner based on the content feature information; and performing the following operations in one iteration process: determining a current round recommendation position according to a selected multimedia content set; determining each candidate multimedia content which has not been sorted in a candidate multimedia content set; acquiring the action income corresponding to each candidate multimedia content based on the content feature information of each multimedia content, the selected multimedia content set and the candidate multimedia content set, and selecting the candidate multimedia content according to the action income and putting the candidate multimedia content into the current round recommendation position. Since the action income of the multimedia content is considered, the recommendation accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to the technical field of artificial intelligence, and provides a content recommendation method and device, an electronic device and a storage medium. BACKGROUND

[0002] A recommendation system is commonly used to recommend multimedia content to a user. If subdivided, the recommendation system generally has four links: recall -> rough sorting -> fine sorting -> rearrangement. The rearrangement module is the last stage of personalized recommendation of the recommendation system. First, the multimedia content sorted by the fine sorting module is collected according to the maximum number limit of each category, and then the recommended position of the collected multimedia content in the exposure list is determined, and finally the recommended multimedia content is displayed to the user. In related technologies, a simple strategy is generally used to determine the recommended position of the multimedia content. Random arrangement, arrangement according to the fine sorting score sequence, etc. are commonly used for the recommended position of the multimedia content. However, the recommendation accuracy of these methods is relatively low. SUMMARY

[0003] The embodiments of the present application provide a content recommendation method, device, electronic device and storage medium to improve the recommendation accuracy of multimedia content.

[0004] The content recommendation method provided by the embodiments of the present application comprises the following steps.

[0005] Obtain content feature information of each multimedia content in a set of multimedia content to be recommended;

[0006] Determine the recommended order of each multimedia content in a cyclic iteration manner based on the content feature information of each multimedia content. In one iteration process, the following operations are performed:

[0007] Determine the recommended position of the current round according to the set of selected multimedia content, wherein the set of selected multimedia content contains each selected multimedia content sorted, and the recommended position of the current round is a new sorting position other than the position of each selected multimedia content sorted.

[0008] Determine each candidate multimedia content not sorted in the set of candidate multimedia content;

[0009] Based on the content feature information of each multimedia content, and the set of selected multimedia content and the set of candidate multimedia content, obtain the action revenue corresponding to each candidate multimedia content, and select at least one candidate multimedia content from the set of candidate multimedia content according to the action revenue and put it into the recommended position of the current round, wherein the action revenue is used to represent the probability of the object performing the target action on the multimedia content.

[0010] The embodiment of the application provides a content recommendation device, comprising:

[0011] An information acquisition unit is configured to acquire content feature information of each multimedia content in a set of multimedia contents to be recommended;

[0012] A recommendation unit is configured to determine a recommendation order of each multimedia content in a cyclic iteration manner based on the content feature information of each multimedia content; wherein the following operations are performed in one iteration process:

[0013] A current recommendation position is determined according to a set of selected multimedia contents, the set of selected multimedia contents comprising each selected multimedia content in a sorted order, and the current recommendation position being a new sorting position other than the position of each selected multimedia content in the sorted order;

[0014] Each candidate multimedia content in a set of candidate multimedia contents is determined;

[0015] Based on the content feature information of each multimedia content, the set of selected multimedia contents and the set of candidate multimedia contents, an action benefit corresponding to each candidate multimedia content is acquired, and at least one candidate multimedia content is selected from the set of candidate multimedia contents according to the action benefit and put into the current recommendation position, wherein the action benefit is used to represent a probability of an object performing a target action on the multimedia content.

[0016] Optionally, the device further comprises:

[0017] A model training unit is configured to perform cyclic iteration training on a rearrangement model according to training samples in a training sample data set, and output the trained rearrangement model when the training is completed; wherein the following operations are performed in one cyclic iteration training process:

[0018] A group of training samples is selected from the training sample data set;

[0019] The content feature information of each sample multimedia content in the group of training samples, and the set of selected sample multimedia contents and the set of candidate sample multimedia contents in the current state and the set of selected sample multimedia contents and the set of candidate sample multimedia contents in the next state are input into the rearrangement model, wherein the set of selected sample multimedia contents and the set of candidate sample multimedia contents are determined based on the real recommendation order;

[0020] The content feature information of each sample multimedia content is extracted based on the embedding part in the rearrangement model, and a second embedding feature vector for the group of training samples is obtained;

[0021] inputting the second embedding feature vector into a prediction part in the rearrangement model, predicting, based on the prediction part, a first estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the current state and a second estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state;

[0022] constructing a loss function based on the first estimated action benefit, the second estimated action benefit, and the corresponding actual action benefit, and adjusting parameters of the rearrangement model based on the constructed loss function.

[0023] Optionally, the model training unit is further configured to:

[0024] selecting an arbitrary training sample from the set of training samples;

[0025] constructing the selected sample multimedia content set in the current state based on each sample multimedia content before the sample multimedia content in the arbitrary training sample and constructing the candidate sample multimedia content set in the current state based on the sample multimedia content in the remaining training samples according to the real recommendation sequence; and

[0026] constructing the selected sample multimedia content set in the next state based on the sample multimedia content in the arbitrary training sample and each sample multimedia content before the sample multimedia content according to the real recommendation sequence and constructing the candidate sample multimedia content set in the next state based on the sample multimedia content in the remaining training samples.

[0027] Optionally, the prediction part includes a first prediction subnetwork and a second prediction subnetwork; and the model training unit is specifically configured to:

[0028] multiplying the second embedding feature vector and a second selected state vector based on the first prediction subnetwork, and obtaining a second context feature vector between each selected sample multimedia content in the selected sample multimedia content set in the current state based on an attention mechanism, wherein each element in the second selected state vector represents whether the multimedia content in the set of training samples corresponding to the current state has a determined recommendation sequence.

[0029] obtaining, based on the first prediction subnetwork, a second candidate feature vector obtained by multiplying the second embedding feature vector and a second candidate state vector, wherein the second candidate feature vector is used to represent content features of each candidate sample multimedia content in the current state, and each element in the second candidate state vector represents whether the multimedia content in the corresponding training sample in the group has a determined recommendation order in the current state;

[0030] extracting, by a fully connected layer in the first prediction subnetwork, a second splicing vector obtained by splicing the second context feature vector and the second candidate feature vector, to determine the first estimated action revenue corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the current state; and

[0031] obtaining, based on the second prediction subnetwork, a third context feature vector between each selected sample multimedia content in the selected sample multimedia content set in the next state by multiplying the second embedding feature vector and a third selected state vector based on an attention mechanism, wherein the third context feature vector is used to represent a context correlation between each selected sample multimedia content in the next state, and each element in the third selected state vector represents whether the multimedia content in the corresponding training sample in the group has a determined recommendation order in the next state;

[0032] obtaining, based on the second prediction subnetwork, a third candidate feature vector obtained by multiplying the third embedding feature vector and a third candidate state vector, wherein the third candidate feature vector is used to represent content features of each candidate sample multimedia content in the next state, and each element in the third candidate state vector represents whether the multimedia content in the corresponding training sample in the group has a determined recommendation order in the next state;

[0033] extracting, by a fully connected layer in the second prediction subnetwork, a third splicing vector obtained by splicing the third context feature vector and the third candidate feature vector, to determine the second estimated action revenue corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state.

[0034] Optionally, the model training unit is specifically configured to:

[0035] constructing a loss function based on a difference between a target estimated action revenue in each first estimated action revenue and a maximum estimated action revenue in each second estimated action revenue, and a corresponding real action revenue, wherein the target estimated action revenue is the first estimated action revenue corresponding to the sample multimedia content in the arbitrary training sample.

[0036] An electronic device provided by an embodiment of the present application comprises a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the content recommendation methods described above.

[0037] An embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of any one of the multimedia content recommendation methods described above.

[0038] An embodiment of the present application provides a computer readable storage medium comprising program code, and when the program product runs on an electronic device, the program code is used to make the electronic device execute the steps of any one of the content recommendation methods described above.

[0039] The present application has the following beneficial effects:

[0040] The embodiment of the present application provides a content recommendation method, device, electronic device and storage medium. Since the embodiment of the present application adopts a circulating structure, the long-term action benefits of each candidate multimedia content at the current position are predicted round by round, and at least one candidate multimedia content is selected based on the action benefit analysis and put into the recommended position of the current round. Since the action benefits of each multimedia content under dynamic conditions are considered, the final recommended accuracy is higher.

[0041] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be understood from the practice of the present application. The purpose and other advantages of the present application can be achieved and obtained by the structure specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0042] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 An optional schematic diagram for an application scenario in an embodiment of the present application;

[0044] Figure 2 A flowchart of a content recommendation method in an embodiment of the present application;

[0045] Figure 3 A structure diagram of a rearrangement model in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a first kind of loop iteration process in the embodiments of the present application;

[0047] Figure 5 A schematic diagram of a second kind of loop iteration process in the embodiments of the present application;

[0048] Figure 6 A schematic diagram of a third kind of loop iteration process in the embodiments of the present application;

[0049] Figure 7 A schematic diagram of a flow of a rearrangement model training method in the embodiments of the present application;

[0050] Figure 8 A schematic diagram of a structure of another rearrangement model in the embodiments of the present application;

[0051] Figure 9 A schematic diagram of a flow of a calculation of an estimated action reward in the embodiments of the present application;

[0052] Figure 10 A schematic diagram of a component structure of a content recommendation device in the embodiments of the present application;

[0053] Figure 11 A schematic diagram of a hardware component structure of a first kind of electronic device applying the embodiments of the present application;

[0054] Figure 12 A schematic diagram of a hardware component structure of a second kind of electronic device applying the embodiments of the present application. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in a clear and complete manner with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments described in the present application document, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the technical solutions of the present application.

[0056] Some concepts involved in the embodiments of the present application will be introduced below.

[0057] Beam search: optimization of the best-first search algorithm. It uses a heuristic function to evaluate the ability of each node it checks. However, beam search can only find the top m nodes that meet the conditions in each depth, and m is a fixed number, indicating the width of the beam. In related technologies, beam search can be used to determine the recommended order of the recommended positions of multimedia content.

[0058] Multimedia content: a man-machine interactive information exchange and communication media combining two or more media. The media includes text, pictures, sound, video, etc. In the embodiments of the present application, the multimedia content can be articles, news, information, videos, music, etc.

[0059] Recommendation system: a tool for automatically connecting users and items, which can help users find information that interests them in an information overload environment, and can also push information to users who are interested in them. Rearrangement is the last stage of the recommendation system, in which the system selects and reorders the candidate multimedia content to determine the position and order of the multimedia content display.

[0060] Action benefit: used to represent the probability of the object performing the target action on the multimedia content. Taking an article as an example, in the article recommendation scenario, the target action can be clicking, liking, browsing for a certain length of time, etc. When a certain article is exposed in a certain position, the user's clicking, liking, forwarding or consuming (browsing) for a certain length of time on the article will generate a certain benefit for the recommendation system, and the same article will obtain different benefits when exposed in different positions. Therefore, the action benefit in the embodiments of the present application is mainly used to represent the probability of the user performing the target action on the recommended article when the recommended article is exposed in a certain position. The greater the probability, the greater the benefit obtained by exposing the recommended article in the position. Correspondingly, the smaller the probability, the smaller the benefit obtained by exposing the recommended article in the position.

[0061] Selected multimedia content: represents the multimedia content whose recommendation position has been determined. In the embodiments of the present application, the loop structure is used to sort each recommended multimedia content in the set of recommended multimedia content. At least one recommendation position can be determined in each loop iteration process. Therefore, at least one multimedia content whose recommendation position is determined is added in each iteration process. The number of selected multimedia content in different iteration processes is different, and there is no selected multimedia content in the first iteration process.

[0062] Candidate multimedia content: represents the multimedia content whose recommendation position has not been determined. Similar to the selected multimedia content, the loop structure is used to sort each recommended multimedia content in the set of recommended multimedia content in the embodiments of the present application. At least one recommendation position can be determined in each loop iteration process. Therefore, at least one multimedia content whose recommendation position is not determined is reduced in each iteration process. The number of candidate multimedia content in different iteration processes is also different, and the candidate multimedia content is all the recommended multimedia content in the first iteration process.

[0063] Content feature information: used to represent the content attribute of the multimedia content, for example, the multimedia content is a news article, the content feature information herein mainly refers to the category of the article, the content described by the article, the author, the number of likes, and the like, in addition, the associated features and environmental features of the article can also be included, and the like, which are not limited here.

[0064] Selected state vector: is determined based on each selected (sample) multimedia content, and each element in the selected state vector represents whether the corresponding (sample) multimedia content has determined the recommendation order. The first selected state vector, the second selected state vector and the third selected state vector in the embodiments of the present application all belong to the selected state vector, wherein the first selected state vector is for the set of multimedia contents to be recommended in the application process, and the second selected state vector and the third selected state vector are for the sample multimedia content in the model training process.

[0065] Candidate state vector: is determined based on each candidate (sample) multimedia content, and each element in the first candidate state vector represents whether the corresponding (sample) multimedia content has not determined the recommendation order. Similar to the selected state vector, the first candidate state vector, the second candidate state vector and the third candidate state vector in the embodiments of the present application all belong to the candidate state vector, wherein the first candidate state vector is also for the set of multimedia contents to be recommended in the application process, and the second candidate state vector and the third candidate state vector are for the sample multimedia content in the model training process.

[0066] The embodiments of the present application relate to artificial intelligence (AI) and machine learning technology, and are designed based on computer vision technology and machine learning (ML) in artificial intelligence.

[0067] Artificial intelligence is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence.

[0068] Artificial intelligence is the study of the design and implementation of intelligent machines, which have the ability to perceive, reason, and make decisions. Artificial intelligence technology mainly includes computer vision technology, natural language processing technology, and machine learning / deep learning. With the development of artificial intelligence technology, artificial intelligence has been applied in many fields, such as smart home, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, autonomous vehicles, autonomous driving, robots, and intelligent medical care. With the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0069] Natural language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. Natural language processing is a science that combines linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, which is the language used in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, and knowledge graph technology.

[0070] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It is a field of study that focuses on how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Unlike data mining, which finds mutual characteristics between large data, machine learning places more emphasis on algorithm design, allowing computers to automatically "learn" rules from data and use the rules to predict unknown data.

[0071] Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. It is applied in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. Reinforcement learning (RL) is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of maximizing rewards or achieving specific goals through learning strategies in the interaction between agents and the environment.

[0072] The embodiment of the present application adopts a reinforcement learning rearrangement model when sorting multimedia content. The method for training the rearrangement model proposed in the embodiment of the present application can be divided into two parts, including a training part and an application part; wherein the training part relates to the technical field of machine learning, in the training part, the rearrangement model is trained through the technology of machine learning, so that the content feature information containing sample multimedia content, and the training sample of the real recommendation order and the real action benefit corresponding to the sample multimedia content given in the embodiment of the present application are used to train the rearrangement model, the output result of the rearrangement model is obtained after the training sample passes through the rearrangement model, and the model parameters are continuously adjusted through the optimization algorithm combined with the output result; the application part is used to sort each multimedia content in the set of multimedia content to be recommended, etc. using the rearrangement model trained in the training part. In addition, it should be noted that the rearrangement model in the embodiment of the present application can be online training or offline training, which is not limited here. In this paper, offline training is taken as an example for illustration.

[0073] The design idea of the embodiment of the present application is briefly introduced as follows:

[0074] In the information-overloaded Internet era, the importance of information recommendation technology for the operation of Internet companies is self-evident, and various websites will use a recommendation system in the background. The recommendation calculates the multimedia content most suitable for recommending to the user according to the user's access characteristics, and then displays these multimedia content to the user for selection. Because the number of multimedia content and the number of users are very large, the recommendation system uses a lot of cloud computing-based data mining to mine user behavior and product information, uses various recommendation algorithms to calculate the recommendation list, and realizes the effect of displaying different content to different users to improve the page conversion rate.

[0075] In the related art, the rearrangement module of the recommendation system is used to calculate the recommendation list to determine the recommendation position of each multimedia content. Generally, a simple strategy is mainly used to determine the recommendation position of the multimedia content, and the recommendation position of the multimedia content is often randomly arranged, arranged in the order of precision ranking, or the approximate optimal arrangement of the news recommendation position is searched using bundle search. Among them, the online bundle search to find the approximate optimal arrangement is not a global optimal arrangement. The recommendation accuracy of these methods is relatively low.

[0076] Therefore, the embodiment of the present application proposes a content recommendation method, device, electronic equipment and storage medium. Since the embodiment of the present application adopts a cyclic structure, the long-term action benefit of each candidate multimedia content in the current position is predicted round by round, and at least one candidate multimedia content is selected based on the action benefit analysis to put into the recommendation position of the current round. Since the action benefits of each multimedia content under dynamic conditions are considered, the final recommendation accuracy is higher.

[0077] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings, it should be understood that the preferred embodiments described herein are only used to explain and illustrate the present application, and are not used to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0078] As shown in Figure 1 , which is an application scenario diagram of the embodiments of the present application. It is an application scenario diagram of the embodiments of the present application. The application scenario diagram includes two terminal devices 110 and one server 120. The terminal device 110 and the server 120 can communicate through a communication network. The user can browse multimedia content through the terminal device 110, and the terminal device 110 can be installed with multimedia content related applications, such as video software, short video software, etc. The application involved in the embodiments of the present application can be software, or a webpage, an applet, etc. The background server is the background server corresponding to the software or the webpage, the applet, etc. The specific type of the client is not limited.

[0079] In an optional implementation, the communication network is a wired network or a wireless network. The terminal device 110 and the server 120 can be directly or indirectly connected through wired or wireless communication, which is not limited herein.

[0080] In the embodiments of the present application, the terminal device 110 is an electronic device used by the user, which can be a personal computer, a mobile phone, a tablet computer, a notebook computer, an e-book reader, a smart home, etc. A computer device with certain computing power and running instant messaging software and websites or social software and websites. Each terminal device 110 is connected to the server 120 through a wireless network. The server 120 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc. Basic cloud computing services such as platform.

[0081] The rearrangement model can be deployed on the server 120 for training. The server 120 can store a large number of training samples for training the rearrangement model. Optionally, after the rearrangement model is trained based on the training method in the embodiments of the present application, the trained rearrangement model can be directly deployed on the server 120 or the terminal device 110. Generally, the rearrangement model is directly deployed on the server 120. In the embodiments of the present application, the rearrangement model is mainly used to sort each multimedia content in the set of multimedia contents to be recommended, determine the recommendation order of each multimedia content, and then applied to various information flow recommendation scenarios such as video and news, and e-commerce recommendation scenarios. For example, the rearrangement module applied to the news recommendation of the mobile client's important news channel and recommendation channel sorts the recalled news, determines the recommendation position of each multimedia content filtered by the rearrangement module according to the multimedia content recommendation method listed in the embodiments of the present application, and then displays the recommendation position to the user, and the like. For other recommendation scenarios, the principle is similar. In this paper, news recommendation is mainly used as an example for illustration.

[0082] In a possible application scenario, the training samples in the present application can be stored using cloud storage technology. Cloud storage is a new concept extended and developed on the basis of the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) is a storage system that cooperatively provides data storage and business access functions by collecting various types of storage devices (storage nodes) in the network through cluster applications, grid technology, and distributed storage file systems, and the like.

[0083] In a possible application scenario, in order to reduce communication delay, the server 120 can be deployed in each region, or in order to balance the load, different servers 120 can be used to serve the corresponding regions of each terminal device 110. The multiple servers 120 share data through a blockchain, and the multiple servers 120 are equivalent to a data sharing system composed of multiple servers 120. For example, the terminal device 110 is located in place a and is connected to the server 120 for communication, and the terminal device 110 is located in place b and is connected to other servers 120 for communication.

[0084] For each server 120 in the data sharing system, there is a node identifier corresponding to the server 120, and each server 120 in the data sharing system can store the node identifiers of other servers 120 in the data sharing system, so as to subsequently broadcast the generated block to other servers 120 in the data sharing system according to the node identifiers of the other servers 120. Each server 120 can maintain a node identifier list as shown in the following table, and store the server 120 name and the node identifier into the node identifier list. The node identifier can be an Internet Protocol (IP) address and any other information capable of identifying the node, and the IP address is taken as an example in Table 1 for illustration only.

[0085] Table 1

[0086] Server name Node identification Node 1 119.115.151.174 Node 2 118.116.189.145 … … Node N 119.124.789.258

[0087] The object state recognition method provided by the example embodiments of the present application will be described below in combination with the above-described application scenarios and with reference to the accompanying drawings. It should be noted that the above-described application scenarios are only shown for the purpose of facilitating understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

[0088] Referring to Figure 2 Fig. 1 shows an implementation flowchart of a content recommendation method provided by an embodiment of the present application, and the specific implementation flow of the method is as follows:

[0089] S21: Obtain content feature information of each multimedia content in the set of multimedia contents to be recommended;

[0090] The content feature information is used to represent the content attributes of the multimedia content, and the present document mainly takes a news article as an example for illustration, and the content feature information herein mainly refers to the category of the article, the content described by the article, the author, the number of likes, and the like, and in addition, can also include the associated features and environmental features of the article, and the like, which are not limited in detail.

[0091] In the embodiments of the present application, each multimedia content in the set of multimedia contents to be recommended refers to news (including photo-text news and video news) that the user may be interested in, which is filtered out for recommendation to the user according to the user's interest characteristics and reading behavior. Specifically, the multimedia contents to be recommended are obtained based on the fine arrangement module, and the fine arrangement module can also be implemented based on a neural network model. For example, the Predict Click-Through Rate (PCTR) of each multimedia content is estimated based on the current user's interest portrait and characteristics, the category and other characteristics of the multimedia content, the environmental characteristics, and the associated characteristics of the displayed multimedia content, the PCTR refers to the score of the model estimated by the user clicking the recommended article, and the fine arrangement sorting is performed according to the PCTR of each multimedia content, and the final multimedia content to be recommended is selected.

[0092] S22: Determine the recommendation order of each multimedia content based on the content feature information of each multimedia content in a cyclic iteration manner; wherein the following operations are performed in one iteration process:

[0093] S221: Determine the current recommendation position according to the selected multimedia content set, the selected multimedia content set contains each selected multimedia content sorted, and the current recommendation position is a new sorting position other than the position of each selected multimedia content sorted;

[0094] S222: Determine each candidate multimedia content that has not been sorted in the candidate multimedia content set;

[0095] S223: Based on the content feature information of each multimedia content, and the selected multimedia content set and the candidate multimedia content set, obtain the action revenue corresponding to each candidate multimedia content, and select at least one candidate multimedia content from the candidate multimedia content set according to the action revenue and put it into the current recommendation position.

[0096] Wherein, the action revenue is used to represent the probability of the object performing the target action on the multimedia content. In the embodiments of the present application, the multimedia content is taken as an article for example, in the article recommendation scene, the target action can represent clicking, or the consumption (browsing) time length reaching a certain time length, or liking, forwarding, etc., which is not limited here. When a certain article is exposed in a certain position, the user's clicking, liking, forwarding, or consuming a certain time length on the article will generate a certain revenue for the recommendation system.

[0097] It should be noted that in the embodiments of the present application, the same article obtains different benefits when exposed in different positions, for example, for article 1, the corresponding action benefit is reward1 when the article is exposed in the recommended position 1, and the corresponding action benefit is reward2 when the article is exposed in the recommended position 2. Therefore, the action benefit in the embodiments of the present application is mainly used to represent the probability of the user performing the target action on the recommended article when the recommended article is exposed in a certain position. The greater the probability, the greater the benefit obtained by exposing the recommended article in the position. Correspondingly, the smaller the probability, the smaller the benefit obtained by exposing the recommended article in the position. The action benefit in step S223 refers to the corresponding action benefit when the article is exposed in the recommended position in this round.

[0098] In the embodiments of the present application, the selected multimedia content represents the multimedia content of the currently determined recommended position. The embodiments of the present application use a loop structure to sort each recommended multimedia content in the set of recommended multimedia content. One recommended position can be determined in one loop iteration process. Therefore, at least one multimedia content of the determined recommended position is added in each iteration process. The number of selected multimedia content in different iteration processes is different, and there is no selected multimedia content in the first iteration process.

[0099] For example, there are 10 articles in the set of recommended multimedia content. The recommended order of the 10 articles is determined based on the loop iteration method. It is assumed that one recommended position is determined in each round. In the first iteration process, the selected multimedia content set is empty, and the corresponding recommended position in this round is the first recommended position. In the second iteration process, there is one selected multimedia content in the selected multimedia content set, for example, {2}, which indicates that the article with id=2 is selected in the first iteration process and located in the first recommended position. The corresponding recommended position in this round is the second recommended position. Similarly, the loop is iterated 9 or 10 times to determine the corresponding multimedia content in each recommended position.

[0100] The candidate multimedia content is corresponding to the selected multimedia content, and is used to represent the multimedia content whose recommended position is not determined. Similar to the selected multimedia content, at least one multimedia content whose recommended position is not determined is reduced in each iteration process, the number of candidate multimedia content in different iteration processes is different, and in the first iteration process, the candidate multimedia content is all the multimedia content to be recommended. In the second iteration process, there are 9 candidate multimedia contents in the candidate multimedia content set, for example, {1, 3, 4, 5, 6, 7, 8, 9, 10}, which indicates that the article with id = 2 is selected to be in the first recommended position in the first iteration process, and so on. After 10 cycles, the candidate multimedia content set is empty, that is, the multimedia content corresponding to each recommended position is determined.

[0101] Specifically, in an iteration process, the action revenue corresponding to each candidate multimedia content is calculated, and at least one candidate multimedia content is selected from the candidate multimedia content set according to the action revenue. Specifically, the number of selected candidate multimedia contents is consistent with the number of recommended positions in this round. The above-mentioned is that one recommended position is selected in each round, and one candidate multimedia content is selected, for example, the one with the maximum action revenue is selected and placed in the recommended position in this round.

[0102] It should be noted that the recommended position to be determined in an iteration process can also be two or more. Taking two as an example, the first two with the maximum action revenue can be selected and placed in the recommended position in this round, and so on. The following mainly takes one as an example for illustration.

[0103] In the above embodiment, the structure of the cycle is adopted to predict the long-term action revenue of each candidate multimedia content in the current position, and at least one candidate multimedia content is selected and placed in the recommended position in this round based on the action revenue analysis. Since the action revenue of the multimedia content is considered, the final recommended accuracy is higher.

[0104] In an optional embodiment, the action revenue corresponding to each candidate multimedia content can be determined based on the following method, which specifically includes the following processes:

[0105] First, a first embedding feature vector determined based on content feature information of each multimedia content is obtained. It should be noted that the first embedding feature vector in the embodiment of the application can be in the form of a matrix, for example, the content feature information of each article can be represented by a 1*100 vector. The first embedding feature vector can be represented by a 10*100 feature matrix or a 100*10 feature matrix.

[0106] In addition, a first selected state vector is determined according to the selected multimedia content set, a first candidate state vector is determined according to the candidate multimedia content set, and then action gains corresponding to each candidate multimedia content are calculated based on the three vectors.

[0107] In the embodiments of the present application, the first selected state vector is determined based on each selected multimedia content, and each element in the first selected state vector represents whether the corresponding multimedia content has a determined recommendation order. For example, in the second iteration process, the selected multimedia content id=2, and the corresponding first selected state vector can be represented as [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], where the element 0 in the vector represents that the recommendation order is not determined, and the element 1 represents that the recommendation order is determined. The first candidate state vector is determined based on each candidate multimedia content, and each element in the first candidate state vector represents whether the corresponding multimedia content has a determined recommendation order. For example, in the second iteration process, the candidate multimedia content id=1, 3, 4, 5, 6, 7, 8, 9, 10, and the corresponding first candidate state vector can be represented as [1, 0, 1, 1, 1, 1, 1, 1, 1, 1], where the element 0 in the vector represents that the recommendation order is determined, and the element 1 represents that the recommendation order is not determined.

[0108] After the above vectors are determined, the action gains corresponding to each candidate multimedia content can be calculated. Specifically, first, context feature extraction is performed based on the first embedding feature vector and the first selected state vector to obtain a first context feature vector between each selected multimedia content, where the first context feature vector is used to represent the context association relationship between each selected multimedia content. In addition, candidate feature extraction is performed based on the first embedding feature vector and the first candidate state vector to obtain a first candidate feature vector of each candidate multimedia content combination, where the first candidate feature vector is used to represent the content features of each candidate multimedia content. Then, based on the first context feature vector and the first candidate feature vector, the action gains corresponding to each candidate multimedia content can be determined.

[0109] It should be noted that the above process can also be implemented based on reinforcement learning. In an optional embodiment, the content feature information of each multimedia content is input into a trained rearrangement model, which is a rearrangement model trained based on reinforcement learning, and specifically includes two parts: an embedding part and a prediction part. Next, the structure of the rearrangement model will be described in detail with reference to FIG. 8. Figure 3 The above process will be described in detail. Referring to FIG. 8, it is a structure diagram of a rearrangement model listed in the embodiments of the present application, where the left part of the dashed line is the embedding part, and the right part is the prediction part. Figure 3

[0110] ​First, the input features of the input rearrangement model are X: [1, 10, feaNum], which represent the content feature information of 10 articles. These 10 articles are represented as article 1: item1, article 2: item2, ..., article 10: item10. Based on the embedding part in the rearrangement model, the content feature information of each article is embedded by looking up (Embedding Lookup) to extract features and obtain the first embedding feature vector (Items Embedding); the first embedding feature vector is input into the prediction part of the rearrangement model, and based on the prediction part, the first embedding feature vector is multiplied by the first selected state vector selected1, and based on the self-attention mechanism, the first context feature vector context1 between each selected article is obtained; based on the prediction part, the first embedding feature vector is multiplied by the first candidate state vector retained1 to obtain the first candidate feature vector candidate obtained by combining each candidate article; and then, through the reduce The sum function compresses context1 into a one-dimensional vector representing the context feature context'. It then copies the one-dimensional vector representing the context feature context' into the same shape as the first candidate feature vector candidate (also known as the candidate article matrix) to obtain context", and concatenates context" and candidate to form the first concatenated vector based on the first context feature vector and the first candidate feature vector. The fully connected layer FC extracts features from the first concatenated vector to determine the action gain Gain corresponding to each candidate article. The model outputs the article placed in the recommended position for this round as the candidate article with the maximum action gain, that is, item = argmax(Gain). This cycle continuously determines the article corresponding to each recommended position to ensure maximum gain.

[0111] Among them, the reduce sum function is summation. Adding reduce in the function name indicates that the dimension will be reduced after summation, that is, the compression process. In the embodiment of the present application, it refers to compression into a one-dimensional vector.

[0112] The following example illustrates the process of loop iteration. Figure 4 , Figure 5 and Figure 6wherein Order represents the recommendation order (also referred to as a recommendation position) of each article to be recommended, and Gain represents the action gain corresponding to the article, and the Deep RL Network is a rearrangement model in the embodiments of the present application, and the full name is Deep Reinforcement Learning Network, i.e., a deep reinforcement learning network. The selected represents the first selected state vector, the retained represents the first candidate state vector, and the Item Embeddings represents the first embedding feature vector.

[0113] Referring to Figure 4 As shown in FIG. 6, it is indicated that in the first iteration process, the action gain Gain of the article with id = 3 in each candidate article is determined to be the largest, i.e., the article with id = 3 is determined to be placed in the recommendation position (i.e., the first recommendation position) of the current round, and then the first candidate state vector and the first selected state vector are updated according to the result. As shown in FIG. 7, the third element in the first selected state vector is 1, indicating that the article with id = 3 is a selected article, and the third element in the first candidate state vector is 0, indicating that the remaining articles except the article with id = 3 are candidate articles, and the recommendation order of the articles is not determined. Figure 4 In the first iteration process, the first selected state set is empty, and each element in the corresponding first selected state vector is 1. Each element in the first candidate state vector corresponding to the first candidate state set is 0. After updating, as shown in FIG. 7, the third element in the first selected state vector is 1, indicating that the article with id = 3 is a selected article, and the third element in the first candidate state vector is 0, indicating that the remaining articles except the article with id = 3 are candidate articles, and the recommendation order of the articles is not determined. Figure 5 As shown in FIG. 7, the third element in the first selected state vector is 1, indicating that the article with id = 3 is a selected article, and the third element in the first candidate state vector is 0, indicating that the remaining articles except the article with id = 3 are candidate articles, and the recommendation order of the articles is not determined.

[0114] Figure 5 As shown in FIG. 8, it is indicated that in the second iteration process, the action gain Gain of the article with id = 7 in each candidate article is determined to be the largest, i.e., the article with id = 7 is determined to be placed in the recommendation position (i.e., the second recommendation position) of the current round, and then the first candidate state vector and the first selected state vector are updated again according to the result. As shown in FIG. 9, the third element and the seventh element in the first selected state vector are 1, indicating that the articles with id = 3 and 7 are selected articles, and the recommendation order of the articles is determined. The third element and the seventh element in the first candidate state vector are 0, indicating that the remaining articles except the articles with id = 3 and 7 are candidate articles, and the recommendation order of the articles is not determined. Figure 6 As shown in FIG. 8, it is indicated that in the second iteration process, the action gain Gain of the article with id = 7 in each candidate article is determined to be the largest, i.e., the article with id = 7 is determined to be placed in the recommendation position (i.e., the second recommendation position) of the current round, and then the first candidate state vector and the first selected state vector are updated again according to the result. As shown in FIG. 9, the third element and the seventh element in the first selected state vector are 1, indicating that the articles with id = 3 and 7 are selected articles, and the recommendation order of the articles is determined. The third element and the seventh element in the first candidate state vector are 0, indicating that the remaining articles except the articles with id = 3 and 7 are candidate articles, and the recommendation order of the articles is not determined.

[0115] Figure 6 As shown in FIG. 10, it is indicated that in the third iteration process, the action gain Gain of the article with id = 4 in each candidate article is determined to be the largest, i.e., the article with id = 4 is determined to be placed in the recommendation position (i.e., the third recommendation position) of the current round, and then the first candidate state vector and the first selected state vector are updated again according to the result. In this way, until the candidate article set is empty, the article corresponding to each recommendation position can be determined.

[0116] The rearrangement model is trained based on a training sample data set, and the training samples in the training sample data set include content feature information of sample multimedia content, and a real recommendation order and a real action benefit corresponding to the sample multimedia content.

[0117] In an optional embodiment, the rearrangement model is trained in the following manner:

[0118] The rearrangement model is trained through cyclic iteration based on the training samples in the training sample data set, and the trained rearrangement model is output when the training is completed. The training is completed when the iteration reaches an upper limit or the model converges, which is not limited herein.

[0119] In the embodiments of the present application, the following operations are performed in one cycle of iteration training:

[0120] S71: A group of training samples is selected from the training sample data set;

[0121] S72: The content feature information of each sample multimedia content in the group of training samples, the selected sample multimedia content set and the candidate sample multimedia content set in the current state, and the selected sample multimedia content set and the candidate sample multimedia content set in the next state are input into the rearrangement model, wherein the selected sample multimedia content set and the candidate sample multimedia content set are determined based on the real recommendation order;

[0122] S73: The content feature information of each sample multimedia content is extracted based on the embedding part in the rearrangement model to obtain a second embedding feature vector for the group of training samples;

[0123] S74: The second embedding feature vector is input into the prediction part in the rearrangement model, and the first estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the current state and the second estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state are predicted based on the prediction part;

[0124] S75: A loss function is constructed based on the first estimated action benefit, the second estimated action benefit, and the corresponding actual action benefit, and the parameters of the rearrangement model are adjusted based on the constructed loss function.

[0125] Referring to Figure 8 The structure of another rearrangement model in the embodiments of the present application is shown in FIG. 2, wherein, Figure 8The prediction part in the rearrangement model shown includes a first prediction subnetwork and a second prediction subnetwork; wherein the first prediction subnetwork is used for predicting the first estimated action gains of the candidate sample multimedia contents in the current state, and the second prediction subnetwork is used for predicting the second estimated action gains of the candidate sample multimedia contents in the next state.

[0126] When the online model is applied, only the first prediction subnetwork corresponding to the current state is selected for calculation. When initialized, the selected article set is empty, and the candidate article set is [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. The first estimated action gains Gain_i of each candidate article in the current state are calculated, the article corresponding to the maximum Gain_i is selected as the current action, that is, the article corresponding to the maximum Gain_i is added to the recommended position in this round, and then the selected article set and the candidate article set are updated. The above process is repeatedly performed until the candidate article set is empty.

[0127] When the offline model is trained, as shown in the figure, Figure 8 the input features of the rearrangement model are [1, 10, FeatureNum], indicating the content feature information of 10 sample articles. The input features are subjected to embedding lookup based on the embedding part to obtain corresponding second embedding feature vectors (Items Embedding). Then, the estimated action gains are calculated based on the prediction part on the right, including the first estimated action gains of each candidate article in the current state and the second estimated action gains of each candidate article in the next state.

[0128] An optional implementation is that when step S84 is performed, the following steps are specifically included, referring to Figure 9 the figure:

[0129] S91: based on the first prediction subnetwork, the second embedding feature vector is multiplied by the second selected state vector, and then based on the attention mechanism, the second context feature vector between each selected sample multimedia content in the selected sample multimedia content set in the current state is obtained;

[0130] wherein the second context feature vector is context2 in Figure 8 , which is used to represent the context association relationship between each selected sample multimedia content in the current state; and the second selected state vector is selected2 in Figure 8 , and each element in selected2 represents whether the recommendation order of each multimedia content in the corresponding group of training samples in the current state has been determined, and the specific representation is the same as that of the first selected state vector, which is not limited here.

[0131] S92: Obtain a second candidate feature vector obtained by multiplying the second embedding feature vector and the second candidate state vector based on the first predictor network;

[0132] The second candidate feature vector is candidate2 in the second candidate state vector, and candidate2 is used to represent the content features of the candidate sample multimedia content in the current state. Figure 8 Figure 8 The second candidate state vector is retained2 in the second candidate state vector, and each element in retained2 represents whether the multimedia content in the corresponding set of training samples in the current state is not determined to have a recommended order, and the specific representation is the same as that of the first candidate state vector, which is not limited here.

[0133] S93: Through the full connection layer in the first predictor network, the second splicing vector obtained by splicing the second context feature vector and the second candidate feature vector is extracted to determine the first estimated action gain corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the current state.

[0134] In step S93, the second splicing vector obtained by splicing the second context feature vector context2 and the second candidate feature vector candidate2 is obtained in a similar manner to the first splicing vector. First, the second context feature vector context2 needs to be compressed into a one-dimensional vector representing the context feature context2', and further, the compressed context feature context2 is copied into the same shape as the second candidate feature vector context2", and context2" and candidate2 are spliced together to obtain the second splicing vector. Based on the second splicing vector, the feature is extracted through the full connection layer FC shown in FIG. 8, and the first estimated action gain Gain corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the current state is determined. Figure 8

[0135] The above steps S91 to S93 are the processing process of the first predictor network side, and the following steps S94 to S96 are the processing process of the second predictor network side. The two predictor networks can be processed in parallel, and the timing is not limited here.

[0136] S94: Based on the second predictor network, multiply the second embedding feature vector and the third selected state vector, and obtain the third context feature vector between each selected sample multimedia content in the selected sample multimedia content set in the next state based on the attention mechanism;

[0137] The third context feature vector is context3 in the third context feature vector, and context3 is used to represent the context features of the selected sample multimedia content in the next state. Figure 8 ​​context3 in the third context feature vector, used to represent the context relationship between each selected sample multimedia content in the next state; the third selected state vector, i.e. Figure 8 selected3 in the third selected state vector, each element in selected3 respectively represents whether the multimedia content in the corresponding group of training samples in the next state has determined the recommended order, and the specific representation is the same as that of the first selected state vector, which is not specifically limited here.

[0138] S95: based on the second prediction sub-network, obtaining a third candidate feature vector obtained by multiplying the third embedding feature vector and the third candidate state vector;

[0139] wherein the third candidate feature vector, i.e. Figure 8 candidate2 in the third candidate feature vector, used to represent the content features of each candidate sample multimedia content in the next state; the third candidate state vector, i.e. Figure 8 retained2 in the third candidate state vector, each element in retained2 respectively represents whether the multimedia content in the corresponding group of training samples in the next state has determined the recommended order, and the specific representation is the same as that of the first candidate state vector, which is not specifically limited here.

[0140] S96: through the fully connected layer in the second prediction sub-network, performing feature extraction on the third splicing vector obtained based on the splicing of the third context feature vector and the third candidate feature vector, to determine the second estimated action gain corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state.

[0141] Similarly, in step S96, the third splicing vector is also obtained in a similar manner. The third splicing vector obtained based on the splicing of the third context feature vector context3 and the third candidate feature vector candidate3 is also similar to the first splicing vector and the second splicing vector. First, the third context feature vector context3 needs to be compressed into a one-dimensional vector representing the context feature context3', and then the compressed context feature context3 is copied into the same shape as the third candidate feature vector context3", and context3" and candidate3 are spliced together to obtain the third splicing vector. Based on the third splicing vector, feature extraction is performed through the fully connected layer FC shown in FIG. 8, so as to determine the second estimated action gain NextGain corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state. Figure 8

[0142] ​In an online recommendation system, the recommendation system recommends 10 articles to a user at a time, and the user may not see all the articles in the recommendation result at once. The actual number of articles displayed is in the range of [1, 10]. For an actual article sequence displayed at a time, any article in the sequence is selected as the action, the articles displayed before the selected article are the selected articles, and the articles displayed after the selected article are the candidate articles. The selected articles and the candidate articles can represent the current state. The selected article is also the action in the sample. After the action is taken, the selected article is added to the selected article list, and the selected article is removed from the candidate article list to represent the next state. The action reward can be evaluated in multiple ways, such as whether the article represented by the action is clicked or the user consumption time of the article represented by the action.

[0143] In an optional implementation, the process of constructing the selected sample multimedia content set and the candidate sample multimedia content set includes the following operations:

[0144] Any one training sample is selected from a set of training samples. Based on each sample multimedia content before the sample multimedia content in the any one training sample, a selected sample multimedia content set in the current state is constructed, and based on the sample multimedia content in the remaining training samples, a candidate sample multimedia content set in the current state is constructed. Based on the sample multimedia content in the any one training sample and each sample multimedia content before the sample multimedia content, a selected sample multimedia content set in the next state is constructed, and based on the sample multimedia content in the remaining training samples, a candidate sample multimedia content set in the next state is constructed.

[0145] For example, the article sequence recommended at a time is represented by 10 ids, the recommended article list is [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], and the actual display list (actual recommendation order) is [1, 2, 3, 4, 5, 6]. The training sample is as follows:

[0146] Suppose Action represents selecting the 4th article as the action, action = 4. According to the actual display list, there are 3 articles before the article, so in the current state: the selected article list is [1, 2, 3]; the candidate article list is [4, 5, 6, 7, 8, 9, 10]. In the next state: the selected article list is [1, 2, 3, 4]; the candidate article list is [5, 6, 7, 8, 9, 10]. The action reward reward represents whether the 4th article is clicked (or the user consumption time of the 4th article, etc.).

[0147] In Figure 8The representation of the input article in the rearrangement model shown is that each article is composed of multiple features, each feature is represented by a low-dimensional dense vector, and the representation of the article vector has multiple methods, which can be the sum of multiple low-dimensional dense vectors, the mean of multiple low-dimensional dense vectors, or the splicing of multiple low-dimensional dense vectors, which is not specifically limited here.

[0148] In the embodiments of the present application, the network structures of the prediction sub-networks corresponding to the current state and the next state can be completely the same or different. Figure 10 The same is exemplified as an example: first, according to the Embedding vector matrix of the ten recommended articles and the two lists representing the state, the matrix multiplication is calculated to cover the unused vectors; then the context relationship of the articles in the selected article matrix is calculated through self_attention, and a one-dimensional vector representing the context feature is compressed, then the context feature vector is copied into the same shape as the candidate article matrix and spliced together, and then the action reward of each candidate article is estimated through the full connection layer. In the end, the output of the first prediction sub-network (current state network) is the action reward Gain_i of the article represented by action selected from the first estimated action reward of each candidate sample multimedia content. For example, the output Gain_4 in the above embodiment. The output of the second prediction sub-network (next state network) is the maximum article action reward max(NextGain) selected from the second estimated action reward of each candidate sample multimedia content. For example, the NextGain corresponding to the 8th article is the largest, and the second estimated action reward corresponding to the 8th article is output.

[0149] The process of adjusting the model parameters is described in detail below:

[0150] An optional implementation is to construct a loss function based on the first estimated action reward, the second estimated action reward, and the actual action reward in a set of training samples, and the specific process is as follows:

[0151] Based on the difference between the target estimated action reward in each first estimated action reward and the maximum estimated action reward in each second estimated action reward, and the corresponding real action reward, a loss function is constructed, wherein the target estimated action reward is the first estimated action reward corresponding to the sample multimedia content in the selected training sample, i.e. Gain_i, and in the above embodiment, i=4.

[0152] Based on the above description, the calculation formula of the loss function is as follows:

[0153] Loss=square(γ*max(NextGain)+Reward_i-Gain_i).

[0154] wherein, γ = 1 represents that the subsequent reward is as important as the current reward; γ < 1 represents subsequent decay, reducing the weight of the subsequent reward. In the embodiments of the present application, γ takes a value between 0 and 1, which is not specifically limited herein. Reward_i represents the immediate reward obtained by the candidate multimedia content i at the current position, which can be represented as the difference between max(NextGain) and Gain_i, or other calculation methods, etc. The optimization method in the embodiments of the present application can use common Stochastic Gradient Descent (SGD) or adaptive moment estimation (ADAM) optimization methods.

[0155] In addition, it should be noted that for the first prediction sub-network and the second prediction sub-network, when the parameters of the two networks are the same, the networks directly share the parameters with each other; of course, the two sets of parameters can also be completely different, and the parameters are optimized by gradient back propagation respectively; or the parameters can be shared, the network parameters of the next state are updated with a delay, the model parameters of the current state are adjusted during training, the parameters of the current state network are copied to the next state network every certain period of time, etc., which are not specifically limited herein.

[0156] Based on the same inventive concept, the embodiments of the present application also provide a content recommendation device. As shown in Figure 1 The content recommendation device 1000 can include:

[0157] An information acquisition unit 1001 is configured to acquire content feature information of each multimedia content in a set of multimedia contents to be recommended;

[0158] A recommendation unit 1002 is configured to determine a recommendation order of each multimedia content in a loop iteration manner based on the content feature information of each multimedia content; wherein, in a round of iteration, the following operations are performed:

[0159] determining a recommended position in the current round based on the set of selected multimedia contents, wherein the set of selected multimedia contents includes each selected multimedia content in the order, and the recommended position in the current round is a new ordered position other than the position of each selected multimedia content in the order;

[0160] determining each candidate multimedia content in the set of candidate multimedia contents which has not been ordered;

[0161] Based on the content feature information of each multimedia content, and the selected multimedia content set and the candidate multimedia content set, an action benefit corresponding to each candidate multimedia content is obtained, and at least one candidate multimedia content is selected from the candidate multimedia content set according to the action benefit and put into the current round of recommendation position, wherein the action benefit is used to represent the probability of the object performing the target action on the multimedia content.

[0162] Optionally, the recommendation unit 1002 is specifically configured to:

[0163] Obtain a first embedding feature vector determined based on the content feature information of each multimedia content;

[0164] Perform context feature extraction based on the first embedding feature vector and a first selected state vector to obtain a first context feature vector between each selected multimedia content, wherein the first context feature vector is used to represent the context association relationship between each selected multimedia content; the first selected state vector is determined based on each selected multimedia content, and each element in the first selected state vector represents whether the corresponding multimedia content has determined the recommendation order; and

[0165] Perform candidate feature extraction based on the first embedding feature vector and a first candidate state vector to obtain a first candidate feature vector obtained by each candidate multimedia content combination, wherein the first candidate feature vector is used to represent the content features of each candidate multimedia content; the first candidate state vector is determined based on each candidate multimedia content, and each element in the first candidate state vector represents whether the corresponding multimedia content has determined the recommendation order;

[0166] Determine an action benefit corresponding to each candidate multimedia content based on the first context feature vector and the first candidate feature vector.

[0167] Optionally, the recommendation unit 1002 is specifically configured to:

[0168] Input the content feature information of each multimedia content into the trained rearrangement model;

[0169] Perform feature extraction on the content feature information of each multimedia content based on the embedding part in the rearrangement model to obtain a first embedding feature vector;

[0170] The rearrangement model is trained based on a training sample data set, and the training sample in the training sample data set includes content feature information of a sample multimedia content, and a real recommendation order and a real action benefit corresponding to the sample multimedia content.

[0171] Optionally, the rearrangement model further includes a prediction part; and the recommendation unit 1002 is specifically configured to:

[0172] inputting the first embedding feature vector into a prediction part in the rearrangement model, multiplying the first embedding feature vector with a first selected state vector based on the prediction part to obtain a first context feature vector between each selected multimedia content;

[0173] performing feature extraction on the first embedding feature vector and the first candidate state vector to obtain a first candidate feature vector of each candidate multimedia content combination, specifically including:

[0174] multiplying the first embedding feature vector with the first candidate state vector based on the prediction part to obtain the first candidate feature vector of each candidate multimedia content combination;

[0175] determining action benefits corresponding to each candidate multimedia content based on the first context feature vector and the first candidate feature vector, specifically including:

[0176] performing feature extraction on a first splicing vector obtained by splicing the first context feature vector and the first candidate feature vector based on a full connection layer to determine the action benefits corresponding to each candidate multimedia content.

[0177] Optionally, the apparatus further includes:

[0178] a model training unit 1003 configured to perform cyclic iteration training on the rearrangement model according to training samples in a training sample data set, and output the trained rearrangement model when the training is completed; wherein the following operations are performed in one cyclic iteration training process:

[0179] selecting a group of training samples from the training sample data set;

[0180] inputting content feature information of each sample multimedia content in the group of training samples, and a selected sample multimedia content set and a candidate sample multimedia content set in a current state, and a selected sample multimedia content set and a candidate sample multimedia content set in a next state into the rearrangement model, wherein the selected sample multimedia content set and the candidate sample multimedia content set are determined based on a real recommendation order;

[0181] performing feature extraction on the content feature information of each sample multimedia content based on an embedding part in the rearrangement model to obtain a second embedding feature vector for the group of training samples;

[0182] inputting the second embedding feature vector into a prediction part in the rearrangement model, and predicting, based on the prediction part, a first estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the current state, and a second estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state;

[0183] construct a loss function based on the first estimated action reward, the second estimated action reward, and the corresponding actual action reward, and adjust parameters of the rearrangement model based on the constructed loss function.

[0184] Optionally, the model training unit 1003 is further configured to:

[0185] select any one training sample from the set of training samples;

[0186] construct, according to the real recommendation order, a set of selected sample multimedia contents in the current state based on each sample multimedia content before the sample multimedia content in the any one training sample, and construct, based on the sample multimedia content in the remaining training sample, a set of candidate sample multimedia contents in the current state; and

[0187] construct, according to the real recommendation order, a set of selected sample multimedia contents in the next state based on the sample multimedia content in the any one training sample and each sample multimedia content before the sample multimedia content, and construct, based on the sample multimedia content in the remaining training sample, a set of candidate sample multimedia contents in the next state.

[0188] Optionally, the prediction part includes a first prediction subnetwork and a second prediction subnetwork; and the model training unit 1003 is specifically configured to:

[0189] based on the first prediction subnetwork, multiply the second embedding feature vector and the second selected state vector to obtain, based on an attention mechanism, a second context feature vector between each selected sample multimedia content in the set of selected sample multimedia contents in the current state, wherein the second context feature vector is used to represent a context association relationship between each selected sample multimedia content in the current state; and each element in the second selected state vector represents whether the multimedia content in the corresponding set of training samples in the current state has determined the recommendation order;

[0190] based on the first prediction subnetwork, obtain a second candidate feature vector obtained by multiplying the second embedding feature vector and a second candidate state vector, wherein the second candidate feature vector is used to represent the content features of each candidate sample multimedia content in the current state; and each element in the second candidate state vector represents whether the multimedia content in the corresponding set of training samples in the current state has determined the recommendation order;

[0191] perform feature extraction on a second splicing vector obtained by splicing the second context feature vector and the second candidate feature vector through a fully connected layer in the first prediction subnetwork to determine the first estimated action reward corresponding to each candidate sample multimedia content in the set of candidate sample multimedia contents in the current state; and

[0192] The second embedding feature vector is multiplied with the third selected state vector based on the second predictor network, and a third context feature vector between each selected sample multimedia content in the selected sample multimedia content set in the next state is obtained based on an attention mechanism, where the third context feature vector is used to represent the context correlation between the selected sample multimedia contents in the next state; each element in the third selected state vector respectively represents whether the multimedia content in the corresponding group of training samples in the next state has a determined recommendation order;

[0193] The third embedding feature vector is multiplied with the third candidate state vector based on the second predictor network to obtain a third candidate feature vector, where the third candidate feature vector is used to represent the content features of the candidate sample multimedia contents in the next state; each element in the third candidate state vector respectively represents whether the multimedia content in the corresponding group of training samples in the next state has a determined recommendation order.

[0194] The third splicing vector obtained by splicing the third context feature vector and the third candidate feature vector is subjected to feature extraction through the full connection layer in the second predictor network to determine the second estimated action revenue corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state.

[0195] Optionally, the model training unit 1003 is specifically configured to:

[0196] A loss function is constructed based on the difference between the target estimated action revenue in each first estimated action revenue and the maximum estimated action revenue in each second estimated action revenue, and the corresponding real action revenue, where the target estimated action revenue is the first estimated action revenue corresponding to the sample multimedia content in any one training sample.

[0197] For the sake of brevity, each part is described as a module (or unit) according to function. Of course, the functions of each module (or unit) can be implemented in the same or multiple software or hardware in the implementation of the present application.

[0198] After introducing the content recommendation method and device of the example embodiment of the present application, next, the content recommendation device according to another example embodiment of the present application is introduced.

[0199] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combined with hardware and software, which can be collectively referred to as "circuit", "module" or "system" here.

[0200] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. The electronic device can be used for recommending multimedia content. In one embodiment, the electronic device can be a server, such as Figure 11 In this embodiment, the structure of the electronic device can be as follows: Figure 11 As shown, it includes a memory 1101 , a communication module 1103 and one or more processors 1102 .

[0201] Memory 1101 is used to store computer programs executed by processor 1102. Memory 1101 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0202] Memory 1101 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1101 may be a combination of the aforementioned memories.

[0203] The processor 1102 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1102 is configured to implement the above-mentioned content recommendation method when calling the computer program stored in the memory 1101 .

[0204] The communication module 1103 is used to communicate with terminal devices and other servers.

[0205] The specific connection medium between the memory 1101, the communication module 1103 and the processor 1102 is not limited in the embodiment of the present application. Figure 11 In the embodiment, the memory 1101 and the processor 1102 are connected via a bus 1104. Figure 11 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus 1104 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration,Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0206] The memory 1101 stores a computer storage medium, which stores computer executable instructions. The computer executable instructions are used to implement the content recommendation method of the embodiment of the present application. The processor 1102 is used to execute the above content recommendation method, such as Figure 1 shown.

[0207] In another embodiment, the electronic device may also be other electronic devices, such as Figure 12 The terminal device 110 shown in FIG. In this embodiment, the structure of the electronic device can be as follows: Figure 2 As shown, it includes: a communication component 1210, a memory 1220, a display unit 1230, a camera 1240, a sensor 1250, an audio circuit 1260, a Bluetooth module 1270, a processor 1280 and other components.

[0208] The communication component 1210 is used to communicate with the server. In some embodiments, it can include a WiFi (Wireless Fidelity) module. The WiFi module is a short-range wireless transmission technology. Electronic devices can help users send and receive information through the WiFi module.

[0209] The memory 1220 can be used to store software programs and data. The processor 1280 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1220. The memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1220 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 1220 can store an operating system and various application programs, and may also store code for executing the content recommendation method of the embodiment of the present application.

[0210] The display unit 1230 can also be used to display information input by the user or provided to the user, as well as a graphical user interface (GUI) for displaying various menus of the terminal device 110. Specifically, the display unit 1230 may include a display screen 1232 disposed on the front of the terminal device 110. The display screen 1232 may be configured in the form of a liquid crystal display, a light-emitting diode, etc. The display unit 1230 can be used to display the video images of the video client in the embodiments of the present application.

[0211] The display unit 1230 can also be used to receive inputted digital or character information, generate signal input related to user settings and function control of the terminal device 110, and specifically, the display unit 1230 can include a touch screen 1231 arranged on the front of the terminal device 110, which can collect touch operations of a user thereon or therearound, such as clicking a button, dragging a scroll box, and the like.

[0212] The touch screen 1231 can be overlaid on the display screen 1232, or the touch screen 1231 can be integrated with the display screen 1232 to realize the input and output functions of the terminal device 110, and after integration, the touch screen 1231 and the display screen 1232 can be simply referred to as a touch display screen. The display unit 1230 in the present application can display application programs and corresponding operation steps.

[0213] The camera 1240 can be used to capture still images, and a user can upload comments on images captured by the camera 1240 through a video client. The camera 1240 can be one or multiple. An object generates an optical image through a lens and projects the optical image to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 1280 to convert it into a digital image signal.

[0214] The terminal device can also include at least one sensor 1250, such as an acceleration sensor 1251, a distance sensor 1252, a fingerprint sensor 1253, and a temperature sensor 1254. The terminal device can also be configured with a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, a motion sensor, and other sensors.

[0215] The audio circuit 1260, the speaker 1261, and the microphone 1262 can provide an audio interface between a user and the terminal device 110. The audio circuit 1260 can convert received audio data into an electrical signal and transmit it to the speaker 1261, which converts it into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1262 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1260 and converted into audio data, which is then output to the communication component 1210 for transmission to another terminal device 110, for example, or to the storage 1220 for further processing.

[0216] The Bluetooth module 1270 is used to exchange information with other Bluetooth devices having a Bluetooth module through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1270 to exchange data.

[0217] The processor 1280 is the control center of the terminal device, which uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1220 and calling data stored in the memory 1220. In some embodiments, the processor 1280 may include one or more processing units; the processor 1280 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1280. In the present application, the processor 1280 can run the operating system, application programs, user interface display and touch response, as well as the content recommendation method of the embodiment of the present application. In addition, the processor 1280 is coupled to the display unit 1230.

[0218] In some possible implementations, various aspects of the content recommendation method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the content recommendation method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: ​ Follow the steps shown in .

[0219] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0220] The program product of the embodiments of the present application can adopt a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a computing device. However, the program product of the present application is not limited to this, and in the present document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, device or apparatus.

[0221] The readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, in which a readable program code is borne. Such a propagated data signal can take on various forms, including but not limited to electro-magnetic signal, optical signal or any suitable combination thereof. The readable signal medium can also be any readable medium that can send, propagate or transfer a program for use by or in connection with a command execution system, device or apparatus, other than the readable storage medium.

[0222] The program code contained in the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0223] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by relevant hardware instructed by programs, and the aforementioned programs can be stored in a computer readable storage medium, and when the programs are executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes mobile storage equipment, read-only memory (ROM), random access memory (RAM), magnetic disc or optical disc and various media that can store program codes.

[0224] Alternatively, the integrated units of the embodiments of the present application, if realized in the form of software function modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, and include several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes mobile storage equipment, ROM, RAM, magnetic disc or optical disc and various media that can store program codes.

[0225] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the spirit and scope of the application.

[0226] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A content recommendation method, characterized in that: The method includes: Based on the content feature information of each multimedia content in the multimedia content set to be recommended, a recommendation order of each multimedia content is determined in a cyclic iteration manner; wherein the following operations are performed during one round of iteration: determining a recommendation position for this round based on a selected multimedia content set, wherein the selected multimedia content set includes sorted selected multimedia content, and the recommendation position for this round is a new sorted position other than the positions of the currently sorted selected multimedia content; determining, in the candidate multimedia content set, individual candidate multimedia contents that have not yet been sorted; calculating, based on a first embedded feature vector determined based on content feature information of each multimedia content, a first selected state vector determined based on the set of selected multimedia content, and a first candidate state vector determined based on the set of candidate multimedia content, an action benefit corresponding to each candidate multimedia content; each element in the first selected state vector indicates whether a recommendation order has been determined for the corresponding multimedia content; and each element in the first candidate state vector indicates whether a recommendation order has not been determined for the corresponding multimedia content; At least one candidate multimedia content is selected from the candidate multimedia content set according to the action benefit and placed in the recommendation position of this round; the action benefit represents the probability that the object performs a target action on the multimedia content.

2. The method according to claim 1, wherein The calculating, based on the first embedded feature vector determined based on the content feature information of each multimedia content, the first selected state vector determined based on the selected multimedia content set, and the first candidate state vector determined based on the candidate multimedia content set, of the action benefits corresponding to each candidate multimedia content specifically includes: Obtaining a first embedded feature vector determined based on content feature information of each multimedia content, a first selected state vector determined based on the selected multimedia content set, and a candidate state vector determined based on the candidate multimedia content set; Performing context feature extraction based on the first embedded feature vector and the first selected state vector to obtain a first context feature vector between the selected multimedia contents, wherein the first context feature vector is used to represent the context association relationship between the selected multimedia contents; and performing candidate feature extraction based on the first embedded feature vector and the first candidate state vector to obtain a first candidate feature vector obtained by combining the candidate multimedia contents, wherein the first candidate feature vector is used to characterize content features of the candidate multimedia contents; Based on the first context feature vector and the first candidate feature vector, action benefits corresponding to the respective candidate multimedia contents are determined.

3. The method according to claim 2, wherein The obtaining of the first embedded feature vector determined based on the content feature information of each multimedia content specifically includes: Inputting the content feature information of each multimedia content into the trained re-ranking model; Based on the embedding part in the rearrangement model, feature extraction is performed on content feature information of each multimedia content to obtain the first embedding feature vector; The re-ranking model is trained based on a training sample data set, wherein the training samples in the training sample data set include content feature information of sample multimedia content, as well as the real recommendation order and real action benefits corresponding to the sample multimedia content.

4. The method according to claim 3, wherein The rearrangement model further includes a prediction part; the context feature extraction based on the first embedded feature vector and the first selected state vector to obtain the first context feature vector between the selected multimedia contents specifically includes: inputting the first embedded feature vector into a prediction part of the rearrangement model, and multiplying the first embedded feature vector by a first selected state vector based on the prediction part to obtain a first context feature vector between the selected multimedia contents; The performing feature extraction based on the first embedded feature vector and the first candidate state vector to obtain the first candidate feature vector obtained by combining the candidate multimedia contents specifically includes: multiplying the first embedded feature vector and the first candidate state vector based on the prediction part to obtain a first candidate feature vector obtained by combining the candidate multimedia contents; The determining, based on the first context feature vector and the first candidate feature vector, the action benefits corresponding to the respective candidate multimedia contents specifically includes: A first splicing vector obtained by splicing the first context feature vector and the first candidate feature vector is subjected to feature extraction through a fully connected layer to determine the action benefits corresponding to each candidate multimedia content.

5. The method according to claim 3 or 4, wherein: The rearrangement model is trained in the following way: Performing cyclic iterative training on the rearrangement model according to the training samples in the training sample data set, and outputting the trained rearrangement model after the training is completed; wherein the following operations are performed during one cyclic iterative training process: Selecting a set of training samples from the training sample data set; Inputting content feature information of each sample multimedia content in the set of training samples, as well as a selected sample multimedia content set and a candidate sample multimedia content set in a current state, and a selected sample multimedia content set and a candidate sample multimedia content set in a next state, into a re-ranking model, wherein the selected sample multimedia content set and the candidate sample multimedia content set are both determined based on the true recommendation order; Based on the embedding part in the rearrangement model, feature extraction is performed on content feature information of each sample multimedia content to obtain a second embedding feature vector for the set of training samples; Inputting the second embedded feature vector into a prediction part of the rearrangement model, and predicting, based on the prediction part, a first estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in a current state, and a second estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in a next state; A loss function is constructed based on the first estimated action benefit, the second estimated action benefit, and the corresponding actual action benefit, and parameters of the rearrangement model are adjusted based on the constructed loss function.

6. The method according to claim 5, wherein The process of constructing the selected sample multimedia content set and the candidate sample multimedia content set includes the following operations: Selecting any one training sample from the set of training samples; Constructing, according to the true recommendation order, a set of selected sample multimedia contents in the current state based on each sample multimedia content preceding the sample multimedia content in any one of the training samples, and constructing a set of candidate sample multimedia contents in the current state based on the sample multimedia content in the remaining training samples; as well as According to the true recommendation order, based on the sample multimedia content in any one of the training samples and the sample multimedia content before the sample multimedia content, the selected sample multimedia content set in the next state is constructed, and based on the sample multimedia content in the remaining training samples, the candidate sample multimedia content set in the next state is constructed.

7. The method according to claim 5, wherein The prediction part includes a first prediction subnetwork and a second prediction subnetwork; the prediction based on the prediction part obtains a first estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the current state, and a second estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state, specifically including: Based on the first prediction subnetwork, after multiplying the second embedded feature vector by the second selected state vector, a second context feature vector between each selected sample multimedia content in the set of selected sample multimedia content in the current state is obtained based on the attention mechanism, wherein the second context feature vector is used to characterize the contextual association relationship between each selected sample multimedia content in the current state; each element in the second selected state vector respectively indicates whether the multimedia content in the corresponding set of training samples in the current state has been determined in a recommendation order; Based on the first prediction subnetwork, obtaining a second candidate feature vector obtained by multiplying the second embedded feature vector and the second candidate state vector, wherein the second candidate feature vector is used to represent content features of each candidate sample multimedia content in the current state; each element in the second candidate state vector respectively indicates whether the multimedia content in the corresponding set of training samples in the current state has not yet determined a recommendation order; performing feature extraction on a second spliced ​​vector obtained by splicing the second context feature vector and the second candidate feature vector through a fully connected layer in the first prediction subnetwork, and determining a first estimated action benefit corresponding to each candidate sample multimedia content in the set of candidate sample multimedia content in the current state; and Based on the second prediction subnetwork, after multiplying the second embedded feature vector by the third selected state vector, a third context feature vector between each selected sample multimedia content in the set of selected sample multimedia content in the next state is obtained based on the attention mechanism, wherein the third context feature vector is used to characterize the contextual association relationship between each selected sample multimedia content in the next state; each element in the third selected state vector respectively indicates whether the multimedia content in the corresponding set of training samples in the next state has determined a recommendation order; Based on the second prediction subnetwork, obtaining a third candidate feature vector obtained by multiplying the third embedded feature vector and the third candidate state vector, wherein the third candidate feature vector is used to represent content features of each candidate sample multimedia content in the next state; each element in the third candidate state vector respectively indicates whether the multimedia content in the set of training samples corresponding to the next state has not yet determined a recommendation order; Through the fully connected layer in the second prediction subnetwork, feature extraction is performed on a third spliced ​​vector obtained by splicing the third context feature vector and the third candidate feature vector, and a second estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in the next state is determined.

8. The method according to claim 6, wherein The constructing of a loss function based on the first estimated action benefit, the second estimated action benefit, and the actual action benefits in the set of training samples specifically includes: A loss function is constructed based on the difference between the target estimated action benefit in each first estimated action benefit, the maximum estimated action benefit in each second estimated action benefit, and the corresponding true action benefit, wherein the target estimated action benefit is the first estimated action benefit corresponding to the sample multimedia content in any one of the training samples.

9. A content recommendation device, characterized in that: include: The recommendation unit is configured to determine the recommendation order of each multimedia content in the set of multimedia content to be recommended in a cyclic iterative manner based on the content feature information of each multimedia content in the set of multimedia content to be recommended; wherein the following operations are performed during one round of iteration: determining a recommendation position for this round based on a selected multimedia content set, wherein the selected multimedia content set includes sorted selected multimedia content, and the recommendation position for this round is a new sorted position other than the positions of the currently sorted selected multimedia content; determining, in the candidate multimedia content set, individual candidate multimedia contents that have not yet been sorted; calculating, based on a first embedded feature vector determined based on content feature information of each multimedia content, a first selected state vector determined based on the set of selected multimedia content, and a first candidate state vector determined based on the set of candidate multimedia content, an action benefit corresponding to each candidate multimedia content; each element in the first selected state vector indicates whether a recommendation order has been determined for the corresponding multimedia content; and each element in the first candidate state vector indicates whether a recommendation order has not been determined for the corresponding multimedia content; selecting at least one candidate multimedia content from the candidate multimedia content set according to the action benefit, and placing the candidate multimedia content into the recommendation position of this round; The action benefit represents the probability that the object performs a target action on the multimedia content.

10. The device according to claim 9, wherein The recommendation unit is specifically used for: Obtaining a first embedded feature vector determined based on content feature information of each multimedia content; Context features are extracted based on the first embedded feature vector and the first selected state vector to obtain a first context feature vector between the selected multimedia contents, wherein the first context feature vector is used to represent the contextual association relationship between the selected multimedia contents; the first selected state vector is determined based on the selected multimedia contents, and each element in the first selected state vector indicates whether a recommendation order has been determined for the corresponding multimedia contents; and performing candidate feature extraction based on the first embedded feature vector and the first candidate state vector to obtain a first candidate feature vector obtained by combining the candidate multimedia contents, wherein the first candidate feature vector is used to characterize content features of the candidate multimedia contents; the first candidate state vector is determined based on the candidate multimedia contents, and each element in the first candidate state vector indicates whether a recommendation order of the corresponding multimedia content has not been determined; Based on the first context feature vector and the first candidate feature vector, action benefits corresponding to the respective candidate multimedia contents are determined.

11. The device according to claim 10, wherein The recommendation unit is specifically used for: Inputting the content feature information of each multimedia content into the trained re-ranking model; Based on the embedding part in the rearrangement model, feature extraction is performed on content feature information of each multimedia content to obtain the first embedding feature vector; The re-ranking model is trained based on a training sample data set, wherein the training samples in the training sample data set include content feature information of sample multimedia content, as well as the real recommendation order and real action benefits corresponding to the sample multimedia content.

12. The device according to claim 11, wherein The rearrangement model also includes a prediction part; the recommendation unit is specifically used to: inputting the first embedded feature vector into a prediction part of the rearrangement model, and multiplying the first embedded feature vector by a first selected state vector based on the prediction part to obtain a first context feature vector between the selected multimedia contents; The performing feature extraction based on the first embedded feature vector and the first candidate state vector to obtain the first candidate feature vector obtained by combining the candidate multimedia contents specifically includes: multiplying the first embedded feature vector and the first candidate state vector based on the prediction part to obtain a first candidate feature vector obtained by combining the candidate multimedia contents; The determining, based on the first context feature vector and the first candidate feature vector, the action benefits corresponding to the respective candidate multimedia contents specifically includes: A first splicing vector obtained by splicing the first context feature vector and the first candidate feature vector is subjected to feature extraction through a fully connected layer to determine the action benefits corresponding to each candidate multimedia content.

13. The device according to claim 11 or 12, characterized in that The device further comprises: The model training unit is configured to perform cyclic iterative training on the rearrangement model according to the training samples in the training sample data set, and output the trained rearrangement model when the training is completed; wherein the following operations are performed during one cyclic iterative training process: Selecting a set of training samples from the training sample data set; Inputting content feature information of each sample multimedia content in the set of training samples, as well as a selected sample multimedia content set and a candidate sample multimedia content set in a current state, and a selected sample multimedia content set and a candidate sample multimedia content set in a next state, into a re-ranking model, wherein the selected sample multimedia content set and the candidate sample multimedia content set are both determined based on the true recommendation order; Based on the embedding part in the rearrangement model, feature extraction is performed on content feature information of each sample multimedia content to obtain a second embedding feature vector for the set of training samples; Inputting the second embedded feature vector into a prediction part of the rearrangement model, and predicting, based on the prediction part, a first estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in a current state, and a second estimated action benefit corresponding to each candidate sample multimedia content in the candidate sample multimedia content set in a next state; A loss function is constructed based on the first estimated action benefit, the second estimated action benefit, and the corresponding actual action benefit, and parameters of the rearrangement model are adjusted based on the constructed loss function.

14. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 8.

15. A computer-readable storage medium, characterized in that It includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of any one of the methods described in claims 1 to 8.

Images