Multimedia resource recommendation method, device, storage medium and electronic device

Abstract

The embodiments of the present application provide a method, device, storage medium and electronic device for recommending multimedia resources, wherein the method includes: predicting the first account characteristics of the target account and the reference characteristics of each multimedia resource included in the first browsing information based on the first browsing information; predicting the second account characteristics of the target account based on the second browsing information and the reference characteristics; predicting the target browsing characteristics of the target account based on the first account characteristics and the second account characteristics; and recommending target multimedia resources that match the target browsing characteristics to the target account. Through this application, the problem of low matching between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources is solved, thereby achieving the effect of improving the matching between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia resource recommendation, and in particular, to a method, device, storage medium and electronic device for recommending multimedia resources. Background Art

[0002] As the amount of information on the Internet grows explosively, users have more and more ways to obtain information. Multimedia resources, due to their rich forms of expression, intuitive information expression, and clear subject content, have become increasingly important in information retrieval and network communication.

[0003] In order to facilitate users to obtain the required multimedia resources, numerous multimedia resource recommendation engines have emerged, and their essence is to recommend multimedia resources to users based on their preferences for multimedia resources. In order to achieve the function of recommending multimedia resources based on users' preferences for multimedia resources, the relevant technology predicts the user's current preference for multimedia resources by learning the temporal relationship between the multimedia resources that the user has recently browsed. The multimedia resource preferences determined in this way only focus on the user's browsing of multimedia resources in a short period of time, and cannot adapt to the user's more complex multimedia resource recommendation needs. Therefore, the multimedia resources recommended to users cannot fully meet the user's current actual preferences for multimedia resources. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and electronic device for recommending multimedia resources, so as to at least solve the problem in the related art that the multimedia resources recommended for a target account and the target account's current resource preference for the multimedia resources have a low matching degree.

[0005] According to an embodiment of the present application, a method for recommending multimedia resources is provided, comprising:

[0006] Predicting a first account feature of a target account and a reference feature of each multimedia resource included in the first browsing information based on the first browsing information, wherein the first browsing information is used to indicate a temporal relationship between multimedia resources browsed by the target account and a reference account associated with the target account within a first time period before a current moment, the first account feature is used to indicate a browsing preference of the target account for multimedia resources under the influence of the reference account, and the reference feature is used to indicate a resource access relationship of the corresponding multimedia resource in the first browsing information;

[0007] Predicting a second account feature of the target account according to the second browsing information and the reference feature, wherein the second browsing information is used to indicate a browsing situation of multimedia resources of the target account in a second time period, the first time period is longer than the second time period and the first time period includes the second time period, and the second account feature is used to indicate an influence of browsing the multimedia resources in the second browsing information on the browsing preference of the target account for multimedia resources;

[0008] Predicting a target browsing feature of the target account according to the first account feature and the second account feature, wherein the target browsing feature is used to indicate a multimedia resource browsing preference of the target account at a current moment;

[0009] A target multimedia resource matching the target browsing feature is recommended to the target account.

[0010] Optionally, predicting the first account feature of the target account and the reference feature of each multimedia resource included in the first browsing information according to the first browsing information includes:

[0011] Constructing a first relationship graph according to the first browsing information, wherein the first relationship graph includes a first vertex and a first edge between first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, and a vertex corresponding to the multimedia resource included in the first browsing information, and the first edge includes an edge for representing an association relationship between accounts, an edge of an access relationship between an account and a multimedia resource, and an edge of an access timing relationship between multimedia resources;

[0012] Extracting target vertex features of a target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence, wherein the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is used to generate vertex features of the target vertex according to vertex features of a reference vertex in the current first relationship graph, and updating features of the target vertex in the first relationship graph using the vertex features of the target vertex extracted by the current feature extraction layer, wherein the reference vertex is a vertex in the first relationship graph connected to the target vertex through the first edge;

[0013] The target vertex feature of the target vertex corresponding to the target account is determined as the first account feature, and the target vertex feature of the target vertex corresponding to each multimedia resource included in the first browsing information is determined as the reference feature.

[0014] Optionally, extracting the target vertex feature of the target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence includes:

[0015] Calling a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records the reference vertices in the first relationship graph whose association degree with the target vertex is greater than the target association degree, and the target feature extraction layer is an extraction layer among the multiple feature extraction layers that is to perform feature extraction on the target vertex in the first relationship graph;

[0016] Calling the target feature extraction layer to extract features of the vertex features currently possessed by the reference vertices included in the target vertex set, and obtaining the reference vertex features of the target vertices output by the target feature extraction layer;

[0017] In a case where the target feature extraction layer is the last layer of a plurality of sequentially connected feature extraction layers, the reference vertex feature is determined as the target vertex feature.

[0018] Optionally, in the embodiment of the present application, when the feature extraction layer is the first layer, the vertex features can be extracted using an existing feature extraction model. For example, the features of user vertices can be extracted using existing deep learning technologies such as Transformer, CNN, etc. from the text, pictures, and other information provided by the user when registering to extract a high-dimensional vector as the user's features. The same is true for the vertices corresponding to multimedia resources, except that the existing technologies used may be different, such as ViTTransformer, etc.

[0019] Optionally, the calling the target feature extraction layer to extract features of the vertex features currently possessed by the vertices included in the target vertex set to obtain reference vertex features of the target vertices output by the target feature extraction layer includes:

[0020] Performing feature aggregation on the second vertex feature currently possessed by the second vertex in the first relationship graph according to the first vertex feature currently possessed by the target vertex in the first relationship graph to obtain a first aggregated feature, and aggregating the third vertex feature currently possessed by the third vertex in the first relationship graph according to the first vertex feature of the target vertex to obtain a second aggregated feature, wherein the second vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an incident end, and the third vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an exit end;

[0021] The first aggregated features and the second aggregated features are subjected to feature fusion to obtain the reference vertex features.

[0022] Optionally, the performing feature aggregation on a first vertex feature currently possessed by the target vertex in the first relationship graph and a second vertex feature currently possessed by the second vertex in the first relationship graph to obtain a first aggregated feature includes:

[0023] Calculating a first similarity between the first vertex feature and the second vertex feature;

[0024] Converting a first weight parameter of each of the second vertices according to the first similarity;

[0025] The first weight parameter is used to perform a weighted sum calculation on the second vertex features to obtain the first aggregate feature.

[0026] Optionally, aggregating the third vertex features currently possessed by the third vertex in the first relationship graph according to the first vertex feature of the target vertex to obtain a second aggregate feature includes:

[0027] Calculating a second similarity between the first vertex feature and the third vertex feature;

[0028] Converting a second weight parameter of each of the third vertices according to the second similarity;

[0029] The second weight parameter is used to perform weighted sum calculation on the third vertex feature to obtain the second aggregate feature.

[0030] Optionally, the calling the target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph includes:

[0031] Extracting an initial vertex set of the target vertex from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph connected to the target vertex through the first edge;

[0032] A candidate number of candidate vertices is screened out from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies the target dependency relationship, and the candidate number is less than the number of vertices included in the initial vertex set.

[0033] Optionally, the step of selecting a candidate number of candidate vertices from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set includes:

[0034] Using the vertex features of each vertex included in the first vertex set of the current screening round, calculating the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change of the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set;

[0035] Extracting vertices whose increment values ​​are greater than or equal to the target increment value from the first vertex set and adding them to the second vertex set;

[0036] When the number of candidate vertices included in the second vertex set to which vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, the third vertex set is determined as the target vertex set.

[0037] Optionally, constructing a first relationship graph according to the first browsing information includes:

[0038] Creating a fourth vertex corresponding to the target account, a fifth vertex corresponding to the reference account, and a sixth vertex corresponding to each multimedia resource included in the first browsing information, wherein the first vertex includes the fourth vertex, the fifth vertex, and the sixth vertex;

[0039] A second edge is created between the corresponding fourth vertex and the fifth vertex according to the association relationship between the target account and the reference account, a third edge is created between the corresponding fifth vertex according to the association relationship between the reference accounts, a fourth edge is created between the corresponding fourth vertex and the sixth vertex according to the access relationship of the target account to the multimedia resource, a fifth edge is added between the corresponding fifth vertex and the sixth vertex according to the access relationship of the reference account to the multimedia resource, and an initial directed edge is created between the sixth vertices according to the access sequence of the account to the multimedia resource in the first browsing information, wherein the sixth vertex corresponding to the multimedia resource with an earlier account access sequence in the first browsing information is connected to the output end of the initial directed edge, and the sixth vertex corresponding to the multimedia resource with a later account access sequence in the first browsing information is connected to the input end of the initial directed edge, and the first edge includes the second edge, the third edge, the fourth edge, the fifth edge and the initial directed edge.

[0040] Optionally, predicting the second account feature of the target account according to the second browsing information and the reference feature includes:

[0041] extracting a first resource feature of each multimedia resource in the second browsing information from the reference feature;

[0042] Filtering a plurality of first multimedia resources from the multimedia resources included in the second browsing information according to the first resource feature, wherein the first multimedia resources are multimedia resources in the second browsing information whose dependency relationships with other multimedia resources other than the first multimedia resource satisfy target dependency relationship conditions;

[0043] The second account feature is predicted based on the first resource feature corresponding to the first multimedia resource.

[0044] Optionally, predicting the second account feature according to the first resource feature corresponding to the first multimedia resource includes:

[0045] Acquire a position feature of each of the first multimedia resources among the plurality of the first multimedia resources, wherein the position feature is used to indicate a temporal relationship between the first multimedia resources in the second browsing information;

[0046] Concatenate the location feature of each of the first multimedia resources with the first resource feature to obtain a second resource feature;

[0047] Inputting a plurality of the second resource features into a target prediction model according to a temporal relationship of the first multimedia resource to obtain a third resource feature corresponding to each of the second resource features, wherein the target prediction model is used to extract an association relationship between the plurality of the second resource features, and output the third resource feature for each of the second resource features according to the association relationship between the second resource features, the third resource feature being used to indicate a resource feature of the current multimedia resource under the influence of the second resource features of other multimedia resources except the current multimedia resource among the plurality of the first multimedia resources;

[0048] The second account characteristic is predicted based on a plurality of the third resource characteristics.

[0049] Optionally, the step of inputting a plurality of the second resource features into a target prediction model according to the temporal relationship of the first multimedia resource to obtain a third resource feature corresponding to each of the second resource features includes:

[0050] Inputting the plurality of second resource features into a feature extraction layer including a plurality of attention modules according to the temporal relationship of the first multimedia resource, wherein the plurality of attention modules are used to extract initial relationship features indicating the feature relationship between the second resource features from a plurality of feature dimensions, and the feature extraction layer is used to perform feature merging on the plurality of initial relationship features output by the plurality of attention modules to obtain a target relationship feature;

[0051] The target relationship feature is input into a feature prediction layer to obtain a third resource feature output by the feature prediction layer corresponding to each of the second resource features, wherein the target prediction model includes the feature extraction layer and the feature prediction layer connected in sequence.

[0052] Optionally, predicting the second account feature according to the plurality of third resource features includes one of the following:

[0053] Determine the third resource feature corresponding to the multimedia resource that is at the last position in the temporal order among the plurality of the first multimedia resources as the second account feature;

[0054] Extracting the maximum feature component from the feature components of each of the third resource features in multiple vector dimensional spaces; merging the maximum feature components of multiple third resource features to obtain the second account feature;

[0055] According to the characteristic relationship between the plurality of the third resource characteristics, a corresponding characteristic weight is assigned to each of the third resource characteristics; and the characteristic weight is used to perform a weighted sum calculation on the plurality of the third resource characteristics to obtain the second account characteristic;

[0056] An average value is calculated for a plurality of the third resource characteristics to obtain the second account characteristic.

[0057] Optionally, the selecting a plurality of first multimedia resources from the multimedia resources included in the second browsing information according to the first resource feature includes:

[0058] Calculating a reference resource parameter of each of the multimedia resources in the second browsing information using the first resource feature, wherein the reference resource parameter is used to indicate a degree of dependency between the current multimedia resource in the second browsing information and other multimedia resources except the current multimedia resource;

[0059] The first multimedia resources whose reference resource parameters are greater than or equal to the target resource parameters are screened out from the second browsing information.

[0060] Optionally, predicting the target browsing feature of the target account according to the first account feature and the second account feature includes:

[0061] Concatenate the first account feature and the second account feature to obtain a concatenated feature;

[0062] The splicing features are fused using a feature fusion model to obtain the target browsing features output by the feature fusion model, wherein the conversion relationship between the splicing features and the browsing features is recorded in the feature fusion model.

[0063] Optionally, the recommending a target multimedia resource matching the target browsing feature to the target account includes:

[0064] Calculating target similarity between the target browsing feature and target resource features of each multimedia resource in the multimedia resource library;

[0065] Filtering the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library;

[0066] The target multimedia resource is recommended to the target account.

[0067] Optionally, the calculating the target similarity between the target browsing feature and the target resource feature of each multimedia resource in the multimedia resource library includes:

[0068] Generate a fourth resource feature of the multimedia resource in the first browsing information and a fifth resource feature of the multimedia resource in the multimedia resource library using a feature generation model;

[0069] Detecting feature difference information between the fourth resource feature and the fifth resource feature;

[0070] Using the feature adjustment amount corresponding to the feature difference information, feature adjustment is performed on the reference feature of the multimedia resource in the first browsing information to obtain the target resource feature, wherein the reference feature is used to indicate the account feature of the account browsing the multimedia resource in the first browsing information;

[0071] The target similarity between the target browsing feature and the target resource feature is calculated.

[0072] According to another embodiment of the present application, a multimedia resource recommendation device is provided, comprising:

[0073] a first prediction module, configured to predict a first account feature of a target account and a reference feature of each multimedia resource included in the first browsing information according to first browsing information, wherein the first browsing information is used to indicate a temporal relationship between multimedia resources browsed by the target account and a reference account associated with the target account within a first time period before a current moment, the first account feature is used to indicate a browsing preference of the target account for multimedia resources under the influence of the reference account, and the reference feature is used to indicate a resource access relationship of the corresponding multimedia resource in the first browsing information;

[0074] a second prediction module, configured to predict a second account feature of the target account based on second browsing information and the reference feature, wherein the second browsing information is used to indicate a browsing situation of multimedia resources of the target account in a second time period, the first time period is longer than the second time period and the first time period includes the second time period, and the second account feature is used to indicate an influence of browsing the multimedia resources in the second browsing information on the browsing preference of the target account for multimedia resources;

[0075] A third prediction module, configured to predict a target browsing feature of the target account based on the first account feature and the second account feature, wherein the target browsing feature is used to indicate a multimedia resource browsing preference of the target account at a current moment;

[0076] The recommendation module is used to recommend target multimedia resources matching the target browsing characteristics to the target account.

[0077] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0078] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0079] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0080] Through the present application, by using the first browsing information for prediction, a first account feature for indicating the browsing preference of the target account for multimedia resources under the influence of the reference account and a reference feature indicating the resource access relationship of the multimedia resource in the first browsing information are obtained, and then the second account feature of the target account is predicted according to the second browsing information and the reference feature, and the feature indicating the influence of the multimedia resource in the second browsing information on the browsing preference of the target account is obtained, and then the target browsing feature of the target account predicted by using the first account feature and the second account feature is integrated with both the resource browsing preference feature under the influence of the social relationship of the account and the resource browsing preference feature under the influence of the multimedia resources accessed by the target account itself in the second time period, so that the target browsing feature is closer to the target account's actual current access preference for multimedia resources, and then the target multimedia resource matching the target browsing feature pushed to the target account can better meet the target account's current demand for multimedia resources. The problem of low matching between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources can be solved, so as to improve the matching effect between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a hardware structure block diagram of a server device of a multimedia resource recommendation method according to an embodiment of the present application;

[0082] Figure 2 is a flowchart of a method for recommending multimedia resources according to an embodiment of the present application;

[0083] Figure 3 is an optional video recommendation schematic diagram according to an embodiment of the present application;

[0084] Figure 4 is a diagram of an optional FPGA heterogeneous accelerator architecture according to an embodiment of the present application;

[0085] Figure 5 is a schematic diagram of an optional recommended device hardware according to an embodiment of the present application;

[0086] Figure 6 It is a structural block diagram of a multimedia resource recommendation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0087] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0088] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0089] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a multimedia resource recommendation method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0090] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for recommending multimedia resources in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0091] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0092] In this embodiment, a method for recommending multimedia resources is provided. Figure 2 is a flowchart of a method for recommending multimedia resources according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0093] Step S202, predicting a first account feature of a target account and a reference feature of each multimedia resource included in the first browsing information based on the first browsing information, wherein the first browsing information is used to indicate a temporal relationship between multimedia resources browsed by the target account and a reference account associated with the target account within a first time period before a current moment, the first account feature is used to indicate a browsing preference of the target account for multimedia resources under the influence of the reference account, and the reference feature is used to indicate a resource access relationship of the corresponding multimedia resource in the first browsing information;

[0094] Step S204, predicting a second account feature of the target account based on the second browsing information and the reference feature, wherein the second browsing information is used to indicate a browsing situation of multimedia resources of the target account in a second time period, the first time period is longer than the second time period and the first time period includes the second time period, and the second account feature is used to indicate an influence of browsing the multimedia resources in the second browsing information on the browsing preference of the target account for multimedia resources;

[0095] Step S206, predicting a target browsing feature of the target account according to the first account feature and the second account feature, wherein the target browsing feature is used to indicate a multimedia resource browsing preference of the target account at the current moment;

[0096] Step S208: recommending target multimedia resources matching the target browsing characteristics to the target account.

[0097] Through the above steps, by using the first browsing information for prediction, the first account feature indicating the browsing preference of the target account for multimedia resources under the influence of the reference account and the reference feature indicating the resource access relationship of the multimedia resources in the first browsing information are obtained, and then the second account feature of the target account is predicted according to the second browsing information and the reference feature, and the feature indicating the influence of the multimedia resources in the second browsing information on the browsing preference of the target account is obtained, and then the target browsing feature of the target account predicted by using the first account feature and the second account feature is integrated with the factors of the resource browsing preference feature under the influence of the social relationship of the account, and the factors of the resource browsing preference feature under the influence of the multimedia resources accessed by the target account itself in the second time period are integrated, so that the target browsing feature is closer to the target account's actual current access preference for multimedia resources, and then the target multimedia resources matched with the target browsing feature pushed to the target account can better meet the target account's current demand for multimedia resources. The problem of low matching between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources can be solved, so as to improve the matching effect between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources.

[0098] The execution subject of the above steps may be a server, a terminal, etc. used to run the above multimedia resource push method, but is not limited thereto.

[0099] Optionally, in the embodiment of the present application, multimedia resources refer to resources that can transmit information in various forms, including text, pictures, audio, video, etc. These resources can be used on various media platforms, such as the Internet, television, movies, radio, etc.

[0100] In the technical solution provided in the above step S202, since the first account feature is the multimedia resource access preference of the target account under the influence of the reference account, the first account feature of the target account can be reflected by the access preference of the reference account to the multimedia resource. Since the reference feature indicates the feature of the resource access relationship of the multimedia resource in the first browsing information, the access relationship of the account to the multimedia resource can be reflected by the account feature, and then the reference feature is reflected. Then, the method of predicting the first account feature and the reference feature of each multimedia resource in the first browsing information according to the first browsing information can be: predicting the multimedia resource browsing feature of the reference account based on the access sequence feature of the multimedia resource browsed by the reference account, and then calculating the influence weight of each reference account on the multimedia resource browsing preference of the target account based on the similarity relationship between the multimedia resource accessed by the target account and the multimedia resource accessed by the reference account, and then performing weighted summation on the multimedia resource browsing preference features of the reference account based on the influence weight, so as to obtain the first account feature of the target account. Then, the candidate account that has accessed the current multimedia resource is searched from the reference account, and then the multimedia resource browsing preference features of the candidate account and the first account feature of the target account are feature-merged, so as to obtain the reference feature of each multimedia resource. When merging the multimedia resource browsing preference features of the candidate account and the first account features of the target account, the attention mechanism can be used to assign account weights to the target account and the candidate account according to their importance to the current multimedia resources, thereby using the account weights to perform a weighted summation of the multimedia resource browsing preference features and the first preference features of the target account.

[0101] Optionally, in the embodiment of the present application, the prediction of the first account feature and the reference feature can also refer to the graph neural network and submodule theory to design a heterogeneous directed graph attention neural network (HDGAT) to effectively learn vertex feature representation by fine-grained modeling of different types, directions, and strengths of relationships between different vertices. The embodiment of the present application takes the multimedia resource as a video resource as an example. The network includes The layer heterogeneous directed graph attention layer, each layer includes two functional components: neighbor sampling and aggregation update. The former is based on the submodule theory, and selects a set of basic neighbor subsets that are non-redundant, representative, and diverse from the neighbor set of each type of central vertex. The aggregation update component is based on the attention mechanism, and learns different weights for different neighbors selected for each type of central vertex to alleviate the imbalance. To this end, we maintain a user (account)-video (multimedia resource) heterogeneous directed graph It is used to update the embedding vector representation of users or videos. The update process requires information from the neighboring vertices of users and videos. and Video In the The learned hidden layer representations are and . For users , its neighbor set Contains two types of vertices: user vertices and video vertices; for videos , its neighbor set It also contains two types of vertices: video vertices and user vertices, so its neighbor set .

[0102] Neighbor sampling: Neighbor sampling refers to sampling from each type of central vertex The neighbor set of In the above example, we select a set of basic neighbor subsets that are non-redundant, representative, and diverse. , and satisfy the cardinality of the subset No more than When the neighbors of the central vertex have multiple types, it is required that there is at least one vertex of each type. In order to reduce the complexity of the network model and optimize the model application scenario, it is advisable to take the basic neighbor subset of the neighbor set of all vertices The upper limit of the cardinality is The above problem can be abstracted into an optimization problem: the optimization target is the basic neighbor subset The representativeness and diversity of the elements in; the constraint is the basic neighbor subset The base number does not exceed , and there is at least one vertex of each type.

[0103] A submodule function is a function defined on a collection, usually used to measure the "value" or "importance" of the collection. is called a submodule function if for any set and any element , the following conditions are met:

[0104] ,

[0105] Intuitively, this means that the marginal gain of adding elements to a smaller set decreases as the size of the set increases. This embodiment proposes a covering function that comprehensively considers the topological space distance and the feature space distance as a submodule function. Intuitively, a subset with a higher function value indicates that for each item in the base set, there is a similar item in the selected subset. In other words, the selected subset is very diverse and representative of the base set. Together with the base neighbor subset Constraints, we will center the vertex The neighbor sampling problem modeling is expressed as follows:

[0106] ,

[0107]

[0108] The above two formulas represent the objective function and constraint conditions of the optimization problem respectively; the central vertex can be either a user vertex or a video vertex; Represents the feature similarity measure between two vertices, which can be Euclidean distance, cosine similarity, etc. " "Judge whether the two operand types are consistent and return a Boolean variable (true or false); Indicates forced type conversion, converting a Boolean variable to an integer variable; without loss of generality, this application uses Gaussian distance as the metric vertex and The feature similarity between:

[0109] ,

[0110] in, represents the norm of a vector, is the bandwidth parameter of the Gaussian kernel. Of course, other similarity measurement methods may also be applied, such as Gaussian distance, Manhattan distance, etc., which are not limited in this embodiment.

[0111] Maximizing the submodule function under cardinality constraints is an NP-hard problem, which can usually be solved using a greedy algorithm. The embodiment of the present application proposes a Subset Extraction algorithm based on Greedy Strategy with Approximation (SEGSA). Since the neighbor sampling problem solving process of different center vertex types is similar, it is advisable to use the center vertex as the approximate solution. Taking the time as an example, the proposed approximate solution algorithm flow is as follows:

[0112] 1. Initialization: will be used to record the basic neighbor subset Initialized to an empty set; will be used to record the unselected vertex set (maybe called the candidate set) Initialize to ;

[0113] 2. Iterative greedy selection: (a). For each vertex in the set of unselected vertices , calculated according to the following formula and added The increment of the posterior objective function:

[0114]

[0115] (b) Select the vertex that can maximize the increment and satisfy the constraints, and remove it from Delete and add ; If there are multiple vertices that meet the requirements, one of them is randomly selected.

[0116] (c). Update the basic neighbor subset and the candidate set: ; ;

[0117] (d). Check whether the constraints are satisfied; if so, exit the loop;

[0118] 3. Output result: get user The sampled base neighbor subset .

[0119] Similarly, we can also use the REGSA algorithm to solve the video The sampled base neighbor subset It should be pointed out that our framework is applicable to any submodule functions, such as mutual information function, vertex cover function, edge cover function and facility location function.

[0120] Through the subset extraction approximate solution algorithm based on the greedy strategy, we Redundant neighbors and noise in the social space and / or item space are removed to obtain a set of basic neighbor subsets that are non-redundant, representative, and diverse, laying the foundation for effectively extracting user embeddings and item embeddings enhanced with social relations and transfer relations.

[0121] Aggregate Update: User-video Heterogeneous Directed Graph is a directed graph containing two types of vertices, users and videos, and three types of edges: social edges between users, interaction edges between users and videos, and transfer edges between videos. It is used to model friend relationships in social space (undirected), transfer relationships in project space (directed), and interaction relationships between social space and project space (undirected). The connecting edges between user vertices and their neighbors are all undirected edges; while the connecting edges between video vertices and their neighbor vertices include directed edges and undirected edges. As mentioned earlier, an undirected edge is equivalent to a pair of directed edges in opposite directions. Therefore, when the features of the central vertex are aggregated and updated, if there is an undirected edge between it and its neighbors, it is regarded as a bidirectional edge. After such equivalent processing, all connecting edges of each vertex are directed edges, which will simplify the introduction of the basic principles of the directed graph attention neural network proposed in this application.

[0122] For user-video heterogeneous directed graph Each vertex in , assuming and any of its neighbor vertices The The hidden layer representations of the layer heterogeneous directed graph attention neural network layer output are Dimensional vector and ; According to the principle of attention, The aggregation update operation of the layer-heterogeneous directed graph attention neural network layer can be divided into the following three steps:

[0123] 1. Incoming neighbor aggregation: For each vertex , calculate the aggregate features of its incident neighbor vertices:

[0124]

[0125] in, is the vertex For the incident neighbor vertices The attention weight of Represents the intersection of the incident neighbor set and the basic neighbor subset, ensuring that we only perform convolution operations on representative and diverse basic incident neighbors. It is calculated by the following formula:

[0126] ,

[0127] ,

[0128] in, Represents a vertex and The weight of the connection edge between them; is the activation function; Represents the concatenation operation of a vector.

[0129] 2. Outgoing neighbor aggregation: For each vertex , calculate the aggregate features of its incident neighbor vertices:

[0130] ,

[0131] in, is the vertex For outgoing neighbor vertices The attention weight of represents the intersection of the outgoing neighbor set and the basic neighbor subset, ensuring that we only perform convolution operations on representative and diverse basic outgoing neighbors. The calculation principle is similar, and the formula will not be repeated here.

[0132] 3. Feature transformation: The incoming neighbor features and outgoing neighbor features are linearly transformed and activated respectively to obtain the updated vertex representation.

[0133]

[0134] in, represents a nonlinear activation function; All of them are model parameters to be learned.

[0135] when At that time, Layer Heterogeneous Directed Graph Attention Neural Network Layer Based on User Feature Matrix and the video feature matrix is the input. When the heterogeneous directed graph attention network outputs the user embedding vector representation enhanced with social semantics and transfer relations and video embedding vector representation .in, Denotes the dimension of the user and video embedding vector representation.

[0136] The above heterogeneous directed graph embedding captures global knowledge such as friend relationships in the social space, transfer relationships in the project space, and all historical conversations between users and projects at one time, greatly improving the accuracy and efficiency of modeling users' long-term interest preferences.

[0137] In the technical solution provided in the above step S204, the second account feature can be obtained by extracting the feature of the association relationship between the resource features of the multimedia resources included in the second browsing information, that is, the second account feature includes the influence relationship between the multimedia resources included in the second browsing information, and the influence relationship can include a temporal relationship feature that characterizes the access temporal relationship of the multimedia resources, or the influence relationship can also include a complex transfer relationship feature that characterizes the adjacent temporal positions or the long-distance temporal positions of the multimedia resources, or can also include a temporal relationship feature and a transfer relationship feature at the same time. Therefore, in this embodiment, the method of predicting the second account feature of the target account according to the second browsing information and the reference feature can be: using the reference feature to configure the resource feature of each multimedia resource in the second browsing information, and then using the temporal relationship extraction model to extract the temporal relationship feature from the second browsing information, thereby obtaining the second account feature; or using the transfer relationship extraction model to extract the transfer relationship feature from the second browsing information as the second account feature; or using the temporal relationship extraction model to extract the temporal relationship feature from the second browsing information, and using the transfer relationship extraction model to extract the transfer relationship feature from the second browsing information, and merging the temporal relationship feature and the transfer relationship feature to obtain the second account feature.

[0138] Optionally, in an embodiment of the present application, the second account feature can simultaneously incorporate the temporal relationship features and transfer relationship features of the multimedia resources included in the second browsing information, that is, the temporal relationship features of the multimedia resources in the second browsing information can be learned by designing an intra-session temporal relationship learning network, and the transfer relationship features of the multimedia resources in the second browsing information can be learned by designing an intra-session transfer relationship learning network, and the features output by the temporal relationship learning network and the transfer relationship learning network are merged to obtain the second account feature.

[0139] The intra-session temporal relationship learning network learns each user Active sessions The network mainly consists of two parts: session preprocessor and time-aware capsule network. Active sessions The first one is responsible for removing redundant or invalid videos; the second one is responsible for learning the temporal relationship between videos from the preprocessed sessions to obtain session representation.

[0140] Session preprocessing program: This program can be implemented based on the neighbor sampling function of submodule theory. To simplify the design of time-aware capsule network, the cardinality of the basic session obtained by sampling is fixed to . Then the session sampling problem based on submodule theory can be modeled as follows:

[0141] ,

[0142] ,

[0143] The submodule function can be any of the above formulas, or any other function that can measure the "value" or "importance" of a set, which is not limited in this embodiment. Similarly, the above problem can be solved by the REGSA algorithm, which will not be described here. Active sessions The fixed length obtained by the session preprocessor is Basic activity session . Because the preprocessor may disrupt the underlying active session The video access timing in Reorder the timing in and record the sorted basic activity session as .

[0144] Time-aware capsule network: In view of the fact that traditional capsule networks are difficult to learn the temporal relationship between entities, this embodiment adds a position embedding layer in front of the traditional capsule network to obtain a time-aware capsule network. It combines the ability of the position embedding layer to learn the interaction timing between users and videos and the ability of the capsule network to learn hierarchical expression, thereby enhancing the accuracy of conversation representation.

[0145] For each user Processed underlying activity session ,video The position is defined as:

[0146] ,

[0147] For each discrete value , we assign a unique trainable parameter vector As a video Then embed it with the video Hidden representation of Add and perform linear transformation to generate Time-aware video capsules:

[0148] ,

[0149] in, is the parameter matrix to be learned. of The temporal-aware video capsules are stacked to form , and then input it into the GLU (Gated Linear Unit) model to capture the long-term dependencies between these temporal-aware video capsules and obtain temporal-enhanced video capsules :

[0150]

[0151] in, and are parameters to be learned; is a nonlinear activation function. The temporal enhanced video capsule Input into the attention model to initialize the coupling coefficient between the temporal-aware video capsule and the conversation capsule :

[0152] ,

[0153] ,

[0154] ,

[0155] in, represents the number of iterations, express Compared with the zero-value initialization adopted by the original capsule network, our improvement optimizes the capsule network's ability to perceive key information in the input data, thereby enhancing the network's representation ability.

[0156] Then the weighted sum of the prediction results is obtained by the following formula: :

[0157]

[0158] in, Indicates that The dimensions are copied to match Then, the prediction results are weighted and Compress it into a normalized vector and keep its direction unchanged to get the session capsule :

[0159]

[0160] in, is an exponential function; Represents the second norm of the vector. Then update the coupling coefficient between the temporal perception video capsule and the session capsule and enter the next iteration:

[0161]

[0162] go through In iterations, As a user Basic activity session Embedding vector representation of temporal relations (I stands for intra, S stands for sequential). The iterative update process is guided by our proposed simulated annealing-based dynamic routing algorithm (SADR). It can overcome the problem that the routing algorithm in the original capsule network is sensitive to hyperparameters and lacks global optimal solution guarantee. Specifically, in each iteration In the process of updating the coupling coefficient between the time-series-aware capsule and the session capsule, the coefficient is perturbed as follows:

[0163]

[0164] ,

[0165] in, is a vector where each element is sampled from a standard normal distribution; is the temperature; the normalized function Use the minimum-maximum method to transform the perturbed coupling coefficient vector The elements of are scaled to the range [0,1]:

[0166] ,

[0167] in, Denotes the coupling coefficient vector No. The optimal coupling coefficient is calculated according to the following formula and the post-disturbance coupling coefficient Energy:

[0168]

[0169] According to the Metropolis criterion, the perturbed coefficients are accepted with a certain probability:

[0170]

[0171] If the perturbed coupling coefficient is accepted, the optimal coupling coefficient and optimal energy are updated according to the following formula:

[0172]

[0173] Then update the temperature according to the following formula:

[0174]

[0175] in, is the attenuation parameter. The above process is iteratively performed until the temperature gradually decreases, and the coupling coefficient is optimized to achieve better information transfer in the network. The pseudo code of our proposed SADR algorithm is shown below.

[0176] 1. Initialize environmental parameters: Initialize maximum temperature , minimum temperature and the optimal coupling coefficient .

[0177] 2. When When , iterative optimization:

[0178] a for the current coupling coefficient The implementation temperature is The random perturbation operation of ;

[0179] b is the coefficient after disturbance Perform normalization operation and get ;

[0180] c Calculate the optimal coupling coefficient and the post-disturbance coupling coefficient energy;

[0181] dAccording to the Metropolis criterion, the coefficient after perturbation is accepted with a certain probability , and update the optimal coupling coefficient and optimal energy.

[0182] eAccording to Reduce temperature ,in is the decay rate.

[0183] 3. Output results: Return to the optimal coupling coefficient after the cycle ends .

[0184] It should be pointed out that the above dynamic routing algorithm can also be designed based on other heuristic methods, such as genetic algorithm, particle swarm algorithm, ant colony algorithm, taboo search, and simulated annealing.

[0185] The intra-session temporal relationship learning network uses the session preprocessor to learn from each user Active sessions The redundancy and noise are removed from the data, which improves the data quality of the training network model and lays the foundation for the subsequent time capsule-aware network to accurately extract the time relationship points within the session. The time-aware capsule network is an improvement on the traditional capsule network, which is mainly reflected in the following three aspects: (1) By adding a position embedding layer, the deficiency of the traditional capsule network that cannot extract the time relationship between videos within a session is solved. (2) The capsule network coupling coefficient initialization method based on the attention mechanism optimizes the capsule network's perception of key information in the input data and enhances the representation ability of the capsule network; (3) Based on the simulated annealing dynamic routing algorithm, the problem of the routing algorithm in the original capsule network being sensitive to hyperparameters and lacking the guarantee of the global optimal solution is overcome, further enhancing the representation ability of the capsule network.

[0186] In-session transfer relation learning network,For the in-session transfer relation learning network, we propose an in-session transfer relation learning network model based on a weighted directed graph autoencoder, which is used to learn the high-order transfer relations between different videos in each user’s current basic session. First, we need to design a weighted directed activity session graph construction algorithm to model the video sequences in the user’s activity session and the transfer relations between them as a weighted directed activity session graph.

[0187] The weighted directed activity session graph is constructed. First, each user Active sessions Processed underlying activity session Modeled as a weighted directed graph In this graph, each vertex Represents a video, each directed edge Indicates user In the base activity session Finished watching the video After watching the video ; Represents a directed edge The weight of , which describes the semantic distance or similarity between video pairs. Basic activity session As an example, we introduce the Weighted and Directed Intra-Session Graph Construction Algorithm (WISGC). Assume , the following are the specific steps of the algorithm:

[0188] 1. Vertex set determination: Session All different videos that appear in the session are the vertex set of the session graph. (For the above session sequence, the vertex set is {A, B, C}, where the same video is only counted once)

[0189] 2. Edge counting and weight calculation: For adjacent video pairs in a session, count the number of directed edges between them. Calculate the weight of each directed edge by dividing the number of edge appearances by the out-degree of the source vertex, that is:

[0190] (30)

[0191] in, Represents a session Any pair of directed edges in The weight of Represents a directed edge In session The number of times it appears in Represents a session China-Israel Video is the number of directed edges of the source point. The specific steps are as follows:

[0192] a. Count the number of occurrences of adjacent video pairs in the session sequence to form edge counts; (< A, B >: appears once; < B, A >: appears twice; < A, C >: appears once; < C, B >: appears once);

[0193] b. Calculate the out-degree of the vertex, that is, the number of edges that each vertex serves as the starting vertex; ( (There are two outgoing edges: A→B and A→C); (There are two outgoing edges: B→A and B→C); (There is one outgoing edge: C→B));

[0194] c. Calculate the weight of each edge based on the number of occurrences and out-degree; (w(A→B)=1 / 2=0.5; w(B→A)=2 / 2=1; w(A→C) =1 / 2 = 0.5; w(C→B)=1 / 1=1);

[0195] d. Construct a weighted directed graph: Construct a weighted directed graph using a set of vertices and a set of weighted edges

[0196] .

[0197] 4. Reconstruction error calculation. Use the reconstructed adjacency matrix With the original adjacency matrix The difference between is taken as the reconstruction error. The commonly used loss function is the mean square error loss function:

[0198]

[0199] in, Represents the loss function value of the weighted directed graph autoencoder network.

[0200] The design of the weighted directed graph autoencoder is based on the weighted directed graph convolutional neural network model. It comprehensively considers the features of neighbor vertices, the direction and weight of the connecting edges, and performs graph convolution operations on the features of the central vertex to map it from the high-dimensional feature space back to the low-dimensional feature space, and then conversely maps it from the low-dimensional feature space back to the high-dimensional feature space. The adjacency matrix is ​​predicted with the help of the adjacency matrix reconstruction operation, and the reconstruction error between the reconstructed adjacency matrix and the original adjacency matrix is ​​constrained to be small enough to achieve unsupervised learning. This design can more effectively capture the complex transfer relationship and information transmission between adjacent vertices and distant vertices in the directed graph, greatly improving the expressiveness and applicability of the model. At the same time, it also reduces the dependence on expensive labeled data.

[0201] In the technical solution provided in the above step S206, the target browsing feature can be predicted by merging the first account feature and the second account feature, or an attention mechanism can be used to assign weights to the first account feature and the second account feature according to their importance to predicting the target preference feature of the target account, thereby using the assigned weights to perform a weighted summation of the first account feature and the second account feature to obtain the target browsing feature.

[0202] In the technical solution provided in the above step S208, the target multimedia resource may be a multimedia resource having a matching degree greater than the target matching degree obtained by matching the multimedia resources in the multimedia resource library based on the characteristics of the target account. The target multimedia resource may be one or more. When there are multiple target multimedia resources, the multiple target multimedia resources may be sorted according to the matching degree with the characteristics of the target account.

[0203] As an optional embodiment, predicting the first account feature of the target account and the reference feature of each multimedia resource included in the first browsing information according to the first browsing information includes:

[0204] Constructing a first relationship graph according to the first browsing information, wherein the first relationship graph includes a first vertex and a first edge between first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, and a vertex corresponding to the multimedia resource included in the first browsing information, and the first edge includes an edge for representing an association relationship between accounts, an edge of an access relationship between an account and a multimedia resource, and an edge of an access timing relationship between multimedia resources;

[0205] Extracting target vertex features of a target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence, wherein the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is used to generate vertex features of the target vertex according to vertex features of a reference vertex in the current first relationship graph, and updating features of the target vertex in the first relationship graph using the vertex features of the target vertex extracted by the current feature extraction layer, wherein the reference vertex is a vertex in the first relationship graph connected to the target vertex through the first edge;

[0206] The target vertex feature of the target vertex corresponding to the target account is determined as the first account feature, and the target vertex feature of the target vertex corresponding to each multimedia resource included in the first browsing information is determined as the reference feature.

[0207] Optionally, in an embodiment of the present application, the feature extraction model is used to iteratively update the relationship between vertices in the first relationship graph through multiple sequentially connected feature extraction layers, that is, each feature extraction layer is used to update the features of the current vertex according to the features of vertices that have an associated relationship with the current vertex, thereby randomly iterating the feature extraction layer, so that the features of the current vertex can be influenced by the features of vertices that are farther away.

[0208] Optionally, in an embodiment of the present application, each feature extraction layer can perform a graph convolution operation on each central vertex, and then learn different weights for the graph convolution results of different central vertices based on the attention mechanism to alleviate the imbalance.

[0209] Through the above content, by using a feature extraction model including multiple sequentially connected feature extraction layers to perform multiple rounds of iterative extraction on the vertex features of the target vertex in the first relationship graph, the feature relationship between vertices with a large distance is extracted, thereby improving the feature information dimension of the target vertex feature.

[0210] As an optional embodiment, extracting the target vertex feature of the target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence includes:

[0211] Calling a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records the reference vertices in the first relationship graph whose association degree with the target vertex is greater than the target association degree, and the target feature extraction layer is an extraction layer among the multiple feature extraction layers that is to perform feature extraction on the target vertex in the first relationship graph;

[0212] Calling the target feature extraction layer to extract features of the vertex features currently possessed by the reference vertices included in the target vertex set, and obtaining the reference vertex features of the target vertices output by the target feature extraction layer;

[0213] In a case where the target feature extraction layer is the last layer of a plurality of sequentially connected feature extraction layers, the reference vertex feature is determined as the target vertex feature.

[0214] Optionally, in an embodiment of the present application, in order to ensure the completeness of the information in the vertex set, the target vertex set needs to include all types of vertices, that is, both vertices corresponding to accounts and vertices corresponding to multimedia resources.

[0215] Optionally, in an embodiment of the present application, in order to improve the reliability of information when extracting the target vertex set, maximum value extraction can be used for extraction, that is, the vertex with the highest correlation with the current target vertex and the greatest benefit in reflecting the characteristics of the target vertex is extracted from the first relationship graph. Therefore, in an embodiment of the present application, a Subset Extraction algorithm based on Greedy Strategy with Approximation (SEGSA) can be used to perform neighbor sampling in the first relationship graph, and when performing neighbor sampling, the importance relationship between vertices can be calculated according to the submodule function theory.

[0216] Through the above content, by calling the target feature extraction layer, global knowledge such as friend relationships in the social space and all historical interaction records between users and multimedia are captured, which greatly improves the accuracy and efficiency of modeling users' long-term interest preferences.

[0217] As an optional embodiment, the calling of the target feature extraction layer to extract features of the vertex features currently possessed by the vertices included in the target vertex set to obtain reference vertex features of the target vertex output by the target feature extraction layer includes:

[0218] Performing feature aggregation on the second vertex feature currently possessed by the second vertex in the first relationship graph according to the first vertex feature currently possessed by the target vertex in the first relationship graph to obtain a first aggregated feature, and aggregating the third vertex feature currently possessed by the third vertex in the first relationship graph according to the first vertex feature of the target vertex to obtain a second aggregated feature, wherein the second vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an incident end, and the third vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an exit end;

[0219] The first aggregated features and the second aggregated features are subjected to feature fusion to obtain the reference vertex features.

[0220] Through the above content, when generating the reference vertex feature, the features of the outgoing neighbor set and the features of the incident neighbor set are aggregated respectively to obtain the first aggregate feature and the second aggregate feature, and the reference vertex feature is obtained by merging the first aggregate feature and the second aggregate feature, thereby improving the reliability of the reference vertex feature.

[0221] As an optional embodiment, the step of performing feature aggregation on a first vertex feature currently possessed by the target vertex in the first relationship graph and a second vertex feature currently possessed by the second vertex in the first relationship graph to obtain a first aggregated feature includes:

[0222] Calculating a first similarity between the first vertex feature and the second vertex feature;

[0223] Converting a first weight parameter of each of the second vertices according to the first similarity;

[0224] The first weight parameter is used to perform a weighted sum calculation on the second vertex features to obtain the first aggregate feature.

[0225] Optionally, in the embodiment of the present application, the first weight parameter of each second vertex converted according to the first similarity may be calculated by the following formula:

[0226] ,

[0227] ,

[0228] in, Represents a vertex and The weight of the connection edge between them; is the activation function; Represents the concatenation operation of a vector.

[0229] Through the above content, by calculating the first similarity between the first vertex feature and the second vertex feature based on the attention mechanism, and then assigning the corresponding feature to each second vertex, the aggregation of vertices according to weights is achieved, thereby improving the balanced relationship between different vertex features in the first aggregated feature.

[0230] As an optional embodiment, aggregating the third vertex features currently possessed by the third vertex in the first relationship graph according to the first vertex feature of the target vertex to obtain a second aggregate feature includes:

[0231] Calculating a second similarity between the first vertex feature and the third vertex feature;

[0232] Converting a second weight parameter of each of the third vertices according to the second similarity;

[0233] The second weight parameter is used to perform weighted sum calculation on the third vertex feature to obtain the second aggregate feature.

[0234] Optionally, in the embodiment of the present application, the second weight parameter of each of the third vertices may be converted according to the second similarity by calculating the second weight parameter using the following formula:

[0235] ,

[0236] ,

[0237] in, Represents a vertex and The weight of the connection edge between them; is the activation function; Represents the concatenation operation of a vector.

[0238] Through the above content, by calculating the second similarity of the first vertex feature and the third vertex feature based on the attention mechanism, and then assigning the corresponding one to each third vertex, the aggregation of vertices according to weights is achieved, thereby improving the balanced relationship between different vertex features in the second aggregated feature.

[0239] As an optional embodiment, the calling the target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph includes:

[0240] Extracting an initial vertex set of the target vertex from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph connected to the target vertex through the first edge;

[0241] A candidate number of candidate vertices is screened out from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies the target dependency relationship, and the candidate number is less than the number of vertices included in the initial vertex set.

[0242] As an optional embodiment, the step of selecting a candidate number of candidate vertices from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set includes:

[0243] Using the vertex features of each vertex included in the first vertex set of the current screening round, calculating the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change of the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set;

[0244] Extracting vertices whose increment values ​​are greater than or equal to the target increment value from the first vertex set and adding them to the second vertex set;

[0245] When the number of candidate vertices included in the second vertex set to which vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, the third vertex set is determined as the target vertex set.

[0246] Optionally, in an embodiment of the present application, a candidate number of candidate vertices are screened out from the initial vertex set according to vertex features currently possessed by the vertices in the initial vertex set, and a method for obtaining the target vertex set may be: randomly screen out the candidate number of vertices from the initial vertex set to obtain a first vertex set; calculate vertex set information of the first vertex set, wherein the vertex set information is used to indicate the importance of vertex features possessed by the vertices in the first vertex set; extract an alternative vertex from the remaining vertices in the initial vertex set; use the vertex features currently possessed by the alternative vertex to calculate the incremental value of the alternative vertex to the vertex set information; if the incremental value meets the target incremental value condition, use the alternative vertex to update the vertices in the first vertex set to obtain a second vertex set; detect the number of remaining vertices in the initial vertex set; and if it is determined that there are no remaining vertices in the initial vertex set, determine the second vertex set as the target vertex set.

[0247] Optionally, in the embodiment of the present application, the target vertex set is screened by using a Subset Extraction algorithm based on Greedy Strategy with Approximation (SEGSA), for each vertex in the first relationship graph Redundant neighbors and noise in the social space and / or item space are removed to obtain a set of basic neighbor subsets that are non-redundant, representative, and diverse, laying the foundation for effectively extracting user embeddings and multimedia resource embeddings (i.e., features) enhanced with social relations.

[0248] As an optional embodiment, constructing a first relationship graph according to the first browsing information includes:

[0249] Creating a fourth vertex corresponding to the target account, a fifth vertex corresponding to the reference account, and a sixth vertex corresponding to each multimedia resource included in the first browsing information, wherein the first vertex includes the fourth vertex, the fifth vertex, and the sixth vertex;

[0250] A second edge is created between the corresponding fourth vertex and the fifth vertex according to the association relationship between the target account and the reference account, a third edge is created between the corresponding fifth vertex according to the association relationship between the reference accounts, a fourth edge is created between the corresponding fourth vertex and the sixth vertex according to the access relationship of the target account to the multimedia resource, a fifth edge is added between the corresponding fifth vertex and the sixth vertex according to the access relationship of the reference account to the multimedia resource, and an initial directed edge is created between the sixth vertices according to the access sequence of the account to the multimedia resource in the first browsing information, wherein the sixth vertex corresponding to the multimedia resource with an earlier account access sequence in the first browsing information is connected to the output end of the initial directed edge, and the sixth vertex corresponding to the multimedia resource with a later account access sequence in the first browsing information is connected to the input end of the initial directed edge, and the first edge includes the second edge, the third edge, the fourth edge, the fifth edge and the initial directed edge.

[0251] As an optional embodiment, predicting the second account feature of the target account according to the second browsing information and the reference feature includes:

[0252] extracting a first resource feature of each multimedia resource in the second browsing information from the reference feature;

[0253] Filtering a plurality of first multimedia resources from the multimedia resources included in the second browsing information according to the first resource feature, wherein the first multimedia resources are multimedia resources in the second browsing information whose dependency relationships with other multimedia resources other than the first multimedia resource satisfy target dependency relationship conditions;

[0254] The second account feature is predicted based on the first resource feature corresponding to the first multimedia resource.

[0255] Optionally, in the embodiment of the present application, the screening logic of the first multimedia resource can be abstracted into an optimization problem. The purpose of the optimization is to make the screened first multimedia resources more representative and diverse. Therefore, a sampling method based on submodule theory can be adopted. The session sampling problem based on submodule theory can be modeled as follows:

[0256] ,

[0257] ,

[0258] The above problem can be solved by REGSA algorithm, which will not be described here. Current session The fixed length obtained by the session preprocessor is The basis of the current session .

[0259] Optionally, in the embodiment of the present application, when predicting the second account feature, it is necessary to consider the temporal proximity relationship and the remote influence relationship between the first multimedia resources that need to be incorporated into the second account feature. Therefore, a feature extraction model with such a feature extraction function can be used to extract the first resource feature corresponding to the first multimedia resource, thereby obtaining the second account feature.

[0260] Through the above content, by screening out multiple first multimedia resources from the multimedia resources included in the second browsing information according to the dependency relationship between the multimedia resources, the multimedia resources with lower "value" included in the second browsing information are screened, thereby ensuring the reliability of the first multimedia resources for the second account characteristics. The accuracy of the second account characteristics is guaranteed.

[0261] As an optional embodiment, predicting the second account feature according to the first resource feature corresponding to the first multimedia resource includes:

[0262] Acquire a position feature of each of the first multimedia resources among the plurality of the first multimedia resources, wherein the position feature is used to indicate a temporal relationship between the first multimedia resources in the second browsing information;

[0263] Concatenate the location feature of each of the first multimedia resources with the first resource feature to obtain a second resource feature;

[0264] Inputting a plurality of the second resource features into a target prediction model according to a temporal relationship of the first multimedia resource to obtain a third resource feature corresponding to each of the second resource features, wherein the target prediction model is used to extract an association relationship between the plurality of the second resource features, and output the third resource feature for each of the second resource features according to the association relationship between the second resource features, the third resource feature being used to indicate a resource feature of the current multimedia resource under the influence of the second resource features of other multimedia resources except the current multimedia resource among the plurality of the first multimedia resources;

[0265] The second account characteristic is predicted based on a plurality of the third resource characteristics.

[0266] Optionally, in an embodiment of the present application, when predicting the third resource feature, it is necessary to consider the temporal sequence of the first multimedia resource, and a position code may be added thereto according to the following formula, so as to allow the model to distinguish the first multimedia resources at different positions of the basic current session:

[0267] ,

[0268] For each discrete value , we assign a unique trainable parameter vector As a video Then it is combined with the video The embedding vector representation of Add and perform linear transformation to get The sequence-aware embedding vector representation of the video:

[0269] ,

[0270] in, is the parameter matrix to be learned. Represents the dimension of the order-aware embedding vector.

[0271] Optionally, in an embodiment of the present application, the target prediction model may be, but is not limited to, a model for extracting features based on the temporal relationship between multimedia resources in a multimedia resource sequence. The target prediction model may be, but is not limited to, a Transformer model. Transformer is a powerful attention mechanism model suitable for processing sequence data and can effectively capture long-distance dependencies in the sequence. Such dependencies include both the sequential relationship between videos within a session and the transfer relationship between videos within a session.

[0272] Through the above content, the location feature and the first resource feature are feature spliced, and the spliced ​​second resource feature is input into a target prediction model for extracting the association relationship between multiple second resource features, and outputting the third resource feature for each second resource feature according to the association relationship between the second resource features, thereby effectively capturing the long-distance dependency and transfer relationship between the second resource features, and extracting accurate user dynamic interest preferences.

[0273] As an optional embodiment, the inputting of multiple second resource features into a target prediction model according to the temporal relationship of the first multimedia resource to obtain a third resource feature corresponding to each second resource feature includes:

[0274] Inputting the plurality of second resource features into a feature extraction layer including a plurality of attention modules according to the temporal relationship of the first multimedia resource, wherein the plurality of attention modules are used to extract initial relationship features indicating the feature relationship between the second resource features from a plurality of feature dimensions, and the feature extraction layer is used to perform feature merging on the plurality of initial relationship features output by the plurality of attention modules to obtain a target relationship feature;

[0275] The target relationship feature is input into a feature prediction layer to obtain a third resource feature output by the feature prediction layer corresponding to each of the second resource features, wherein the target prediction model includes the feature extraction layer and the feature prediction layer connected in sequence.

[0276] Optionally, in an embodiment of the present application, each attention module and the single-head self-attention mechanism extracts the feature relationship between the second resource features. Generally, the single-head self-attention mechanism can be defined as:

[0277]

[0278] in, , and The input matrices representing query, key, and value respectively; Indicates the number of columns of the key matrix.

[0279] Optionally, in an embodiment of the present application, multiple attention modules are used to extract features based on a multi-head attention mechanism, and the self-attention mechanism includes self-attention heads, which in parallel perform Conducting conversational representation learning:

[0280] ,

[0281] ,

[0282] ,

[0283] in, , , and are parameter matrices to be learned; and They are and Along the last dimension A division.

[0284] Through the above content, by setting the feature extraction layers of multiple attention modules, features for indicating the feature relationship between the second resource features are extracted from multiple feature dimensions, thereby increasing the dimension of the feature space included in the third resource features.

[0285] As an optional embodiment, predicting the second account feature according to the plurality of the third resource features includes one of the following:

[0286] Determine the third resource feature corresponding to the multimedia resource that is at the last position in the temporal order among the plurality of the first multimedia resources as the second account feature;

[0287] Extracting the maximum feature component from the feature components of each of the third resource features in multiple vector dimensional spaces; merging the maximum feature components of multiple third resource features to obtain the second account feature;

[0288] According to the characteristic relationship between the plurality of the third resource characteristics, a corresponding characteristic weight is assigned to each of the third resource characteristics; and the characteristic weight is used to perform a weighted sum calculation on the plurality of the third resource characteristics to obtain the second account characteristic;

[0289] An average value is calculated for a plurality of the third resource characteristics to obtain the second account characteristic.

[0290] As an optional embodiment, the selecting a plurality of first multimedia resources from the multimedia resources included in the second browsing information according to the first resource feature includes:

[0291] Calculating a reference resource parameter of each of the multimedia resources in the second browsing information using the first resource feature, wherein the reference resource parameter is used to indicate a degree of dependency between the current multimedia resource in the second browsing information and other multimedia resources except the current multimedia resource;

[0292] The first multimedia resources whose reference resource parameters are greater than or equal to the target resource parameters are screened out from the second browsing information.

[0293] Optionally, in the embodiment of the present application, the reference resource parameter may be, but is not limited to, calculated based on the submodule theory for the first resource feature corresponding to the multimedia resource. The session sampling problem based on the submodule theory may be modeled as follows:

[0294] ,

[0295] ,

[0296] The above problem can be solved by the REGSA algorithm proposed above, which will not be described here.

[0297] Current session The fixed length obtained by the session preprocessor is The basis of the current session .

[0298] Through the above content, by calculating the reference resource parameters used to characterize the dependency between multimedia resources, the multimedia resources with lower "value" included in the second browsing information are screened, thereby ensuring the reliability of the first multimedia resource for the second account feature and the accuracy of the second account feature.

[0299] As an optional embodiment, predicting the target browsing feature of the target account according to the first account feature and the second account feature includes:

[0300] Concatenate the first account feature and the second account feature to obtain a concatenated feature;

[0301] The splicing features are fused using a feature fusion model to obtain the target browsing features output by the feature fusion model, wherein the conversion relationship between the splicing features and the browsing features is recorded in the feature fusion model.

[0302] As an optional embodiment, the recommending a target multimedia resource matching the target browsing feature to the target account includes:

[0303] Calculating target similarity between the target browsing feature and target resource features of each multimedia resource in the multimedia resource library;

[0304] Filtering the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library;

[0305] The target multimedia resource is recommended to the target account.

[0306] As an optional embodiment, the calculating the target similarity between the target browsing feature and the target resource feature of each multimedia resource in the multimedia resource library includes:

[0307] Generate a fourth resource feature of the multimedia resource in the first browsing information and a fifth resource feature of the multimedia resource in the multimedia resource library using a feature generation model;

[0308] Detecting feature difference information between the fourth resource feature and the fifth resource feature;

[0309] Using the feature adjustment amount corresponding to the feature difference information, feature adjustment is performed on the reference feature of the multimedia resource in the first browsing information to obtain the target resource feature, wherein the reference feature is used to indicate the account feature of the account browsing the multimedia resource in the first browsing information;

[0310] The target similarity between the target browsing feature and the target resource feature is calculated.

[0311] When recommending multimedia resources, a recommendation method based on sessions (i.e., multimedia resource browsing information of an account within a certain time period (such as the second time period)) is often used. This recommendation method has the following problems: (a) The session contains not only the temporal information of user behavior, but also the transfer information between adjacent items. Both types of information have an important impact on the performance of the recommendation algorithm, and it is inappropriate to consider any one factor alone. (b) There is a large amount of noise data in the session, such as items clicked by mistake or items clicked repeatedly. These noises seriously affect the modeling of user interest preferences, and thus affect the accuracy of recommendations. (c) There are many redundant calculations, and additional sessions from social friends need to be processed to make recommendations for the current session. (d) The underlying hardware deployed by the system usually uses GPUs for training and inference, which has low computing efficiency, high power consumption, and low energy efficiency.

[0312] In view of the above shortcomings, this application proposes a video recommendation system based on Social-semantics and Transition-relationship enhanced Transformer (VRSTT). Figure 3 is an optional video recommendation schematic diagram according to an embodiment of the present application, such as Figure 3As shown, it consists of a heterogeneous directed graph construction component, a heterogeneous directed graph embedding component, a personalized session embedding component, a user interest preference fusion component, and a prediction component. Among them, the heterogeneous directed graph construction component is responsible for constructing a user-video heterogeneous directed graph containing two vertices and three connecting edges (relationships) from the friendship relationship between users in the social space, the transfer relationship between videos in the project space, and the interaction relationship between the social space and the project space. In the heterogeneous directed graph embedding component, this application designs a neighbor sampling algorithm REGSA (Subset Extraction algorithm based on Greedy Strategy with Approximation) based on submodule theory to sample the neighbors of each vertex in the user-video heterogeneous directed graph to remove redundant information and noise; a heterogeneous directed graph convolutional neural network based on the attention mechanism is designed to extract user embedding and video embedding with enhanced social semantics and transfer relationships from the sampled heterogeneous directed graph. The personalized session embedding component uses the Transformer encoder to learn the temporal relationship and transfer relationship between videos from the preprocessed user's current session, and extracts the session representation as the user's dynamic interest preference. It is combined with the user embedding as the user's long-term interest preference to obtain the user's comprehensive interest preference through the user interest preference fusion component. The prediction component calculates the probability of the user watching all videos that have not been visited based on the learned user's comprehensive interest preference, and returns the video or group of videos with the highest probability to the user as the recommendation result. To improve the energy efficiency of training and inference, we also designed a general heterogeneous accelerator based on FPGA for heterogeneous directed graph convolutional neural networks and Transformer encoders.

[0313] The proposed recommendation method comprehensively considers the influence of social relations in social space, transition relations in project space, and the temporal relationship between user and project interaction in the two spaces on user interest preferences. It innovatively proposes a heterogeneous directed graph construction algorithm to characterize the three dynamic interaction relationships between the two entities of user and video in the video recommendation system; innovatively proposes a neighbor sampling algorithm based on submodule theory to remove information redundancy and noise in the social space, project space, and the interaction process between the two spaces; designs a heterogeneous directed graph attention network to mine the complex interaction relationship between users and videos in social space and project space, and learns user embedding and video embedding enhanced by social semantics and transition relations. Among them, the user embedding enhanced by social semantics and transition relations accurately describes the user's long-term interest preferences. Innovatively proposes a personalized session embedding network based on Transformer encoder to learn the temporal and transition relations between videos from the user's current session that removes redundancy and noise, and extracts the user's dynamic interest preferences. After being integrated with the user's long-term interest preferences, it obtains the user's comprehensive interest preferences enhanced by social relations, transition relations, and temporal relations, which more accurately and personalizedly describes the user's interests and can greatly improve the accuracy of video recommendations. The proposed method only uses the user's historical conversation to generate social semantics and transfer relationship enhanced user embedding in the heterogeneous directed graph embedding process. It avoids repeated learning of social friends' historical conversations to integrate social influence when making recommendations for the current session, greatly reducing the computational complexity of the network model and significantly improving the network training and reasoning efficiency. In addition, we have also designed a general heterogeneous accelerator based on FPGA to improve the training and reasoning efficiency of heterogeneous directed graph convolutional neural networks and Transformer encoders, and can significantly save energy consumption and reduce costs.

[0314] First, we introduce the basic symbols used in this application and the problem to be solved. Specifically, all vectors and scalars are represented by bold lowercase letters (such as ) and lowercase letters (such as ) and use bold, italic, and uppercase letters to denote matrices (e.g. ); Use italic capital letters to represent tuples (such as ); capital cursive letters to denote sets (e.g. ); In particular, use uppercase Euclid Math Two font to represent the range of variables (such as the and the set of integers ).

[0315] let and Respectively represent the user set and video set in the recommendation system VRSTT. There is a friend relationship between users; there is an interactive relationship between users and videos (when users Visited the video ,say and There are interactive relationships). Graphs are usually used to model the relationships between these entities.

[0316] Definition 1 (Graph) is a tuple consisting of a vertex set and a connecting edge set, which is used to describe the relationship between pairs of vertices. It is usually denoted as Among them, the vertex set Non-empty, each element represents a vertex; connected edge set Represents the connection relationship between a pair of vertices, and each element represents a connecting edge.

[0317] In addition to representing the graph as a set of two-tuples, we can also use the adjacency matrix Represents the association relationship between vertices. Row, No. Column Elements Represents a vertex and The relationship between them. Vertices directly connected by connecting edges For its first-order neighbors, all first-order neighbors form the vertex Neighborhood set , so there is . Vertex Neighborhood Set Cardinality Vertex The degree of .

[0318] Depending on whether the connecting edges have weights, graphs can be divided into weighted graphs and unweighted graphs. For weighted graphs, each edge All with weight , which is used to describe the similarity between a pair of vertices connected by the edge. Adjacency matrix of weighted graph Middle Row, No. Column Elements Represents a vertex and The similarity between and When there are connecting edges between ;otherwise For an unweighted graph, when and When there are connecting edges between ;otherwise .

[0319] Depending on whether the connecting edges have directions, the graph can be divided into directed graphs and undirected graphs, and recorded and The directed and undirected edges between them are and .when When it is a directed graph, the vertices Neighborhood set It can be divided into the incident neighbor set and the outgoing neighbor set . So there is .in, , . Accordingly, the vertex The degree is further divided into in-degree and out-degree , and are defined as and .

[0320] According to the combination principle, graphs can also be divided into weighted directed graphs, weighted undirected graphs, unweighted undirected graphs and unweighted directed graphs. Figure 4 kind.

[0321] Definition 2 (Social Graph) is a graph that models the friendship between users. It consists of a set of user vertices and a set of connected edges that represent friendship between users. It is usually expressed as .in, Represents a collection of friend relationships between users.

[0322] In the social graph In each user Both Dimensional Features , this feature can be extracted from the user's own description information through pre-trained deep learning models such as Bert, Word2Vec, etc. The features of all users constitute the user feature matrix .user There are several friends, and the set of these friends is , and call them From neighbors in the social space. Since the friend relationship is a symmetric relationship, that is, if the user Yes Friends, then Too friends, thus the social graph is an undirected graph. Obviously, for any undirected graph, we can convert it into a directed graph by replacing each of its edges with a pair of directed edges in opposite directions.

[0323] In addition, each user It is also associated with a length of Sequence of conversations

[0324] .in, Indicates the current moment; Indicates user In time period The possible behaviors include video clicks, favorites, and ratings. They form a time-ordered session, and the video Viewed before , and said and There is a transfer relationship, denoted as The length of the session is recorded as ,The session lengths of different users may be the same or different. ,The interaction relationship between users and videos is usually modeled with the help of ,user-video bipartite graph.

[0325] Definition 3 (User-video Bipartite Graph) is a graph that models the interaction relationship between users and videos. It consists of a set of user vertices and a set of video vertices as well as a set of connected edges representing the interaction relationship between users and videos. It is usually expressed as . in, represents a vertex set; represents the set of user-video interaction edges.

[0326] In the user-video bipartite graph In each user The neighbor set of is composed of all videos that the user has watched, which can be recorded as , then ; Each video The neighbor set of is only composed of all users who have watched the video. Let's remember these neighbor sets from the social space as , then . Although the fact that the interaction action is directional, the user Videos they watched The connecting edge between them should be a directed edge, but in order to facilitate the subsequent propagation and fusion of the friend relationship in the social space and the transfer relationship in the project space in the two spaces through the interactive relationship, this application considers the user-video bipartite graph to be an undirected graph.

[0327] For each video , extract the visual, audio and text features of the video through existing deep learning technologies such as ResMNet, SoundNet and GloVe, and fuse them to obtain Dimensional Features , the features of all videos form the video feature matrix .

[0328] Definition 4 (Video Transition Graph) is a directed graph that models the transition relationship between videos. It consists of a set of video vertices and a set of directed connected edges that describe the transition relationship between videos. It is usually expressed as . in, is the set of vertices consisting of all videos; Represents the transition relationship between videos.

[0329] In the video recommendation system, a video transition graph can be constructed based on the historical access records of all users. Specifically, for each user Sequence of conversations , find all consecutively accessed video pairs and ,in is the starting video of the visit. is the video that is accessed later. Each video and Both represent a directed edge , the directed edges obtained from all video pairs of all users constitute the directed edge set of the video transfer relationship graph .video Neighborhood set By the incident neighbor set and the outgoing neighbor set Together they constitute .in, , .

[0330] Definition 5 (User-video Heterogeneous Directed Graph) is a directed graph containing two types of vertices: users and videos, and three types of edges: social edges between users, interaction edges between users and videos, and transfer edges between videos. It is used to model friend relationships in the social space (undirected), transfer relationships in the project space (directed), and interaction relationships between the social space and the project space (undirected).

[0331] The user-video heterogeneous directed graph can be viewed as a fusion of the social graph, the user-video bipartite graph, and the video transfer relationship graph. Formally, the user-video heterogeneous directed graph can be represented as a four-tuple . in, and Represent the vertex set and the connected edge set respectively; and Vertex type mapping function and connection edge type mapping function respectively; and They are vertex type sets and edge type sets respectively. Contains two types of vertices: user and video; It includes the social relationship edges between users (social), the interaction relationship edges between users and videos (interaction), and the transition relationship edges between videos (transition). According to Definitions 2 and 3, users Neighborhood set ; For videos For example, its neighbors include user vertices in the social space and video vertices in the project space, so its neighbor set .

[0332] In order to construct a user-video heterogeneous directed graph from the social relationships between users, the transfer relationships between items, and the interactive relationships between users and items in the video recommendation system, this application proposes a User-video Heterogeneous Directed Graph Construction Algorithm (UHDGC). The specific process is as follows:

[0333] 1. Initialization: Create an empty user-video heterogeneous directed graph , and define a set of vertex types and the set of connecting edge types ;

[0334] 2. Add vertices: Traverse all users and videos in the video recommendation system, and add each user and each video as a single vertex to the vertex set in the graph. and add them to the user vertex set according to the vertex type. and video vertex set , build vertex type mapping function ;

[0335] 3. Add edges: According to the social relationships between users, the interaction relationships between users and videos, and the transfer relationships between videos, add corresponding edges to the graph. Specifically:

[0336] (a) For the social relationship edges between users, traverse the social relationship data, for each pair of friend relationships, if the user Is a user 's friends, then in the picture Add a line from the vertex Point to the vertex A social edge and a link from vertex Point to the vertex The social side will and Add to the connected edge set respectively middle;

[0337] (b) For the interaction edge between a user and a video, we traverse all the historical sessions of all users. For a record of interaction between a user and a video, if the user With Video If there is interaction, then in the figure Add a line from the vertex Point to the vertex The interaction edges of Add to connected edge set middle;

[0338] (c) For the transition relationship between videos, we traverse all the historical sessions of all users and find the transition relationship between each adjacent video pair. and , then in the figure Add a line from the vertex Point to the vertex The transfer edge of middle;

[0339] (d) Construct a connection edge type mapping function based on the type of each edge ;

[0340] 4. Return result: Return the user-video heterogeneous directed graph .

[0341] The technical problem solved by this application is the problem of social-aware video recommendation based on conversation. Its definition is as follows:

[0342] Definition 6 (Session-based Social-aware Video Recommendation Problem) Given a user set , video collection and each user Historical conversation sequence and social networks .in, Indicates user The task of the session-based social recommendation problem is to predict the user’s video viewing behavior history and social information based on the above video viewing behavior history and social information. In the new session, that is, the current session In the next time step Most likely to visit videos , and the video has been visited before, it satisfies . Common recommendation algorithms return a A recommendation list of videos, which are ordered from high to low according to the probability of users watching them. recommend.

[0343] The following is a detailed description of each invention part of the VRSTT:

[0344] Heterogeneous Directed Graph Attention Network:

[0345] Generally speaking, there are problems of information imbalance, redundancy and failure in video recommendation systems:

[0346] (1) There are usually more interactive relationships between users and videos than social relationships. Direct vertex aggregation may lead to information imbalance between social knowledge and historical interactions, resulting in a weaker impact on social relationships.

[0347] (2) There are often some redundant or invalid relationships in the interaction between users and videos. The following are some situations and examples that may cause redundancy or invalidity in the interaction between users and videos: (a) Invalid or repeated click problems of some videos: Some videos may be considered invalid by users due to poor content quality, not to the user's taste, or other reasons. For example, some videos may be low-quality advertisements or very unpopular content. (b) Repeated clicks: Users may click on the same video multiple times, which may be because the user forgets that he has watched the video or makes a mistake. Repeated clicks may cause the recommendation system to over-recommend the same video, affecting the user experience.

[0348] (3) There may be some redundant or invalid relationships in the social relationship network. The following are some situations and examples that may lead to redundancy or invalidity in social relationships: (a) Invalid social connections: Some social connections may be invalid because they do not reflect real social relationships or interactions. For example, on some social platforms, users may form connections due to misoperation or meaningless attention, and such connections do not represent actual social relationships. (b) Redundant social interactions: Some social interactions may be redundant because they do not provide additional information or contribute value. For example, users frequently have similar interactions or repeated exchanges, but these interactions do not add new insights or information. (c) Outdated or invalid social relationships: Some social relationships may be outdated or invalid because they reflect past connections or interactions but no longer have actual meaning or value. For example, the interactions between users may decrease or stop over time, but these relationships still exist in the social network. (d) False or fictitious social relationships: In some cases, social relationships may be false or fictitious because they are formed based on false information or fake behavior. For example, some users may artificially increase the number of their social connections by purchasing followers or using bots, which are not real or valid. (e) Irrelevant or meaningless social interactions: Sometimes the interactions in social connections may be irrelevant or meaningless because they are not related to the user’s real interests or purposes. For example, some social interactions may be insignificant comments or likes without real communication or interaction.

[0349] From (1) to (3), we can see that both the user-video interaction network and the social relationship network have information redundancy or invalidity problems. These problems may affect the modeling and understanding of user interests, item representations, and social relationships, reduce recommendation diversity and accuracy, and thus affect user experience.

[0350] To solve the above problems, this application draws on the graph neural network and submodule theory to design a heterogeneous directed graph attention neural network (HDGAT) to effectively learn vertex feature representation by fine-grained modeling of relationships of different types, directions, and strengths between different vertices. The network contains The layer heterogeneous directed graph attention layer, each layer includes two functional components: neighbor sampling and aggregation update. The former is based on the submodule theory, and selects a set of basic neighbor subsets that are non-redundant, representative, and diverse from the neighbor set of each type of central vertex. The aggregation update component is based on the attention mechanism, and learns different weights for different neighbors selected by each type of central vertex to alleviate the imbalance. To this end, we maintain a user-video heterogeneous directed graph It is used to update the embedding vector representation of users or videos. The update process requires information from the neighboring vertices of users and videos. and Video In the The learned hidden layer representations are and For users , its neighbor set Contains two types of vertices: user vertices and video vertices; for videos , its neighbor set It also contains two types of vertices: video vertices and user vertices, so its neighbor set .

[0351] Neighbor sampling: Neighbor sampling refers to sampling from each type of central vertex The neighbor set of In the above example, we select a set of basic neighbor subsets that are non-redundant, representative, and diverse. , and satisfy the cardinality of the subset No more than When the neighbors of the central vertex have multiple types, it is required that there is at least one vertex of each type. In order to reduce the complexity of the network model and optimize the model application scenario, it is advisable to take the basic neighbor subset of the neighbor set of all vertices The upper limit of the cardinality is The above problem can be abstracted into an optimization problem: the optimization target is the basic neighbor subset The representativeness and diversity of the elements in; the constraint is the basic neighbor subset The base number does not exceed , and there is at least one vertex of each type.

[0352] A submodule function is a function defined on a collection, usually used to measure the "value" or "importance" of the collection. is called a submodule function if for any set and any element , the following conditions are met:

[0353] , (1)

[0354] Intuitively, this means that the marginal gain of adding elements to a smaller set decreases as the size of the set increases. Submodular functions have many useful properties in optimization problems, such as being able to approximate the optimal solution through greedy algorithms and being able to be used to maximize or minimize some objective function. Common submodular functions include covering functions and mutual information functions.

[0355] Inspired by the submodular function, this application proposes a covering function that comprehensively considers the topological space distance and the feature space distance as the submodular function. Intuitively, a subset with a higher function value indicates that for each item in the base set, there is a similar item in the selected subset. In other words, the selected subset is very diverse and representative of the base set. Together with the base neighbor subset Constraints, we will center the vertex The neighbor sampling problem modeling is expressed as follows:

[0356] (2)

[0357] (3)

[0358] Among them, formulas (2) and (3) represent the objective function and constraint conditions of the optimization problem respectively; the central vertex can be either a user vertex or a video vertex; Represents the feature similarity measure between two vertices, which can be Euclidean distance, cosine similarity, etc. " "Judge whether the two operand types are consistent and return a Boolean variable (true or false); Indicates forced type conversion, converting a Boolean variable to an integer variable; without loss of generality, this application uses Gaussian distance as the metric vertex and The feature similarity between:

[0359] , (4)

[0360] in, represents the norm of a vector, is the bandwidth parameter of the Gaussian kernel. Of course, other similarity measurement methods can also be applied, such as Gaussian distance, Manhattan distance, etc., which are not limited in this application.

[0361] Maximizing the submodule function under cardinality constraints is an NP-hard problem, which can usually be solved using a greedy algorithm. This application proposes a Subset Extraction algorithm based on Greedy Strategy with Approximation (SEGSA). Since the neighbor sampling problem solving process of different center vertex types is similar, we can use the center vertex as an example. Taking the time as an example, the proposed approximate solution algorithm flow is as follows:

[0362] 1 Initialization: will be used to record the basic neighbor subset Initialized to an empty set; will be used to record the unselected vertex set (maybe called the candidate set) Initialize to ;

[0363] 2-iteration greedy selection:

[0364] (a). For each vertex in the unselected vertex set , calculated according to the following formula and added The increment of the posterior objective function:

[0365] , (5)

[0366] (b) Select the vertex that maximizes the increment and satisfies the constraints. , and change it from Delete and add

[0367] ; If there are multiple vertices that meet the requirements, one of them is randomly selected.

[0368] (c). Update the basic neighbor subset and the candidate set: ; ;

[0369] (d). Check whether the constraints are satisfied; if so, exit the loop;

[0370] 3 Output result: Get user The sampled base neighbor subset .

[0371] Similarly, we can also use the REGSA algorithm to solve the video The sampled base neighbor subset It should be pointed out that our framework is applicable to any submodule functions, such as mutual information function, vertex cover function, edge cover function and facility location function.

[0372] Through the subset extraction approximate solution algorithm based on the greedy strategy, we Redundant neighbors and noise in the social space and / or item space are removed to obtain a set of basic neighbor subsets that are non-redundant, representative, and diverse, laying the foundation for effectively extracting user embeddings and item embeddings enhanced with social relations and transfer relations.

[0373] Aggregate update: According to definition (5), the user-video heterogeneous directed graph includes two types of vertices: user vertices and video vertices. The connecting edges between user vertices and their neighbors are all undirected edges; while the connecting edges between video vertices and their neighbor vertices include directed edges and undirected edges. As mentioned above, an undirected edge is equivalent to a pair of directed edges in opposite directions. Therefore, when the features of the central vertex are aggregated and updated, if there is an undirected edge between it and its neighbors, it is regarded as a bidirectional edge. After such equivalent processing, all the connecting edges of each vertex are directed edges, which will simplify the introduction of the basic principles of the directed graph attention neural network proposed in this application.

[0374] For user-video heterogeneous directed graph Each vertex in , assuming and any of its neighbor vertices The The hidden layer representations of the layer heterogeneous directed graph attention neural network layer output are Dimensional vector and ; According to the principle of attention, The aggregation update operation of the layer-heterogeneous directed graph attention neural network layer can be divided into the following three steps:

[0375] 1 Incoming neighbor aggregation: For each vertex , calculate the aggregate features of its incident neighbor vertices:

[0376] , (6)

[0377] in, is the vertex For the incident neighbor vertices The attention weight of Represents the intersection of the incident neighbor set and the basic neighbor subset, ensuring that we only perform convolution operations on representative and diverse basic incident neighbors. It is calculated by the following formula:

[0378] , (7)

[0379] , (8)

[0380] in, Represents a vertex and The weight of the connection edge between them; is the activation function; Represents the concatenation operation of a vector.

[0381] 2 Outgoing neighbor aggregation: For each vertex , calculate the aggregate features of its incident neighbor vertices:

[0382] , (9)

[0383] in, is the vertex For outgoing neighbor vertices The attention weight of represents the intersection of the outgoing neighbor set and the basic neighbor subset, ensuring that we only perform convolution operations on representative and diverse basic outgoing neighbors. The calculation principle is similar, and the formula will not be repeated here.

[0384] 3. Feature transformation: The incoming neighbor features and outgoing neighbor features are linearly transformed and activated respectively to obtain the updated vertex representation.

[0385] , (10)

[0386] in, represents a nonlinear activation function; All of them are model parameters to be learned.

[0387] when At that time, Layer Heterogeneous Directed Graph Attention Neural Network Layer Based on User Feature Matrix and the video feature matrix is the input. When the heterogeneous directed graph attention network outputs the user embedding vector representation enhanced with social semantics and transfer relations and video embedding vector representation .in, Denotes the dimension of the user and video embedding vector representation.

[0388] The above heterogeneous directed graph embedding captures global knowledge such as friend relationships in social space, transfer relationships in project space, and all historical conversations between users and projects at one time, greatly improving the accuracy and efficiency of modeling users' long-term interest preferences. However, the above heterogeneous directed graph embedding does not capture the contextual information of a specific session. However, for a session-based personalized video recommendation system, user behavior in the current session is very important for capturing users' dynamic interests. To this end, this application also designs a personalized session embedding component to extract dynamic interest preferences from the user's current session. It involves two tasks: The first task is to perform an operation called embedding lookup to extract relevant embeddings of the current user and all items in the current session from the heterogeneous graph embedding component. These extracted heterogeneous embeddings include: (1) the current user The heterogeneous graph embedding of , and (2) the heterogeneous graph embedding of all items in the current session, represented as The second task is to perform an operation called personalized session embedding extraction to compute a personalized session-specific embedding based on the extracted heterogeneous graph embedding. Specifically, this operation is a function Θ that takes as input the user embedding and the embeddings of all videos in the current session and computes a personalized session-specific embedding .Right now Any existing conversational recommendation model can be easily embedded into our framework. First, existing personalized conversational embedding models can be directly embedded into our function Θ since heterogeneous graph embeddings can serve as the user and item embeddings required by these models.

[0389] The personalized session embedding component learns the sequential and transition relations within the session: The personalized session component generates personalized session-specific embeddings from multiple perspectives, capturing the user's dynamic interest preferences and the contextual attributes of the items. It involves two tasks: The first task is to perform an operation called embedding lookup to extract the relevant embeddings of the current user and all the items in the current session from the heterogeneous directed graph embedding component. These extracted heterogeneous embeddings include: (1) the current user The heterogeneous directed graph embedding of , and (2) the heterogeneous directed graph embedding of all items in the current session, represented as The second task is to perform an operation called personalized session embedding extraction, which computes a personalized session embedding based on the extracted heterogeneous directed graph embedding. This operation is a function Θ that takes as input the user embedding and the embeddings of all videos in the current session and computes a personalized session embedding ,Right now .

[0390] Transformer is a powerful attention mechanism model suitable for processing sequence data. It can effectively capture long-distance dependencies in the sequence, which include both the order relationship between videos in a session and the transfer relationship between videos in a session. In view of this, this application proposes a Transformer encoder-based Session Representation Learning NeuralNetwork (TSRLN), which consists of a session preprocessor and a Transformer encoder. The former is responsible for obtaining the sequence information from each user. Current session The first one removes redundant or invalid videos and obtains a basic session of fixed length; the second one is responsible for learning the sequential relationship between videos from the preprocessed session to obtain the session representation, that is, the user's dynamic interest preference.

[0391] Conversation preprocessor: The Transformer model performs well in natural language processing and sequence modeling, but its original design is mainly used to process fixed-length sequences, such as natural language sentences. In conversational recommendation systems, the length of a session can be dynamic and the number of items in each session can be different. This makes it challenging to directly apply the standard Transformer to conversation representation extraction. To this end, we design a conversation preprocessor that can be implemented based on the neighbor sampling function of the submodule theory. The difference is that we agree that the cardinality of the sampled base session is fixed to Instead of just setting an upper limit on the length, the session sampling problem based on submodule theory can be modeled as follows:

[0392] , (11)

[0393] , (12)

[0394] The above problem can be solved by the REGSA algorithm proposed above, which will not be described here. Current session The fixed length obtained by the session preprocessor is The basis of the current session . Because the preprocessor may disrupt the current session The order in which the videos are accessed in Rearrange the order in and remember the sorted basic current session as .

[0395] The Transformer encoder learns the sequential and transition relations within a session. , let’s assume that the last layer of the heterogeneous directed graph attention neural network HDGAT is The social semantics enhanced embedding vector output by the layer is expressed as ; Add position encoding according to the following formula to allow the Transformer model to distinguish videos at different positions in the current session:

[0396] . (13)

[0397] For each discrete value , we assign a unique trainable parameter vector As a video Then it is combined with the video The embedding vector representation of Add and perform linear transformation to get The sequence-aware embedding vector representation of the video:

[0398] , (14)

[0399] in, is the parameter matrix to be learned. Represents the dimension of the order-aware embedding vector. Base current session of The sequence-aware embedding vectors are stacked to form , feeding them into the Transformer encoder to extract the base current session representation.

[0400] 1 Single-head self-attention mechanism. Generally, the single-head self-attention mechanism can be defined as:

[0401] , (15)

[0402] in, , and The input matrices representing query, key, and value respectively; Indicates the number of columns of the key matrix.

[0403] 2 Multi-head self-attention mechanism. The self-attention mechanism we proposed includes self-attention heads, which in parallel perform Conducting conversational representation learning:

[0404] , (16)

[0405] , (17)

[0406] , (18)

[0407] in, are parameter matrices to be learned; , and They are and Along the last dimension A division.

[0408] 3 Output session representation. For user The basis of the current session is , let’s assume that the sequence output by the Transformer encoder is .in, Represents the length of the output sequence, which is the same as the length of the input sequence They may be equal or different. To extract the embedding vector representation of the entire session and use it as the user's dynamic interest preference , this application proposes the following four methods for extracting session representations: (a) End method. The output of the last time step of the above sequence is used as the embedding vector representation of the entire session, then:

[0409] , (19)

[0410] in, Indicates taking the last element of the sequence. (b) Mean method. This method uses a weighted average of all vectors in the sequence:

[0411] (20)

[0412] (c) Maximum method. This method takes the maximum value of all vectors in the sequence and returns a new vector:

[0413] , (twenty one)

[0414] (d) Attention method. This method is based on the attention mechanism and dynamically obtains the weighted average of all vectors in the sequence by learning the attention weights. The steps of this method are as follows:

[0415] , (twenty two)

[0416] , (twenty three)

[0417] , (twenty four)

[0418] in, is the parameter matrix to be learned, is the activation function (such as ReLU), is a vector Score, is a vector The attention weight.

[0419] The above-mentioned personalized conversation embedding component uses the conversation preprocessor to remove information redundancy and noise in the conversation, retaining diverse and representative videos, laying the foundation for the subsequent Transformer encoder-based conversation representation network to extract accurate user dynamic interest preferences; and the Transformer encoder-based conversation representation network effectively captures the long-distance dependencies and transfer relationships in the current conversation after preprocessing, and can extract accurate user dynamic interest preferences.

[0420] User interest preference fusion component: The user embedding output by the heterogeneous directed graph embedding component integrates knowledge such as the influence of friends in the social space, the transfer relationship between videos in the project space, and all historical conversations between users and videos, which can be regarded as the user's long-term interest preference. Many previous studies have shown that the user's long-term interest preference is useful for predicting the next video that the user will watch. To this end, we designed a user interest preference fusion component to fuse the user's long-term interest preference output by the heterogeneous directed graph embedding component and the user's dynamic interest preference output by the personalized session embedding component to obtain the user's final interest preference representation. :

[0421] , (25)

[0422] in, Indicates the user's long-term interest preferences; are the network parameters to be learned; “[;]” represents the vector connection and splicing operation.

[0423] The above-mentioned user interest preference fusion component integrates user long-term interest preferences and user dynamic interest preferences to obtain user comprehensive interest preferences that fully consider knowledge such as the influence of friends in the social space and the transfer relationship between videos in the project space, greatly improving the accuracy of video recommendations.

[0424] Video recommendation: To generate the probability distribution of the video at the next time step, for each video with an embedded , calculate its social semantics and transfer relationship enhanced embedding vector representation And the user's final interest preference expression The similarity score between:

[0425] , (26)

[0426] The score vector for all videos Then, the score vector is normalized by softmax to obtain the probability distribution:

[0427] , (27)

[0428] Among them, the probability distribution Characterizes the user The probability of watching each video in the next time step of the current session. Sort the probability distribution from high to low and return the previous The high-probability videos are selected as candidate sets for the next time step video. To avoid repeated recommendations, the returned videos can be constrained to be videos that the user has not watched.

[0429] Model training method:

[0430] set up Represents the true probability distribution of the next video, which is a vector using one-hot encoding. Loss function Defined as predicted value With the true value The cross entropy between:

[0431] . (28)

[0432] Then, according to specific strategies such as Stochastic Gradient Descent (SGD), Momentum Gradient Descent (MGD), Nesterov Momentum, AdaGrad, RMSprop and Adam (Adaptive Moment Estimation) or Batch Gradient Descent (BGD), the parameters of the proposed VRSTT network are modified and updated to optimize the loss function value until the loss function reaches the minimum value or the number of training times reaches the specified maximum value, the training ends.

[0433] FPGA-based universal heterogeneous accelerator design: We use FPGA to design a universal heterogeneous accelerator for heterogeneous directed graph convolutional neural networks and Transformer encoders. In order to maximize the reuse of FPGA computing and storage resources, this application uniformly maps the graph convolution in the heterogeneous directed graph convolutional neural network and the multi-head self-attention mechanism in the Transformer encoder, so that the two types of operations can be completed using the same computing array. In terms of timing structure design, reconfigurable bit-serial matrix multiplication overlay (BISMO) is used. In terms of spatial structure, a three-dimensional pulsating array structure can be used.

[0434] Figure 4 is an optional FPGA heterogeneous accelerator architecture diagram according to an embodiment of the present application, such as Figure 4The FPGA-based general heterogeneous accelerator proposed in this paper is mainly composed of a scheduling module, a general computing module, a nonlinear and normalization module, a data operation module, an off-chip memory, an input cache, and an output cache. The interconnection relationship between different modules is shown in FIG. Figure 4 As shown. The off-chip memory is used to store the instructions and data sent by the CPU to the heterogeneous accelerator via the PCIE bus; the scheduling module decodes the received instructions and distributes the decoded instructions to the data operation module, the nonlinear and normalization module or the general computing module; the general computing module is responsible for executing the graph convolution operation in the heterogeneous graph convolutional neural network and the multi-head self-attention operation in the Transformer encoder, and its calculation results are finally output to the off-chip storage via the output cache; the data operation module is used to receive scheduling core control signals, state feedback, data handling and process state control.

[0435] The main workflow of the proposed general heterogeneous accelerator is as follows: the host computer first loads the instruction set, weight data, feature data and / or graph topology data, writes them into the off-chip memory of the heterogeneous accelerator, and then starts running. After receiving the start command, the scheduling core pre-fetches some instructions from the off-chip memory to the instruction cache, then starts to read and decode the instructions from it, and dispatches the instructions to the general computing module, nonlinear and normalization module, data operation module, etc. for execution according to the different types of instructions, until the program end (END) instruction sent by the host is encountered, and the operation ends.

[0436] The target device of the general heterogeneous accelerator designed in this application is F26A FPGA. The accelerator is synthesized using Vivado v2021.1, and the system operating frequency is 200 MHz.

[0437] Due to the low power consumption, low latency, and customizability of FPGA, this application customized a general acceleration architecture by analyzing the core operations of heterogeneous directed graph convolutional neural networks and Transformer encoders, which can greatly improve the training and reasoning efficiency of the proposed video recommendation system, significantly save energy consumption, and reduce costs.

[0438] Compared with the existing video recommendation methods, the video recommendation system based on social semantics and transfer relationship enhanced Transformer (VRSTT) proposed in this paper has the following beneficial effects: (1) It comprehensively considers the influence of social relations in social space, video transfer relations in project space and the interaction between users and videos on user interest preferences, more accurately describes user interest preferences, and greatly improves the accuracy of video recommendation. (2) With the help of the subset extraction approximate solution algorithm based on the greedy strategy, the information redundancy and noise in the heterogeneous directed graph and the user's current session are removed, while improving the accuracy of user and video embedding and increasing the diversity of video recommendation. (3) The proposed method only uses the user's historical session to generate the user embedding enhanced with social semantics and transfer relations in the process of heterogeneous directed graph embedding. When making recommendations for the current session, it avoids the repeated learning of social friends' historical sessions to integrate social influences, which greatly reduces the computational complexity of the network model. (4) With the help of low-latency, low-power FPGA, we designed a general-purpose heterogeneous accelerator for heterogeneous directed graph convolutional neural networks and Transformer encoders, which can significantly improve the training and inference efficiency of heterogeneous directed graph convolutional neural networks and Transformer encoders, and can significantly save energy consumption and reduce costs.

[0439] Figure 5 is an optional recommended device hardware schematic diagram according to an embodiment of the present application, such as Figure 5 As shown, the host computer in the figure is responsible for running Figure 3 The user directed graph construction component in the figure is used to obtain a user-video heterogeneous directed graph. The heterogeneous accelerator in the figure is responsible for running the heterogeneous directed graph convolutional neural network, the personalized session embedding component, the user interest preference fusion component and the prediction component. The implementation method can be: loading the video recommendation network model and the video recommendation data set from the host's disk to the main memory; further moving the video recommendation network model in the main memory to the hardware accelerator memory; the host preprocesses the video recommendation data set, constructs a user-video heterogeneous directed graph from the user's social network and browsing history, and divides the user browsing data into a training set, a test set and a validation set according to the time sequence and 0.6:0.2:0.2; the host sends a video recommendation network model training instruction to the hardware accelerator, and the hardware accelerator loads the training set and validation set data from the main memory in small batches to complete the training of the video recommendation network; the hardware accelerator continues to read the test set data from the host main memory, completes the performance test of the video recommendation network model, and returns the test results to the host; after receiving the recommendation results from the hardware accelerator, the host displays them to the user in a certain form.

[0440] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0441] In the present embodiment, a multimedia resource recommendation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0442] Figure 6 is a structural block diagram of a multimedia resource recommendation device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0443] a first prediction module, configured to predict a first account feature of a target account and a reference feature of each multimedia resource included in the first browsing information according to first browsing information, wherein the first browsing information is used to indicate a temporal relationship between multimedia resources browsed by the target account and a reference account associated with the target account within a first time period before a current moment, the first account feature is used to indicate a browsing preference of the target account for multimedia resources under the influence of the reference account, and the reference feature is used to indicate a resource access relationship of the corresponding multimedia resource in the first browsing information;

[0444] a second prediction module, configured to predict a second account feature of the target account based on second browsing information and the reference feature, wherein the second browsing information is used to indicate a browsing situation of multimedia resources of the target account in a second time period, the first time period is longer than the second time period and the first time period includes the second time period, and the second account feature is used to indicate an influence of browsing the multimedia resources in the second browsing information on the browsing preference of the target account for multimedia resources;

[0445] A third prediction module, configured to predict a target browsing feature of the target account based on the first account feature and the second account feature, wherein the target browsing feature is used to indicate a multimedia resource browsing preference of the target account at a current moment;

[0446] The recommendation module is used to recommend target multimedia resources matching the target browsing characteristics to the target account.

[0447] Through the above content, by using the first browsing information for prediction, the first account feature indicating the target account's browsing preference for multimedia resources under the influence of the reference account and the reference feature indicating the resource access relationship of the multimedia resources in the first browsing information are obtained, and then the second account feature of the target account is predicted according to the second browsing information and the reference feature, and the feature indicating the influence of the multimedia resources in the second browsing information on the browsing preference of the target account is obtained, and then the target browsing feature of the target account predicted by using the first account feature and the second account feature is integrated with the factors of the resource browsing preference feature under the influence of the social relationship of the account, and the factors of the resource browsing preference feature under the influence of the multimedia resources accessed by the target account itself in the second time period are integrated, so that the target browsing feature is closer to the target account's actual current access preference for multimedia resources, and then the target multimedia resources matched with the target browsing feature pushed to the target account can better meet the target account's current demand for multimedia resources. The problem of low matching between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources can be solved, so as to achieve the effect of improving the matching between the multimedia resources recommended for the target account and the target account's current resource preference for multimedia resources.

[0448] Optionally, the first prediction module includes:

[0449] A construction unit, configured to construct a first relationship graph according to the first browsing information, wherein the first relationship graph includes a first vertex and a first edge between first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, and a vertex corresponding to the multimedia resource included in the first browsing information, and the first edge includes an edge for representing an association relationship between accounts, an edge for an access relationship between an account and a multimedia resource, and an edge for an access timing relationship between multimedia resources;

[0450] A first extraction unit is configured to extract target vertex features of a target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence, wherein the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is configured to generate vertex features of the target vertex according to vertex features of a reference vertex in the current first relationship graph, and to update features of the target vertex in the first relationship graph using the vertex features of the target vertex extracted by the current feature extraction layer, wherein the reference vertex is a vertex in the first relationship graph connected to the target vertex through the first edge;

[0451] A determination unit is used to determine the target vertex feature of the target vertex corresponding to the target account as the first account feature, and to determine the target vertex feature of the target vertex corresponding to each multimedia resource included in the first browsing information as the reference feature.

[0452] Optionally, the first extraction unit is used to:

[0453] Calling a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records the reference vertices in the first relationship graph whose association degree with the target vertex is greater than the target association degree, and the target feature extraction layer is an extraction layer among the multiple feature extraction layers that is to perform feature extraction on the target vertex in the first relationship graph;

[0454] Calling the target feature extraction layer to extract features of the vertex features currently possessed by the reference vertices included in the target vertex set, and obtaining the reference vertex features of the target vertices output by the target feature extraction layer;

[0455] In a case where the target feature extraction layer is the last layer of a plurality of sequentially connected feature extraction layers, the reference vertex feature is determined as the target vertex feature.

[0456] Optionally, the first extraction unit is used to:

[0457] Performing feature aggregation on the second vertex feature currently possessed by the second vertex in the first relationship graph according to the first vertex feature currently possessed by the target vertex in the first relationship graph to obtain a first aggregated feature, and aggregating the third vertex feature currently possessed by the third vertex in the first relationship graph according to the first vertex feature of the target vertex to obtain a second aggregated feature, wherein the second vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an incident end, and the third vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an exit end;

[0458] The first aggregated features and the second aggregated features are subjected to feature fusion to obtain the reference vertex features.

[0459] Optionally, the first extraction unit is used to:

[0460] Calculating a first similarity between the first vertex feature and the second vertex feature;

[0461] Converting a first weight parameter of each of the second vertices according to the first similarity;

[0462] The first weight parameter is used to perform a weighted sum calculation on the second vertex features to obtain the first aggregate feature.

[0463] Optionally, the first extraction unit is used to:

[0464] Calculating a second similarity between the first vertex feature and the third vertex feature;

[0465] Converting a second weight parameter of each of the third vertices according to the second similarity;

[0466] The second weight parameter is used to perform weighted sum calculation on the third vertex feature to obtain the second aggregate feature.

[0467] Optionally, the first extraction unit is used to:

[0468] Extracting an initial vertex set of the target vertex from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph connected to the target vertex through the first edge;

[0469] A candidate number of candidate vertices is screened out from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies the target dependency relationship, and the candidate number is less than the number of vertices included in the initial vertex set.

[0470] Optionally, the first extraction unit is used to:

[0471] Using the vertex features of each vertex included in the first vertex set of the current screening round, calculating the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change of the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set;

[0472] Extracting vertices whose increment values ​​are greater than or equal to the target increment value from the first vertex set and adding them to the second vertex set;

[0473] When the number of candidate vertices included in the second vertex set to which vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, the third vertex set is determined as the target vertex set.

[0474] Optionally, the construction unit is used to:

[0475] Creating a fourth vertex corresponding to the target account, a fifth vertex corresponding to the reference account, and a sixth vertex corresponding to each multimedia resource included in the first browsing information, wherein the first vertex includes the fourth vertex, the fifth vertex, and the sixth vertex;

[0476] A second edge is created between the corresponding fourth vertex and the fifth vertex according to the association relationship between the target account and the reference account, a third edge is created between the corresponding fifth vertex according to the association relationship between the reference accounts, a fourth edge is created between the corresponding fourth vertex and the sixth vertex according to the access relationship of the target account to the multimedia resource, a fifth edge is added between the corresponding fifth vertex and the sixth vertex according to the access relationship of the reference account to the multimedia resource, and an initial directed edge is created between the sixth vertices according to the access sequence of the account to the multimedia resource in the first browsing information, wherein the sixth vertex corresponding to the multimedia resource with an earlier account access sequence in the first browsing information is connected to the output end of the initial directed edge, and the sixth vertex corresponding to the multimedia resource with a later account access sequence in the first browsing information is connected to the input end of the initial directed edge, and the first edge includes the second edge, the third edge, the fourth edge, the fifth edge and the initial directed edge.

[0477] Optionally, the second prediction module includes:

[0478] A second extraction module, configured to extract a first resource feature of each multimedia resource in the second browsing information from the reference feature;

[0479] a first screening unit, configured to screen out a plurality of first multimedia resources from the multimedia resources included in the second browsing information according to the first resource feature, wherein the first multimedia resources are multimedia resources in the second browsing information whose dependency relationships with other multimedia resources other than the first multimedia resource meet target dependency relationship conditions;

[0480] A prediction unit is used to predict the second account feature according to the first resource feature corresponding to the first multimedia resource.

[0481] Optionally, the prediction unit is used to:

[0482] Acquire a position feature of each of the first multimedia resources among the plurality of the first multimedia resources, wherein the position feature is used to indicate a temporal relationship between the first multimedia resources in the second browsing information;

[0483] Concatenate the location feature of each of the first multimedia resources with the first resource feature to obtain a second resource feature;

[0484] Inputting a plurality of the second resource features into a target prediction model according to a temporal relationship of the first multimedia resource to obtain a third resource feature corresponding to each of the second resource features, wherein the target prediction model is used to extract an association relationship between the plurality of the second resource features, and output the third resource feature for each of the second resource features according to the association relationship between the second resource features, the third resource feature being used to indicate a resource feature of the current multimedia resource under the influence of the second resource features of other multimedia resources except the current multimedia resource among the plurality of the first multimedia resources;

[0485] The second account characteristic is predicted based on a plurality of the third resource characteristics.

[0486] Optionally, the prediction unit is used to:

[0487] Inputting the plurality of second resource features into a feature extraction layer including a plurality of attention modules according to the temporal relationship of the first multimedia resource, wherein the plurality of attention modules are used to extract initial relationship features indicating the feature relationship between the second resource features from a plurality of feature dimensions, and the feature extraction layer is used to perform feature merging on the plurality of initial relationship features output by the plurality of attention modules to obtain a target relationship feature;

[0488] The target relationship feature is input into a feature prediction layer to obtain a third resource feature output by the feature prediction layer corresponding to each of the second resource features, wherein the target prediction model includes the feature extraction layer and the feature prediction layer connected in sequence.

[0489] Optionally, the prediction unit is used to perform one of the following operations:

[0490] Determine the third resource feature corresponding to the multimedia resource that is at the last position in the temporal order among the plurality of the first multimedia resources as the second account feature;

[0491] Extracting the maximum feature component from the feature components of each of the third resource features in multiple vector dimensional spaces; merging the maximum feature components of multiple third resource features to obtain the second account feature;

[0492] According to the characteristic relationship between the plurality of the third resource characteristics, a corresponding characteristic weight is assigned to each of the third resource characteristics; and the characteristic weight is used to perform a weighted sum calculation on the plurality of the third resource characteristics to obtain the second account characteristic;

[0493] An average value is calculated for a plurality of the third resource characteristics to obtain the second account characteristic.

[0494] Optionally, the first screening unit is used to:

[0495] Calculating a reference resource parameter of each of the multimedia resources in the second browsing information using the first resource feature, wherein the reference resource parameter is used to indicate a degree of dependency between the current multimedia resource in the second browsing information and other multimedia resources except the current multimedia resource;

[0496] The first multimedia resources whose reference resource parameters are greater than or equal to the target resource parameters are screened out from the second browsing information.

[0497] Optionally, the third prediction module includes:

[0498] A concatenation unit, configured to concatenate the first account feature and the second account feature to obtain a concatenated feature;

[0499] The fusion unit is used to perform feature fusion on the splicing features using a feature fusion model to obtain the target browsing features output by the feature fusion model, wherein the feature fusion model records the conversion relationship between the splicing features and the browsing features.

[0500] Optionally, the recommendation module includes:

[0501] A calculation unit, used for calculating the target similarity between the target browsing feature and the target resource feature of each multimedia resource in the multimedia resource library;

[0502] A second screening unit, configured to screen out the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library;

[0503] The recommendation unit is used to recommend the target multimedia resource to the target account.

[0504] Optionally, the calculating the target similarity between the target browsing feature and the target resource feature of each multimedia resource in the multimedia resource library includes:

[0505] Generate a fourth resource feature of the multimedia resource in the first browsing information and a fifth resource feature of the multimedia resource in the multimedia resource library using a feature generation model;

[0506] Detecting feature difference information between the fourth resource feature and the fifth resource feature;

[0507] Using the feature adjustment amount corresponding to the feature difference information, feature adjustment is performed on the reference feature of the multimedia resource in the first browsing information to obtain the target resource feature, wherein the reference feature is used to indicate the account feature of the account browsing the multimedia resource in the first browsing information;

[0508] The target similarity between the target browsing feature and the target resource feature is calculated.

[0509] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0510] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0511] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0512] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0513] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0514] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0515] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0516] The embodiments of the present application also provide a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0517] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0518] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0519] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for recommending multimedia resources, characterized in that: include: Predicting a first account feature of a target account and a reference feature of each multimedia resource included in the first browsing information based on the first browsing information, wherein the first browsing information is used to indicate a temporal relationship between multimedia resources browsed by the target account and a reference account associated with the target account within a first time period before a current moment, the first account feature is used to indicate the browsing preference of the target account for multimedia resources under the influence of the reference account, the reference feature is used to indicate a resource access relationship of the corresponding multimedia resource in the first browsing information, and the reference feature is used to characterize an access relationship of an account to a multimedia resource through the resource access feature possessed by the account; Predicting a second account feature of the target account according to the second browsing information and the reference feature, wherein the second browsing information is used to indicate a browsing situation of multimedia resources of the target account in a second time period, the first time period is longer than the second time period and the first time period includes the second time period, and the second account feature is used to indicate an influence of browsing the multimedia resources in the second browsing information on the browsing preference of the target account for multimedia resources; Predicting a target browsing feature of the target account according to the first account feature and the second account feature, wherein the target browsing feature is used to indicate a multimedia resource browsing preference of the target account at a current moment; A target multimedia resource matching the target browsing feature is recommended to the target account.

2. The method according to claim 1, characterized in that The predicting, based on the first browsing information, the first account feature of the target account and the reference feature of each multimedia resource included in the first browsing information includes: Constructing a first relationship graph according to the first browsing information, wherein the first relationship graph includes a first vertex and a first edge between first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, and a vertex corresponding to the multimedia resource included in the first browsing information, and the first edge includes an edge for representing an association relationship between accounts, an edge of an access relationship between an account and a multimedia resource, and an edge of an access timing relationship between multimedia resources; Extracting target vertex features of a target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence, wherein the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is used to generate vertex features of the target vertex according to vertex features of a reference vertex in the current first relationship graph, and updating features of the target vertex in the first relationship graph using the vertex features of the target vertex extracted by the current feature extraction layer, wherein the reference vertex is a vertex in the first relationship graph connected to the target vertex through the first edge; The target vertex feature of the target vertex corresponding to the target account is determined as the first account feature, and the target vertex feature of the target vertex corresponding to each multimedia resource included in the first browsing information is determined as the reference feature.

3. The method according to claim 2, characterized in that The extracting the target vertex feature of the target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence comprises: Calling a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records the reference vertices in the first relationship graph whose association degree with the target vertex is greater than the target association degree, and the target feature extraction layer is an extraction layer among the multiple feature extraction layers that is to perform feature extraction on the target vertex in the first relationship graph; Calling the target feature extraction layer to extract features of the vertex features currently possessed by the reference vertices included in the target vertex set, and obtaining the reference vertex features of the target vertices output by the target feature extraction layer; In a case where the target feature extraction layer is the last layer of a plurality of sequentially connected feature extraction layers, the reference vertex feature is determined as the target vertex feature.

4. The method according to claim 3, characterized in that The calling of the target feature extraction layer to extract features of the vertex features currently possessed by the vertices included in the target vertex set to obtain reference vertex features of the target vertices output by the target feature extraction layer includes: Performing feature aggregation on the second vertex feature currently possessed by the second vertex in the first relationship graph according to the first vertex feature currently possessed by the target vertex in the first relationship graph to obtain a first aggregated feature, and aggregating the third vertex feature currently possessed by the third vertex in the first relationship graph according to the first vertex feature of the target vertex to obtain a second aggregated feature, wherein the second vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an incident end, and the third vertex is a vertex in the first relationship graph that is included in the target vertex set and has the target vertex as an exit end; The first aggregated features and the second aggregated features are subjected to feature fusion to obtain the reference vertex features.

5. The method according to claim 4, characterized in that The step of performing feature aggregation on the first vertex feature currently possessed by the target vertex in the first relationship graph and the second vertex feature currently possessed by the second vertex in the first relationship graph to obtain a first aggregated feature includes: Calculating a first similarity between the first vertex feature and the second vertex feature; Converting a first weight parameter of each of the second vertices according to the first similarity; The first weight parameter is used to perform a weighted sum calculation on the second vertex features to obtain the first aggregate feature.

6. The method according to claim 4, characterized in that The step of aggregating the third vertex feature currently possessed by the third vertex in the first relationship graph according to the first vertex feature of the target vertex to obtain a second aggregate feature includes: Calculating a second similarity between the first vertex feature and the third vertex feature; Converting a second weight parameter of each of the third vertices according to the second similarity; The second weight parameter is used to perform weighted sum calculation on the third vertex feature to obtain the second aggregate feature.

7. The method according to claim 3, characterized in that The calling of the target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph includes: Extracting an initial vertex set of the target vertex from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph connected to the target vertex through the first edge; A candidate number of candidate vertices is screened out from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies the target dependency relationship, and the candidate number is less than the number of vertices included in the initial vertex set.

8. The method according to claim 7, characterized in that The step of selecting a candidate number of candidate vertices from the initial vertex set according to the vertex features currently possessed by the vertices in the initial vertex set to obtain the target vertex set includes: Using the vertex features of each vertex included in the first vertex set of the current screening round, calculating the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change of the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set; Extracting vertices whose increment values ​​are greater than or equal to the target increment value from the first vertex set and adding them to the second vertex set; When the number of candidate vertices included in the second vertex set to which vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, the second vertex set is determined as the target vertex set.

9. The method according to claim 2, characterized in that: The step of constructing a first relationship graph according to the first browsing information includes: Creating a fourth vertex corresponding to the target account, a fifth vertex corresponding to the reference account, and a sixth vertex corresponding to each multimedia resource included in the first browsing information, wherein the first vertices include the fourth vertex, the fifth vertex, and the sixth vertex; A second edge is created between the corresponding fourth vertex and the fifth vertex according to the association relationship between the target account and the reference account, a third edge is created between the corresponding fifth vertex according to the association relationship between the reference accounts, a fourth edge is created between the corresponding fourth vertex and the sixth vertex according to the access relationship of the target account to the multimedia resource, a fifth edge is added between the corresponding fifth vertex and the sixth vertex according to the access relationship of the reference account to the multimedia resource, and an initial directed edge is created between the sixth vertices according to the access sequence of the account to the multimedia resource in the first browsing information, wherein the sixth vertex corresponding to the multimedia resource with an earlier account access sequence in the first browsing information is connected to the output end of the initial directed edge, and the sixth vertex corresponding to the multimedia resource with a later account access sequence in the first browsing information is connected to the input end of the initial directed edge, and the first edge includes the second edge, the third edge, the fourth edge, the fifth edge and the initial directed edge.

10. The method according to claim 1, characterized in that The predicting the second account feature of the target account according to the second browsing information and the reference feature includes: extracting a first resource feature of each multimedia resource in the second browsing information from the reference feature; Filtering a plurality of first multimedia resources from the multimedia resources included in the second browsing information according to the first resource feature, wherein the first multimedia resources are multimedia resources in the second browsing information whose dependency relationships with other multimedia resources other than the first multimedia resource satisfy target dependency relationship conditions; The second account feature is predicted based on the first resource feature corresponding to the first multimedia resource.

11. The method according to claim 10, characterized in that The predicting the second account feature according to the first resource feature corresponding to the first multimedia resource includes: Acquire a position feature of each of the first multimedia resources among the plurality of the first multimedia resources, wherein the position feature is used to indicate a temporal relationship between the first multimedia resources in the second browsing information; Concatenate the location feature of each of the first multimedia resources with the first resource feature to obtain a second resource feature; Inputting a plurality of the second resource features into a target prediction model according to a temporal relationship of the first multimedia resource to obtain a third resource feature corresponding to each of the second resource features, wherein the target prediction model is used to extract an association relationship between the plurality of the second resource features, and output the third resource feature for each of the second resource features according to the association relationship between the second resource features, the third resource feature being used to indicate a resource feature of the current multimedia resource under the influence of the second resource features of other multimedia resources except the current multimedia resource among the plurality of the first multimedia resources; The second account characteristic is predicted based on a plurality of the third resource characteristics.

12. The method according to claim 11, characterized in that The step of inputting a plurality of the second resource features into a target prediction model according to the temporal relationship of the first multimedia resource to obtain a third resource feature corresponding to each of the second resource features includes: Inputting the plurality of second resource features into a feature extraction layer including a plurality of attention modules according to the temporal relationship of the first multimedia resource, wherein the plurality of attention modules are used to extract initial relationship features indicating the feature relationship between the second resource features from a plurality of feature dimensions, and the feature extraction layer is used to perform feature merging on the plurality of initial relationship features output by the plurality of attention modules to obtain a target relationship feature; The target relationship feature is input into a feature prediction layer to obtain a third resource feature output by the feature prediction layer corresponding to each of the second resource features, wherein the target prediction model includes the feature extraction layer and the feature prediction layer connected in sequence.

13. The method according to claim 11, characterized in that The predicting the second account feature according to the plurality of the third resource features comprises one of the following: Determine the third resource feature corresponding to the multimedia resource that is at the last position in the temporal order among the plurality of the first multimedia resources as the second account feature; Extracting the maximum feature component from the feature components of each of the third resource features in multiple vector dimensional spaces; merging the maximum feature components of multiple third resource features to obtain the second account feature; According to the characteristic relationship between the plurality of the third resource characteristics, a corresponding characteristic weight is assigned to each of the third resource characteristics; and the characteristic weight is used to perform a weighted sum calculation on the plurality of the third resource characteristics to obtain the second account characteristic; An average value is calculated for a plurality of the third resource characteristics to obtain the second account characteristic.

14. The method according to claim 10, characterized in that The step of selecting a plurality of first multimedia resources from the multimedia resources included in the second browsing information according to the first resource feature includes: Calculating a reference resource parameter of each of the multimedia resources in the second browsing information using the first resource feature, wherein the reference resource parameter is used to indicate a degree of dependency between the current multimedia resource in the second browsing information and other multimedia resources except the current multimedia resource; The first multimedia resources whose reference resource parameters are greater than or equal to the target resource parameters are screened out from the second browsing information.

15. The method according to claim 1, characterized in that The predicting the target browsing feature of the target account according to the first account feature and the second account feature includes: Concatenate the first account feature and the second account feature to obtain a concatenated feature; The splicing features are fused using a feature fusion model to obtain the target browsing features output by the feature fusion model, wherein the conversion relationship between the splicing features and the browsing features is recorded in the feature fusion model.

16. The method according to claim 1, characterized in that The recommending a target multimedia resource matching the target browsing feature to the target account includes: Calculating target similarity between the target browsing feature and target resource features of each multimedia resource in the multimedia resource library; Filtering the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library; The target multimedia resource is recommended to the target account.

17. The method according to claim 16, characterized in that The calculating of the target similarity between the target browsing feature and the target resource feature of each multimedia resource in the multimedia resource library comprises: Generate a fourth resource feature of the multimedia resource in the first browsing information and a fifth resource feature of the multimedia resource in the multimedia resource library using a feature generation model; Detecting feature difference information between the fourth resource feature and the fifth resource feature; Using the feature adjustment amount corresponding to the feature difference information, feature adjustment is performed on the reference feature of the multimedia resource in the first browsing information to obtain the target resource feature, wherein the reference feature is used to indicate the account feature of the account browsing the multimedia resource in the first browsing information; The target similarity between the target browsing feature and the target resource feature is calculated.

18. A multimedia resource recommendation device, characterized in that: include: A first prediction module, configured to predict a first account feature of a target account and a reference feature of each multimedia resource included in the first browsing information based on first browsing information, wherein the first browsing information is used to indicate a temporal relationship between multimedia resources browsed by the target account and a reference account associated with the target account within a first time period before a current moment, the first account feature is used to indicate the browsing preference of the target account for multimedia resources under the influence of the reference account, the reference feature is used to indicate a resource access relationship of the corresponding multimedia resource in the first browsing information, and the reference feature is used to characterize an access relationship of an account to multimedia resources through the resource access feature possessed by the account; a second prediction module, configured to predict a second account feature of the target account based on second browsing information and the reference feature, wherein the second browsing information is used to indicate a browsing situation of multimedia resources of the target account in a second time period, the first time period is longer than the second time period and the first time period includes the second time period, and the second account feature is used to indicate an influence of browsing the multimedia resources in the second browsing information on the browsing preference of the target account for multimedia resources; A third prediction module, configured to predict a target browsing feature of the target account based on the first account feature and the second account feature, wherein the target browsing feature is used to indicate a multimedia resource browsing preference of the target account at a current moment; The recommendation module is used to recommend target multimedia resources matching the target browsing characteristics to the target account.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 17 when executed by a processor.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 17 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 17 are implemented.

Images