Multimedia resource push method, device, storage medium and electronic device

Abstract

The embodiments of the present application provide a method, device, storage medium and electronic device for pushing multimedia resources, wherein the method includes: predicting a first resource feature and a first preference feature of the target account based on the first access information of the target account and the second access information of the reference account; predicting the second preference feature of the target account based on the first resource feature and the third access information; predicting the target preference feature of the target account using the first preference feature and the second preference feature; and pushing a target multimedia resource that matches the target preference feature to the target account. Through the present application, the problem of low matching between the multimedia resources pushed to the target account and the target account's current resource preference for the multimedia resources is solved, thereby achieving the effect of improving the matching between the multimedia resources pushed to the target account and the target account's current resource preference for the multimedia resources.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia resource recommendation, and more specifically, to a method, device, storage medium and electronic device for pushing multimedia resources. Background Art

[0002] As the amount of information on the Internet grows explosively, users have more and more ways to obtain information. Multimedia resources, due to their rich forms of expression, intuitive information expression, and clear subject content, have become increasingly important in information retrieval and network communication.

[0003] In order to facilitate users to obtain the required multimedia resources, numerous multimedia resource recommendation engines have emerged, and their essence is to recommend multimedia resources to users based on their preferences for multimedia resources. In order to achieve the function of recommending multimedia resources based on users' preferences for multimedia resources, the relevant technology predicts the user's current preference for multimedia resources by learning the temporal relationship between the multimedia resources that the user has recently browsed. The multimedia resource preferences determined in this way only focus on the user's browsing of multimedia resources in a short period of time, and cannot adapt to the user's more complex multimedia resource recommendation needs. Therefore, the multimedia resources recommended to users cannot fully meet the user's current actual preferences for multimedia resources. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and electronic device for pushing multimedia resources, so as to at least solve the problem in the related art that the multimedia resources pushed to the target account and the target account's current resource preference for the multimedia resources have a low matching degree.

[0005] According to an embodiment of the present application, a method for pushing multimedia resources is provided, comprising: predicting a first resource feature and a first preference feature of the target account based on first access information of the target account and second access information of a reference account, wherein the reference account is an account associated with the target account, the first access information is used to indicate the multimedia resource access situation of the target account in multiple reference time periods before the current moment, the second access information is used to indicate the multimedia resource access situation of the reference account in multiple reference time periods, the first preference feature is the multimedia resource access preference of the target account under the influence of the second access information of the reference account, and the first resource feature is used to indicate the multimedia resource access situation included in the first access information. The method comprises the following steps: first, predicting a second preference feature of the target account according to the first resource feature and the third access information, wherein the third access information is used to indicate the access of the target account to multimedia resources within a target time period, and the target time period is the time period closest to the current moment among the multiple reference time periods, and the second preference feature is used to indicate the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period; using the first preference feature and the second preference feature to predict the target preference feature of the target account, wherein the target preference feature is used to indicate the multimedia resource access preference of the target account at the current moment; and pushing target multimedia resources matching the target preference feature to the target account.

[0006] Optionally, predicting the first resource feature and the first preference feature of the target account based on the first access information of the target account and the second access information of the reference account includes: constructing a first relationship graph based on the first browsing information and the second browsing information, wherein the first relationship graph includes a first vertex and a first edge between the first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, a vertex corresponding to each multimedia resource included in the first access information, and a vertex corresponding to each multimedia resource included in the second access information, the first edge is used to characterize the association relationship between accounts or the access relationship of accounts to multimedia resources; extracting the first relationship using a feature extraction model including a plurality of feature extraction layers connected in sequence The target vertex features of the target vertex in the graph, wherein the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is used to generate the vertex features of the target vertex according to the vertex features of the reference vertex in the current first relationship graph, and use the vertex features of the target vertex extracted by the current feature extraction layer to update the features of the target vertex in the first relationship graph, the reference vertex is the vertex in the first relationship graph connected to the target vertex through the first edge; the target vertex features of the target vertex corresponding to the target account are determined as the first preference features, and the target vertex features of the target vertex corresponding to the multimedia resource included in the first access information are determined as the first resource features.

[0007] Optionally, the use of a feature extraction model including multiple feature extraction layers connected in sequence to extract target vertex features of target vertices in the first relationship graph includes: calling a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records vertices in the first relationship graph whose correlation with the target vertex is greater than a target correlation, and the target feature extraction layer is an extraction layer among the multiple feature extraction layers for performing feature extraction on the target vertices in the first relationship graph; calling the target feature extraction layer to perform feature extraction on vertex features currently possessed by vertices included in the target vertex set to obtain reference vertex features of the target vertex output by the target feature extraction layer; and when the target feature extraction layer is the last layer among the multiple feature extraction layers connected in sequence, determining the reference vertex features as the target vertex features.

[0008] Optionally, calling the target feature extraction layer to perform feature extraction on vertex features currently possessed by vertices included in the target vertex set to obtain reference vertex features of the target vertex output by the target feature extraction layer includes: calculating an initial degree of association between the target vertex and each of the second vertices based on vertex features currently possessed by the target vertex in the first relationship graph and vertex features currently possessed by the second vertices, wherein the second vertex is any vertex included in the target vertex set; and performing feature aggregation on the vertices in the target vertex set based on the initial degree of association to obtain the reference vertex features.

[0009] Optionally, the feature aggregation of vertices in the target vertex set according to the initial association degree to obtain the reference vertex feature includes: assigning an attention weight to each second vertex according to the initial association degree; weighting the vertex features of the corresponding second vertices using the attention weight to obtain a weighted vertex feature of each second vertex; and feature aggregation of the weighted vertex features of each second vertex in the target vertex set to obtain the reference vertex feature.

[0010] Optionally, the calling target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph includes: extracting an initial vertex set of the target vertices from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph that are connected to the target vertex through the first edge; screening a candidate number of candidate vertices from the initial vertex set according to vertex features currently possessed by the vertices in the initial vertex set to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies a target dependency condition.

[0011] Optionally, the method of screening out a candidate number of candidate vertices from the initial vertex set according to the vertex features currently possessed by the vertices in the initial vertex set to obtain the target vertex set includes: using the vertex features possessed by each vertex included in the first vertex set in the current screening round to calculate the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change in the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set; extracting vertices whose incremental values ​​are greater than or equal to the target incremental value from the first vertex set and adding them to the second vertex set; when the number of candidate vertices included in the second vertex set to which the vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, determining the second vertex set as the target vertex set.

[0012] Optionally, the calculation of vertex set information of the first vertex set includes one of the following: calculating vertex cover information of the first vertex set by using a vertex cover function, wherein the vertex cover information is used to indicate the proportion of edges in an edge cover graph of the first vertex set; calculating edge cover information of the first vertex set by using an edge cover function, wherein the edge cover information is used to indicate the proportion of vertices in the vertex cover graph of the first vertex set.

[0013] Optionally, predicting the second preference feature of the target account based on the first resource feature and the third access information includes: predicting a resource timing feature of the target account within the target time period based on the first resource feature and the third access information, and predicting a resource transfer feature of the target account within the target time period based on the first resource feature and the third access information, wherein the resource timing feature is used to indicate the temporal change relationship of the target account's browsing preference for multimedia resources within the target time period, and the resource transfer feature is used to indicate the browsing preference of the target account under the influence of various multimedia resources browsed within the target time period; the resource timing feature and the resource transfer feature are feature spliced ​​to obtain the second preference feature.

[0014] Optionally, predicting the resource timing characteristics of the target account within the target time period based on the first resource characteristic and the third access information includes: extracting the second resource characteristic of each reference multimedia resource in the third access information from the first resource characteristic; performing feature extraction on the second resource characteristic and the timing characteristic of each reference multimedia resource to obtain the third resource characteristic of each reference multimedia resource, wherein the timing characteristic is used to indicate the timing position of the reference multimedia resource in the third access information; calling a capsule network to perform feature processing on the third resource characteristic to obtain the resource timing characteristic, wherein the feature processing is used to indicate performing multiple rounds of weight iterations on the third resource characteristic through the capsule network to obtain the resource weight of each third resource characteristic, and performing feature merging on the third resource characteristics according to the resource weight.

[0015] Optionally, the calling of the capsule network to perform feature processing on the third resource feature to obtain the resource timing feature includes: calling the capsule network in the target weight iteration round to add a perturbation parameter to the reference resource weight output by the reference weight iteration round to obtain a candidate resource weight, wherein the target weight iteration round is the round for the current weight iteration to be performed, and the reference weight iteration round is the iteration round before the target weight iteration round in multiple rounds of weight iteration; calculating a first weight potential of the reference resource weight and a second weight potential of the candidate resource weight, wherein the weight potential is used to indicate the degree of matching between the resource weight and the third access information; determining a difference between the first weight potential and the second weight potential; when the difference value is used to indicate that the candidate resource weight meets the target update condition, determining the candidate resource weight as the weight output by the capsule network; when it is determined that the target weight iteration round is the last iteration round in the multiple rounds of weight iteration, determining the candidate resource weight as the resource weight.

[0016] Optionally, before calling the capsule network to perform feature processing on the third resource feature to obtain the resource timing feature, the method also includes: performing feature conversion on the third resource feature of each multimedia resource in the third access information to obtain a first spatial feature of the third resource feature in a first feature representation space, and a second spatial feature of the third resource feature in a second feature representation space; calculating the feature similarity between the first spatial feature and the second spatial feature of the third resource feature; and allocating the resource weight of each of the multiple third resource features according to the feature similarity.

[0017] Optionally, the feature extraction of the second resource feature and the timing feature of each of the reference multimedia resources to obtain the third resource feature of each of the reference multimedia resources includes: feature splicing the second resource feature and the timing feature of the reference multimedia resource to obtain a fourth resource feature; inputting a plurality of the fourth resource features into the target prediction model according to the time sequence of the reference multimedia resources within the target time period to obtain a fifth resource feature of each of the reference multimedia resources, wherein the fifth resource feature is a feature under the influence of the fourth resource feature of other multimedia resources among the plurality of reference multimedia resources except the current multimedia resource.

[0018] Optionally, predicting the resource transfer characteristics of the target account within the target time period based on the first resource characteristic and the third access information includes: extracting the second resource characteristic of each reference multimedia resource in the third access information from the first resource characteristic; constructing a second relationship graph based on the second resource characteristic and the third access information, wherein the second relationship graph records a plurality of third vertices and target directed edges connecting the third vertices, each of the third vertices and the reference multimedia resource having the second resource characteristic is set in a one-to-one correspondence, and the target directed edges record the association relationship between the reference multimedia resources; extracting the target hidden layer characteristics of each of the reference multimedia resources from the second relationship graph, wherein the target hidden layer characteristics are used to indicate the influence of the corresponding reference multimedia resource on the browsing preference of the target account; predicting the resource transfer characteristics of the target account based on the plurality of target hidden layer characteristics.

[0019] Optionally, extracting the target hidden layer features of each of the reference multimedia resources from the second relationship graph includes: extracting the initial vertex features of each of the third vertices from the second relationship graph and inputting them into sequentially connected multi-layer encoders to obtain initial hidden features output by the multi-layer encoders, wherein the initial vertex features include the second resource features of the reference multimedia resource corresponding to the third vertex and the association relationship between the vertices connected to the current third vertex through the target directed edge, and each layer of the encoders is used to extract the influence relationship between the initial vertex features; inputting the initial hidden features into sequentially connected multi-layer decoders to obtain the target hidden layer features, wherein each layer of the decoders is used to extract the influence of the reference multimedia resource corresponding to each of the third vertices on the browsing preference of the target account from the initial hidden features.

[0020] Optionally, constructing a second relationship graph based on the second resource feature and the third access information includes: creating the third vertex corresponding to the second resource feature of each of the reference multimedia resources; creating a target directed edge between the third vertices according to the access timing relationship between the reference multimedia resources in the third access information, wherein the input end of the target directed edge is connected to the vertex with an earlier occurrence timing, and the output end of the target directed edge is connected to the vertex with a later occurrence timing; configuring a target weight parameter of each target directed edge according to the target number of occurrences of the timing relationship indicated by each target directed edge in the third access information, to obtain the second relationship graph, wherein the target weight parameter is used to indicate the similarity relationship between the reference multimedia resource corresponding to the current third vertex and the reference multimedia resource corresponding to the vertex connected by the current target directed edge.

[0021] Optionally, configuring the target weight parameter of each target directed edge according to the target number of times the timing relationship indicated by each target directed edge appears in the third access information includes: obtaining a reference number of times all directed edges with the current third vertex as the output end appear in the third access information; determining the quotient obtained by dividing the target number by the reference number as the target weight parameter of the corresponding directed edge; and configuring the corresponding target directed edge using the target weight parameter.

[0022] Optionally, predicting the resource transfer feature of the target account based on multiple target hidden layer features includes one of the following: determining the mean of the multiple target hidden layer features as the resource transfer feature; determining the maximum feature value among the multiple target hidden layer features as the resource transfer feature; assigning a corresponding feature weight to each target hidden layer feature based on the feature relationship between the target hidden layer features; and performing weighted sum calculation on the target hidden layer features using the feature weights to obtain the resource transfer feature.

[0023] Optionally, the use of the first preference feature and the second preference feature to predict the target preference feature of the target account includes: performing feature splicing on the first preference feature and the second preference feature to obtain a spliced ​​feature; performing feature fusion on the spliced ​​feature using a feature fusion model to obtain the target preference feature output by the feature fusion model, wherein the feature fusion model records the conversion relationship between the spliced ​​feature and the preference feature.

[0024] Optionally, pushing the target multimedia resources matching the target preference characteristics to the target account includes: calculating the target similarity between the target preference characteristics and reference resource characteristics of each multimedia resource in the multimedia resource library; screening out the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library; and displaying the target multimedia resources to the target account.

[0025] Optionally, the calculation of the target similarity between the target preference feature and the reference resource features of each multimedia resource in the multimedia resource library includes: using a feature generation model to generate initial features of the multimedia resources in the third access information and reference features of the multimedia resources in the multimedia resource library; detecting feature difference information between the initial features and the reference features; using a feature adjustment amount corresponding to the feature difference information to perform feature adjustment on the second resource features of the multimedia resources in the third access information to obtain a target resource feature, wherein the target resource feature is used to indicate an account feature of an account that browses the multimedia resources in the third access information; and calculating the target similarity between the target preference feature and the target resource feature.

[0026] According to another embodiment of the present application, a multimedia resource push device is provided, including: a first prediction module, used to predict a first resource feature and a first preference feature of the target account based on first access information of the target account and second access information of a reference account, wherein the reference account is an account associated with the target account, the first access information is used to indicate the multimedia resource access status of the target account in multiple reference time periods before the current moment, the second access information is used to indicate the multimedia resource access status of the reference account in multiple reference time periods, the first preference feature is the multimedia resource access preference of the target account under the influence of the second access information of the reference account, and the first resource feature is used to indicate the multimedia resource access preference of the account accessing the multimedia resource included in the first access information. Features; a second prediction module, used to predict the second preference feature of the target account based on the first resource feature and the third access information, wherein the third access information is used to indicate the access of the target account to multimedia resources within a target time period, the target time period is the time period closest to the current moment among the multiple reference time periods, and the second preference feature is used to indicate the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period; a third prediction module, used to predict the target preference feature of the target account using the first preference feature and the second preference feature, wherein the target preference feature is used to indicate the multimedia resource access preference of the target account at the current moment; a push module, used to push target multimedia resources matching the target preference feature to the target account.

[0027] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0028] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0029] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0030] Through the present application, by predicting based on the first access information of the target account and the second access information of the reference account, a first resource feature indicating the feature of the account accessing the multimedia resources included in the first access information and a first preference feature of the multimedia resource access preference of the target account under the influence of the second access information of the reference account are obtained, and then a prediction is made based on the first resource feature and the third resource feature to obtain a second preference feature indicating the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period, and then the target preference feature of the target account predicted by using the first preference feature and the second preference feature incorporates both the factors of the resource browsing preference features under the influence of the social relationships of the account and the factors of the resource browsing preference features under the influence of the multimedia resources accessed by the target account itself within the target time period, so that the target preference feature is closer to the target account's actual current access preference for multimedia resources, and then the target multimedia resources matched with the target preference feature pushed to the target account can better meet the target account's current demand for multimedia resources. It can solve the problem of low matching between the multimedia resources pushed to the target account and the target account's current resource preference for multimedia resources, thereby achieving the effect of improving the matching between the multimedia resources pushed to the target account and the target account's current resource preference for multimedia resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a hardware structure block diagram of a server device of a multimedia resource push method according to an embodiment of the present application;

[0032] Figure 2 is a flowchart of a method for pushing multimedia resources according to an embodiment of the present application;

[0033] Figure 3 is an optional video recommendation schematic diagram according to an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of an optional capsule network according to an embodiment of the present application;

[0035] Figure 5 is an optional FPGA heterogeneous accelerator architecture diagram according to an embodiment of the present application;

[0036] Figure 6 is a schematic diagram of an optional recommended device hardware according to an embodiment of the present application;

[0037] Figure 7 It is a structural block diagram of a multimedia resource push device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0040] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a multimedia resource push method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0041] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for pushing multimedia resources in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] In this embodiment, a method for pushing multimedia resources is provided. Figure 2 is a flowchart of a method for pushing multimedia resources according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0044] Step S202, predicting a first resource feature and a first preference feature of the target account based on first access information of the target account and second access information of a reference account, wherein the reference account is an account associated with the target account, the first access information is used to indicate multimedia resource access by the target account in multiple reference time periods before the current moment, the second access information is used to indicate multimedia resource access by the reference account in multiple reference time periods, the first preference feature is the multimedia resource access preference of the target account under the influence of the second access information of the reference account, and the first resource feature is used to indicate the feature of the account accessing the multimedia resource included in the first access information;

[0045] Step S204, predicting a second preference feature of the target account based on the first resource feature and the third access information, wherein the third access information is used to indicate the access of the target account to multimedia resources within a target time period, the target time period being the time period closest to the current moment among the multiple reference time periods, and the second preference feature is used to indicate the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period;

[0046] Step S206, using the first preference feature and the second preference feature to predict a target preference feature of the target account, wherein the target preference feature is used to indicate the multimedia resource access preference of the target account at the current moment;

[0047] Step S208: Pushing target multimedia resources matching the target preference characteristics to the target account.

[0048] Through the above steps, by predicting based on the first access information of the target account and the second access information of the reference account, a first resource feature indicating the feature of the account accessing the multimedia resources included in the first access information and a first preference feature of the multimedia resource access preference of the target account under the influence of the second access information of the reference account are obtained, and then prediction is made based on the first resource feature and the third resource feature to obtain a second preference feature indicating the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period, and then the target preference feature of the target account predicted by using the first preference feature and the second preference feature incorporates both the factors of the resource browsing preference features under the influence of the social relationships of the account and the factors of the resource browsing preference features under the influence of the multimedia resources accessed by the target account itself within the target time period, so that the target preference feature is closer to the target account's actual current access preference for multimedia resources, and then the target multimedia resources matched with the target preference feature pushed to the target account can better meet the target account's current demand for multimedia resources. It can solve the problem of low matching between the multimedia resources pushed to the target account and the target account's current resource preference for multimedia resources, thereby achieving the effect of improving the matching between the multimedia resources pushed to the target account and the target account's current resource preference for multimedia resources.

[0049] The execution subject of the above steps may be a server, a terminal, etc. used to run the above multimedia resource push method, but is not limited thereto.

[0050] Optionally, in the embodiment of the present application, multimedia resources refer to resources that can transmit information in various forms, including text, pictures, audio, video, etc. These resources can be used on various media platforms, such as the Internet, television, movies, radio, etc.

[0051] In the technical solution provided in the above step S202, since the first preference feature is the multimedia resource access preference of the target account under the influence of the second access information of the reference account, the first preference feature of the target account can be reflected by the access preference of the reference account to the multimedia resource, and since the first resource feature indicates the feature of the account accessing the multimedia resource included in the first access information, the first resource feature can be reflected by the account feature, and then the method of predicting the first resource feature and the second preference feature according to the first access information and the second access information can be: predicting the multimedia resource browsing feature of each reference account based on the second access information, and then calculating the influence weight of each reference account on the multimedia resource browsing preference of the target account based on the similarity relationship between the first access information and the second access information, and then weighted summing the multimedia resource browsing preference features of the reference account based on the influence weight, so as to obtain the first preference feature of the target account. Then, the candidate account that has accessed each multimedia resource included in the first access information is searched from the reference account, and then the multimedia resource browsing preference features of the candidate account and the first preference features of the target account are feature-merged, so as to obtain the first resource feature of each multimedia resource. When merging the multimedia resource browsing preference features of the candidate account and the first preference features of the target account, the attention mechanism can be used to assign account weights to the target account and the candidate account according to their importance to the current multimedia resources, thereby using the account weights to perform a weighted summation of the multimedia resource browsing preference features and the first preference features of the target account.

[0052] In the technical solution provided in the above step S204, the second preference feature can be obtained by extracting the feature of the association relationship between the resource features of the multimedia resources included in the third access information, that is, the second preference feature includes the influence relationship between the multimedia resources included in the third access information, and the influence relationship can include the temporal relationship feature that characterizes the access temporal relationship of the multimedia resources, or the influence relationship can also include the complex transfer relationship feature that characterizes the adjacent temporal position or the long-distance temporal position of the multimedia resources, or can also include the temporal relationship feature and the transfer relationship feature at the same time. Therefore, in this embodiment, the method of predicting the second preference feature according to the first resource feature and the third access information can be: using the first resource feature to determine the resource feature of each multimedia resource of the third access information for configuration, and then using the temporal relationship extraction model to extract the temporal relationship feature from the third access information, thereby obtaining the second preference feature; or using the transfer relationship extraction model to extract the transfer relationship feature from the third access information as the second preference feature; or using the temporal relationship extraction model to extract the temporal relationship feature from the third access information, and using the transfer relationship extraction model to extract the transfer relationship feature from the third access information, and merging the temporal relationship feature and the transfer relationship feature to obtain the second preference feature.

[0053] Optionally, in an embodiment of the present application, the second preference feature may be a feature that simultaneously incorporates the temporal relationship feature and the transition relationship feature of the multimedia resources included in the third access information. Transformer is a powerful attention mechanism model that is suitable for processing sequence data and can effectively capture long-distance dependencies in the sequence. This dependency includes both the sequential relationship between videos within a session and the transition relationship between videos within a session. In view of this, this embodiment proposes a transformer encoder-based session (multimedia resource browsing record within a time period, such as a target time period or a reference time period) representation learning network (TSRLN), which consists of a session preprocessor and a Transformer encoder. The former is responsible for obtaining the session information from each user (account). Current session The redundant or invalid multimedia resources are removed from the preprocessed session to obtain a basic session of fixed length; the latter is responsible for learning the sequential relationship between multimedia from the preprocessed session to obtain the session representation, that is, the user's dynamic interest preference. The embodiment of the present application takes the multimedia resource as a video resource as an example, first uses the first resource feature to configure the resource feature of each multimedia resource in the third access information, and then uses the model to extract features from the third access information configured with the resource feature, specifically including the following contents:

[0054] Conversation preprocessor: The Transformer model performs well in natural language processing and sequence modeling, but its original design is mainly used to process fixed-length sequences, such as natural language sentences. In conversational recommendation systems, the length of sessions can be dynamic and noisy, and the number of items in each session can be different. This makes it challenging to directly apply the standard Transformer to conversation representation extraction. To this end, we design a conversation preprocessor that can be implemented based on the neighbor sampling function of submodule theory. The difference is that we agree that the cardinality of the sampled basic sessions is fixed to Instead of just setting an upper limit on the length, the session sampling problem based on submodule theory can be modeled as follows:

[0055] ,

[0056] ,

[0057] The above problem can be solved by AGSNS algorithm, which will not be described here. Current session The fixed length obtained by the session preprocessor is The basis of the current session . Because the preprocessor may disrupt the current session The order in which the videos are accessed in Rearrange the order in and remember the sorted basic current session as .

[0058] The Transformer encoder learns the sequential and transition relations within a session. , let’s assume that the last layer of the heterogeneous directed graph attention neural network HDGAT is The social semantics enhanced embedding vector output by the layer is expressed as ; Add position encoding according to the following formula to allow the Transformer model to distinguish videos at different positions in the current session:

[0059]

[0060] For each discrete value , we assign a unique trainable parameter vector As a video Then it is combined with the video The embedding vector representation of Add and perform linear transformation to get The sequence-aware embedding vector representation of the video:

[0061] ,

[0062] in, is the parameter matrix to be learned; Represents the dimension of the order-aware embedding vector. Base current session of The sequence-aware embedding vectors are stacked to form , feeding them into the Transformer encoder to extract the base current session representation.

[0063] 1) Single-head self-attention mechanism. Generally, the single-head self-attention mechanism can be defined as:

[0064] ,

[0065] in, , and The input matrices representing query, key, and value respectively; Indicates the number of columns of the key matrix.

[0066] 2) Multi-head self-attention mechanism. The self-attention mechanism we proposed includes self-attention heads, which in parallel perform Conducting conversational representation learning:

[0067]

[0068]

[0069]

[0070] in, , , and are parameter matrices to be learned; , and They are , and Along the last dimension A division.

[0071] 3) Output session representation. For user The basis of the current session is , let’s assume that the sequence output by the Transformer encoder is .in, Represents the length of the output sequence, which is the same as the length of the input sequence They may be equal or different. To extract the embedding vector representation of the entire session and use it as the user's dynamic interest preference , this embodiment proposes the following four methods for extracting session representations: (a) End method. The output of the last time step of the above sequence is used as the embedding vector representation of the entire session, then:

[0072]

[0073] in, Indicates taking the last element of the sequence. (b) Mean method. This method uses a weighted average of all vectors in the sequence:

[0074]

[0075] (c) Maximum method. This method takes the maximum value of all vectors in the sequence and returns a new vector:

[0076]

[0077] (d) Attention method. This method is based on the attention mechanism and dynamically obtains the weighted average of all vectors in the sequence by learning the attention weights. The steps of this method are as follows:

[0078]

[0079]

[0080]

[0081] in, is the parameter matrix to be learned, is the activation function (such as ReLU), is a vector Score is a vector The attention weight.

[0082] In the technical solution provided in the above step S206, the prediction method of the target preference feature can directly merge the first preference feature and the second preference feature, or can also adopt an attention mechanism to assign weights to the first preference feature and the second preference feature according to their importance to predicting the target preference feature of the target account, thereby using the assigned weights to perform a weighted summation of the first preference feature and the second preference feature to obtain the target preference feature.

[0083] In the technical solution provided in the above step S208, the target multimedia resource may be a multimedia resource having a matching degree greater than the target matching degree obtained by matching the multimedia resources in the multimedia resource library based on the characteristics of the target account. The target multimedia resource may be one or more. When there are multiple target multimedia resources, the multiple target multimedia resources may be sorted according to the matching degree with the characteristics of the target account.

[0084] As an optional embodiment, predicting the first resource feature and the first preference feature of the target account according to the first access information of the target account and the second access information of the reference account includes:

[0085] Constructing a first relationship graph according to the first browsing information and the second browsing information, wherein the first relationship graph includes a first vertex and a first edge between first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, a vertex corresponding to each multimedia resource included in the first access information, and a vertex corresponding to each multimedia resource included in the second access information, and the first edge is used to represent an association relationship between accounts or an access relationship between accounts and multimedia resources;

[0086] Extracting target vertex features of a target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence, wherein the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is used to generate vertex features of the target vertex according to vertex features of a reference vertex in the current first relationship graph, and updating features of the target vertex in the first relationship graph using the vertex features of the target vertex extracted by the current feature extraction layer, wherein the reference vertex is a vertex in the first relationship graph connected to the target vertex through the first edge;

[0087] The target vertex feature of the target vertex corresponding to the target account is determined as the first preference feature, and the target vertex feature of the target vertex corresponding to the multimedia resource included in the first access information is determined as the first resource feature.

[0088] Optionally, in the embodiment of the present application, the feature extraction model may be a heterogeneous graph convolutional neural network to effectively learn the relationship and feature representation between different types of vertices. The network includes Layer heterogeneous graph convolutional neural network layer, each layer includes two functional components: neighbor sampling and aggregation update. The former is based on the submodule theory, and selects a set of basic neighbor subsets that are non-redundant, representative, and diverse from the neighbor set of each type of central vertex. The aggregation update component performs graph convolution operations on each type of central vertex by type, and then learns different weights for the graph convolution results of different types of central vertices based on the attention mechanism to alleviate the imbalance. To this end, we maintain a user-video heterogeneous graph (i.e., the first relationship graph) It is used to update the embedding vector representation of users (i.e. accounts) or videos (i.e. multimedia resources). The update process requires information from the neighboring vertices of users and videos. Let’s assume that user and Video In the The learned hidden layer representations are and For users , its neighbor set Contains two types of vertices: user vertices and video vertices; for videos , its neighbor set Contains only one type of vertex: the user vertex that accesses it, and thus its neighbor set .

[0089] Neighbor sampling: Neighbor sampling refers to sampling from each type of central vertex The neighbor set In the above example, we select a set of basic neighbor subsets that are non-redundant, representative, and diverse. , and satisfy the cardinality of the subset No more than When the neighbors of the central vertex have multiple types, it is required that there is at least one vertex of each type. In order to reduce the complexity of the network model and optimize the model application scenario, it is advisable to take the basic neighbor subset of the neighbor set of all vertices The upper limit of the cardinality is The above problem can be abstracted into an optimization problem: the optimization target is the basic neighbor subset The representativeness and diversity of the elements in; the constraint is the basic neighbor subset The base number does not exceed , and there is at least one vertex of each type.

[0090] A submodule function is a function defined on a collection, usually used to measure the "value" or "importance" of the collection. is called a submodule function if for any set and any element , the following conditions are met:

[0091]

[0092] Intuitively, this means that the marginal gain of adding elements to a smaller set decreases as the size of the set increases. Submodular functions have many useful properties in optimization problems, such as being able to approximate the optimal solution through greedy algorithms and being able to be used to maximize or minimize some objective function. Common submodular functions include mutual information functions.

[0093] Inspired by the submodule function, this embodiment models the neighbor sampling problem of the central vertex as a cardinality-constrained subset selection problem: given a user-video heterogeneous graph , from each central vertex The egocentric subgraph of Choose a base number not exceeding Subset of , and there is at least one vertex of each type, and the submodule function achieves the maximum value. This problem can be formally expressed as:

[0094] ,

[0095]

[0096] The formulas represent the objective function and constraints of the optimization problem respectively; the center vertex can be either a user vertex or a video vertex; Determines whether the two operand types are consistent and returns a Boolean variable (true or false); Indicates forced type conversion, converting a Boolean variable to an integer variable.

[0097] Mutual Information Function: Mutual Information is an important indicator used in information theory to measure the correlation between two random variables. It can be used to describe the degree of mutual dependence or the amount of shared information between two random variables. Suppose there are two random variables and , whose joint probability distribution is , and the marginal probability distributions are and , mutual information It can be defined as:

[0098] ,

[0099] in, and They are random variables and If and Independent of each other, , at this time the mutual information In other words, a mutual information of zero indicates that the two random variables are completely independent. and If there is a dependency relationship between The larger the mutual information value, the and The higher the degree of correlation between them, or the more information they share. The calculation of the mutual information function can be regarded as a measure of the degree of information sharing between two random variables, and can be applied in many fields, including machine learning, information retrieval, signal processing, etc. In machine learning, mutual information is often used in tasks such as feature selection, cluster analysis, and semi-supervised learning to help evaluate the correlation and information between features. For egocentric graphs , based on the mutual information principle, the submodule function is defined as:

[0100] ,

[0101] in, Is a collection About Vertex Sets The complement of , .

[0102] Submodule functions It can be the mutual information function defined by the above formula, or any other function that can measure the "value" or "importance" of a set, which is not limited in this embodiment. The above cardinality-constrained maximization submodule function problem is an NP-hard problem, which can usually be solved using a greedy algorithm. This embodiment proposes an approximation algorithm based on GreedyStrategy for Neighbor Sampling (AGSNS). Since the solution process of neighbor sampling problems of different central vertex types is similar, it is advisable to use the central vertex as the approximation algorithm. Taking the time as an example, the proposed approximate solution algorithm flow is as follows:

[0103] 1. Initialization: will be used to record the basic neighbor subset Initialized to an empty set; will be used to record the unselected vertex set (maybe called the candidate set) Initialize to ;

[0104] 2. Iterative greedy selection:

[0105] (a). For each vertex in the unselected vertex set , calculated according to the following formula and added The increment of the posterior objective function:

[0106]

[0107] (b) Select the vertex that maximizes the increment and satisfies the constraints. , and change it from Delete and add ; If there are multiple vertices that meet the requirements, one of them is randomly selected.

[0108] (c). Update the basic neighbor subset and the candidate set: ; ;

[0109] (d). Check whether the constraints are satisfied; if so, exit the loop;

[0110] 3. Output result: get user The sampled base neighbor subset .

[0111] Similarly, we can also use the AGSNS algorithm to solve the video The sampled base neighbor subset It should be pointed out that our framework is applicable to any submodule functions, such as mutual information function, vertex cover function, edge cover function and facility location function.

[0112] Through the neighbor sampling approximate solution algorithm based on the greedy strategy, we Redundant neighbors and noise in the social space and / or item space are removed to obtain a set of basic neighbor subsets that are non-redundant, representative, and diverse, laying the foundation for effectively extracting user embeddings and item embeddings enhanced with social relationships.

[0113] Aggregate Update: For User-Video Heterogeneous Graph , whose vertex type set is , a set of connection edge types For each edge type ,

[0114] When the center vertex When assuming that the user and any of its neighbor vertices The The hidden layer representations of the output of the layer-heterogeneous graph convolutional neural network are Dimensional vector and , these neighbor vertices include both user vertices and video vertices. , No. The operations of the layer-heterogeneous graph convolutional neural network layer can be formally described as follows:

[0115] 1. Implement graph convolution operations for all neighbor vertices of type "user":

[0116]

[0117] 2. For all neighbor vertices of type "video":

[0118] ,

[0119] 3. Based on the attention mechanism, dynamically assign weights to the aggregation results of the two types of vertices and merge them:

[0120] ,

[0121] ,

[0122] in, Indicates user The set of all user vertices in the intersection of the neighbor set and the basic neighbor subset; Indicates user The set of all video vertices in the intersection of the neighbor set and the base neighbor subset; represents a nonlinear activation function, represents the power operation of natural exponents; All belong to the parameter matrix to be learned.

[0123] When the center vertex When, according to the user-video heterogeneous graph (first relationship graph) As defined, it has only one type of neighbor, the user vertex. and any of its neighbor vertices The The hidden layer representations of the output of the layer-heterogeneous graph convolutional neural network are Dimensional vector and .because The neighbor vertex of The operation of a layer of a heterogeneous graph convolutional neural network can be defined as:

[0124] ,

[0125] in, All belong to the parameter matrix to be learned.

[0126] when At that time, Layer-heterogeneous graph convolutional neural network layer based on user feature matrix and the video feature matrix is the input. When , the heterogeneous graph convolutional neural network outputs the user embedding vector representation enhanced by social semantics and video embedding vector representation .in, Denotes the dimension of the user and video embedding vector representation.

[0127] Optionally, in an embodiment of the present application, the feature extraction model is used to iteratively update the relationship between vertices in the first relationship graph through multiple sequentially connected feature extraction layers, that is, each feature extraction layer is used to update the features of the current vertex according to the features of vertices that have an associated relationship with the current vertex, so that with the iteration of the feature extraction layer, the features of the current vertex can be influenced by the features of vertices that are farther away.

[0128] Optionally, in an embodiment of the present application, each feature extraction layer can perform a graph convolution operation on each central vertex, and then learn different weights for the graph convolution results of different central vertices based on the attention mechanism to alleviate the imbalance.

[0129] Through the above content, by using a feature extraction model including multiple sequentially connected feature extraction layers to perform multiple rounds of iterative extraction on the vertex features of the target vertex in the first relationship graph, the feature relationship between vertices with a large distance is extracted, thereby improving the feature information dimension of the target vertex feature.

[0130] As an optional embodiment, extracting the target vertex feature of the target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence includes:

[0131] Calling a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records vertices in the first relationship graph whose association degree with the target vertex is greater than a target association degree, and the target feature extraction layer is an extraction layer among the multiple feature extraction layers that is to perform feature extraction on the target vertex in the first relationship graph;

[0132] Calling the target feature extraction layer to extract features of the vertices currently included in the target vertex set, and obtaining reference vertex features of the target vertices output by the target feature extraction layer;

[0133] In a case where the target feature extraction layer is the last layer of a plurality of sequentially connected feature extraction layers, the reference vertex feature is determined as the target vertex feature.

[0134] Optionally, in the embodiment of the present application, when the feature extraction layer is the first layer, the vertex features can be extracted using an existing feature extraction model. For example, the features of user vertices can be extracted using existing deep learning technologies such as Transformer, CNN, etc. from the text, pictures, and other information provided by the user when registering to extract a high-dimensional vector as the user's features. The same is true for the vertices corresponding to multimedia resources, except that the existing technologies used may be different, such as ViTTransformer, etc.

[0135] Optionally, in an embodiment of the present application, when the target vertex is a vertex of an account, the vertices in the target vertex set need to include two types of vertices: vertices corresponding to multimedia resources and vertices corresponding to the account.

[0136] Optionally, in an embodiment of the present application, in order to improve the reliability of information when extracting the target vertex set, maximum value extraction can be used for extraction, that is, the vertex with the highest correlation with the current target vertex and the greatest benefit in reflecting the characteristics of the target vertex is extracted from the first relationship graph. Therefore, in an embodiment of the present application, a greedy strategy can be used to perform neighbor sampling in the first relationship graph, and the importance relationship between vertices can be calculated according to the submodule function theory when performing neighbor sampling.

[0137] Optionally, in the embodiment of the present application, the target feature extraction layer is called to extract the vertex features currently possessed by the vertices included in the target vertex set, and the idea of ​​obtaining the reference vertex features of the target vertices output by the target feature extraction layer may be: since the aggregation update process of different central vertex types is similar, it may be appropriate to use the central vertex as the reference vertex feature. As an example, we introduce the basic principle of directed graph attention neural network. and any of its neighbor vertices The The hidden layer representations of the output of the layer-directed graph attention neural network are Dimensional vector and ; According to the principle of attention, The operation of the layer-heterogeneous graph attention layer can be formally described as:

[0138] ,

[0139] ,

[0140] ,

[0141] ,

[0142] ,

[0143] in, represents a nonlinear activation function; , , are all model parameters to be learned. At that time, Layer heterogeneous graph attention layer based on user feature matrix and the video feature matrix is the input. When the heterogeneous graph attention neural network outputs the user embedding vector representation enhanced by social semantics and video embedding vector representation .in, Denotes the dimension of the user and video embedding vector representation.

[0144] Through the above content, by calling the target feature extraction layer, global knowledge such as friend relationships in the social space and all historical interaction records between users and multimedia are captured, which greatly improves the accuracy and efficiency of modeling users' long-term interest preferences.

[0145] As an optional embodiment, the calling the target feature extraction layer to extract features of the vertex features currently possessed by the vertices included in the target vertex set to obtain the reference vertex features of the target vertices output by the target feature extraction layer includes:

[0146] Calculating an initial degree of association between the target vertex and each of the second vertices according to the vertex features currently possessed by the target vertex and the vertex features currently possessed by the second vertices in the first relationship graph, wherein the second vertex is any vertex included in the target vertex set;

[0147] The features of the vertices in the target vertex set are aggregated according to the initial association degree to obtain the reference vertex features.

[0148] Optionally, in the embodiment of the present application, the initial association degree between vertices may be, but is not limited to, calculated based on the feature similarity corresponding to the vertices.

[0149] Optionally, in an embodiment of the present application, the method of performing feature aggregation on vertices according to the initial correlation degree may be based on the idea of ​​an attention mechanism, assigning a vertex weight to each vertex according to the initial correlation degree, and then using the vertex weight to weighted sum the vertex features of the second vertex to obtain a reference vertex feature.

[0150] As an optional embodiment, performing feature aggregation on vertices in the target vertex set according to the initial association degree to obtain the reference vertex feature includes:

[0151] assigning an attention weight to each of the second vertices according to the initial relevance;

[0152] Using the attention weight to weight the vertex features of the corresponding second vertices, to obtain a weighted vertex feature of each second vertex;

[0153] Feature aggregation is performed on the weighted vertex feature of each second vertex in the target vertex set to obtain the reference vertex feature.

[0154] Through the above content, the attention mechanism is used to assign attention weights to each second vertex according to the initial correlation, thereby effectively solving the problem of information imbalance.

[0155] As an optional embodiment, the calling the target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph includes:

[0156] Extracting an initial vertex set of the target vertex from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph connected to the target vertex through the first edge;

[0157] A candidate number of candidate vertices are screened out from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies a target dependency relationship condition.

[0158] As an optional embodiment, the step of selecting a candidate number of candidate vertices from the initial vertex set according to the current vertex features of the vertices in the initial vertex set to obtain the target vertex set includes:

[0159] Using the vertex features of each vertex included in the first vertex set of the current screening round, calculating the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change of the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set;

[0160] Extracting vertices whose increment values ​​are greater than or equal to the target increment value from the first vertex set and adding them to the second vertex set;

[0161] When the number of candidate vertices included in the second vertex set to which vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, the second vertex set is determined as the target vertex set.

[0162] Optionally, in an embodiment of the present application, a candidate number of candidate vertices are screened out from the initial vertex set according to the vertex features currently possessed by the vertices in the initial vertex set, and a method for obtaining the target vertex set can also be: randomly screen out the candidate number of vertices from the initial vertex set to obtain a first vertex set; calculate vertex set information of the first vertex set, wherein the vertex set information is used to indicate the importance of the vertex features possessed by the vertices in the first vertex set; extract an alternative vertex from the remaining vertices in the initial vertex set; use the vertex features currently possessed by the alternative vertex to calculate the incremental value of the alternative vertex to the vertex set information; if the incremental value meets the target incremental value condition, use the alternative vertex to update the vertices in the first vertex set to obtain a second vertex set; detect the number of remaining vertices in the initial vertex set; and if it is determined that there are no remaining vertices in the initial vertex set, determine the second vertex set as the target vertex set.

[0163] Optionally, in the embodiment of the present application, the target vertex set is screened by using a neighbor sampling approximate solution theory based on a greedy strategy, for each vertex in the first relationship graph Redundant neighbors and noise in the social space and / or item space are removed to obtain a set of basic neighbor subsets that are non-redundant, representative, and diverse, laying the foundation for effectively extracting user embeddings and multimedia resource embeddings (i.e., features) enhanced with social relations.

[0164] As an optional embodiment, the calculating the vertex set information of the first vertex set includes one of the following:

[0165] Calculate vertex cover information of the first vertex set by using a vertex cover function, wherein the vertex cover information is used to indicate the proportion of edges in an edge cover graph of the first vertex set;

[0166] The edge coverage information of the first vertex set is calculated by using an edge coverage function, wherein the edge coverage information is used to indicate the proportion of vertices in a vertex coverage graph of the first vertex set.

[0167] Optionally, in the embodiment of the present application, the egocentric map , the covering function is usually used to get from the vertex set Select a subset , to cover the edges or vertices in the graph, which are divided into edge covering functions and vertex covering functions accordingly. They are defined as follows:

[0168] 1. Edge Covering Function : The edge cover function measures the set The proportion of edges in the graph that are covered by the edges in the graph. It can be defined as:

[0169] ,

[0170] in, is the vertex A subset of a set.

[0171] 2. Vertex Cover Function : Vertex cover function measures the set The fraction of vertices in the vertex cover graph. It can be defined as:

[0172] .

[0173] As an optional embodiment, predicting the second preference feature of the target account according to the first resource feature and the third access information includes:

[0174] Predicting a resource timing feature of the target account within the target time period based on the first resource feature and the third access information, and predicting a resource transfer feature of the target account within the target time period based on the first resource feature and the third access information, wherein the resource timing feature is used to indicate a temporal change relationship of the target account's browsing preference for multimedia resources within the target time period, and the resource transfer feature is used to indicate the browsing preference of the target account under the influence of each multimedia resource browsed within the target time period;

[0175] The resource timing feature and the resource transfer feature are concatenated to obtain the second preference feature.

[0176] Optionally, in an embodiment of the present application, the resource timing feature is the multimedia resource preference feature of the target account under the influence of the timing relationship of the multimedia resources browsed in the current target time period, and the method for predicting the timing relationship feature can be to aggregate the resource features corresponding to the multimedia resources in chronological order.

[0177] Optionally, in an embodiment of the present application, the multimedia browsing information browsed by the target account at each moment in the target time period has different effects on the browsing resource preferences of the target account at the current moment, that is, the multimedia resources browsed at different moments may have a complex dependency relationship such as an incentive or inhibitory effect on the resource browsing preferences of the user at the current moment, that is, the influence of the multimedia resource browsing preferences of the user at different moments on the user's current resource browsing preferences may have a complex dependency relationship such as an incentive or inhibitory effect, and by predicting the resource transfer characteristics of the target account based on the third browsing information, the multimedia resource browsing preferences under the incentive or inhibitory effect of the multimedia information browsed by the target account in each reference time period are obtained. The resource transfer characteristics can be understood as the characteristics of the high-level structural information (i.e., high-order feature relationships) between the multimedia resource browsing information at each moment in the target time period.

[0178] Through the above content, by using the third access information to respectively predict the resource transfer characteristics and resource timing characteristics of the target account within the target time period, and performing feature splicing on the resource transfer characteristics and resource timing characteristics, the second preference characteristics are obtained, so that the obtained preference characteristics incorporate characteristic factors of multiple dimensions such as timing relationship and transfer relationship, thereby more accurately understanding the user's preference characteristics.

[0179] As an optional embodiment, predicting the resource timing characteristics of the target account in the target time period according to the first resource characteristics and the third access information includes:

[0180] extracting a second resource feature of each reference multimedia resource in the third access information from the first resource feature;

[0181] Extracting the second resource feature and the timing feature of each of the reference multimedia resources to obtain a third resource feature of each of the reference multimedia resources, wherein the timing feature is used to indicate the timing position of the reference multimedia resource in the third access information;

[0182] Calling a capsule network to perform feature processing on the third resource feature to obtain the resource timing feature, wherein the feature processing is used to indicate that multiple rounds of weight iterations are performed on the third resource feature through the capsule network to obtain a resource weight of each of the third resource features, and the third resource features are feature merged according to the resource weights.

[0183] Optionally, in an embodiment of the present application, a multi-level capsule network is used to perform multiple rounds of iterative updates on the resource weights of the third resource features, thereby ensuring the reliability of the weights.

[0184] Optionally, in the embodiment of the present application, the second resource feature and the time series feature of each of the reference multimedia resources are extracted to obtain the third resource feature of each of the reference multimedia resources by: extracting the second resource feature and the time series feature of each of the reference multimedia resources from the discrete value. , we assign a unique trainable parameter vector As a video Then embed it with the video Hidden representation of Add and perform linear transformation to generate Time-aware video capsules:

[0185] ,

[0186] in, is the parameter matrix to be learned. of The temporal-aware video capsules are stacked to form , and then input it into the GLU (Gated Linear Unit) model to capture the long-term dependencies between these temporal-aware video capsules and obtain temporal-enhanced video capsules :

[0187] ,

[0188] in, and are parameters to be learned; is a non-linear activation function.

[0189] Through the above content, the resource characteristics of multiple multimedia resources included in the third access information are weighted and merged to obtain the resource timing characteristics. In order to ensure the accuracy and reliability of the obtained resource characteristic weights, the weights of the third resource characteristics are iteratively updated for multiple rounds using a multi-level capsule network to ensure the reliability of the weights.

[0190] As an optional embodiment, the calling of the capsule network to perform feature processing on the third resource feature to obtain the resource time series feature includes:

[0191] In a target weight iteration round, the capsule network is called to add a perturbation parameter to the reference resource weight output by the reference weight iteration round to obtain a candidate resource weight, wherein the target weight iteration round is the current round of weight iteration to be performed, and the reference weight iteration round is the iteration round before the target weight iteration round in multiple rounds of weight iteration;

[0192] Calculating a first weight potential of the reference resource weight and a second weight potential of the candidate resource weight, wherein the weight potential is used to indicate a matching degree between the resource weight and the third access information;

[0193] Determining a difference between the first weighted potential energy and the second weighted potential energy;

[0194] When the difference value is used to indicate that the candidate resource weight satisfies a target update condition, determining the candidate resource weight as a weight output by the capsule network;

[0195] In the case where it is determined that the target weight iteration round is the last iteration round in the multiple rounds of weight iterations, the candidate resource weight is determined as the resource weight.

[0196] Optionally, in the embodiment of the present application, the iterative update process of the capsule network can be guided by a simulated annealing-based dynamic routing algorithm (SADR). It can overcome the problem that the routing algorithm in the original capsule network is sensitive to hyperparameters and lacks global optimal solution guarantee. Specifically, in each iteration In the process of updating the coupling coefficient between the time-series-aware capsule and the session capsule, the coefficient is perturbed as follows:

[0197] ,

[0198] ,

[0199] in, is a vector where each element is sampled from a standard normal distribution; is the temperature; the normalized function Use the minimum-maximum method to transform the perturbed coupling coefficient vector The elements of are scaled to the range [0,1]:

[0200] ,

[0201] in, Denotes the coupling coefficient vector No. The optimal coupling coefficient is calculated according to the following formula and the post-disturbance coupling coefficient Energy:

[0202] ,

[0203] According to the Metropolis criterion, the perturbed coefficients are accepted with a certain probability:

[0204] .

[0205] If the perturbed coupling coefficient is accepted, the optimal coupling coefficient and optimal energy are updated according to the following formula:

[0206] ,

[0207] Then update the temperature according to the following formula:

[0208] ,

[0209] in, is the attenuation parameter. The above process is iteratively performed until the temperature gradually decreases, and the coupling coefficient is optimized to achieve better information transfer in the network. The pseudo code of our proposed SADR algorithm is shown below.

[0210] 1. Initialize environmental parameters: Initialize maximum temperature , minimum temperature and the optimal coupling coefficient .

[0211] 2. When When , iterative optimization:

[0212] a) For the current coupling coefficient The implementation temperature is The random perturbation operation of ;

[0213] b) The coefficients after disturbance Perform normalization operation and get ;

[0214] c) Calculate the optimal coupling coefficient and the post-disturbance coupling coefficient energy;

[0215] d)According to the Metropolis criterion, the coefficients after perturbation are accepted with a certain probability , and update the optimal coupling coefficient and optimal energy.

[0216] e) According to Reduce temperature ,in is the decay rate.

[0217] 3. Output results: Return to the optimal coupling coefficient after the cycle ends .

[0218] Through the above content, the capsule network is iteratively updated by using the simulated annealing dynamic routing algorithm, thereby overcoming the problems of the routing algorithm in the original capsule network being sensitive to hyperparameters and lacking global optimal solution guarantee.

[0219] Optionally, in an embodiment of the present application, the essence of resource weight is the contribution value of multiple third resource features to the resource timing features of the final output. Each third resource feature is a capsule, and the final resource timing feature is also a capsule. These weights are the connection coefficients between the two layers of capsules. These two layers of capsules, coefficients, and parameters together constitute a capsule network.

[0220] As an optional embodiment, before calling the capsule network to perform feature processing on the third resource feature to obtain the resource timing feature, the method further includes:

[0221] Performing feature conversion on the third resource feature of each multimedia resource in the third access information to obtain a first spatial feature of the third resource feature in a first feature representation space and a second spatial feature of the third resource feature in a second feature representation space;

[0222] Calculating feature similarity between the first spatial feature and the second spatial feature of the third resource feature;

[0223] The resource weight of each of the third resource features in the plurality of the third resource features is allocated according to the feature similarity.

[0224] Optionally, in the embodiment of the present application, in order to avoid setting the initial value of the feature weight of the capsule network to zero and starting weight iteration from zero, the temporal enhancement video capsule (third resource feature) Input into the attention model to initialize the coupling coefficient between the temporal-aware video capsule and the conversation capsule :

[0225] (17)

[0226] (18)

[0227] , (19)

[0228] in, represents the number of iterations, express Number of columns

[0229] Through the above content, by calculating the similarity between the features of the third resource under different spatial feature dimensions, the resource weight is allocated according to the similarity, that is, the resource weight of the capsule network is configured by the attention model. Compared with the zero-value initialization used by the original capsule network, our improvement optimizes the capsule network's perception of key information in the input data, thereby enhancing the network's representation ability.

[0230] As an optional embodiment, the extracting the second resource feature and the timing feature of each of the reference multimedia resources to obtain the third resource feature of each of the reference multimedia resources includes:

[0231] Concatenate the second resource feature and the time sequence feature of the reference multimedia resource to obtain a fourth resource feature;

[0232] Input the multiple fourth resource features into the target prediction model in the time sequence of the reference multimedia resources within the target time period to obtain the fifth resource feature of each reference multimedia resource, wherein the fifth resource feature is a feature under the influence of the fourth resource feature of other multimedia resources among the multiple reference multimedia resources except the current multimedia resource.

[0233] Optionally, in an embodiment of the present application, the target prediction model may be a GLU (Gated Linear Unit) model.

[0234] Through the above content, the fourth resource feature is obtained by splicing the second resource feature and the time series feature, and then multiple fourth resource features are input into the target prediction model in time series, so as to capture the long-term dependency between these fourth resource features.

[0235] As an optional embodiment, predicting the resource transfer characteristics of the target account within the target time period according to the first resource characteristics and the third access information includes:

[0236] Extracting a second resource feature of each reference multimedia resource in the third access information from the first resource feature;

[0237] Constructing a second relationship graph according to the second resource feature and the third access information, wherein the second relationship graph records a plurality of third vertices and target directed edges connecting the third vertices, each of the third vertices is set in one-to-one correspondence with the reference multimedia resource having the second resource feature, and the target directed edges record association relationships between the reference multimedia resources;

[0238] Extracting a target hidden layer feature of each of the reference multimedia resources from the second relationship graph, wherein the target hidden layer feature is used to indicate the influence of the corresponding reference multimedia resource on the browsing preference of the target account;

[0239] The resource transfer feature of the target account is predicted based on the multiple target hidden layer features.

[0240] Optionally, in an embodiment of the present application, the method of predicting the resource transfer feature based on multiple target hidden layer features can be to average the multiple target hidden layer features to obtain the resource transfer feature, or to extract the largest one from the multiple target hidden layer features as the resource transfer feature, or to assign weights to each target hidden layer feature based on the attention mechanism and use the weights to perform weighted summation of the target hidden layer features to obtain the resource transfer feature. This scheme does not limit this.

[0241] Optionally, in the embodiment of the present application, in order to capture the high-level structural information between the multimedia resources included in the third access information, a network model for learning the intra-session transition relationship based on the weighted directed graph autoencoder can be used to learn the high-order transition relationship between different videos in each user's current basic session. First, we need to design a weighted directed activity session graph construction algorithm to model the video sequences in the user's activity session and the transition relationship between them as a weighted directed activity session graph. Construction of the weighted directed activity session graph (second relationship graph). First, each user

[0242] Active sessions Processed underlying activity session Modeled as a weighted directed graph In this graph, each vertex Represents a video (i.e., the multimedia resource in the third access information), each directed edge Indicates user In the base activity session Finished watching the video After watching the video ; Represents a directed edge The weight of , which describes the semantic distance or similarity between video pairs. Basic activity session As an example, we introduce the Weighted and Directed Intra-Session Graph Construction Algorithm (WISGC). Assume , the following are the specific steps of the algorithm:

[0243] 1. Vertex set determination: Session All different videos that appear in the session are the vertex set of the session graph. (For the above session sequence, the vertex set is , where the same video is counted only once).

[0244] 2. Edge counting and weight calculation: For adjacent video pairs in a session, count the number of directed edges between them. Calculate the weight of each directed edge by dividing the number of edge appearances by the out-degree of the source vertex, that is:

[0245] ,

[0246] in, Represents a session Any pair of directed edges in The weight of Represents a directed edge In session The number of times it appears in Represents a session China-Israel Video is the number of directed edges of the source point. The specific steps are as follows:

[0247] a. Count the number of occurrences of adjacent video pairs in the session sequence to form edge counts; ( : appears once;<B, A > : appears twice; < A, C >: appears once; < C, B >: appears once);

[0248] b. Calculate the out-degree of the vertex, that is, the number of edges that each vertex serves as the starting vertex; ( (There are two outgoing edges: and ); (There are two outgoing edges: and ); (There is one outgoing edge: ))

[0249] c. Calculate the weight of each edge based on the number of occurrences and out-degree; (w( )=1 / 2=0.5; w( )=2 / 2=1; w( ) =1 / 2 = 0.5; w( )=1 / 1=1);

[0250] d. Construct a weighted directed graph: Construct a weighted directed graph using a set of vertices and a set of weighted edges

[0251] .

[0252] Design of weighted directed graph autoencoder. This embodiment designs a weighted and directed graph autoencoder (WDGAE) to effectively learn the high-order transfer relationship between different video vertices in the session. The network contains Layer weighted directed graph encoder, The graph pooling layer is composed of a weighted directed graph decoder and a graph pooling layer. Each weighted directed graph autoencoder is responsible for mapping the hidden layer representation of each vertex in the weighted directed activity conversation graph to a low-dimensional space and outputting the low-dimensional representation vector of the vertex; each weighted directed graph decoder reconstructs the low-dimensional representation vector of the vertex into the predicted value of the adjacency matrix and outputs the reconstructed adjacency matrix. The graph pooling layer performs a pooling operation on the hidden layer representation of the vertices output by the last weighted directed graph decoder to extract the graph-level representation and obtain the embedded vector representation of the entire weighted directed activity conversation graph.

[0253] To this end, we provide users Basic activity session Maintain a weighted directed activity conversation graph , which is used to update the vertex embedding vector representation and the active session representation. The update process requires information such as vertex features and the topological structure of the graph. According to the network structure of the weighted directed graph autoencoder, the following introduces the network design details in three parts: encoder, decoder, and graph pooling layer.

[0254] 1. Encoder design. Suppose the user Basic activity session Any video in In the The hidden layer learned by the layer graph encoder is represented as Dimensional vector . The weighted directed graph encoder layer performs graph convolution operations on each video and outputs the The video hidden layer of the layer is represented as:

[0255] ,

[0256] in, Is the basic activity session All videos in The matrix composed of layer hidden layer representation; is the adjacency matrix with self-connection, where represents the identity matrix, all diagonal elements are 1 and the rest are zero; yes The degree matrix of is the parameter matrix to be learned, and satisfies The above operations gradually map video features from high-dimensional space to low-dimensional space.

[0257] 2. Decoder design. The operation of the weighted directed graph decoder layer is similar. The weighted directed graph decoder layer performs graph convolution operations on each video and outputs the The video hidden layer representation of the layer is:

[0258] ,

[0259] in, is the parameter matrix to be learned, and satisfies The above operations gradually map the video features from the low-dimensional space back to the high-dimensional space. In addition, the last weighted directed graph decoder layer also performs an adjacency matrix reconstruction operation to generate neighbor matrix prediction values ​​based on the hidden layer representation of the vertex. :

[0260] ,

[0261] 3. Graph pooling layer design. The graph pooling layer is a method for fusing all vertex features in a graph into an embedded vector representation. This method aims to capture the global features of the graph and can be used for tasks such as graph classification, graph-level prediction, and graph generation. In order to extract the representation of intra-session transfer relations from the weighted directed activity session graph, this embodiment proposes three graph pooling methods: global average pooling, global maximum pooling, and attention pooling. , let’s remember that any video output by the last layer of weighted directed graph attention layer is The hidden layer is represented as , record the weighted directed activity session graph The embedding vector is represented as (I stands for intra, T stands for transitional), that is, the user Basic Activity Session The intra-session transfer relation embedding vector representation of . Then the three pooling methods are introduced as follows:

[0262] (a). Global Average Pooling (GAP): Global Average Pooling is a simple and effective graph-level pooling method that averages the hidden layer representations of all vertices to obtain the representation of the entire weighted directed activity session graph. :

[0263] ,

[0264] Similarly, we can get the global maximum pooling (Global Max Pooling, GMP):

[0265] ,

[0266] in, It means to take the maximum value of all input vectors by component and return a new vector.

[0267] (b) Attention Pooling: Attention pooling is a pooling method based on the attention mechanism. It dynamically aggregates vertex features by learning attention weights. The steps of this method are as follows:

[0268] ,

[0269] ,

[0270] ,

[0271] in, is the parameter matrix to be learned, is the activation function (such as ReLU), is the vertex The score, is the vertex The attention weight.

[0272] 4. Reconstruction error calculation. Use the reconstructed adjacency matrix With the original adjacency matrix The difference between is taken as the reconstruction error. The commonly used loss function is the mean square error loss function:

[0273] ,

[0274] in, Represents the loss function value of the weighted directed graph autoencoder network.

[0275] Through the above content, through the design of the weighted directed graph autoencoder, based on the weighted directed graph convolutional neural network model, the neighbor vertex features, the direction and weight of the connecting edges are comprehensively considered, and the graph convolution operation is performed on the central vertex features to map them from the high-dimensional feature space back to the low-dimensional feature space, and then from the low-dimensional feature space back to the high-dimensional feature space. The adjacency matrix is ​​predicted with the help of the adjacency matrix reconstruction operation, and the reconstruction error between the reconstructed adjacency matrix and the original adjacency matrix is ​​constrained to be small enough to achieve unsupervised learning. This design can more effectively capture the complex transfer relationship and information transmission between adjacent vertices and distant vertices in the directed graph, greatly improving the expressiveness and applicability of the model. At the same time, it also reduces the dependence on expensive labeled data.

[0276] As an optional embodiment, extracting the target hidden layer feature of each reference multimedia resource from the second relationship graph includes:

[0277] Extracting initial vertex features of each third vertex from the second relationship graph and inputting them into the sequentially connected multi-layer encoders to obtain initial hidden features output by the multi-layer encoders, wherein the initial vertex features include the second resource features of the reference multimedia resource corresponding to the third vertex and the association relationship between the vertices connected to the current third vertex through the target directed edge, and each layer of the encoders is used to extract the influence relationship between the initial vertex features;

[0278] The initial hidden features are input into multiple layers of decoders connected in sequence to obtain the target hidden layer features, wherein each layer of the decoder is used to extract the influence of the reference multimedia resource corresponding to each third vertex on the browsing preference of the target account from the initial hidden features.

[0279] As an optional embodiment, constructing a second relationship graph according to the second resource feature and the third access information includes:

[0280] Creating the third vertex corresponding to the second resource feature of each of the reference multimedia resources;

[0281] Creating a target directed edge between the third vertices according to the access timing relationship between the reference multimedia resources in the third access information, wherein the input end of the target directed edge is connected to the vertex with an earlier occurrence timing, and the output end of the target directed edge is connected to the vertex with a later occurrence timing;

[0282] The target weight parameter of each target directed edge is configured according to the target number of occurrences of the timing relationship indicated by each target directed edge in the third access information to obtain the second relationship graph, wherein the target weight parameter is used to indicate the similarity relationship between the reference multimedia resource corresponding to the current third vertex and the reference multimedia resource corresponding to the vertex connected by the current target directed edge.

[0283] Through the above content, by creating the second relationship graph, the association relationship and transfer relationship between the third access information are more intuitively represented.

[0284] As an optional embodiment, configuring the target weight parameter of each target directed edge according to the target number of occurrences of the timing relationship indicated by each target directed edge in the third access information includes:

[0285] Obtain a reference number of times that all directed edges with the current third vertex as an outgoing end appear in the third access information;

[0286] Determine a quotient obtained by dividing the target number by the reference number as the target weight parameter of the corresponding directed edge;

[0287] The target weight parameter is used to configure the corresponding target directed edge.

[0288] As an optional embodiment, predicting the resource transfer feature of the target account according to the plurality of target hidden layer features includes one of the following:

[0289] Determining a mean value of a plurality of the target hidden layer features as the resource transfer feature;

[0290] Determine the maximum value of the features among the plurality of target hidden layer features as the resource transfer feature;

[0291] A corresponding feature weight is assigned to each of the target hidden layer features according to the feature relationship between the target hidden layer features; and the target hidden layer features are weighted and summed using the feature weights to obtain the resource transfer feature.

[0292] As an optional embodiment, the using the first preference feature and the second preference feature to predict the target preference feature of the target account includes:

[0293] Concatenate the first preference feature and the second preference feature to obtain a concatenated feature;

[0294] The splicing features are fused using a feature fusion model to obtain the target preference features output by the feature fusion model, wherein the conversion relationship between the splicing features and the preference features is recorded in the feature fusion model.

[0295] As an optional embodiment, the pushing of the target multimedia resource matching the target preference feature to the target account includes:

[0296] Calculating target similarity between the target preference feature and reference resource features of each multimedia resource in the multimedia resource library;

[0297] Filtering the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library;

[0298] The target multimedia resource is displayed to the target account.

[0299] As an optional embodiment, the calculating the target similarity between the target preference feature and the reference resource feature of each multimedia resource in the multimedia resource library includes:

[0300] Generate initial features of the multimedia resources in the third access information and reference features of the multimedia resources in the multimedia resource library using a feature generation model;

[0301] Detecting feature difference information between the initial feature and the reference feature;

[0302] Using the feature adjustment amount corresponding to the feature difference information, feature-adjusting the second resource feature of the multimedia resource in the third access information to obtain a target resource feature, wherein the target resource feature is used to indicate an account feature of an account that browses the multimedia resource in the third access information;

[0303] The target similarity between the target preference feature and the target resource feature is calculated.

[0304] When recommending multimedia resources, a recommendation method based on sessions (i.e., multimedia resource browsing information of an account within a reference time period) is often used. This recommendation method has the following problems: (a) The session contains not only the temporal information of user behavior, but also the complex transfer relationship between adjacent or distant items. Both types of information have an important impact on the performance of the recommendation algorithm, and it is inappropriate to consider either factor alone. Current methods usually only simulate the transfer relationship between consecutive items in the session and ignore the transition between other items in the context. (b) There is a large amount of noisy data in the session, such as items that are clicked by mistake or items that are clicked repeatedly. These noises seriously affect the modeling of user interest preferences, and thus affect the accuracy of recommendations. (c) There are many redundant calculations, and additional sessions from social friends need to be processed to make recommendations for the current session. (d) The underlying hardware deployed in the system usually uses GPUs for training and inference, which has low computing efficiency, high power consumption, and low energy efficiency.

[0305] In view of the above shortcomings, this embodiment proposes a social, temporal, and transitional semantic-aware Video Recommendation system based on Active Sessions (SVRAS). Figure 3 is an optional video recommendation schematic diagram according to an embodiment of the present application, such as Figure 3As shown, it consists of a heterogeneous graph construction component, a heterogeneous graph embedding component, a personalized session embedding component, a user interest preference fusion component, and a prediction component. Among them, the heterogeneous graph construction component is responsible for constructing a user-video heterogeneous graph containing two vertices and two connecting edges (relationships) from the friendship relationship between users in the social space and the interaction relationship between the social space and the project space. In the heterogeneous graph embedding component, this embodiment designs a neighbor sampling algorithm AGSNS (Approximation algorithm based on GreedyStrategy for Neighbor Sampling) based on submodule theory to sample the neighbors of each vertex in the user-video heterogeneous graph to remove redundant information and noise; a heterogeneous graph attention neural network is designed to extract social semantics enhanced user embedding and video embedding from the sampled heterogeneous graph. The personalized session embedding component is further divided into an intra-session temporal relationship learning network and an intra-session transfer relationship learning network. The former uses the self-developed time-aware capsule network to learn the time relationship between videos in a session to extract the embedded vector representation of the time relationship within the session; the latter uses the self-developed weighted directed graph autoencoder to extract features from the weighted directed activity session graph obtained by the weighted directed graph construction algorithm for the active session, and obtains the embedded vector representation of the transfer relationship within the session. The embedded vector representation of the time relationship within the session, the embedded vector representation of the transfer relationship within the session, and the user embedding as the user's long-term interest preference are used to obtain the user's comprehensive interest preference through the user interest preference fusion component. The prediction component calculates the probability of the user watching all videos that have not been visited based on the learned user comprehensive interest preferences, and returns the video or group of videos with the highest probability to the user as the recommendation result. In order to improve the energy efficiency of training and inference, we have also designed a general-purpose heterogeneous accelerator based on FPGA for heterogeneous graph attention neural networks, time-aware capsule networks, and weighted directed graph autoencoders.

[0306] The proposed recommendation method comprehensively considers the social relations in the social space, the temporal relationship of the interaction between users and videos in the active session, and the influence of the high-order transfer relationship between videos on the user's interest preference. It innovatively proposes a user-video heterogeneous graph construction algorithm to model the two interaction relationships between users and videos in the video recommendation system; innovatively proposes a neighbor sampling algorithm based on submodule theory to remove information redundancy and noise in the social space and project space; designs a heterogeneous graph attention neural network to mine the complex interaction relationship between users and videos in the social space and project space, and learns user embedding and video embedding enhanced by social semantics. Among them, the user embedding enhanced by social semantics accurately depicts the user's long-term interest preferences; the intra-session temporal relationship learning component based on the time-aware capsule network is innovatively proposed to learn the temporal relationship between videos from the user activity session without redundancy and noise, and extract the temporal relationship embedding vector representation within the session; the weighted directed activity session graph construction algorithm is innovatively proposed to convert the user activity session into a weighted directed graph to model the high-order transfer relationship between videos; the intra-session transfer relationship learning component based on the weighted directed graph autoencoder is innovatively proposed to extract features from the weighted directed activity session graph to obtain the intra-session transfer relationship embedding vector representation. The intra-session temporal relationship embedding vector representation and the intra-session transfer relationship embedding vector representation respectively depict the user's dynamic interest preferences from the temporal and transfer relationships between videos within the session. After being integrated with the user's long-term interest preferences, they obtain the user's comprehensive interest preferences enhanced by social, temporal and transfer semantics, which more accurately and personally depicts the user's interests and can greatly improve the accuracy of video recommendations. The proposed method only uses historical user conversations to generate social semantics to enhance user and video embeddings during heterogeneous graph embedding. It avoids repeated learning of social friends' historical conversations to integrate social influence when making recommendations for activity sessions, greatly reducing the computational complexity of the network model. In addition, we have designed a general-purpose heterogeneous accelerator based on FPGA to improve the training and reasoning efficiency of heterogeneous graph attention neural networks, time-aware capsule networks, and weighted directed graph autoencoders, and can significantly save energy consumption and reduce costs.

[0307] First, we introduce the basic symbols used in this embodiment and the problem to be solved. Specifically, all vectors and scalars are represented by bold lowercase letters (such as ) and lowercase letters (such as ) and use bold, italic, uppercase letters to denote matrices (e.g. ); Use italic capital letters to represent tuples (such as ); capital cursive letters to denote sets (e.g. ); In particular, use uppercase Euclid Math Two font to represent the range of variables (such as the and the set of integers ).

[0308] let and Respectively represent the user set and video set in the recommendation system SVRAS. There is a friend relationship between users; there is an interactive relationship between users and videos (when users Visited the video ,say and There are interactive relationships). Graphs are usually used to model the relationships between these entities.

[0309] Definition 1 (Graph) is a tuple consisting of a vertex set and a connecting edge set, which is used to describe the relationship between pairs of vertices. It is usually denoted as Among them, the vertex set Non-empty, each element represents a vertex; connected edge set Represents the connection relationship between a pair of vertices, and each element represents a connecting edge.

[0310] In addition to representing the graph as a set of two-tuples, we can also use the adjacency matrix Represents the association relationship between vertices. Row, No. Column Elements Represents a vertex and The relationship between them. Vertices directly connected by connecting edges For its first-order neighbors, all first-order neighbors form the vertex Neighborhood set , so there is .vertex Neighborhood Set Cardinality Vertex The degree of .

[0311] Depending on whether the connecting edges have weights, graphs can be divided into weighted graphs and unweighted graphs. For weighted graphs, each edge All with weight , which is used to describe the similarity between a pair of vertices connected by the edge. Adjacency matrix of weighted graph Middle Row, No. Column Elements Represents a vertex and The similarity between and When there are connecting edges between ;otherwise For an unweighted graph, when and When there are connecting edges between ;otherwise .

[0312] Depending on whether the connecting edges have directions, the graph can be divided into directed graphs and undirected graphs, and recorded and The directed and undirected edges between them are and .when When it is a directed graph, the vertices Neighborhood set It can be divided into the incident neighbor set and the outgoing neighbor set . So there is .in, , . Accordingly, the vertex The degree is further divided into in-degree and out-degree , and are defined as and .

[0313] According to the combination principle, graphs can also be divided into weighted directed graphs, weighted undirected graphs, unweighted undirected graphs and unweighted directed graphs. Figure 4 kind.

[0314] Definition 2 (Ego Graph) is used to describe the relationship between a vertex (called the central vertex) and its first-order neighbor vertices in the graph. Given an undirected graph , each vertex of Egocentric map Can be defined as The subgraph with the central vertex , including the central node And all the neighboring vertices directly connected to it and the edges between them. Among them, , . Correspondingly, the egocentric graph of each vertex in the directed graph can also be defined, which is omitted here.

[0315] Definition 3 (Social Graph) is a graph that models the friendship between users. It consists of a set of user vertices and a set of connected edges that represent friendship between users. It is usually expressed as .in, Represents a collection of friend relationships between users.

[0316] In the social graph In each user Both Dimensional Features , this feature can be extracted from the user's own description information through pre-trained deep learning models such as Bert, Word2Vec, etc. The features of all users constitute the user feature matrix .user There are several friends, and the set of these friends is , and call them From neighbors in the social space. Since the friend relationship is a symmetric relationship, that is, if the user Yes Friends, then Too friends, thus the social graph is an undirected graph. Obviously, for any undirected graph, we can convert it into a directed graph by replacing each of its edges with a pair of directed edges in opposite directions.

[0317] In addition, each user It is also associated with a length of The conversation sequence c .in, Indicates the current moment; Indicates user In time period The possible behaviors include video clicks, favorites, and ratings. They form a time-ordered session, and the video Viewed before , and said and There is a transfer relationship, denoted as The length of the session is recorded as ,The session lengths of different users may be the same or different. ,The interaction relationship between users and videos is usually modeled with the help of ,user-video bipartite graph.

[0318] Definition 4 (User-video Bipartite Graph) is a graph that models the interaction relationship between users and videos. It consists of a set of user vertices and a set of video vertices as well as a set of connected edges representing the interaction relationship between users and videos. It is usually expressed as . in, represents a vertex set; represents the set of user-video interaction edges.

[0319] In the user-video bipartite graph In each user The neighbor set of is composed of all videos that the user has watched, which can be recorded as , then ; Each video The neighbor set of is only composed of all users who have watched the video. Let's remember these neighbor sets from the social space as , then . Although the fact that the interaction action is directional, the user Videos they watched The connecting edge between them should be a directed edge, but in order to facilitate the subsequent propagation and fusion of the friend relationship in the social space and the transfer relationship in the project space in the two spaces through the interactive relationship, this embodiment considers that the user-video bipartite graph is an undirected graph.

[0320] For each video , extract the visual, audio and text features of the video through existing deep learning technologies such as ResMNet, SoundNet and GloVe, and fuse them to obtain Dimensional Features , the features of all videos form the video feature matrix .

[0321] Definition 5 (Video Transition Graph) is a directed graph that models the transition relationship between videos. It consists of a set of video vertices and a set of directed connected edges that describe the transition relationship between videos. It is usually expressed as . in, is the set of vertices consisting of all videos; Represents the transition relationship between videos.

[0322] In the video recommendation system, a video transition graph can be constructed based on the historical access records of all users. Specifically, for each user Sequence of conversations , find all consecutively accessed video pairs and ,in is the starting video of the visit. is the video that is accessed later. Each video and Both represent a directed edge , the directed edges obtained from all video pairs of all users constitute the directed edge set of the video transfer relationship graph .video Neighborhood set By the incident neighbor set and the outgoing neighbor set Together they constitute .in, , .

[0323] Definition 6 (User-video Heterogeneous Graph) is a graph containing two types of vertices: users and videos, and two types of edges: social edges between users and interaction edges between users and videos. It is used to model the friendship relationship in the social space and the interaction relationship in the interaction space.

[0324] The user-video heterogeneous graph can be formed by merging the social graph and the user-video bipartite graph. Formally, the user-video heterogeneous graph can be represented as a four-tuple .in, and Represent the vertex set and the connected edge set respectively; and Vertex type mapping function and connection edge type mapping function respectively; and They are vertex type sets and edge type sets respectively. Contains two types of vertices: user and video; It includes the social relationship edges between users (social) and the interaction relationship edges between users and videos (interaction). According to Definitions 2 and 3, users Neighborhood set ; For video For example, its neighbors are only user-type vertices, so its neighbor set .

[0325] In order to construct a user-video heterogeneous graph from the social relationships between users and the interactive relationships between users and items in the video recommendation system, this embodiment proposes a user-video heterogeneous graph construction algorithm (User-video Heterogeneous Graph Construction Algorithm, UHGC), and the specific process is as follows:

[0326] 1. Initialization: Create an empty user-video heterogeneous graph , and define a set of vertex types and the set of connecting edge types ;

[0327] 2. Add vertices: Traverse all users and videos in the video recommendation system, and add each user and each video as a single vertex to the vertex set in the graph. and add them to the user vertex set according to the vertex type. and video vertex set , build vertex type mapping function ;

[0328] 3. Add edges: According to the social relationships between users and the interactive relationships between users and videos, add corresponding edges to the graph. Specifically:

[0329] (a) For the social relationship edges between users, traverse the social relationship data, for each pair of friend relationships, if the user Is a user 's friends, then in the picture Midpoint and Add an undirected social edge between the two. Add to connected edge set middle;

[0330] (b) For the interaction edge between a user and a video, we traverse all the historical sessions of all users. For a record of interaction between a user and a video, if the user With Video If there is interaction, then in the figure Midpoint and Add an undirected interaction edge between the two and Add to connected edge set middle;

[0331] (c) Construct a connection edge type mapping function based on the type of each edge ;

[0332] 4. Return result: Return the user-video heterogeneous directed graph .

[0333] The technical problem solved by this embodiment is the problem of social-aware video recommendation based on conversation. It is defined as follows:

[0334] Definition 7 (Session-based Socially Aware Video Recommendation Problem) Given a user set , video collection and each user Historical conversation sequence and social networks .in, Indicates user The task of the session-based social recommendation problem is to predict the user’s video viewing behavior history and social information based on the above video viewing behavior history and social information. In the new session is the active session In the next time step Most likely to visit videos , and the video has been visited before, it satisfies . Common recommendation algorithms return a A recommendation list of videos, which are ordered from high to low according to the probability of users watching them. recommend.

[0335] The following is a detailed description of each invention part of the SVRAS:

[0336] Heterogeneous Graph Attention Network: Generally speaking, there are problems of information imbalance, redundancy and failure in video recommendation systems:

[0337] 1. The interaction relationships between users and videos are usually more than social relationships. Direct vertex aggregation may lead to information imbalance between social knowledge and historical interactions, resulting in weaker impact on social relationships.

[0338] 2. There are often some redundant or invalid relationships in the interaction between users and videos. The following are some situations and examples that may cause redundancy or invalidity in the interaction between users and videos: (a) Invalid or repeated click problems of some videos: Some videos may be considered invalid by users due to poor content quality, not to the user's taste, or other reasons. For example, some videos may be low-quality advertisements or very unpopular content. (b) Repeated clicks: Users may click on the same video multiple times, which may be because the user forgets that he has watched the video or makes a mistake. Repeated clicks may cause the recommendation system to over-recommend the same video, affecting the user experience.

[0339] 3. There may be some redundant or invalid relationships in the social relationship network. The following are some situations and examples that may lead to redundancy or invalidity in social relationships: (a) Invalid social connections: Some social connections may be invalid because they do not reflect real social relationships or interactions. For example, on some social platforms, users may form connections due to misoperation or meaningless attention, and such connections do not represent actual social relationships. (b) Redundant social interactions: Some social interactions may be redundant because they do not provide additional information or contribute value. For example, users frequently have similar interactions or repeated exchanges, but these interactions do not add new insights or information. (c) Outdated or invalid social relationships: Some social relationships may be outdated or invalid because they reflect past connections or interactions but no longer have actual meaning or value. For example, the interactions between users may decrease or stop over time, but these relationships still exist in the social network. (d) False or fictitious social relationships: In some cases, social relationships may be false or fictitious because they are formed based on false information or fake behavior. For example, some users may artificially increase the number of their social connections by purchasing followers or using bots, which are not real or valid. (e) Irrelevant or meaningless social interactions: Sometimes the interactions in social connections may be irrelevant or meaningless because they are not related to the user’s real interests or purposes. For example, some social interactions may be insignificant comments or likes without real communication or interaction.

[0340] From the above problems, we can see that both the user-video interaction network and the social relationship network have information imbalance, redundancy or invalidity problems. These problems may affect the modeling and understanding of user interests, item representations and social relationships, reduce recommendation diversity and accuracy, and thus affect user experience.

[0341] To solve the above problems, this embodiment draws on the graph neural network and submodule theory to design a heterogeneous graph attention neural network (HGAT) to effectively learn the relationship and feature representation between different types of vertices. The network contains The layer heterogeneous graph attention layer, each layer includes two functional components: neighbor sampling and aggregation update. The former is based on the submodule theory, and selects a set of basic neighbor subsets that are non-redundant, representative, and diverse from the neighbor set of each type of central vertex. The aggregation update component is based on the attention mechanism, and learns different weights for different neighbors selected for each type of central vertex to alleviate the imbalance. To this end, we maintain a user-video heterogeneous graph It is used to update the embedding vector representation of users or videos. The update process requires information from the neighboring vertices of users and videos. and Video In the The learned hidden layer representations are and For users , its neighbor set Contains two types of vertices: user vertices and video vertices; for videos , its neighbor set Contains only one type of vertex: the user vertex that accesses it, and thus its neighbor set .

[0342] Neighbor sampling: Neighbor sampling refers to sampling from each type of central vertex The neighbor set of In the above example, we select a set of basic neighbor subsets that are non-redundant, representative, and diverse. , and satisfy the cardinality of the subset No more than When the neighbors of the central vertex have multiple types, it is required that there is at least one vertex of each type. In order to reduce the complexity of the network model and optimize the model application scenario, it is advisable to take the basic neighbor subset of the neighbor set of all vertices The upper limit of the cardinality is The above problem can be abstracted into an optimization problem: the optimization target is the basic neighbor subset The representativeness and diversity of the elements in; the constraint is the basic neighbor subset The base number does not exceed , and there is at least one vertex of each type.

[0343] A submodule function is a function defined on a collection, usually used to measure the "value" or "importance" of the collection. is called a submodule function if for any set and any element , the following conditions are met:

[0344] , (1)

[0345] Intuitively, this means that the marginal gain of adding elements to a smaller set decreases as the size of the set increases. Submodular functions have many useful properties in optimization problems, such as being able to approximate optimal solutions via greedy algorithms and being able to be used to maximize or minimize some objective function.

[0346] Inspired by the submodule function, this embodiment models the neighbor sampling problem of the central vertex as a cardinality-constrained subset selection problem: given a user-video heterogeneous graph , from each central vertex The egocentric subgraph of Choose a base number not exceeding Subset of , and there is at least one vertex of each type, and the submodule function achieves the maximum value. This problem can be formally expressed as:

[0347] , (2)

[0348] (3)

[0349] Wherein, formulas (2) and (3) represent the objective function and constraint conditions of the optimization problem respectively; the central vertex can be either a user vertex or a video vertex; "Judge whether the two operand types are consistent and return a Boolean variable (true or false); Indicates forced type conversion, converting a Boolean variable to an integer variable.

[0350] Common submodule functions include coverage functions. , the covering function is usually used to get from the vertex set Select a subset , to cover the edges or vertices in the graph, which are divided into edge covering functions and vertex covering functions accordingly. They are defined as follows:

[0351] Edge Cover Function : The edge cover function measures the set The proportion of edges in the graph that are covered by the edges in the graph. It can be defined as:

[0352] , (4)

[0353] in, is the vertex A subset of a set.

[0354] Vertex Cover Function : Vertex cover function measures the set The proportion of vertices in the vertex cover graph. It can be defined as:

[0355] , (5)

[0356] Submodule functions It can be any one of the above formulas (5)-(6), or any other energy.

[0357] 1) Initialization: will be used to record the basic neighbor subset Initialized to an empty set; will be used to record the unselected vertex set (maybe called the candidate set) Initialize to ;

[0358] 2) Iterative greedy selection:

[0359] (a). For each vertex in the unselected vertex set , calculated according to the following formula and added The increment of the posterior objective function:

[0360] (6)

[0361] (b) Select the vertex that maximizes the increment and satisfies the constraints. , and change it from Delete and add ; If there are multiple vertices that meet the requirements, one of them is randomly selected.

[0362] (c). Update the basic neighbor subset and the candidate set: ; ;

[0363] (d). Check whether the constraints are satisfied; if so, exit the loop;

[0364] 3) Output result: Get user The sampled base neighbor subset .

[0365] Similarly, we can also use the AGSNS algorithm to solve the video The sampled base neighbor subset It should be pointed out that our framework is applicable to any submodule functions, such as mutual information function, vertex cover function, edge cover function and facility location function.

[0366] Through the neighbor sampling approximate solution algorithm based on the greedy strategy, we Redundant neighbors and noise in the social space and / or item space are removed to obtain a set of basic neighbor subsets that are non-redundant, representative, and diverse, laying the foundation for effectively extracting user embeddings and item embeddings enhanced with social relationships.

[0367] Aggregation update: Since the aggregation update process of different center vertex types is similar, we might as well take the center vertex as As an example, we introduce the basic principle of directed graph attention neural network. and any of its neighbor vertices The The hidden layer representations of the output of the layer-directed graph attention neural network are Dimensional vector and ; According to the principle of attention, The operation of the layer-heterogeneous graph attention layer can be formally described as:

[0368] , (7)

[0369] , (8)

[0370] , (9)

[0371] , (10)

[0372] , (11)

[0373] in, represents a nonlinear activation function; , , are all model parameters to be learned. At that time, Layer heterogeneous graph attention layer based on user feature matrix and the video feature matrix is the input. When the heterogeneous graph attention neural network outputs the user embedding vector representation enhanced by social semantics and video embedding vector representation .in, Denotes the dimension of the user and video embedding vector representation.

[0374] The above user-video heterogeneous graph embedding captures global knowledge such as friend relationships in the social space and all historical interaction records between users and videos at one time, greatly improving the accuracy and efficiency of modeling users' long-term interest preferences. In addition, we designed a vertex-type graph convolution operation for heterogeneous graphs, and dynamically allocated weights between different types of vertices based on the attention mechanism, effectively solving the information imbalance problem in the video recommendation system. However, the above heterogeneous graph embedding does not capture the contextual information of a specific session. However, for a personalized video recommendation system based on a session, user behavior in an active session is very important for capturing users' dynamic interests.

[0375] To this end, this embodiment designs a personalized session embedding component to extract user dynamic interest preferences from user activity sessions, which is mainly composed of an inter-session temporal relationship learning network and an inter-session transfer relationship network. The following will introduce each network one by one in two subsections.

[0376] In-session temporal relationship learning network: The in-session temporal relationship learning network learns each user Active sessions The network mainly consists of two parts: session preprocessor and time-aware capsule network. Active sessions The first one is responsible for removing redundant or invalid videos; the second one is responsible for learning the temporal relationship between videos from the preprocessed sessions to obtain session representation.

[0377] Session preprocessing procedure: This procedure can be implemented with the help of the neighbor sampling function based on submodule theory proposed above. To simplify the design of time-aware capsule network, the cardinality of the basic session obtained by sampling is fixed to . Then the session sampling problem based on submodule theory can be modeled as follows:

[0378] , (12)

[0379] , (13)

[0380] The submodule function can be any one of the above formulas (5)-(6), or any other function that can measure the "value" or "importance" of a set, which is not limited in this embodiment. Similarly, the above problem can be solved by the AGSNS algorithm proposed above, which will not be repeated here. Active sessions The fixed length obtained by the session preprocessor is Basic activity session . Because the preprocessor may disrupt the underlying active session The video access timing in Reorder the timing in and record the sorted basic activity session as .

[0381] Timing-aware capsule network: Figure 4This is an optional capsule network diagram according to an embodiment of the present application. In view of the fact that traditional capsule networks are difficult to learn the temporal relationship between entities, this embodiment adds a position embedding layer in front of the traditional capsule network to obtain a temporal-aware capsule network. It combines the ability of the position embedding layer to learn the interaction timing between users and videos and the ability of the capsule network to learn hierarchical expression, thereby enhancing the accuracy of conversation representation.

[0382] For each user Processed underlying activity session ,video The position is defined as:

[0383] . (14)

[0384] For each discrete value , we assign a unique trainable parameter vector As a video Then embed it with the video Hidden representation of Add and perform linear transformation to generate Time-aware video capsules:

[0385] , (15)

[0386] in, is the parameter matrix to be learned. of The temporal-aware video capsules are stacked to form , and then input it into the GLU (Gated Linear Unit) model to capture the long-term dependencies between these temporal-aware video capsules and obtain temporal-enhanced video capsules :

[0387] , (16)

[0388] in, and are parameters to be learned; is a nonlinear activation function. The temporal enhanced video capsule Input into the attention model to initialize the coupling coefficient between the temporal-aware video capsule and the conversation capsule :

[0389] , (17)

[0390] , (18)

[0391] , (19)

[0392] in, represents the number of iterations, express Compared with the zero-value initialization adopted by the original capsule network, our improvement optimizes the capsule network's ability to perceive key information in the input data, thereby enhancing the network's representation ability.

[0393] Then the weighted sum of the prediction results is obtained by the following formula: :

[0394] , (20)

[0395] in, Indicates that The dimensions are copied to match Then, the prediction results are weighted and Compress it into a normalized vector and keep its direction unchanged to get the session capsule :

[0396] , (twenty one)

[0397] in, is an exponential function; Represents the second norm of the vector. Then update the coupling coefficient between the temporal perception video capsule and the session capsule and enter the next iteration:

[0398] (twenty two)

[0399] go through In iterations, As a user Basic activity session Embedding vector representation of temporal relations (I stands for intra, S stands for sequential). The iterative update process is guided by our proposed simulated annealing-based dynamic routing algorithm (SADR). It can overcome the problem that the routing algorithm in the original capsule network is sensitive to hyperparameters and lacks global optimal solution guarantee. Specifically, in each iteration In the process of updating the coupling coefficient between the time-series-aware capsule and the session capsule, the coefficient is perturbed as follows:

[0400] , (twenty three)

[0401] , (twenty four)

[0402] in, is a vector where each element is sampled from a standard normal distribution; is the temperature; the normalized function Use the minimum-maximum method to transform the perturbed coupling coefficient vector The elements of are scaled to the range [0,1]:

[0403] , (25)

[0404] in, Denotes the coupling coefficient vector No. The optimal coupling coefficient is calculated according to the following formula and the post-disturbance coupling coefficient Energy:

[0405] , (26)

[0406] According to the Metropolis criterion, the perturbed coefficients are accepted with a certain probability:

[0407] (27)

[0408] If the perturbed coupling coefficient is accepted, the optimal coupling coefficient and optimal energy are updated according to the following formula:

[0409] , (28)

[0410] Then update the temperature according to the following formula:

[0411] , (29)

[0412] in, is the attenuation parameter. The above process (Formulas (23) to (29)) is iteratively executed until the temperature gradually decreases, and the coupling coefficient is optimized to achieve better information transfer in the network. The pseudo code of the proposed SADR algorithm is shown below.

[0413] 1) Initialize environmental parameters: Initialize maximum temperature , minimum temperature and the optimal coupling coefficient .

[0414] when When , iterative optimization:

[0415] For the current coupling coefficient The implementation temperature is The random perturbation operation of ;

[0416] The coefficient after perturbation Perform normalization operation and get ;

[0417] Calculate the optimal coupling coefficient and the post-disturbance coupling coefficient energy;

[0418] According to the Metropolis criterion, the coefficients after perturbation are accepted with a certain probability. , and update the optimal coupling coefficient and optimal energy.

[0419] 2) Follow Reduce temperature ,in is the decay rate.

[0420] 3) Output result: Return the optimal coupling coefficient after the cycle ends .

[0421] It should be pointed out that the above dynamic routing algorithm can also be designed based on other heuristic methods, such as genetic algorithm, particle swarm algorithm, ant colony algorithm, taboo search, and simulated annealing.

[0422] The intra-session temporal relationship learning network uses the session preprocessor to learn from each user Active sessions The redundancy and noise are removed from the data, which improves the data quality of the training network model and lays the foundation for the subsequent time capsule-aware network to accurately extract the time relationship points within the session. The time-aware capsule network is an improvement on the traditional capsule network, which is mainly reflected in the following three aspects: (1) By adding a position embedding layer, the deficiency of the traditional capsule network that cannot extract the time relationship between videos within a session is solved. (2) The capsule network coupling coefficient initialization method based on the attention mechanism optimizes the capsule network's perception of key information in the input data and enhances the representation ability of the capsule network; (3) Based on the simulated annealing dynamic routing algorithm, the problem of the routing algorithm in the original capsule network being sensitive to hyperparameters and lacking the guarantee of the global optimal solution is overcome, further enhancing the representation ability of the capsule network.

[0423] Intra-session transfer relation learning network: The intra-session temporal learning network effectively utilizes the temporal relationship between videos in the active session, but fails to capture the high-level structural information between videos. To this end, we propose an intra-session transfer relation learning network model based on a weighted directed graph autoencoder to learn the high-order transfer relations between different videos in each user's current base session. First, we need to design a weighted directed active session graph construction algorithm to model the video sequences in the user's active session and the transfer relations between them as a weighted directed active session graph.

[0424] The weighted directed activity session graph is constructed. First, each user Active sessions Processed underlying activity session Modeled as a weighted directed graph In this graph, each vertex Represents a video, each directed edge Indicates user In the base activity session Finished watching the video After watching the video ; Represents a directed edge The weight of , which describes the semantic distance or similarity between video pairs. Basic activity session As an example, we introduce the Weighted and Directed Intra-Session Graph Construction Algorithm (WISGC). Assume , the following are the specific steps of the algorithm:

[0425] 1. Vertex set determination: Session All different videos that appear in the session are the vertex set of the session graph. (For the above session sequence, the vertex set is , where the same video is only counted once)

[0426] 2. Edge counting and weight calculation: For adjacent video pairs in a session, count the number of directed edges between them. Calculate the weight of each directed edge by dividing the number of edge appearances by the out-degree of the source vertex, that is:

[0427] , (30)

[0428] in, Represents a session Any pair of directed edges in The weight of Represents a directed edge In session The number of times it appears in Represents a session China-Israel Video is the number of directed edges of the source point. The specific steps are as follows:

[0429] a. Count the number of occurrences of adjacent video pairs in the session sequence to form edge counts; (< A, B >: appears once; < B, A >: appears twice; < A, C >: appears once; < C, B >: appears once);

[0430] b. Calculate the out-degree of the vertex, that is, the number of edges that each vertex serves as the starting vertex; ( (There are two outgoing edges: and ); (There are two outgoing edges: and ); (There is one outgoing edge: ))

[0431] c. Calculate the weight of each edge based on the number of occurrences and out-degree; (w( )=1 / 2=0.5; w( )=2 / 2=1; w( ) =1 / 2 = 0.5; w( )=1 / 1=1);

[0432] d. Construct a weighted directed graph: Construct a weighted directed graph using a set of vertices and a set of weighted edges

[0433] .

[0434] Design of weighted directed graph autoencoder. This embodiment designs a weighted and directed graph autoencoder (WDGAE) to effectively learn the high-order transfer relationship between different video vertices in the session. The network contains Layer weighted directed graph encoder, The graph pooling layer is composed of a weighted directed graph decoder and a graph pooling layer. Each weighted directed graph autoencoder is responsible for mapping the hidden layer representation of each vertex in the weighted directed activity conversation graph to a low-dimensional space and outputting the low-dimensional representation vector of the vertex; each weighted directed graph decoder reconstructs the low-dimensional representation vector of the vertex into the predicted value of the adjacency matrix and outputs the reconstructed adjacency matrix. The graph pooling layer performs a pooling operation on the hidden layer representation of the vertices output by the last weighted directed graph decoder to extract the graph-level representation and obtain the embedded vector representation of the entire weighted directed activity conversation graph.

[0435] To this end, we provide users Basic activity session Maintain a weighted directed activity conversation graph , which is used to update the vertex embedding vector representation and the active session representation. The update process requires information such as vertex features and the topological structure of the graph. According to the network structure of the weighted directed graph autoencoder, the following introduces the network design details in three parts: encoder, decoder, and graph pooling layer.

[0436] 1. Encoder design. Suppose the user Basic activity session Any video in In the The hidden layer learned by the layer graph encoder is represented as Dimensional vector . The weighted directed graph encoder layer performs graph convolution operations on each video and outputs the The video hidden layer representation of the layer is:

[0437] , (31)

[0438] in, Is the basic activity session All videos in The matrix composed of layer hidden layer representation; is the adjacency matrix with self-connection, where represents the identity matrix, all diagonal elements are 1 and the rest are zero; yes The degree matrix of is the parameter matrix to be learned, and satisfies The above operations gradually map video features from high-dimensional space to low-dimensional space.

[0439] 2. Decoder design. The operation of the weighted directed graph decoder layer is similar. The weighted directed graph decoder layer performs graph convolution operations on each video and outputs the The video hidden layer representation of the layer is:

[0440] , (32)

[0441] in, is the parameter matrix to be learned, and satisfies The above operations gradually map the video features from the low-dimensional space back to the high-dimensional space. In addition, the last weighted directed graph decoder layer also performs an adjacency matrix reconstruction operation to generate neighbor matrix prediction values ​​based on the hidden layer representation of the vertex. :

[0442] , (33)

[0443] 3. Graph pooling layer design. The graph pooling layer is a method for fusing all vertex features in a graph into an embedded vector representation. This method aims to capture the global features of the graph and can be used for tasks such as graph classification, graph-level prediction, and graph generation. In order to extract the representation of intra-session transfer relations from the weighted directed activity session graph, this embodiment proposes three graph pooling methods: global average pooling, global maximum pooling, and attention pooling. , let’s remember that any video output by the last layer of weighted directed graph attention layer is The hidden layer is represented as , record the weighted directed activity session graph The embedding vector is represented as (I stands for intra, T stands for transitional), that is, the user Basic Activity Session The intra-session transfer relation embedding vector representation of . Then the three pooling methods are introduced as follows:

[0444] (a). Global Average Pooling (GAP): Global Average Pooling is a simple and effective graph-level pooling method that averages the hidden layer representations of all vertices to obtain the representation of the entire weighted directed activity session graph. :

[0445] , (34)

[0446] Similarly, we can get the global maximum pooling (Global Max Pooling, GMP):

[0447] , (35)

[0448] in, It means to take the maximum value of all input vectors by component and return a new vector.

[0449] (b) Attention Pooling: Attention pooling is a pooling method based on the attention mechanism. It dynamically aggregates vertex features by learning attention weights. The steps of this method are as follows:

[0450] , (36)

[0451] (37)

[0452] , (38)

[0453] in, is the parameter matrix to be learned, is the activation function (such as ReLU), is the vertex The score, is the vertex The attention weight.

[0454] 4. Reconstruction error calculation. Use the reconstructed adjacency matrix With the original adjacency matrix The difference between is taken as the reconstruction error. The commonly used loss function is the mean square error loss function:

[0455] , (39)

[0456] in, Represents the loss function value of the weighted directed graph autoencoder network.

[0457] The design of the weighted directed graph autoencoder is based on the weighted directed graph convolutional neural network model. It comprehensively considers the features of neighbor vertices, the direction and weight of the connecting edges, and performs graph convolution operations on the features of the central vertex to map it from the high-dimensional feature space back to the low-dimensional feature space, and then conversely maps it from the low-dimensional feature space back to the high-dimensional feature space. The adjacency matrix is ​​predicted with the help of the adjacency matrix reconstruction operation, and the reconstruction error between the reconstructed adjacency matrix and the original adjacency matrix is ​​constrained to be small enough to achieve unsupervised learning. This design can more effectively capture the complex transfer relationship and information transmission between adjacent vertices and distant vertices in the directed graph, greatly improving the expressiveness and applicability of the model. At the same time, it also reduces the dependence on expensive labeled data.

[0458] User interest preference fusion component: The user embedding output by the heterogeneous graph embedding component integrates the knowledge of friend influence in the social space and all historical conversations between users and videos, which can be regarded as the user's long-term interest preference. Many previous studies have shown that user long-term interest preferences are useful for predicting the next video that users will watch. To this end, we designed a user interest preference fusion component to fuse the user's long-term interest preference output by the heterogeneous directed graph embedding component with the intra-session temporal relationship embedding vector representation and intra-session transfer relationship embedding vector representation output by the personalized session embedding component to obtain the user's comprehensive interest preference representation. :

[0459] , (40)

[0460] in, are the network parameters to be learned; Represents a vector concatenation operation.

[0461] The above-mentioned user interest preference fusion component integrates user long-term interest preferences and personalized session embedding to obtain the user's comprehensive interest preferences that fully consider the influence of friends in the social space, the temporal relationship and transfer relationship between videos contained in the active session, and more accurately portrays the user's behavior characteristics.

[0462] Video recommendation: To generate the probability distribution of the video at the next time step, for each video with an embedded , calculate the similarity score between its embedding vector representation and the user's final interest preference representation:

[0463] , (41)

[0464] The score vector for all videos Then, the score vector is normalized by softmax to obtain the probability distribution:

[0465] , (42)

[0466] Among them, the probability distribution Characterizes the user The probability of watching each video in the next time step of the active session. Sort the probability distribution from high to low and return the previous The high-probability videos are selected as candidate sets for the next time step video. To avoid repeated recommendations, the returned videos can be constrained to be videos that the user has not watched.

[0467] Model training method: represents the true probability distribution of the next video, which is a vector using one-hot encoding. Defined as predicted value With the true value The cross entropy between:

[0468] . (43)

[0469] Then the loss function of the entire network is It is equal to the sum of the prediction error loss function and the reconstruction error loss function:

[0470] . (44)

[0471] Then, according to specific strategies such as Stochastic Gradient Descent (SGD), Momentum Gradient Descent (MGD), Nesterov Momentum, AdaGrad, RMSprop and Adam (Adaptive Moment Estimation) or Batch Gradient Descent (BGD), the parameters of the proposed SVRAS network are modified and updated to optimize the loss function value until the loss function reaches the minimum value or the number of training times reaches the specified maximum value, the training ends.

[0472] Design of a general heterogeneous accelerator based on FPGA: A general heterogeneous accelerator for heterogeneous graph attention neural networks, time-aware capsule networks, and weighted directed graph autoencoders was designed using Inspur F10A FPGA. In order to maximize the reuse of FPGA computing and storage resources, this application uniformly maps the graph convolution of the heterogeneous graph attention neural network, the dynamic routing operation of the time-aware capsule network, and the encoding, decoding, and pooling operations in the weighted directed graph autoencoder, so that multiple types of operations can be completed using the same computing array. In terms of timing structure design, reconfigurable bit-serial matrix multiplication overlay (BISMO) is used. In terms of spatial structure, a three-dimensional systolic array structure is adopted.

[0473] Figure 5 is an optional FPGA heterogeneous accelerator architecture diagram according to an embodiment of the present application, such as Figure 5 As shown in FIG. 1 , the FPGA-based general heterogeneous accelerator proposed in this embodiment mainly consists of a scheduling module, a general computing module, a nonlinear and normalization module, a pooling module, a data operation module, an off-chip memory, an input cache, and an output cache. The interconnection relationship between different modules is shown in FIG. Figure 5As shown. Among them, the off-chip memory is used to store the instructions and data sent by the CPU to the heterogeneous accelerator via the PCIE bus; the scheduling unit decodes the received instructions and distributes the decoded instructions to the data operation module, the nonlinear and normalization module, the general computing module or the pooling module; the general computing module is responsible for performing the graph convolution of the heterogeneous graph attention neural network, the dynamic routing operation of the time-series-aware capsule network, and the encoding and decoding in the weighted directed graph autoencoder; the pooling module is responsible for performing the pooling operation in the weighted directed graph autoencoder; its results are finally output to the off-chip memory via the output cache; the data operation module is used to receive the scheduling core control signal, state feedback, data handling and process state control.

[0474] The main workflow of the proposed general heterogeneous accelerator is as follows: the host computer first loads the instruction set, weight data, feature data and / or graph topology data, writes them into the off-chip memory of the heterogeneous accelerator, and then starts running. After receiving the start command, the scheduling core pre-fetches some instructions from the off-chip memory to the instruction cache, then starts to read and decode the instructions from it, and dispatches the instructions to the general computing module, nonlinear and normalization module, pooling module, data operation module, etc. for execution according to the different types of instructions, until the program end (END) instruction issued by the host is encountered, and the operation ends.

[0475] The target device of the general heterogeneous accelerator designed in this embodiment is F26A FPGA. The accelerator is synthesized using Vivado v2021.1, and the system operating frequency is 200 MHz.

[0476] Due to the low power consumption, low latency, and customizability of FPGA, this embodiment customizes a general acceleration architecture by analyzing the core operations of heterogeneous directed graph convolutional neural networks and time-aware capsule networks, which can greatly improve the training and reasoning efficiency of the proposed video recommendation system, significantly save energy consumption, and reduce costs.

[0477] Compared with the existing video recommendation methods, the video recommendation system based on social, temporal and transition relationship awareness of activity sessions proposed in this embodiment has the following beneficial effects: (1) It comprehensively considers the influence of social relationships in social space and temporal relationships and transition relationships hidden in historical interactive sessions between users and videos on user interest preferences, more accurately describes user interest preferences, and greatly improves the accuracy of video recommendation. (2) With the help of the neighbor sampling approximate solution algorithm based on the greedy strategy, the information redundancy and noise in the heterogeneous graph and user activity session are removed, while improving the accuracy of user and video embedding, the diversity of video recommendation is increased. (3) The coupling coefficient initialization method and the iterative calculation method of the coupling coefficient are improved, which greatly improves the perception ability of the temporal-aware capsule network to the key information in the activity session data, thereby enhancing the sequential representation ability of the network. (4) The design of the weighted directed graph autoencoder can more effectively capture the complex transition relationship and information transmission between adjacent vertices and distant vertices in the directed graph, greatly improving the expression ability and applicability of the model. (5) The proposed method only uses the user's historical conversation to generate social semantics enhanced user embedding once in the heterogeneous graph embedding process, avoiding repeated learning of social friends' activity conversations to integrate social influence when recommending user activity conversations, greatly reducing the computational complexity of the network model. (6) With the help of low-latency, low-power FPGA, we designed a general heterogeneous accelerator for heterogeneous graph attention neural networks, time-aware capsule networks, and weighted directed graph autoencoders, which can significantly improve the training and inference efficiency of heterogeneous graph attention neural networks, time-aware capsule networks, and weighted directed graph autoencoders, and can significantly save energy consumption and reduce costs.

[0478] Figure 6 is a schematic diagram of an optional recommended device hardware according to an embodiment of the present application, such as Figure 6 As shown, the host computer in the figure is responsible for running Figure 3The user directed graph construction component in the figure is used to obtain a user-video heterogeneous directed graph. The heterogeneous accelerator in the figure is responsible for running the heterogeneous directed graph convolutional neural network, the personalized session embedding component, the user interest preference fusion component and the prediction component. The implementation method can be: loading the video recommendation network model and the video recommendation data set from the host's disk to the main memory; further moving the video recommendation network model in the main memory to the hardware accelerator memory; the host preprocesses the video recommendation data set, constructs a user-video heterogeneous directed graph from the user's social network and browsing history, and divides the user browsing data into a training set, a test set and a validation set according to the time sequence and 0.6:0.2:0.2; the host sends a video recommendation network model training instruction to the hardware accelerator, and the hardware accelerator loads the training set and validation set data from the main memory in small batches to complete the training of the video recommendation network; the hardware accelerator continues to read the test set data from the host main memory, completes the performance test of the video recommendation network model, and returns the test results to the host; after receiving the recommendation results from the hardware accelerator, the host displays them to the user in a certain form.

[0479] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0480] In this embodiment, a multimedia resource push device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0481] Figure 7 is a structural block diagram of a multimedia resource push device according to an embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0482] A first prediction module, configured to predict a first resource feature and a first preference feature of the target account based on first access information of the target account and second access information of a reference account, wherein the reference account is an account associated with the target account, the first access information is used to indicate multimedia resource access by the target account in multiple reference time periods before a current moment, the second access information is used to indicate multimedia resource access by the reference account in multiple reference time periods, the first preference feature is the multimedia resource access preference of the target account under the influence of the second access information of the reference account, and the first resource feature is used to indicate a feature of an account that accesses the multimedia resource included in the first access information;

[0483] a second prediction module, configured to predict a second preference feature of the target account based on the first resource feature and the third access information, wherein the third access information is used to indicate the access of the target account to multimedia resources within a target time period, the target time period being the time period closest to the current moment among the multiple reference time periods, and the second preference feature is used to indicate the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period;

[0484] A third prediction module, configured to predict a target preference feature of the target account using the first preference feature and the second preference feature, wherein the target preference feature is used to indicate the multimedia resource access preference of the target account at a current moment;

[0485] The push module is used to push the target multimedia resources matching the target preference characteristics to the target account.

[0486] Through the above-mentioned device, by predicting based on the first access information of the target account and the second access information of the reference account, a first resource feature indicating the feature of the account accessing the multimedia resources included in the first access information and a first preference feature of the multimedia resource access preference of the target account under the influence of the second access information of the reference account are obtained, and then prediction is made based on the first resource feature and the third resource feature to obtain a second preference feature indicating the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period, and then the target preference feature of the target account predicted by using the first preference feature and the second preference feature incorporates both the factors of the resource browsing preference features under the influence of the social relationships of the account and the factors of the resource browsing preference features under the influence of the multimedia resources accessed by the target account itself within the target time period, so that the target preference feature is closer to the target account's actual current access preference for multimedia resources, and then the target multimedia resources matched with the target preference feature pushed to the target account can better meet the target account's current demand for multimedia resources. It can solve the problem of low matching between the multimedia resources pushed to the target account and the target account's current resource preference for multimedia resources, thereby achieving the effect of improving the matching between the multimedia resources pushed to the target account and the target account's current resource preference for multimedia resources.

[0487] Optionally, the first prediction module includes: a construction unit, which is used to construct a first relationship graph according to the first browsing information and the second browsing information, wherein the first relationship graph includes a first vertex and a first edge between the first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, a vertex corresponding to each multimedia resource included in the first access information, and a vertex corresponding to each multimedia resource included in the second access information, and the first edge is used to represent the association relationship between accounts or the access relationship of an account to a multimedia resource; an extraction unit, which is used to extract target vertex features of the target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence, In the embodiment, the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is used to generate vertex features of the target vertex according to vertex features of a reference vertex in the current first relationship graph, and use the vertex features of the target vertex extracted by the current feature extraction layer to update the features of the target vertex in the first relationship graph, and the reference vertex is a vertex in the first relationship graph connected to the target vertex through the first edge; a determination unit is used to determine the target vertex feature of the target vertex corresponding to the target account as the first preference feature, and determine the target vertex feature of the target vertex corresponding to the multimedia resource included in the first access information as the first resource feature.

[0488] Optionally, the extraction unit is used to: call a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records vertices in the first relationship graph whose correlation degree with the target vertex is greater than a target correlation degree, and the target feature extraction layer is an extraction layer among multiple feature extraction layers that is to perform feature extraction on the target vertices in the first relationship graph; call the target feature extraction layer to perform feature extraction on vertex features currently possessed by the vertices included in the target vertex set to obtain reference vertex features of the target vertex output by the target feature extraction layer; and when the target feature extraction layer is the last layer among multiple feature extraction layers connected in sequence, determine the reference vertex features as the target vertex features.

[0489] Optionally, the extraction unit is used to: calculate the initial association degree between the target vertex and each of the second vertices according to the vertex features currently possessed by the target vertex and the vertex features currently possessed by the second vertices in the first relationship graph, wherein the second vertex is any vertex included in the target vertex set;

[0490] The features of the vertices in the target vertex set are aggregated according to the initial association degree to obtain the reference vertex features.

[0491] Optionally, the extraction unit is used to: assign an attention weight to each second vertex according to the initial association degree; use the attention weight to weight the vertex features of the corresponding second vertices to obtain weighted vertex features of each second vertex; and perform feature aggregation on the weighted vertex features of each second vertex in the target vertex set to obtain the reference vertex features.

[0492] Optionally, the extraction unit is used to: extract an initial vertex set of the target vertex from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph that are connected to the target vertex through the first edge; filter out a candidate number of candidate vertices from the initial vertex set according to vertex features currently possessed by the vertices in the initial vertex set, to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies a target dependency condition.

[0493] Optionally, the extraction unit is used to: use the vertex features of each vertex included in the first vertex set of the current screening round to calculate the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change of the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set;

[0494] Extracting vertices whose increment values ​​are greater than or equal to the target increment value from the first vertex set and adding them to the second vertex set;

[0495] When the number of candidate vertices included in the second vertex set to which vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, the second vertex set is determined as the target vertex set.

[0496] Optionally, the extraction unit is used to: calculate vertex cover information of the first vertex set by using a vertex cover function, wherein the vertex cover information is used to indicate the proportion of edges in an edge cover graph of the first vertex set;

[0497] The edge coverage information of the first vertex set is calculated by using an edge coverage function, wherein the edge coverage information is used to indicate the proportion of vertices in a vertex coverage graph of the first vertex set.

[0498] Optionally, the second prediction module includes: a prediction unit, used to predict the resource timing characteristics of the target account within the target time period based on the first resource characteristics and the third access information, and predict the resource transfer characteristics of the target account within the target time period based on the first resource characteristics and the third access information, wherein the resource timing characteristics are used to indicate the temporal change relationship of the target account's browsing preference for multimedia resources within the target time period, and the resource transfer characteristics are used to indicate the browsing preference of the target account under the influence of each multimedia resource browsed within the target time period; a first splicing unit, used to perform feature splicing on the resource timing characteristics and the resource transfer characteristics to obtain the second preference characteristics.

[0499] Optionally, the prediction unit is used to: extract the second resource feature of each reference multimedia resource in the third access information from the first resource feature; perform feature extraction on the second resource feature and timing feature of each reference multimedia resource to obtain the third resource feature of each reference multimedia resource, wherein the timing feature is used to indicate the timing position of the reference multimedia resource in the third access information; call the capsule network to perform feature processing on the third resource feature to obtain the resource timing feature, wherein the feature processing is used to indicate performing multiple rounds of weight iterations on the third resource feature through the capsule network to obtain the resource weight of each third resource feature, and perform feature merging on the third resource features according to the resource weight.

[0500] Optionally, the prediction unit is used to: call the capsule network in a target weight iteration round to add a perturbation parameter to the reference resource weight output by a reference weight iteration round to obtain a candidate resource weight, wherein the target weight iteration round is the current round of weight iteration to be performed, and the reference weight iteration round is the iteration round in multiple rounds of weight iteration that is located before the target weight iteration round; calculate a first weight potential of the reference resource weight and a second weight potential of the candidate resource weight, wherein the weight potential is used to indicate a degree of matching between the resource weight and the third access information; determine a difference between the first weight potential and the second weight potential; when the difference value is used to indicate that the candidate resource weight meets a target update condition, determine the candidate resource weight as the weight output by the capsule network; when it is determined that the target weight iteration round is the last iteration round in the multiple rounds of weight iteration, determine the candidate resource weight as the resource weight.

[0501] Optionally, the device also includes: a conversion module, used to perform feature conversion on the third resource feature of each multimedia resource in the third access information before calling the capsule network to perform feature processing on the third resource feature to obtain the resource timing feature, so as to obtain a first spatial feature of the third resource feature in the first feature representation space and a second spatial feature of the third resource feature in the second feature representation space; a calculation module, used to calculate the feature similarity between the first spatial feature and the second spatial feature of the third resource feature; and an allocation module, used to allocate the resource weight of each of the multiple third resource features according to the feature similarity.

[0502] Optionally, the prediction unit is used to: perform feature splicing on the second resource feature and the timing feature of the reference multimedia resource to obtain a fourth resource feature; input multiple fourth resource features into the target prediction model in the time sequence of the reference multimedia resources within the target time period to obtain a fifth resource feature of each of the reference multimedia resources, wherein the fifth resource feature is a feature under the influence of the fourth resource feature of other multimedia resources among the multiple reference multimedia resources except the current multimedia resource.

[0503] Optionally, the prediction unit is used to: extract the second resource feature of each reference multimedia resource in the third access information from the first resource feature; construct a second relationship graph based on the second resource feature and the third access information, wherein the second relationship graph records multiple third vertices and target directed edges connecting the third vertices, each of the third vertices and the reference multimedia resource with the second resource feature is set in a one-to-one correspondence, and the target directed edges record the association relationship between the reference multimedia resources; extract the target hidden layer feature of each reference multimedia resource from the second relationship graph, wherein the target hidden layer feature is used to indicate the influence of the corresponding reference multimedia resource on the browsing preference of the target account; and predict the resource transfer characteristics of the target account based on multiple target hidden layer features.

[0504] Optionally, the prediction unit is used to: extract the initial vertex features of each of the third vertices from the second relationship graph and input them into sequentially connected multi-layer encoders to obtain initial hidden features output by the multi-layer encoders, wherein the initial vertex features include the second resource features of the reference multimedia resource corresponding to the third vertex and the association relationship between the vertices connected to the current third vertex through the target directed edge, and each layer of the encoder is used to extract the influence relationship between the initial vertex features; input the initial hidden features into sequentially connected multi-layer decoders to obtain the target hidden layer features, wherein each layer of the decoder is used to extract the influence of the reference multimedia resource corresponding to each of the third vertices on the browsing preference of the target account from the initial hidden features.

[0505] Optionally, the prediction unit is used to: create the third vertex corresponding to the second resource feature of each of the reference multimedia resources; create a target directed edge between the third vertices according to the access timing relationship between the reference multimedia resources in the third access information, wherein the input end of the target directed edge is connected to the vertex with an earlier occurrence timing, and the output end of the target directed edge is connected to the vertex with a later occurrence timing; configure the target weight parameter of each target directed edge according to the target number of occurrences of the timing relationship indicated by each target directed edge in the third access information, to obtain the second relationship graph, wherein the target weight parameter is used to indicate the similarity relationship between the reference multimedia resource corresponding to the current third vertex and the reference multimedia resource corresponding to the vertex connected by the current target directed edge.

[0506] Optionally, the prediction unit is used to: obtain a reference number of times that all directed edges with the current third vertex as the output end appear in the third access information; determine the quotient obtained by dividing the target number by the reference number as the target weight parameter of the corresponding directed edge; and use the target weight parameter to configure the corresponding target directed edge.

[0507] Optionally, the prediction unit is used to: determine the mean of multiple target hidden layer features as the resource transfer feature; determine the maximum feature value among the multiple target hidden layer features as the resource transfer feature; assign a corresponding feature weight to each target hidden layer feature according to the feature relationship between the target hidden layer features; and use the feature weights to perform weighted sum calculation on the target hidden layer features to obtain the resource transfer feature.

[0508] Optionally, the first prediction module includes: a second splicing unit, used to perform feature splicing on the first preference feature and the second preference feature to obtain a spliced ​​feature; and a fusion unit, used to perform feature fusion on the spliced ​​feature using a feature fusion model to obtain the target preference feature output by the feature fusion model, wherein the feature fusion model records the conversion relationship between the splicing feature and the preference feature.

[0509] Optionally, the push module includes: a calculation unit, used to calculate the target similarity between the target preference characteristics and the reference resource characteristics of each multimedia resource in the multimedia resource library; a screening unit, used to screen out the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library; and a display unit, used to display the target multimedia resources to the target account.

[0510] Optionally, the computing unit is used to: use a feature generation model to generate initial features of the multimedia resources in the third access information and reference features of the multimedia resources in the multimedia resource library; detect feature difference information between the initial features and the reference features; use a feature adjustment amount corresponding to the feature difference information to perform feature adjustment on the second resource features of the multimedia resources in the third access information to obtain target resource features, wherein the target resource features are used to indicate account features of an account that browses the multimedia resources in the third access information; and calculate the target similarity between the target preference features and the target resource features.

[0511] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0512] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0513] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0514] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0515] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0516] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0517] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0518] The embodiments of the present application also provide a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0519] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0520] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0521] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for pushing multimedia resources, characterized in that: include: Predicting a first resource feature and a first preference feature of the target account based on first access information of the target account and second access information of a reference account, wherein the reference account is an account associated with the target account, the first access information is used to indicate multimedia resource access by the target account in multiple reference time periods before a current moment, the second access information is used to indicate multimedia resource access by the reference account in multiple reference time periods, the first preference feature is the multimedia resource access preference of the target account under the influence of the second access information of the reference account, and the first resource feature is used to indicate the feature of an account that accesses the multimedia resource included in the first access information; Predicting a second preference feature of the target account based on the first resource feature and the third access information, wherein the third access information is used to indicate the access of the target account to multimedia resources within a target time period, the target time period being the time period closest to the current moment among the multiple reference time periods, and the second preference feature is used to indicate the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period; Predicting a target preference feature of the target account using the first preference feature and the second preference feature, wherein the target preference feature is used to indicate the multimedia resource access preference of the target account at the current moment; Pushing target multimedia resources matching the target preference characteristics to the target account.

2. The method according to claim 1, characterized in that The predicting the first resource feature and the first preference feature of the target account according to the first access information of the target account and the second access information of the reference account includes: Constructing a first relationship graph according to the first browsing information and the second browsing information, wherein the first relationship graph includes a first vertex and a first edge between first vertices, the first vertex includes a vertex corresponding to the target account, a vertex corresponding to the reference account, a vertex corresponding to each multimedia resource included in the first access information, and a vertex corresponding to each multimedia resource included in the second access information, and the first edge is used to represent an association relationship between accounts or an access relationship between accounts and multimedia resources; Extracting target vertex features of a target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence, wherein the target vertex is any vertex in the first relationship graph, each of the feature extraction layers is used to generate vertex features of the target vertex according to vertex features of a reference vertex in the current first relationship graph, and updating features of the target vertex in the first relationship graph using the vertex features of the target vertex extracted by the current feature extraction layer, wherein the reference vertex is a vertex in the first relationship graph connected to the target vertex through the first edge; The target vertex feature of the target vertex corresponding to the target account is determined as the first preference feature, and the target vertex feature of the target vertex corresponding to the multimedia resource included in the first access information is determined as the first resource feature.

3. The method according to claim 2, characterized in that The extracting the target vertex feature of the target vertex in the first relationship graph using a feature extraction model including a plurality of feature extraction layers connected in sequence comprises: Calling a target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph, wherein the target vertex set records vertices in the first relationship graph whose association degree with the target vertex is greater than a target association degree, and the target feature extraction layer is an extraction layer among the multiple feature extraction layers that is to perform feature extraction on the target vertex in the first relationship graph; Calling the target feature extraction layer to extract features of the vertices currently included in the target vertex set, and obtaining reference vertex features of the target vertices output by the target feature extraction layer; In a case where the target feature extraction layer is the last layer of a plurality of sequentially connected feature extraction layers, the reference vertex feature is determined as the target vertex feature.

4. The method according to claim 3, characterized in that The calling of the target feature extraction layer to extract features of the vertex features currently possessed by the vertices included in the target vertex set to obtain reference vertex features of the target vertices output by the target feature extraction layer includes: Calculating an initial degree of association between the target vertex and each of the second vertices according to the vertex features currently possessed by the target vertex and the vertex features currently possessed by the second vertices in the first relationship graph, wherein the second vertex is any vertex included in the target vertex set; The features of the vertices in the target vertex set are aggregated according to the initial association degree to obtain the reference vertex features.

5. The method according to claim 4, characterized in that The step of performing feature aggregation on the vertices in the target vertex set according to the initial association degree to obtain the reference vertex features includes: assigning an attention weight to each of the second vertices according to the initial relevance; Using the attention weight to weight the vertex features of the corresponding second vertices, to obtain a weighted vertex feature of each second vertex; Feature aggregation is performed on the weighted vertex feature of each second vertex in the target vertex set to obtain the reference vertex feature.

6. The method according to claim 3, characterized in that The calling of the target feature extraction layer to extract a target vertex set of target vertices from the first relationship graph includes: Extracting an initial vertex set of the target vertex from the first relationship graph, wherein the initial vertex set includes vertices in the first relationship graph connected to the target vertex through the first edge; A candidate number of candidate vertices are screened out from the initial vertex set according to vertex features currently possessed by vertices in the initial vertex set to obtain the target vertex set, wherein the dependency relationship between the vertices in the target vertex set and the target vertex satisfies a target dependency relationship condition.

7. The method according to claim 6, characterized in that The step of selecting a candidate number of candidate vertices from the initial vertex set according to the vertex features currently possessed by the vertices in the initial vertex set to obtain the target vertex set includes: Using the vertex features of each vertex included in the first vertex set of the current screening round, calculating the incremental value of each vertex to the vertex set information, wherein the vertex set information is used to indicate the importance of the vertex features of all vertices in the second vertex set, and the incremental value is used to indicate the amount of change of the vertex set information of the second vertex set after the corresponding vertex in the first vertex set is added to the second vertex set, and the initial vertex set includes vertices in the first vertex set and vertices in the second vertex set; Extracting vertices whose increment values ​​are greater than or equal to the target increment value from the first vertex set and adding them to the second vertex set; When the number of candidate vertices included in the second vertex set to which vertices are added is the candidate number, and the candidate vertices include both vertices corresponding to accounts and vertices corresponding to multimedia resources, the second vertex set is determined as the target vertex set.

8. The method according to claim 7, characterized in that The calculating the vertex set information of the first vertex set includes one of the following: Calculate vertex cover information of the first vertex set by using a vertex cover function, wherein the vertex cover information is used to indicate the proportion of edges in an edge cover graph of the first vertex set; The edge coverage information of the first vertex set is calculated by using an edge coverage function, wherein the edge coverage information is used to indicate the proportion of vertices in a vertex coverage graph of the first vertex set.

9. The method according to claim 1, characterized in that: The predicting the second preference feature of the target account according to the first resource feature and the third access information includes: Predicting a resource timing feature of the target account within the target time period based on the first resource feature and the third access information, and predicting a resource transfer feature of the target account within the target time period based on the first resource feature and the third access information, wherein the resource timing feature is used to indicate a temporal change relationship of the target account's browsing preference for multimedia resources within the target time period, and the resource transfer feature is used to indicate the browsing preference of the target account under the influence of each multimedia resource browsed within the target time period; The resource timing feature and the resource transfer feature are concatenated to obtain the second preference feature.

10. The method according to claim 9, characterized in that The predicting the resource timing characteristics of the target account in the target time period according to the first resource characteristics and the third access information includes: extracting a second resource feature of each reference multimedia resource in the third access information from the first resource feature; Extracting the second resource feature and the timing feature of each of the reference multimedia resources to obtain a third resource feature of each of the reference multimedia resources, wherein the timing feature is used to indicate the timing position of the reference multimedia resource in the third access information; Calling a capsule network to perform feature processing on the third resource feature to obtain the resource timing feature, wherein the feature processing is used to indicate that multiple rounds of weight iterations are performed on the third resource feature through the capsule network to obtain a resource weight of each of the third resource features, and the third resource features are feature merged according to the resource weights.

11. The method according to claim 10, characterized in that The calling of the capsule network to perform feature processing on the third resource feature to obtain the resource time series feature includes: In a target weight iteration round, the capsule network is called to add a perturbation parameter to the reference resource weight output by the reference weight iteration round to obtain a candidate resource weight, wherein the target weight iteration round is the current round of weight iteration to be performed, and the reference weight iteration round is the iteration round before the target weight iteration round in multiple rounds of weight iteration; Calculating a first weight potential of the reference resource weight and a second weight potential of the candidate resource weight, wherein the weight potential is used to indicate a matching degree between the resource weight and the third access information; Determining a difference between the first weighted potential energy and the second weighted potential energy; When the difference value is used to indicate that the candidate resource weight satisfies a target update condition, determining the candidate resource weight as a weight output by the capsule network; In the case where it is determined that the target weight iteration round is the last iteration round in the multiple rounds of weight iterations, the candidate resource weight is determined as the resource weight.

12. The method according to claim 10, characterized in that Before calling the capsule network to perform feature processing on the third resource feature to obtain the resource time series feature, the method further includes: Performing feature conversion on the third resource feature of each multimedia resource in the third access information to obtain a first spatial feature of the third resource feature in a first feature representation space and a second spatial feature of the third resource feature in a second feature representation space; Calculating feature similarity between the first spatial feature and the second spatial feature of the third resource feature; The resource weight of each of the third resource features in the plurality of the third resource features is allocated according to the feature similarity.

13. The method according to claim 10, characterized in that The extracting the second resource feature and the time sequence feature of each of the reference multimedia resources to obtain the third resource feature of each of the reference multimedia resources includes: Concatenate the second resource feature and the time sequence feature of the reference multimedia resource to obtain a fourth resource feature; Input the multiple fourth resource features into the target prediction model in the time sequence of the reference multimedia resources within the target time period to obtain the fifth resource feature of each reference multimedia resource, wherein the fifth resource feature is a feature under the influence of the fourth resource feature of other multimedia resources among the multiple reference multimedia resources except the current multimedia resource.

14. The method according to claim 9, characterized in that The predicting the resource transfer characteristic of the target account within the target time period according to the first resource characteristic and the third access information includes: Extracting a second resource feature of each reference multimedia resource in the third access information from the first resource feature; Constructing a second relationship graph according to the second resource feature and the third access information, wherein the second relationship graph records a plurality of third vertices and target directed edges connecting the third vertices, each of the third vertices is set in one-to-one correspondence with the reference multimedia resource having the second resource feature, and the target directed edges record association relationships between the reference multimedia resources; Extracting a target hidden layer feature of each of the reference multimedia resources from the second relationship graph, wherein the target hidden layer feature is used to indicate the influence of the corresponding reference multimedia resource on the browsing preference of the target account; The resource transfer feature of the target account is predicted based on the multiple target hidden layer features.

15. The method according to claim 14, characterized in that The step of extracting a target hidden layer feature of each of the reference multimedia resources from the second relationship graph includes: Extracting initial vertex features of each third vertex from the second relationship graph and inputting them into the sequentially connected multi-layer encoders to obtain initial hidden features output by the multi-layer encoders, wherein the initial vertex features include the second resource features of the reference multimedia resource corresponding to the third vertex and the association relationship between the vertices connected to the current third vertex through the target directed edge, and each layer of the encoders is used to extract the influence relationship between the initial vertex features; The initial hidden features are input into multiple layers of decoders connected in sequence to obtain the target hidden layer features, wherein each layer of the decoder is used to extract the influence of the reference multimedia resource corresponding to each third vertex on the browsing preference of the target account from the initial hidden features.

16. The method according to claim 14, characterized in that The constructing a second relationship graph according to the second resource feature and the third access information includes: Creating the third vertex corresponding to the second resource feature of each of the reference multimedia resources; Creating a target directed edge between the third vertices according to the access timing relationship between the reference multimedia resources in the third access information, wherein the input end of the target directed edge is connected to the vertex with an earlier occurrence timing, and the output end of the target directed edge is connected to the vertex with a later occurrence timing; The target weight parameter of each target directed edge is configured according to the target number of occurrences of the timing relationship indicated by each target directed edge in the third access information to obtain the second relationship graph, wherein the target weight parameter is used to indicate the similarity relationship between the reference multimedia resource corresponding to the current third vertex and the reference multimedia resource corresponding to the vertex connected by the current target directed edge.

17. The method according to claim 16, characterized in that The step of configuring a target weight parameter of each target directed edge according to the target number of occurrences of the timing relationship indicated by each target directed edge in the third access information comprises: Obtain a reference number of times that all directed edges with the current third vertex as an outgoing end appear in the third access information; Determine a quotient obtained by dividing the target number by the reference number as the target weight parameter of the corresponding directed edge; The target weight parameter is used to configure the corresponding target directed edge.

18. The method according to claim 14, characterized in that The predicting the resource transfer feature of the target account according to the plurality of target hidden layer features comprises one of the following: Determining a mean value of a plurality of the target hidden layer features as the resource transfer feature; Determine the maximum value of the features among the plurality of target hidden layer features as the resource transfer feature; A corresponding feature weight is assigned to each of the target hidden layer features according to the feature relationship between the target hidden layer features; and the target hidden layer features are weighted and summed using the feature weights to obtain the resource transfer feature.

19. The method according to claim 1, characterized in that The using the first preference feature and the second preference feature to predict the target preference feature of the target account includes: Concatenate the first preference feature and the second preference feature to obtain a concatenated feature; The splicing features are fused using a feature fusion model to obtain the target preference features output by the feature fusion model, wherein the conversion relationship between the splicing features and the preference features is recorded in the feature fusion model.

20. The method according to claim 1, characterized in that The pushing of the target multimedia resource matching the target preference feature to the target account includes: Calculating target similarity between the target preference feature and reference resource features of each multimedia resource in the multimedia resource library; Filtering the target multimedia resources whose target similarity is greater than or equal to a preset similarity from the multimedia resource library; The target multimedia resource is displayed to the target account.

21. The method according to claim 20, characterized in that The calculating of the target similarity between the target preference feature and the reference resource features of each multimedia resource in the multimedia resource library includes: Generate initial features of the multimedia resources in the third access information and reference features of the multimedia resources in the multimedia resource library using a feature generation model; Detecting feature difference information between the initial feature and the reference feature; Using the feature adjustment amount corresponding to the feature difference information, feature-adjusting the second resource feature of the multimedia resource in the third access information to obtain a target resource feature, wherein the target resource feature is used to indicate an account feature of an account that browses the multimedia resource in the third access information; The target similarity between the target preference feature and the target resource feature is calculated.

22. A multimedia resource push device, characterized in that: include: A first prediction module, configured to predict a first resource feature and a first preference feature of the target account based on first access information of the target account and second access information of a reference account, wherein the reference account is an account associated with the target account, the first access information is used to indicate multimedia resource access by the target account in multiple reference time periods before a current moment, the second access information is used to indicate multimedia resource access by the reference account in multiple reference time periods, the first preference feature is the multimedia resource access preference of the target account under the influence of the second access information of the reference account, and the first resource feature is used to indicate a feature of an account that accesses the multimedia resource included in the first access information; a second prediction module, configured to predict a second preference feature of the target account based on the first resource feature and the third access information, wherein the third access information is used to indicate the access of the target account to multimedia resources within a target time period, the target time period being the time period closest to the current moment among the multiple reference time periods, and the second preference feature is used to indicate the multimedia resource access preference of the target account under the influence of the multimedia resources accessed within the target time period; A third prediction module, configured to predict a target preference feature of the target account using the first preference feature and the second preference feature, wherein the target preference feature is used to indicate the multimedia resource access preference of the target account at a current moment; The push module is used to push the target multimedia resources matching the target preference characteristics to the target account.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 21 when executed by a processor.

24. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 21 are implemented.

25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 21 are implemented.

Images