Short video popularity prediction method, device, electronic device and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a short video popularity prediction method, device, electronic device and storage medium, the method comprising: constructing a multimodal graph and a plurality of unimodal graphs based on a plurality of short videos; performing heterogeneous graph representation learning based on the multimodal graph and each unimodal graph, obtaining a first node representation of a target video node corresponding to the target short video in the multimodal graph and a second node representation of a target video node corresponding to the target short video in each unimodal graph; performing popularity prediction based on the first node representation and each second node representation, and obtaining the popularity of the target short video. The short video popularity prediction method provided by the present invention captures cross-modal information of interactive propagation between multimodalities through a multimodal graph, and retains unimodal original information through a unimodal graph, which helps to reduce the loss of cross-modal information, and then comprehensively predicts the popularity of the short video by modal representations of multiple dimensions, significantly improving the accuracy and efficiency of short video popularity prediction.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a short video popularity prediction method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of the Internet and multimedia technologies, short videos have quickly become popular on major digital platforms. Users can interact with content through sharing, liking, and commenting, generating a large amount of multimodal data. This surge in short videos has brought about the challenge of short video popularity prediction, a task that aims to predict the popularity index of short videos within a predefined period of time in the future. Accurate predictions can provide valuable insights into user behavior, identify trends in information dissemination, and create business opportunities.

[0003] At present, short video popularity prediction mainly relies on two methods: (1) based on traditional feature engineering methods; (2) based on deep learning methods. In the early days, researchers relied on manual feature extraction to handle the classification and prediction tasks of short videos. Although these methods are simple and intuitive, hand-crafted features often bring strong subjectivity and cannot effectively capture the deep semantic information of the data. With the development of deep learning technology, especially the application of deep neural networks, researchers have begun to use multimodal neural network methods to improve the popularity prediction of short videos. These methods have made certain progress in improving the accuracy of features. However, these methods often only focus on modal features, but often ignore the interactive propagation between multiple modalities. Therefore, the existing technology for short video popularity prediction still lacks a certain degree of accuracy. Summary of the invention

[0004] The present invention provides a short video popularity prediction method, device, electronic device and storage medium, which are used to solve the problem that the short video popularity prediction method in the prior art lacks a certain degree of accuracy.

[0005] In a first aspect, the present invention provides a short video popularity prediction method, comprising:

[0006] Based on multiple short videos, construct a multimodal graph and multiple unimodal graphs; the multimodal graph is a heterogeneous graph that retains multiple modal information; the unimodal graph is a heterogeneous graph that retains a single modal information;

[0007] Performing heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph and a second node representation of a target video node corresponding to a target short video in each of the unimodal graphs;

[0008] Popularity prediction is performed based on the first node representation and each of the second node representations to obtain the popularity of the target short video.

[0009] In one embodiment, the multiple unimodal graphs include a first unimodal graph and a second unimodal graph; and constructing a multimodal graph and multiple unimodal graphs based on multiple short videos includes:

[0010] Determine the node types and edge types associated with the plurality of short videos;

[0011] Based on the node type and the edge type, construct a heterogeneous graph;

[0012] Determine the video frame features extracted from each of the short videos as the node representation of the corresponding video frame node in the heterogeneous graph, determine the video title features extracted from each of the short videos as the node representation of the corresponding video title node in the heterogeneous graph, and initialize the node representations of the remaining nodes to obtain a multimodal graph; the remaining nodes are the nodes in the heterogeneous graph other than the video frame nodes and the video title nodes;

[0013] Initializing the node representation of each of the video frame nodes in the multimodal graph to obtain a first unimodal graph, and initializing the node representation of each of the video title nodes in the multimodal graph to obtain a second unimodal graph.

[0014] In one embodiment, performing heterogeneous graph representation learning based on the multimodal graph to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph includes:

[0015] Determine at least one meta-path in the multimodal graph that has a target video node corresponding to the target short video as an end node;

[0016] Calculating neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain an updated first node representation of the target video node on each meta-path;

[0017] Iteratively calculate the neighbor node information of the target video node on each meta-path, and update the node representation of the target video node according to the neighbor node information to obtain the updated first node representation of the target video node on each meta-path, until the number of iterations reaches L times, and obtain the updated first node representation of the target video node on each meta-path in each iteration process.

[0018] In one embodiment, the calculating of neighbor node information of the target video node on each meta-path and updating the node representation of the target video node according to each of the neighbor node information to obtain the updated first node representation of the target video node on each meta-path includes:

[0019] If the number of hops from the start node to the end node in any meta-path is an odd number, the neighbor node information of the target video node on the meta-path is calculated by a first formula, and the node representation of the target video node is updated according to the neighbor node information to obtain a first node representation of the target video node after the update on the meta-path;

[0020] If the number of hops from the starting node to the ending node in any meta-path is an even number, the neighbor node information of the target video node on the meta-path is calculated by the second formula, and the node representation of the target video node is updated according to the neighbor node information to obtain the updated first node representation of the target video node on the meta-path.

[0021] In one embodiment, after performing heterogeneous graph representation learning based on the multimodal graph to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph, and before performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video, the method further includes:

[0022] Determine a plurality of short videos similar to the target short video from the plurality of short videos;

[0023] Determine the updated third node representation of the video nodes corresponding to each of the similar short videos on each meta-path during each iteration; the meta-path of the target video node corresponding to the target short video in the multimodal graph is consistent with the meta-path of the video nodes corresponding to each of the similar short videos in the multimodal graph;

[0024] Calculating the mean of each of the third node representations on each meta-path during the iteration process to obtain a mean result on each meta-path during the iteration process;

[0025] The mean results in the iteration process are determined as the retrieval embedding representation of the target video node on each meta-path in the iteration process.

[0026] In one embodiment, determining a plurality of similar short videos of the target short video from the plurality of short videos includes:

[0027] Calculate the similarity between the retrieval features of the target short video and the retrieval features of the remaining short videos to obtain multiple similarity values; the retrieval features are extracted from the retrieval information obtained by splicing the video title and the video subtitle; the remaining short videos are the remaining short videos in the multiple short videos except the target short video;

[0028] The remaining short videos corresponding to the top K maximum similarity values ​​among the similarity values ​​are determined as similar short videos of the target short video.

[0029] In one embodiment, performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video includes:

[0030] Fusing L first node representations of the target video node on each meta-path in the multimodal graph to obtain a first fused node representation of the target video node on each meta-path in the multimodal graph;

[0031] Fusing L second node representations of the target video node on each meta-path in the first unimodal graph to obtain a second fused node representation of the target video node on each meta-path in the first unimodal graph;

[0032] Fusing L second node representations of the target video node on each meta-path in the second unimodal graph to obtain a third fused node representation of the target video node on each meta-path in the second unimodal graph;

[0033] Fusing L retrieval embedding representations of the target video node on each meta-path in the multimodal graph to obtain a fourth fused node representation of the target video node on each meta-path in the multimodal graph;

[0034] fusing each of the first fusion node representations, each of the second fusion node representations, each of the third fusion node representations, and each of the fourth fusion node representations to obtain a target node representation of the target video node;

[0035] The popularity of the target node representation is predicted to obtain the popularity of the target short video.

[0036] In a second aspect, the present invention further provides a short video popularity prediction device, comprising:

[0037] A construction module, for constructing a multimodal graph and a plurality of unimodal graphs based on a plurality of short videos; the multimodal graph is a heterogeneous graph retaining information of multiple modalities; the unimodal graph is a heterogeneous graph retaining information of a single modality;

[0038] A heterogeneous graph representation learning module, used to perform heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs, to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph and a second node representation of a target video node corresponding to a target short video in each of the unimodal graphs;

[0039] A popularity prediction module is used to perform popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video.

[0040] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the short video popularity prediction methods described above are implemented.

[0041] In a fourth aspect, the present invention further provides a storage medium, comprising a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the short video popularity prediction methods described above are implemented.

[0042] The short video popularity prediction method, device, electronic device and storage medium provided by the present invention construct a multimodal graph, and construct a unimodal graph by selectively shielding the representations of other modalities to retain the unique information of each modality. When heterogeneous graph representation learning is performed based on the multimodal graph and each unimodal graph, the multimodal graph can capture cross-modal information of interactive propagation between multimodalities, and the unimodal graph can retain the original information of the unimodal modality, which helps to reduce the loss of cross-modal information and ensure sufficient modal information, optimize the overall modal representation, and then according to the modal representation of the target video node in the multimodal graph and the modal representation of the target video node in each unimodal graph, the popularity of the short video is comprehensively predicted by the modal representations of multiple dimensions, which significantly improves the accuracy and efficiency of short video popularity prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a flow chart of the short video popularity prediction method provided by the present invention.

[0045] Figure 2 It is a schematic diagram of the architecture of the short video popularity prediction method provided by the present invention.

[0046] Figure 3 It is a schematic diagram of the architecture of the short video popularity prediction provided by the present invention.

[0047] Figure 4 It is a structural schematic diagram of the short video popularity prediction device provided by the present invention.

[0048] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein.

[0051] Combine the following Figure 1-Figure 5 The present invention describes a short video popularity prediction method, device, electronic device and storage medium.

[0052] It should be noted that the short video popularity prediction method provided by the embodiment of the present invention is implemented based on the short video popularity prediction device. The short video popularity prediction method provided by the present invention introduces a heterogeneous graph representation learning method, and captures modal information in a complex heterogeneous graph through a heterogeneous graph neural network. In view of the problem that the existing heterogeneous graph neural network cannot effectively integrate modal information and global information in predicting popularity, an improved method based on a heterogeneous graph neural network is proposed. By constructing a multimodal graph, and optimizing it to a single modal graph through a conversion strategy based on the multimodal graph, and integrating the multimodal graph and the single modal graph for heterogeneous graph representation learning, information loss can be avoided and information redundancy can be reduced. Furthermore, in the heterogeneous graph neural network to implement heterogeneous graph representation learning, pre-calculation and meta-path technology are used to effectively integrate different modal information and improve processing efficiency. In addition, a similarity retrieval mechanism for short video content is introduced to calculate the global information between videos to capture video similarities beyond the graph structure. Therefore, the local information of the target node is captured through the heterogeneous graph neural network, and the global information between videos is captured through the similarity retrieval mechanism. The local information and the global information are integrated, and the popularity of short videos is predicted through the encoder to improve the prediction ability.

[0053] The embodiment of the present invention describes a short video popularity prediction method by taking a short video popularity prediction device as an execution subject as an example.

[0054] Reference Figure 1-Figure 2 , Figure 1is a flow chart of a short video popularity prediction method provided by the present invention, Figure 2 It is a schematic diagram of the architecture of the short video popularity prediction method provided by the present invention.

[0055] like Figure 1 As shown, the method includes the following:

[0056] Step 101: construct a multimodal graph and multiple unimodal graphs based on multiple short videos;

[0057] Step 102: performing heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph and a second node representation of a target video node corresponding to a target short video in each of the unimodal graphs;

[0058] Step 103: Perform popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video.

[0059] Specifically, multiple short videos for which short video popularity prediction is required are obtained. Some of these short videos have similar themes or contents, while some have completely different themes and contents. Based on these short videos, the node types and edge types related to them are determined, and a heterogeneous graph is constructed through the node types and edge types. A heterogeneous graph is a graph data structure with multiple node types and multiple edge types, which is used to model multiple objects of a complex interactive system and their associations. In a heterogeneous graph, nodes of the same type are considered to be a single mode, and nodes of different types are considered to be multiple modes. In the constructed heterogeneous graph, there are at least video nodes, video frame nodes, and video title nodes corresponding to each short video.

[0060] Based on the heterogeneous graph, a multimodal graph is constructed by retaining multiple modal information such as video frame nodes and video title nodes, and based on the heterogeneous graph, a unimodal graph is constructed by retaining only a single modal information of the video frame node, which is referred to as the first unimodal graph in this article, and based on the heterogeneous graph, a unimodal graph is constructed by retaining only a single modal information of the video title node, which is referred to as the second unimodal graph in this article.

[0061] Furthermore, heterogeneous graph representation learning is performed based on the multimodal graph to obtain a first node representation of the target video node corresponding to the target short video in the multimodal graph, and heterogeneous graph representation learning is performed based on the first unimodal graph to obtain a second node representation of the target video node corresponding to the target short video in the first unimodal graph, and heterogeneous graph representation learning is performed based on the second unimodal graph to obtain a second node representation of the target video node corresponding to the target short video in the second unimodal graph. It should be noted that in the process of short video popularity prediction, it is possible to selectively determine which short video needs to be predicted for popularity, or to predict the popularity of all short videos to which the heterogeneous graph is applied. The method of predicting popularity for each short video in this article is the same. Therefore, in order to describe clearly, a popularity prediction process is described from the perspective of a target short video, which is actually not limited to popularity prediction for a specific video.

[0062] In the process of learning heterogeneous graph representation, the heterogeneous graph neural network model is applied to pre-calculate the representation of the target video node by using "propagation-update" on the heterogeneous graph. Therefore, it can be understood that the multimodal graph, the first unimodal graph, and the second unimodal graph are respectively input into the heterogeneous graph neural network model, and the heterogeneous graph neural network uses the "propagation-update" pre-calculation method to calculate the first node representation of the target video node in the multimodal graph, the second node representation of the target video node in the first unimodal graph, and the second node representation of the target video node in the second unimodal graph.

[0063] Furthermore, based on the first node representation of the target video node in the multimodal graph, the second node representation of the target video node in the first unimodal graph, and the second node representation of the target video node in the second unimodal graph, a target node representation of the target video node is obtained, and popularity prediction is performed based on the target node representation of the target video node to obtain the popularity of the target short video. In the above manner, the popularity of any short video can be obtained.

[0064] The short video popularity prediction method provided by the present invention constructs a multimodal graph, and constructs a unimodal graph by selectively shielding the representations of other modalities to retain the unique information of each modality. When heterogeneous graph representation learning is performed based on the multimodal graph and each unimodal graph, the multimodal graph can capture cross-modal information of interactive propagation between multimodalities, and the unimodal graph can retain the original information of the unimodal modality, which helps to reduce the loss of cross-modal information and ensure sufficient modal information, optimize the overall modal representation, and then according to the modal representation of the target video node in the multimodal graph and the modal representation of the target video node in each unimodal graph, the popularity of the short video is comprehensively predicted by the modal representations of multiple dimensions, which significantly improves the accuracy and efficiency of short video popularity prediction.

[0065] In some embodiments, based on step 101, constructing a multimodal graph and a plurality of unimodal graphs based on a plurality of short videos includes:

[0066] Determine the node types and edge types associated with the plurality of short videos;

[0067] Based on the node type and the edge type, construct a heterogeneous graph;

[0068] Determine the video frame features extracted from each of the short videos as the node representation of the corresponding video frame node in the heterogeneous graph, determine the video title features extracted from each of the short videos as the node representation of the corresponding video title node in the heterogeneous graph, and initialize the node representations of the remaining nodes to obtain a multimodal graph; the remaining nodes are the nodes in the heterogeneous graph other than the video frame nodes and the video title nodes;

[0069] Initializing the node representation of each of the video frame nodes in the multimodal graph to obtain a first unimodal graph, and initializing the node representation of each of the video title nodes in the multimodal graph to obtain a second unimodal graph.

[0070] Specifically, the node types and edge types associated with multiple short videos are determined. Generally, the node types associated with short videos include video nodes, user nodes, video frame nodes, video title nodes, video frame subtitle nodes, comment nodes, etc., where a video node is used to represent a single short video, a user node is used to represent a user who uploads or watches a short video, a video frame node is used to represent a video frame extracted from a short video, a video title node is used to represent a title of a short video, a video frame subtitle node is used to represent a descriptive subtitle of a video frame, and a comment node is used to represent a comment on a short video. Typically, the edge types associated with short videos include upload relationships, watch relationships, correspondence relationships, comment relationships, and the like, wherein the upload relationship can be used to connect user nodes and video nodes, indicating that the user has uploaded the video; the watch relationship can be used to connect user nodes and video nodes, indicating that the user has watched the video; the correspondence relationship can be used to connect video nodes and video title nodes, indicating that the video title is the title corresponding to the video; it can also be used to connect video frame nodes and video nodes, indicating that the video frame is extracted from the video; the comment relationship can be used to connect user nodes and comment nodes, indicating that the user has posted a comment; it can also be used to connect comment nodes and video nodes, indicating that the comment is directed to the video. The above are only some examples, which are not limited to this and are set according to actual conditions. Furthermore, a heterogeneous graph is constructed based on the node types and edge types associated with multiple videos.

[0071] It should be noted that when constructing a multimodal graph and a unimodal graph, the modal information of the video frame node and the modal information of the video title node need to be applied. Therefore, before constructing a multimodal graph and a unimodal graph, it is necessary to obtain the feature information of these two nodes.

[0072] The process of obtaining the characteristic information of these two types of nodes is described in detail as follows. Figure 2 As shown, for each short video , sample video frames with a fixed frame number K and a fixed interval, and input these video frames into a pre-trained visual language model, such as the ViT model, to extract video frame features from the video frames , where K refers to the number of video frames, Represents the dimension of the video frame features. At the same time, the title of each short video Input into the pre-trained language model to extract video title features from the short video title , where V represents the number of video titles, Represents the dimension of the video title feature. Each video frame has a corresponding video frame node in the heterogeneous graph, so the video frame features extracted from the video frame also have corresponding video frame nodes in the heterogeneous graph; each video title has a corresponding video title node in the heterogeneous graph, so the video title features extracted from the video title also have corresponding video title nodes.

[0073] After obtaining the feature information of the two nodes, the video frame features extracted from each short video are determined as the node representation of the corresponding video frame node in the heterogeneous graph, the video title features extracted from each short video are determined as the node representation of the corresponding video title node in the heterogeneous graph, and the node representations of the remaining nodes are initialized to obtain a multimodal graph. Figure 2 In the heterogeneous network diagram of short and medium videos, some nodes are gray, indicating that the node representation of this type of node is randomly initialized, while some nodes are colored, indicating that the feature data of this type of node is embedded in its node representation and has modal information. In addition, nodes of different colors belong to different types of modalities.

[0074] Furthermore, the node representation of each video frame node in the multimodal graph is initialized to obtain a first unimodal graph, which means that only the modal information of the video frame node type node is retained in the first unimodal graph, but the network structure of the heterogeneous graph remains unchanged. Similarly, the node representation of each video title node in the multimodal graph is initialized to obtain a second unimodal graph, which means that only the modal information of the video title node type node is retained in the second unimodal graph.

[0075] The embodiment of the present invention constructs a heterogeneous graph through the node types and edge types associated with short videos, and on the basis of this heterogeneous graph, constructs a multimodal graph and multiple unimodal graphs by retaining different modal information, accurately integrating multi-source data. The multimodal graph can capture cross-modal information, and the unimodal graph can retain the original modal information. Then, heterogeneous graph representation learning can be performed based on the multimodal graph and the unimodal graph to capture the interactive information transmitted through short video interactions, and can reduce the loss of cross-modal information in the capture process, thereby significantly improving the effectiveness and efficiency of short video popularity prediction results.

[0076] In some embodiments, based on step 102, heterogeneous graph representation learning is performed based on the multimodal graph to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph, including:

[0077] Determine at least one meta-path in the multimodal graph that has a target video node corresponding to the target short video as an end node;

[0078] Calculating neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain an updated first node representation of the target video node on each meta-path;

[0079] Iteratively calculate the neighbor node information of the target video node on each meta-path, and update the node representation of the target video node according to the neighbor node information to obtain the updated first node representation of the target video node on each meta-path, until the number of iterations reaches L times, and obtain the updated first node representation of the target video node on each meta-path in each iteration process.

[0080] It should be noted that in the heterogeneous graph neural network model, a "propagation-update" scheme is adopted. First, the neighbor node information of the video node is collected from the meta-path, and then the vertex representation is updated through the neighbor node information on the meta-path, that is, the node representation of the video node. The process of updating the vertex representation is iterated until the final output is obtained.

[0081] Specifically, at least one meta-path with a target video node corresponding to a target short video as an end node in the multimodal graph is determined. Usually, there are more than two meta-paths, so the subsequent description process is described from the perspective of multiple meta-paths.

[0082] The neighbor node information of the target video node on each meta-path is calculated, and the node representation of the target video node is updated according to the neighbor node information of the target video node on each meta-path to obtain the updated first node representation of the target video node on each meta-path.

[0083] Iteratively execute the step of calculating the neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each neighbor node information, and obtaining the first node representation of the target video node after the update on each meta-path, until the number of iterations reaches L times, and obtaining the first node representation of the target video node after the update on each meta-path in each iteration process. Among them, L times is the preset number of iterations, which is set according to actual conditions.

[0084] Through the above method, we can infer the process of learning heterogeneous graph representation based on multiple unimodal graphs to obtain the second node representation of the target video node corresponding to the target short video in each unimodal graph. Taking the first unimodal graph as an example, the second unimodal graph also has the same process.

[0085] Specifically, it is as follows: determining at least one meta-path in the first single-modal graph with the target video node corresponding to the target short video as the end node; calculating the neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain the updated second node representation of the target video node on each meta-path; iteratively executing the steps of calculating the neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain the updated second node representation of the target video node on each meta-path, until the number of iterations reaches L times, and obtaining the updated second node representation of the target video node on each meta-path in each iteration process.

[0086] The detailed process is consistent with the heterogeneous graph representation learning of multimodal graphs and will not be elaborated here.

[0087] The embodiments of the present invention can effectively capture cross-modal information by performing heterogeneous graph representation learning on a multimodal graph, and can effectively retain unimodal information by performing heterogeneous graph representation learning on a unimodal graph, so that the model can accurately capture the interactive information transmitted through short video interactions under sufficient information, output the target video node representation, and realize the capture of local information of the target video node by pre-calculation, thereby improving the representation ability of the target video node, so that it can show more accurate node information in subsequent short video popularity prediction, thereby significantly improving the effectiveness and efficiency of the short video popularity prediction results.

[0088] According to the above content, the calculating of the neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain the updated first node representation of the target video node on each meta-path, includes:

[0089] If the number of hops from the start node to the end node in any meta-path is an odd number, the neighbor node information of the target video node on the meta-path is calculated by a first formula, and the node representation of the target video node is updated according to the neighbor node information to obtain a first node representation of the target video node after the update on the meta-path;

[0090] If the number of hops from the starting node to the ending node in any meta-path is an even number, the neighbor node information of the target video node on the meta-path is calculated by the second formula, and the node representation of the target video node is updated according to the neighbor node information to obtain the updated first node representation of the target video node on the meta-path.

[0091] It should be noted that in order to strike a balance between metapath complexity and information retention, an even-odd scheme is used. and the edge type it ends with , defines odd and even relations. An odd relation means that the number of hops on the metapath from the starting node to the target video node is odd, that is, the number of edges between the two nodes is odd; an even relation means that the number of hops on the metapath from the starting node to the target video node is even, that is, the number of edges between the two nodes is even. For example, on a metapath, from the starting node To the end node Only one hop is needed, then they are considered to have an odd relationship, and the meta-path can be expressed as ; On a certain meta-path, from the starting node To the end node It is a symmetrical 2-hop, and it is considered that there is a pair relationship between them. The meta-path can be expressed as ,in yes The reverse of .

[0092] Specifically, in one case, if the number of hops from the start node to the end node in any meta-path is an odd number, the neighbor node information of the target video node on the meta-path is calculated using the first formula.

[0093] The first formula is as follows:

[0094] ;

[0095] in, is the target video node representation matrix generated in the L-1th iteration, Is the edge type The adjacency matrix of Is the edge type The degree matrix of It is the neighbor node information of the target video node on the meta-path with odd relationship. In the first iteration, the target video node representation matrix is ​​the initial value.

[0096] Furthermore, the random projection compression technique is used to compress the representation of different neighbor relationships into a vector as the updated vertex representation. Its main purpose is to reduce the dimension to a constant value while maintaining computational efficiency. For odd relationships, for the target video node The node representation of the update is as follows:

[0097] ;

[0098] in, It is used to indicate the meta path ending at the target video node. is the neighbor node information of the target video node on the meta-path with odd relationship, is the random projection weight matrix, The function for each row vector Perform L2 normalization.

[0099] The random projection weight matrix is ​​generated using the sparse random projection method:

[0100] ;

[0101] in, Represents the sparsity index, which determines the ratio of non-zero elements in the generated random projection matrix. Values ​​of result in more zero values, thus producing a sparser matrix; larger The value will produce more non-zero elements, making the matrix less sparse.

[0102] In another case, if the number of hops from the start node to the end node in any meta-path is an even number, the neighbor node information of the target video node on the meta-path is calculated using the second formula.

[0103] The second formula is as follows:

[0104] ;

[0105] in, is the target video node representation matrix generated in the L-1th iteration, Is the edge type The adjacency matrix of Is the edge type The degree matrix of It is the neighbor node information of the target video node on the meta-path with even relationship. In the first iteration, the target video node representation matrix is ​​the initial value.

[0106] Furthermore, the random projection compression technique is used to compress the representations of different neighbor relationships into a vector as the updated vertex representation. The node representation of the update is as follows:

[0107] ;

[0108] in, It is used to indicate the meta path ending at the target video node. is the neighbor node information of the target video node on the meta-path with even relationship, is the random projection weight matrix, The function for each row vector Perform L2 normalization.

[0109] The random projection weight matrix is ​​generated using the sparse random projection method:

[0110] ;

[0111] in, Represents the sparsity index.

[0112] like Figure 2 As shown, the neighbor node information is collected for the odd relationship and even relationship dimensions respectively, and the target video node representation is updated through the collected neighbor node information, and the iteration is performed L times.

[0113] The embodiment of the present invention adopts an odd-even scheme, which adopts different neighbor node information collection methods for odd relations and even relations, and can strike a balance between meta-path complexity and information retention. At the same time, it utilizes relational neighbor information collection and random projection compression technology to capture local information of the target video node through pre-calculation, maximizes the retention of structural information, reduces information loss, improves data processing speed and reduces computing cost.

[0114] In some embodiments, after performing heterogeneous graph representation learning based on the multimodal graph to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph, and performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video, that is, between step 102 and step 103, the method further includes:

[0115] Determine a plurality of short videos similar to the target short video from the plurality of short videos;

[0116] Determine the updated third node representation of the video nodes corresponding to each of the similar short videos on each meta-path during each iteration; the meta-path of the target video node corresponding to the target short video in the multimodal graph is consistent with the meta-path of the video nodes corresponding to each of the similar short videos in the multimodal graph;

[0117] Calculating the mean of each of the third node representations on each meta-path during the iteration process to obtain a mean result on each meta-path during the iteration process;

[0118] The mean results in the iteration process are determined as the retrieval embedding representation of the target video node on each meta-path in the iteration process.

[0119] It should be noted that, as mentioned above, among multiple short videos, some short videos have similar themes or contents. By selecting similar short videos for information aggregation, the global information between these short videos can be obtained to capture the video similarity beyond the graph structure.

[0120] Specifically, a plurality of similar short videos that are most similar to the target short video are calculated through a similarity algorithm.

[0121] Furthermore, through the above method of calculating the node representation of the target video node corresponding to the target short video, the updated third node representation of the video node corresponding to each similar short video on each meta-path during the L iterations can also be calculated. It should be noted that, since the relationship between the target short video and the similar short videos is fixed, the meta-path of the target video node corresponding to the target short video in the multimodal graph is consistent with the meta-path of the video node corresponding to each similar short video in the multimodal graph.

[0122] The third node representations on each meta-path in each iteration process are averaged to obtain the average result on each meta-path in each iteration process. For example, in the first iteration process, multiple third node representations on path A and multiple third node representations on path B are output, and the multiple third node representations on path A include the third node representation of the video node corresponding to similar short video 1 after being updated on path A, and the third node representation of the video node corresponding to similar short video 2 after being updated on path A; the multiple third node representations on path B include the third node representation of the video node corresponding to similar short video 1 after being updated on path B, and the third node representation of the video node corresponding to similar short video 2 after being updated on path B. Therefore, by performing the average calculation on the multiple third node representations on path A, the average result of path A in the first iteration process can be obtained; by performing the average calculation on the multiple third node representations on path B, the average result of path B in the first iteration process can be obtained.

[0123] The average results in each iteration process are determined as the retrieval embedding representation of the target video node on each meta-path in each iteration process. For example, the average result of path A in the first iteration process is determined as the retrieval embedding representation of the target video node in path A in the first iteration process, and the average result of path B in the first iteration process is determined as the retrieval embedding representation of the target video node in path B in the first iteration process.

[0124] The embodiment of the present invention introduces an efficient content similarity retrieval mechanism for identifying and utilizing short videos with similar contents, and uses the retrieval embedding representation of these similar short videos as supplementary data to enhance the global information processing capability of the prediction model, so that the prediction model can better combine local connections and global semantic information, thereby significantly enhancing the prediction accuracy of the popularity of short videos.

[0125] According to the above content, the step of determining a plurality of similar short videos of the target short video from the plurality of short videos includes:

[0126] Calculate the similarity between the retrieval features of the target short video and the retrieval features of the remaining short videos to obtain multiple similarity values; the retrieval features are extracted from the retrieval information obtained by splicing the video title and the video subtitle; the remaining short videos are the remaining short videos in the multiple short videos except the target short video;

[0127] The remaining short videos corresponding to the top K maximum similarity values ​​among the similarity values ​​are determined as similar short videos of the target short video.

[0128] It should be noted that the similarity between short videos can be calculated by calculating the cosine similarity between the retrieval features of the short videos, thereby screening out the top K short videos that are most similar to the target short video. The retrieval features of the short videos are extracted from the retrieval information obtained by splicing the video title and the video subtitle.

[0129] Specifically, before calculating the similarity, it is necessary to first obtain the retrieval features of each short video.

[0130] Use a visual-language model, such as the BLIP model, to generate descriptive captions for each video frame, represented as , represents the descriptive subtitle of the Kth video frame in the i-th short video, The descriptive subtitle of the i-th short video is composed of the descriptive subtitles of K video frames. The title of the short video Splicing to form search information , input the search information into the pre-trained language model to generate search features ,in Indicates the number of retrieved information, Represents the dimension of the retrieved features.

[0131] After obtaining the retrieval features of each short video, the retrieval features of the target short video and the retrieval features of the remaining short videos are similarly calculated to obtain multiple similarity values, wherein the remaining short videos refer to the remaining short videos in the multiple short videos except the target short video.

[0132] The similarity calculation between the retrieval features of the target short video and the retrieval features of other short videos can be achieved by the following formula:

[0133] ;

[0134] in, is the retrieval feature of the target short video i, is the retrieval feature of the remaining short videos j, is the similarity value between the target short video i and the other short videos j.

[0135] Furthermore, the similarity values ​​are compared numerically, and the first K maximum similarity values ​​are determined according to the numerical comparison results, and the remaining short videos corresponding to the K similarity values ​​are determined as similar short videos of the target short video.

[0136] The embodiment of the present invention calculates multiple similar short videos of the target short video through cosine similarity, which can achieve efficient video retrieval, and then calculates the global information between videos under the similarity retrieval mechanism of short video content to capture video similarities beyond the graph structure.

[0137] In some embodiments, based on step 103, performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video includes:

[0138] Fusing L first node representations of the target video node on each meta-path in the multimodal graph to obtain a first fused node representation of the target video node on each meta-path in the multimodal graph;

[0139] Fusing L second node representations of the target video node on each meta-path in the first unimodal graph to obtain a second fused node representation of the target video node on each meta-path in the first unimodal graph;

[0140] Fusing L second node representations of the target video node on each meta-path in the second unimodal graph to obtain a third fused node representation of the target video node on each meta-path in the second unimodal graph;

[0141] Fusing L retrieval embedding representations of the target video node on each meta-path in the multimodal graph to obtain a fourth fused node representation of the target video node on each meta-path in the multimodal graph;

[0142] fusing each of the first fusion node representations, each of the second fusion node representations, each of the third fusion node representations, and each of the fourth fusion node representations to obtain a target node representation of the target video node;

[0143] The popularity of the target node representation is predicted to obtain the popularity of the target short video.

[0144] The following content is combined Figure 3 , Figure 3 It is a schematic diagram of the architecture of the short video popularity prediction provided by the present invention.

[0145] It should be noted that the above process obtains three types of information, including multi-graph information, single-graph information and retrieval information, specifically, the first node representation of the target video node on each meta-path in the multimodal graph after being updated during L iterations, the second node representation of the target video node on each meta-path in the first unimodal graph after being updated during L iterations, the second node representation of the target video node on each meta-path in the second unimodal graph after being updated during L iterations, and the retrieval embedding representation of the target video node on each meta-path in the multimodal graph during L iterations. Figure 2 As shown in the figure, by integrating the node representation obtained under the multimodal graph, the node representation obtained under the unimodal graph and the retrieval embedding representation obtained under similar short videos, accurate prediction of the popularity of short videos can be achieved.

[0146] In order to predict the popularity of the target short video, it is necessary to integrate these three types of information. Therefore, a relation encoder and a fusion encoder are designed as prediction models to achieve this goal. The prediction model architecture is as follows: Figure 3As shown. The relationship encoder aims to combine different iteration information of the same relationship, that is, different iteration information on the same meta-path, to extract relationship-specific features. Each relationship encoder uses independent parameters, including a one-dimensional convolution (Conv1D) layer, a connection layer, and a multilayer perceptron (MLP). The Conv1D layer integrates the neighbor information collected in different iterations by using the iteration index as the input channel. The output features are concatenated and reduced in dimension by the connection layer and the multilayer perceptron layer, and finally the fused node representation of the same relationship is obtained. The fusion encoder uses the interaction between different relationships to learn vertex representations. It includes a connection layer and a multilayer perceptron layer. The connection layer concatenates the fused node representations of different relationships into a vector, which is then encoded into a popularity prediction result by the multilayer perceptron layer.

[0147] Specifically, the L first node representations of the target video node on each meta-path in the multimodal graph are fused, that is, the output results in the L iterations are fused to obtain the first fused node representation of the target video node on each meta-path in the multimodal graph.

[0148] Similarly, the L second node representations of the target video node on each meta-path in the first unimodal graph are fused, that is, the output results during L iterations are fused to obtain the second fused node representation of the target video node on each meta-path in the first unimodal graph.

[0149] Similarly, the L second node representations of the target video node on each meta-path in the second unimodal graph are fused, that is, the output results during L iterations are fused to obtain the third fused node representation of the target video node on each meta-path in the second unimodal graph.

[0150] Similarly, the L retrieval embedding representations of the target video node on each meta-path in the multimodal graph are fused, that is, the output results in the L iterations are fused to obtain the fourth fused node representation of the target video node on each meta-path in the multimodal graph.

[0151] Further, the first fusion node representations, the second fusion node representations, the third fusion node representations and the fourth fusion node representations are fused, that is, the fusion node representations under different meta-paths are spliced ​​to obtain the final target node representation of the target video node.

[0152] Furthermore, the popularity of the target node representation is predicted to obtain the popularity of the target short video.

[0153] The embodiments of the present invention integrate local information and global information, optimize the performance of the prediction model in multiple dimensions, and improve the adaptability and sensitivity to popular trends. Specifically, different iterative output results of the same relationship are fused through a relationship encoder, and information of different relationships is integrated through a fusion encoder, and finally the accurate short video popularity is output. Therefore, through an efficient and comprehensive solution strategy, it has significant practical application value for accurately predicting the popularity of short videos.

[0154] In order to evaluate the effectiveness of the present invention, a publicly available MicroLens short video dataset is used, and the dataset is divided into two subsets: MicroLens-view and MicroLens-comment. Both subsets use the video frame and video title of the video as vertices of attribute type. For MicroLens-view, an interaction relationship is established through the "user-comment" edge, and the interaction attributes are defined based on the user comments. For MicroLens-comment, the "video-category" interaction attribute is used to classify the video by content, and the popularity is measured by the number of comments on the video. In the latter subset, the "user-comment" data is avoided to prevent data leakage.

[0155] Table 1 compares the proposed method with other methods on Microlens100K-view and Microlens100K-comment data. Three evaluation indicators are used: nMSE, MAE, and SRC.

[0156] Table 1

[0157]

[0158] Among them, the proposed method is compared with nine benchmark models based on different methodological foundations, which are divided into three categories. Feature engineering methods include SVR, a regression method based on support vector machines for predicting continuous target values; and HyFea, a hybrid feature method specifically designed to improve the performance of prediction. Deep learning methods include CLSTM, which combines the advantages of convolutional neural networks and long short-term memory networks to process sequence data; TMALL, which uses multimodal learning to extract and fuse different types of input data; CBAN, which improves feature learning of cross-modal data through attention mechanisms; and MMRA, which enhances the model's prediction of social media trends through multimodal retrieval. Finally, HGNN-based models include RGCN, which uses graph convolutional networks to process complex relational data; HetSANN, which enhances information processing of heterogeneous graphs through adaptive neural networks; and SeHGNN, which improves efficiency through a simplified graph neural network architecture. These comparisons can help verify the performance and efficiency of the proposed models from different dimensions.

[0159] As can be seen from Table 1, the performance of the method proposed in this paper on the above datasets significantly exceeds that of other methods.

[0160] The short video popularity prediction device provided by the present invention is described below. The short video popularity prediction device described below and the short video popularity prediction method described above can refer to each other.

[0161] Reference Figure 4 , Figure 4 It is a structural schematic diagram of the short video popularity prediction device provided by the present invention.

[0162] The short video popularity prediction device comprises:

[0163] The construction module 410 is used to construct a multimodal graph and multiple unimodal graphs based on multiple short videos; the multimodal graph is a heterogeneous graph that retains multiple modal information; the unimodal graph is a heterogeneous graph that retains a single modal information.

[0164] The heterogeneous graph representation learning module 420 is used to perform heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph and a second node representation of a target video node corresponding to a target short video in each of the unimodal graphs.

[0165] The popularity prediction module 430 is used to perform popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video.

[0166] The short video popularity prediction device provided by the present invention constructs a multimodal graph, and constructs a unimodal graph by selectively shielding the representations of other modalities to retain the unique information of each modality. When heterogeneous graph representation learning is performed based on the multimodal graph and each unimodal graph, the multimodal graph can capture the cross-modal information of the interactive propagation between the multimodal modes, and the unimodal graph can retain the original information of the unimodal mode, which helps to reduce the loss of cross-modal information and ensure the sufficiency of modal information, optimize the overall modal representation, and then according to the modal representation of the target video node in the multimodal graph and the modal representation of the target video node in each unimodal graph, the popularity of the short video is comprehensively predicted by the modal representations of multiple dimensions, which significantly improves the accuracy and efficiency of the short video popularity prediction.

[0167] Furthermore, the construction module 410 is also used for:

[0168] Determine the node types and edge types associated with the plurality of short videos;

[0169] Based on the node type and the edge type, construct a heterogeneous graph;

[0170] Determine the video frame features extracted from each of the short videos as the node representation of the corresponding video frame node in the heterogeneous graph, determine the video title features extracted from each of the short videos as the node representation of the corresponding video title node in the heterogeneous graph, and initialize the node representations of the remaining nodes to obtain a multimodal graph; the remaining nodes are the nodes in the heterogeneous graph other than the video frame nodes and the video title nodes;

[0171] Initializing the node representation of each of the video frame nodes in the multimodal graph to obtain a first unimodal graph, and initializing the node representation of each of the video title nodes in the multimodal graph to obtain a second unimodal graph.

[0172] Furthermore, the heterogeneous graph representation learning module 420 is also used for:

[0173] Determine at least one meta-path in the multimodal graph that has a target video node corresponding to the target short video as an end node;

[0174] Calculating neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain an updated first node representation of the target video node on each meta-path;

[0175] Iteratively calculate the neighbor node information of the target video node on each meta-path, and update the node representation of the target video node according to the neighbor node information to obtain the updated first node representation of the target video node on each meta-path, until the number of iterations reaches L times, and obtain the updated first node representation of the target video node on each meta-path in each iteration process.

[0176] Furthermore, the heterogeneous graph representation learning module 420 is also used for:

[0177] If the number of hops from the start node to the end node in any meta-path is an odd number, the neighbor node information of the target video node on the meta-path is calculated by a first formula, and the node representation of the target video node is updated according to the neighbor node information to obtain a first node representation of the target video node after the update on the meta-path;

[0178] If the number of hops from the starting node to the ending node in any meta-path is an even number, the neighbor node information of the target video node on the meta-path is calculated by the second formula, and the node representation of the target video node is updated according to the neighbor node information to obtain the updated first node representation of the target video node on the meta-path.

[0179] Furthermore, the short video popularity prediction device is also used for:

[0180] Determine a plurality of short videos similar to the target short video from the plurality of short videos;

[0181] Determine the updated third node representation of the video nodes corresponding to each of the similar short videos on each meta-path during each iteration; the meta-path of the target video node corresponding to the target short video in the multimodal graph is consistent with the meta-path of the video nodes corresponding to each of the similar short videos in the multimodal graph;

[0182] Calculating the mean of each of the third node representations on each meta-path during the iteration process to obtain a mean result on each meta-path during the iteration process;

[0183] The mean results in the iteration process are determined as the retrieval embedding representation of the target video node on each meta-path in the iteration process.

[0184] Furthermore, the short video popularity prediction device is also used for:

[0185] Calculate the similarity between the retrieval features of the target short video and the retrieval features of the remaining short videos to obtain multiple similarity values; the retrieval features are extracted from the retrieval information obtained by splicing the video title and the video subtitle; the remaining short videos are the remaining short videos in the multiple short videos except the target short video;

[0186] The remaining short videos corresponding to the top K maximum similarity values ​​among the similarity values ​​are determined as similar short videos of the target short video.

[0187] Furthermore, the popularity prediction module 430 is also used for:

[0188] Fusing L first node representations of the target video node on each meta-path in the multimodal graph to obtain a first fused node representation of the target video node on each meta-path in the multimodal graph;

[0189] Fusing L second node representations of the target video node on each meta-path in the first unimodal graph to obtain a second fused node representation of the target video node on each meta-path in the first unimodal graph;

[0190] Fusing L second node representations of the target video node on each meta-path in the second unimodal graph to obtain a third fused node representation of the target video node on each meta-path in the second unimodal graph;

[0191] Fusing L retrieval embedding representations of the target video node on each meta-path in the multimodal graph to obtain a fourth fused node representation of the target video node on each meta-path in the multimodal graph;

[0192] fusing each of the first fusion node representations, each of the second fusion node representations, each of the third fusion node representations, and each of the fourth fusion node representations to obtain a target node representation of the target video node;

[0193] The popularity of the target node representation is predicted to obtain the popularity of the target short video.

[0194] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the short video popularity prediction method, which includes: constructing a multimodal graph and a plurality of unimodal graphs based on a plurality of short videos; the multimodal graph is a heterogeneous graph retaining multiple modal information; the unimodal graph is a heterogeneous graph retaining a single modal information; performing heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs to obtain a first node representation of a target video node corresponding to the target short video in the multimodal graph and a second node representation of a target video node corresponding to the target short video in each of the unimodal graphs; performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video.

[0195] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0196] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the short video popularity prediction method provided by the above-mentioned embodiments, the method including: based on multiple short videos, constructing a multimodal graph and multiple unimodal graphs; the multimodal graph is a heterogeneous graph that retains multiple modal information; the unimodal graph is a heterogeneous graph that retains a single modal information; based on the multimodal graph and each of the unimodal graphs, heterogeneous graph representation learning is performed to obtain a first node representation of a target video node corresponding to the target short video in the multimodal graph and a second node representation of a target video node corresponding to the target short video in each of the unimodal graphs; based on the first node representation and each of the second node representations, popularity prediction is performed to obtain the popularity of the target short video.

[0197] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the short video popularity prediction method provided in the above-mentioned embodiments, the method comprising: constructing a multimodal graph and multiple unimodal graphs based on multiple short videos; the multimodal graph is a heterogeneous graph that retains multiple modal information; the unimodal graph is a heterogeneous graph that retains a single modal information; performing heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs to obtain a first node representation of a target video node corresponding to the target short video in the multimodal graph and a second node representation of a target video node corresponding to the target short video in each of the unimodal graphs; performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video.

[0198] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0199] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting the popularity of short videos, characterized in that: include: Based on multiple short videos, construct a multimodal graph and multiple unimodal graphs; the multimodal graph is a heterogeneous graph that retains multiple modal information; The unimodal graph is a heterogeneous graph that retains information of a single modality; Performing heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph and a second node representation of a target video node corresponding to a target short video in each of the unimodal graphs; Performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video; The step of performing heterogeneous graph representation learning based on the multimodal graph to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph includes: Determine at least one meta-path in the multimodal graph that has a target video node corresponding to the target short video as an end node; Calculating neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain an updated first node representation of the target video node on each meta-path; Iteratively execute the step of calculating the neighbor node information of the target video node on each meta-path until the number of iterations reaches L times, and obtain the updated first node representation of the target video node on each meta-path during each iteration.

2. The short video popularity prediction method according to claim 1, characterized in that: The multiple single-modal graphs include a first single-modal graph and a second single-modal graph; the multi-modal graph and the multiple single-modal graphs are constructed based on the multiple short videos, including: Determine the node types and edge types associated with the plurality of short videos; Based on the node type and the edge type, construct a heterogeneous graph; Determine the video frame features extracted from each of the short videos as the node representation of the corresponding video frame node in the heterogeneous graph, determine the video title features extracted from each of the short videos as the node representation of the corresponding video title node in the heterogeneous graph, and initialize the node representations of the remaining nodes to obtain a multimodal graph; the remaining nodes are the nodes in the heterogeneous graph other than the video frame nodes and the video title nodes; Initializing the node representation of each of the video frame nodes in the multimodal graph to obtain a first unimodal graph, and initializing the node representation of each of the video title nodes in the multimodal graph to obtain a second unimodal graph.

3. The short video popularity prediction method according to claim 1, characterized in that: The calculating the neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain the updated first node representation of the target video node on each meta-path, includes: If the number of hops from the start node to the end node in any meta-path is an odd number, the neighbor node information of the target video node on the meta-path is calculated by a first formula, and the node representation of the target video node is updated according to the neighbor node information to obtain a first node representation of the target video node after the update on the meta-path; If the number of hops from the starting node to the ending node in any meta-path is an even number, the neighbor node information of the target video node on the meta-path is calculated by the second formula, and the node representation of the target video node is updated according to the neighbor node information to obtain the updated first node representation of the target video node on the meta-path.

4. The short video popularity prediction method according to claim 1, characterized in that: After performing heterogeneous graph representation learning based on the multimodal graph to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph, and before performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video, the method further includes: Determine a plurality of short videos similar to the target short video from the plurality of short videos; Determine the updated third node representation of the video nodes corresponding to each of the similar short videos on each meta-path during each iteration; the meta-path of the target video node corresponding to the target short video in the multimodal graph is consistent with the meta-path of the video nodes corresponding to each of the similar short videos in the multimodal graph; Calculating the mean of each of the third node representations on each meta-path during the iteration process to obtain a mean result on each meta-path during the iteration process; The mean results in the iteration process are determined as the retrieval embedding representation of the target video node on each meta-path in the iteration process.

5. The short video popularity prediction method according to claim 4, characterized in that: The determining a plurality of similar short videos of the target short video from the plurality of short videos comprises: Calculate the similarity between the retrieval features of the target short video and the retrieval features of the remaining short videos to obtain multiple similarity values; the retrieval features are extracted from the retrieval information obtained by splicing the video title and the video subtitle; the remaining short videos are the remaining short videos in the multiple short videos except the target short video; The remaining short videos corresponding to the top K maximum similarity values ​​among the similarity values ​​are determined as similar short videos of the target short video.

6. The short video popularity prediction method according to claim 4, characterized in that: The performing popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video includes: Fusing L first node representations of the target video node on each meta-path in the multimodal graph to obtain a first fused node representation of the target video node on each meta-path in the multimodal graph; Fusing L second node representations of the target video node on each meta-path in the first unimodal graph to obtain a second fused node representation of the target video node on each meta-path in the first unimodal graph; Fusing L second node representations of the target video node on each meta-path in the second unimodal graph to obtain a third fused node representation of the target video node on each meta-path in the second unimodal graph; Fusing L retrieval embedding representations of the target video node on each meta-path in the multimodal graph to obtain a fourth fused node representation of the target video node on each meta-path in the multimodal graph; fusing each of the first fusion node representations, each of the second fusion node representations, each of the third fusion node representations, and each of the fourth fusion node representations to obtain a target node representation of the target video node; The popularity of the target node representation is predicted to obtain the popularity of the target short video.

7. A short video popularity prediction device, characterized in that: include: A construction module, used to construct a multimodal graph and multiple unimodal graphs based on multiple short videos; the multimodal graph is a heterogeneous graph that retains multiple modal information; The unimodal graph is a heterogeneous graph that retains information of a single modality; A heterogeneous graph representation learning module, used to perform heterogeneous graph representation learning based on the multimodal graph and each of the unimodal graphs, to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph and a second node representation of a target video node corresponding to a target short video in each of the unimodal graphs; A popularity prediction module, used to perform popularity prediction based on the first node representation and each of the second node representations to obtain the popularity of the target short video; The step of performing heterogeneous graph representation learning based on the multimodal graph to obtain a first node representation of a target video node corresponding to a target short video in the multimodal graph includes: Determine at least one meta-path in the multimodal graph that has a target video node corresponding to the target short video as an end node; Calculating neighbor node information of the target video node on each meta-path, and updating the node representation of the target video node according to each of the neighbor node information, to obtain an updated first node representation of the target video node on each meta-path; Iteratively execute the step of calculating the neighbor node information of the target video node on each meta-path until the number of iterations reaches L times, and obtain the updated first node representation of the target video node on each meta-path during each iteration.

8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the short video popularity prediction method according to any one of claims 1 to 6 are implemented.

9. A storage medium, comprising a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the short video popularity prediction method according to any one of claims 1 to 6 are implemented.

Images