Method and apparatus for predicting video topic peaks

Abstract

A video topic peak prediction method and device are disclosed. The method comprises: obtaining topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph contains topic information of multiple videos; aligning the topic information of the target topic with an input of a large language model to obtain aligned topic information of the target topic; embedding the aligned topic information into a preset prompt for the large language model to obtain input information of the large language model; inputting the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information contains the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; and inputting the enhanced topic information into a linear regression layer to predict a peak value of the target topic.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of multimedia technology, and more particularly, to a method and apparatus for predicting a video topic peak. BACKGROUND

[0002] With the increasing popularity of video platforms, a large number of hot events and phenomena emerge every day. Users participate in popular topic discussions on the platform by publishing, browsing, and commenting on videos, thereby generating a large amount of social interaction data and multi-modal data (such as images, videos, and text). These data interweave with each other and collectively build a complex heterogeneous information network, providing multiple channels for the spread of various types of content. Current research mainly focuses on the fields of recommendation systems, content classification, and video heat prediction. Traditional prediction methods usually focus on the analysis of fine-grained features of a single video to predict its heat peak in the next few hours, however, such isolated content analysis is significantly disturbed by noise and is difficult to capture the macro-level spread patterns. In contrast, studying from the topic level (rather than the single video level) can more effectively reveal the potential patterns of collective user emotions and emerging cultural trends, thereby providing deeper insights into understanding the dynamic evolution of content and user behavior. Therefore, predicting the heat peak of a video topic over a long period of time becomes an important research challenge.

[0003] Current research is dedicated to constructing a video propagation graph to learn representations and make heat predictions, however, such methods mostly rely on learning representations of topic nodes on a heterogeneous propagation graph and directly predicting topic peaks based on the representations, but this fails to effectively mine and utilize deep semantic information contained in multi-modal data itself, resulting in inaccurate prediction results. SUMMARY

[0004] Embodiments of the present disclosure provide a method and apparatus for predicting a video topic peak, which can effectively solve the problem of inaccurate topic peak prediction in the prior art.

[0005] In one general aspect, a method for predicting a video topic peak is provided, including: obtaining topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph contains topic information of a plurality of videos; aligning the topic information of the target topic with an input of a large language model to obtain aligned topic information of the target topic; embedding the aligned topic information into a preset prompt for the large language model to obtain input information of the large language model; inputting the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information contains the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; and inputting the enhanced topic information into a linear regression layer to predict a peak value of the target topic.

[0006] Optionally, the large language model is trained by: processing the initial large language model by using a parameter efficient fine-tuning method to obtain a fine-tuned large language model, wherein a number of adjustable parameters of the fine-tuned large language model is less than a number of adjustable parameters of the initial large language model; and training the fine-tuned large language model and a linear regression layer based on topic information of multiple topics obtained from the heterogeneous graph to obtain a trained large language model.

[0007] Optionally, the fine-tuned large language model and the linear regression layer are trained based on topic information of multiple topics obtained from the heterogeneous graph to obtain a trained large language model, including: aligning the topic information of the multiple topics with inputs of the large language model respectively to obtain respective aligned topic information of the multiple topics; and for each topic of the multiple topics, performing the following processing: embedding the aligned topic information of the current topic into a preset prompt to obtain estimated input information of the fine-tuned large language model; inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; inputting the enhanced estimated topic information into the initial linear regression layer to predict a peak value of the current topic; and adjusting parameters of the fine-tuned large language model and the initial linear regression layer based on a loss of the predicted peak value and a real peak value of each topic to obtain the trained large language model and the linear regression layer.

[0008] Optionally, before obtaining the topic information of the target topic from the heterogeneous graph, the method further includes: obtaining visual text information, user information, and topic information of multiple videos, wherein the visual text information includes visual information and text information; and constructing a heterogeneous graph with videos, users, and topics as nodes based on the visual text information, the user information, and the topic information of the multiple videos; and wherein obtaining the topic information of the target topic from the heterogeneous graph includes: obtaining the topic information of the target topic, visual text information related to the target topic, and user information related to the target topic from the heterogeneous graph, and taking the obtained topic information of the target topic, the visual text information related to the target topic, and the user information related to the target topic as final topic information of the target topic.

[0009] Optionally, the visual text information, the user information, and the topic information of each video of the multiple videos are obtained by: performing equal-interval frame sampling on a current video to obtain multiple frames of images; inputting the multiple frames of images into a video feature extraction model to extract visual information of each frame of image; performing average aggregation processing on the visual information of each frame of image to obtain visual information of the current video; inputting information of a text modality of the current video into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; initializing publisher information of the current video to obtain user information of the current video; and inputting a topic of the text modality of the current video into the text feature extraction model to obtain topic information of the current video.

[0010] Optionally, the topic information of the target topic is aligned with an input of a large language model to obtain aligned topic information of the target topic.

[0011] Optionally, the topic information of the target topic is obtained from the heterogeneous graph, including: inputting the heterogeneous graph into a pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, wherein the heterogeneous graph neural network is trained by: inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain topic information of each topic; for each topic, inputting the topic information of the topic into an initial linear prediction layer to predict a peak value of the topic; based on a loss of the predicted peak value and a real peak value of each topic, adjusting parameters of the initial heterogeneous graph neural network and the initial linear prediction layer to obtain the trained heterogeneous graph neural network and the linear prediction layer.

[0012] In another general aspect, a device for predicting a peak value of a video topic is provided, including: an obtaining unit configured to obtain topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph contains topic information of a plurality of videos; an aligning unit configured to align the topic information of the target topic with an input of a large language model to obtain aligned topic information of the target topic; an embedding unit configured to embed the aligned topic information into a preset prompt for the large language model to obtain input information of the large language model; an enhancing unit configured to input the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information contains the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; and a predicting unit configured to input the enhanced topic information into a linear regression layer to predict a peak value of the target topic.

[0013] Optionally, the device further includes a training unit configured to train the large language model by: processing an initial large language model using a parameter-efficient fine-tuning method to obtain a fine-tuned large language model, wherein a number of adjustable parameters of the fine-tuned large language model is less than a number of adjustable parameters of the initial large language model; and training the fine-tuned large language model and the linear regression layer based on the topic information of the plurality of topics obtained from the heterogeneous graph to obtain a trained large language model.

[0014] Optionally, the training unit is further configured to align the topic information of the plurality of topics with inputs of the large language model respectively to obtain respective aligned topic information of the plurality of topics; for each topic in the plurality of topics, perform the following processing: embedding the aligned topic information of the current topic into the preset prompt to obtain estimated input information of the fine-tuned large language model; inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; inputting the enhanced estimated topic information into the initial linear regression layer to predict a peak value of the current topic; adjusting parameters of the fine-tuned large language model and the initial linear regression layer based on a loss of the predicted peak value and a real peak value of each topic to obtain a trained large language model and linear regression layer.

[0015] Optionally, the obtaining unit is further configured to obtain visual text information, user information, and topic information of a plurality of videos, wherein the visual text information comprises visual information and text information; construct a heterogeneous graph with videos, users, and topics as nodes based on the visual text information, the user information, and the topic information of the plurality of videos; obtain topic information of a target topic, visual text information related to the target topic, and user information related to the target topic from the heterogeneous graph, and take the obtained topic information of the target topic, the visual text information related to the target topic, and the user information related to the target topic as final topic information of the target topic.

[0016] Optionally, the visual text information, the user information, and the topic information of each video in the plurality of videos are obtained by the following methods respectively: performing equal-interval frame sampling on a current video to obtain a plurality of images; inputting the plurality of images into a video feature extraction model to extract visual information of each image; performing average aggregation processing on the visual information of each image to obtain visual information of the current video; inputting information of a text modality of the current video into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; initializing publisher information of the current video to obtain user information of the current video; inputting a topic of the text modality of the current video into the text feature extraction model to obtain topic information of the current video.

[0017] Optionally, the aligning unit is further configured to input the topic information of the target topic into a graph adapter to obtain aligned topic information of the target topic.

[0018] Optionally, the obtaining unit is further configured to input the heterogeneous graph into a pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, and the training unit is further configured to train the heterogeneous graph neural network by: inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain the topic information of each topic; inputting the topic information of each topic into an initial linear prediction layer to predict the peak value of the topic; and adjusting parameters of the initial heterogeneous graph neural network and the initial linear prediction layer based on a loss of the predicted peak value and the real peak value of each topic to obtain the trained heterogeneous graph neural network and the linear prediction layer.

[0019] In another general aspect, a computer-readable storage medium storing instructions is provided, wherein the instructions, when executed by at least one computing device, cause the at least one computing device to perform any of the above methods for predicting a video topic peak value.

[0020] In another general aspect, a system including at least one computing device and at least one storage device storing instructions is provided, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform any of the above methods for predicting a video topic peak value.

[0021] In another general aspect, a computer program product including computer instructions that, when executed by a processor, implement any of the above methods for predicting a video topic peak value.

[0022] The method and device for predicting a video topic peak value according to embodiments of the present disclosure embed topic information of a target topic obtained from a heterogeneous graph into a preset prompt of a large language model, so that the large language model can guide the large language model to enhance the topic information of the target topic based on the embedded topic information and semantic instructions in the preset prompt using prior knowledge of the large language model, and then predict a topic peak value based on the enhanced topic information. The present disclosure combines a heterogeneous graph with a large language model, uses the large language model to obtain external knowledge to assist in predicting a topic peak value, and improves the accuracy of predicting a topic peak value. Therefore, the present disclosure can effectively solve the problem of inaccurate prediction of a topic peak value in the prior art.

[0023] Additional aspects and / or advantages of the general inventive concept will be set forth in part in the description that follows, and in part will be obvious from the description, or can be learned by practice of the general inventive concept. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other objects and features of embodiments of the present disclosure will become more apparent from the following description made with reference to the accompanying drawings, in which:

[0025] Figure 1is a schematic diagram illustrating a short video topic peak prediction (SVTPP) of an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart illustrating a method of predicting a topic peak of an embodiment of the present disclosure;

[0027] Figure 3 is a flowchart illustrating a method of predicting a short video topic peak of an embodiment of the present disclosure;

[0028] Figure 4 is a schematic diagram illustrating a system architecture employed by a method of predicting a short video topic peak of an embodiment of the present disclosure;

[0029] Figure 5 is a block diagram of a prediction apparatus of a video topic peak of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following detailed description is provided to aid in understanding the method, apparatus, and / or system described herein. However, various changes, modifications, and equivalents of the method, apparatus, and / or system described herein will be clear to those skilled in the art after understanding the disclosure provided herein. For example, the order of the operations described herein is merely an example, and is not limited to those set forth herein, but can be changed as will be clear to those skilled in the art after understanding the disclosure provided herein, except for operations that must occur in a specific order. Also, the description of features known in the art can be omitted for the sake of clarity and conciseness.

[0031] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided so as to illustrate some of the many possible ways of implementing the method, apparatus, and / or system described herein, which will be clear to those skilled in the art after understanding the disclosure provided herein.

[0032] As used herein, the term "and / or" includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0033] Although terms such as "first", "second", and "third" can be used herein to describe various components, assemblies, regions, layers, or portions, the components, assemblies, regions, layers, or portions should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or portion from another component, assembly, region, layer, or portion. Thus, a first component, a first assembly, a first region, a first layer, or a first portion referred to in the examples described herein can also be referred to as a second component, a second assembly, a second region, a second layer, or a second portion without departing from the teachings of the examples.

[0034] In the description, when an element (such as a layer, a region, or a substrate) is referred to as being "on" another element, "connected to" or "coupled to" another element, it can be directly on, directly connected to, or directly coupled to the other element, or one or more other elements can be interposed therebetween. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element, there are no other elements interposed therebetween.

[0035] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including" and "has," "having" as used herein, specify the presence of stated features, numbers, operations, components, elements, and / or groups thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, components, elements, and / or groups thereof.

[0036] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs when read in light of the present disclosure. Unless otherwise explicitly defined herein, the terms (such as terms commonly used in general dictionaries) should be interpreted as having meanings consistent with their meanings in the context of the relevant art and in the disclosure, and should not be interpreted ideally or too formally.

[0037] In addition, in the description of the examples, when it is considered that a detailed description of the related structure or function known as such will cause a blurred explanation of the disclosure, such a detailed description will be omitted.

[0038] For video topic heat peak prediction, especially short video topic heat peak prediction (abbreviated as SVTPP), as shown in Figure 1 A topic level heterogeneous graph is generally constructed, and the cumulative heat peak of each topic in a specific time period is predicted, such as the topic peak can be defined as the sum of the heat peaks of all associated videos under the topic, but the prediction of the topic peak in the related art is not ideal.

[0039] To solve the above problems, the present disclosure proposes a large language model enhancement solution, that is, a video topic peak prediction method based on a heterogeneous graph large language model. Specifically, first, topic information of a target topic is extracted from the constructed heterogeneous graph, such as the topic information of the target topic extracted from the constructed heterogeneous graph by a heterogeneous graph neural network (HGNN). Second, align the topic information of the target topic with the input of the large language model, such as using a graph adapter (Graph Adapter) to adapt and map the topic information of the target topic learned by the HGNN to the space of the large language model. Third, embed the aligned topic information into the preset prompt for the large language model to obtain the input information of the large language model. Then, input the input information into the large language model to guide the large language model to generate optimized topic information containing rich semantic and structural information, which is used to support the regression prediction of the final topic peak. The present disclosure fully integrates heterogeneous graphs and multi-modal semantic information, effectively solves the shortcomings of existing methods in mining deep semantic information and modeling complex topic dynamics, and significantly improves the accuracy of long-term heat peak prediction at the video topic level. In addition, to improve the model training efficiency and reduce the parameter update cost, the large language model of the present disclosure uses LoRA technology for fine-tuning.

[0040] It should be noted that the above-mentioned video can be a short video or a long video, and the present disclosure does not limit it. The above-mentioned topic can be any topic, such as "daily record", "walking baby holy place", etc., and the present disclosure does not limit it. The above-mentioned topic peak can be the topic play quantity, the topic forwarding quantity, the topic comment quantity, etc., and the present disclosure does not limit it.

[0041] The prediction method and device for video topic peak of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0042] The present disclosure proposes a prediction method for video topic peak, Figure 2 is a flowchart showing the prediction method for video topic peak of an embodiment of the present disclosure. Referring to Figure 2 , the prediction method for video topic peak includes the following steps:

[0043] In step S201, topic information of a target topic is obtained from a heterogeneous graph, wherein the heterogeneous graph contains topic information of multiple videos.

[0044] As an example, the above-mentioned heterogeneous graph can be constructed based on user information and topic information, or can be constructed based on visual text information and topic information, and the present disclosure does not limit it.

[0045] According to an embodiment of the present disclosure, before obtaining topic information of a target topic from a heterogeneous graph, visual text information, user information and topic information of a plurality of videos can be obtained, wherein the visual text information comprises visual information and text information; based on the visual text information, the user information and the topic information of the plurality of videos, a heterogeneous graph taking videos, users and topics as nodes is constructed; wherein obtaining the topic information of the target topic from the heterogeneous graph can comprise: obtaining the topic information of the target topic, visual text information related to the target topic and user information related to the target topic from the heterogeneous graph, and taking the obtained topic information of the target topic, the visual text information related to the target topic and the user information related to the target topic as the final topic information of the target topic. Through the present embodiment, the present disclosure constructs a heterogeneous graph taking videos, users and topics as nodes based on the visual text information, the user information and the topic information of the plurality of videos, thereby unifying cross-platform propagation dynamics and multi-modal data into a time-series heterogeneous graph, and solving the problem of insufficient modeling of long-term propagation patterns at the topic level in traditional methods.

[0046] As an example, after obtaining the visual text information, the user information and the topic information of the plurality of videos, a heterogeneous graph containing three types of nodes, i.e. video nodes, user nodes and topic nodes, can be constructed based on the visual text information, the user information and the topic information of the plurality of videos, so as to fuse multi-modal data, wherein the video nodes integrate visual information and text information, the user nodes integrate social attributes (i.e. integrate user information), and the topic nodes are associated with cross-platform propagation paths (i.e. integrate topic information), so that cross-platform propagation dynamics and multi-modal content can be unified into a time-series heterogeneous graph, and the problem of insufficient modeling of long-term propagation patterns at the topic level in traditional methods is solved.

[0047] It should be noted that the present disclosure is not limited to visual text information, user information and topic information, and other information can also be obtained to construct a heterogeneous graph, and the present disclosure does not limit this.

[0048] According to an embodiment of the present disclosure, the visual text information, the user information and the topic information of each video in the plurality of videos can be obtained in the following manner: equally spaced frame sampling is performed on a current video to obtain a plurality of images; the plurality of images are input into a video feature extraction model to extract visual information of each image; average aggregation processing is performed on the visual information of each image to obtain visual information of the current video; information of a text modality of the current video is input into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; the publisher information of the current video is initialized to obtain user information of the current video; and a topic of the text modality of the current video is input into the text feature extraction model to obtain topic information of the current video. Through the present embodiment, the visual text information, the user information and the topic information of each video can be obtained conveniently and quickly.

[0049] Specifically, the video feature extraction model and the text feature extraction model can be pre-trained good models. For example, the video feature extraction model can be a pre-trained Vision Transformer (ViT), and the text feature extraction model can be a pre-trained Bert-base-chinese1 (BERT), which is not limited in the present disclosure.

[0050] As an example, for the information of the video modality of each video, visual features can be extracted from the frame sequence of the video. For example, for the information of the video modality of the video The equally spaced frame sampling strategy can be used to extract frames from the video , so that the sampled frame set of the video is , where is the number of sampled frames. It should be noted that the sampling interval can be a fixed interval and 8 frames are uniformly sampled for each video, which is not limited in the present disclosure. After sampling, a video feature extraction model (Vision Transformer, abbreviated as ViT) can be applied to process each sampled frame to extract its frame-level feature vector, that is, the visual information of each frame image. After obtaining the visual information of each sampled frame , the original feature of the video used to construct the heterogeneous graph can be obtained by the following formula:

[0051]

[0052] wherein represents the frame-level feature vector extracted from the i-th frame by applying the ViT model.

[0053] As an example, for the information of the text modality of each video, such as the information of the fields of the content, title and description of the video (except the information of the topic field). For example, for the information of the text modality of the video A text feature extraction model (Bert-base-chinese1, abbreviated as BERT) can be applied to extract the original feature of the video used to construct the heterogeneous graph from these fields:

[0054]

[0055] Here, ​​​represents a feature vector extracted from the embedded splicing result of fields such as content, title and description, that is, the text information of the video.

[0056] As an example, for features that cannot be directly extracted from the text content (for example, the video publisher), random initialization can be performed thereon. For example, the publisher information of the video can be initialized to obtain the user information of the video, and the present disclosure does not limit this.

[0057] As an example, for the information of the topic field in the information of the text modality of each video, a BERT model can be applied to extract the topic information of the video from the information of the topic field, that is, the topic of the video, and the present disclosure does not limit this.

[0058] According to an embodiment of the present disclosure, obtaining the topic information of the target topic from the heterogeneous graph can include: inputting the heterogeneous graph into a pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, wherein the heterogeneous graph neural network can be trained by: inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain the topic information of each topic; for each topic, inputting the topic information of the topic into an initial linear prediction layer to predict the peak value of the topic; and adjusting the parameters of the initial heterogeneous graph neural network and the initial linear prediction layer based on the loss of the predicted peak value and the real peak value of each topic to obtain the trained heterogeneous graph neural network and the linear prediction layer. Through this embodiment, in the training process of the heterogeneous graph neural network (HGNN), the heterogeneous graph that fuses the multi-modal data of multiple videos is utilized, so that the trained HGNN can capture the relationship between various video components and the associated topics, thereby utilizing the trained HGNN to extract the topic information of the required topic, which can improve the accuracy and richness of the extracted topic information.

[0059] As an example, the above-mentioned heterogeneous graph neural network (HGNN) can process the heterogeneous relationship in layers, for example, the HGNN can adopt a 2-layer HetSANN model, wherein the first layer learns local node interaction and the second layer captures global topic evolution, so that a deeper representation of each node in the heterogeneous graph can be learned, and then the topic information is extracted, and the present disclosure does not limit this.

[0060] It should be noted that the HGNN can also assign weights to different edge types through an attention mechanism to strengthen the perception ability of key propagation paths, and the present disclosure does not limit this.

[0061] As an example, assuming that the heterogeneous graph and the original features used to construct the heterogeneous graph (containing the extracted features described above), wherein, represents a node, represents an edge, represents the features of the node, represents the features of the edges, at this time the HGNN model can be represented as:

[0062]

[0063] wherein, represents the topic information of the topic obtained by the HGNN model from the heterogeneous graph.

[0064] The above obtained is input into the linear prediction layer to predict the peak value of the topic :

[0065]

[0066] Then, using the predicted peak value of each topic, the mean square error (MSE) loss function can be obtained as follows:

[0067]

[0068] wherein, represents the predicted peak value of the topic , represents the true peak value of the topic , and respectively represent the weight matrix and the bias term that can be learned by the prediction layer, represents the total number of topics in the heterogeneous graph.

[0069] Then, by minimizing the above loss function, the parameters of the heterogeneous graph neural network and the linear prediction layer can be adjusted to obtain the trained heterogeneous graph neural network and the linear prediction layer.

[0070] As an example, after the heterogeneous graph neural network is trained, the heterogeneous graph is input into the trained heterogeneous graph neural network, and the topic information of each topic in the heterogeneous graph can be obtained, and then the topic information of the target topic is selected from the topic information, and the obtained topic information can be used in the subsequent stage.

[0071] Returning to Figure 2 , in step S202, the topic information of the target topic is aligned with the input of the large language model to obtain the aligned topic information of the target topic.

[0072] According to an embodiment of the present disclosure, the topic information of the target topic can be aligned with the input of the large language model by inputting the topic information of the target topic into a graph adapter to obtain the aligned topic information of the target topic. Through the present embodiment, the graph adapter is used to bridge the graph knowledge and the large language model, to realize the bridging of heterogeneous features to the semantic space, and to project and map the output of the graph neural network as the input of the large language model, to realize the cooperation of semantic understanding and structural reasoning.

[0073] As an example, the HGNN model output can be input into the graph adapter as follows: Further, the graph adapter is input to realize the alignment of the output of the HGNN and the input requirement of the large language model (LLM). Specifically, the graph adapter can be defined as a linear mapping as follows:

[0074]

[0075] wherein, represents the aligned topic information, represents a weight matrix, represents a bias term.

[0076] In step S203, the aligned topic information is embedded into a preset prompt for the large language model to obtain the input information of the large language model.

[0077] Specifically, a prompt can be designed in advance, and the preset prompt can include a special graph token and a task description. The structure of the preset prompt is designed to include the video and its context information, wherein the position of the graph token is replaced by the aligned topic information , that is, the aligned topic information is embedded in the preset prompt. In this way, the input embedding of the large language model (i.e., the above input information) can be constructed by splicing the task description and the graph token as follows:

[0078]

[0079] In step S204, the input information is input into the large language model to obtain the enhanced topic information of the target topic, wherein the enhanced topic information includes the topic information of the target topic and the prior knowledge related to the topic information of the target topic in the large language model.

[0080] As an example, after obtaining the above input information Then, the enhanced topic information can be input into the LLM, and the LLM can obtain enhanced topic information in combination with prior knowledge of the LLM The enhanced topic information can be obtained in the following manner:

[0081]

[0082] According to an embodiment of the present disclosure, the large language model can be trained in the following manner: an initial large language model is processed by using a parameter-efficient fine-tuning method to obtain a fine-tuned large language model, wherein the number of adjustable parameters of the fine-tuned large language model is less than the number of adjustable parameters of the initial large language model; and the fine-tuned large language model and the linear regression layer are trained based on topic information of a plurality of topics obtained from a heterogeneous graph to obtain a trained large language model. Through this embodiment, the parameter-efficient fine-tuning method (such as LoRA) is used to freeze most parameters of the large language model, and only the low-rank matrix and the linear regression layer are fine-tuned, thereby reducing the risk of overfitting while adapting to the peak regression task.

[0083] As an example, before training the LLM, a parameter-efficient fine-tuning (PEFT) method, in particular LoRA, can be used to process the large language model to alleviate the problem of overfitting commonly seen in large language models. Specifically, the LoRA technique introduces a low-rank matrix into the large language model, i.e., freezes most parameters of the large language model, and only allows efficient adaptation with fewer trainable parameters, thereby preserving previous pre-training knowledge and improving generalization performance.

[0084] It should be noted that the training process of the large language model is similar to that of the HGNN model, and mean square error (MSE) loss can also be used to adjust the parameters of the large language model and the linear regression layer, which is not limited by the present disclosure. Moreover, the present disclosure combines the graph adapter with the LoRA-based PEFT to achieve efficient fine-tuning, and reduces the risk of overfitting while maintaining performance.

[0085] According to embodiments of this disclosure, the above-mentioned training of a fine-tuned large language model and a linear regression layer based on topic information obtained from a heterogeneous graph to obtain a trained large language model may include: aligning the topic information of multiple topics with the input of the large language model to obtain aligned topic information for each of the multiple topics; for each of the multiple topics, performing the following processing: embedding the aligned topic information of the current topic with a preset prompt to obtain the estimated input information of the fine-tuned large language model; inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information for the current topic; inputting the enhanced estimated topic information into the initial linear regression layer to predict the peak value of the current topic; and adjusting the parameters of the fine-tuned large language model and the initial linear regression layer based on the loss between the predicted peak value and the actual peak value for each topic to obtain the trained large language model and the linear regression layer. Through this embodiment, the large language model can be trained conveniently and accurately.

[0086] As an example, after obtaining topic information for multiple topics from a heterogeneous graph, this topic information can be aligned with the input of a large language model. Then, for each topic, the aligned topic information is embedded into a preset prompt to obtain the corresponding input information. After inputting this input information into an LLM, the enhanced topic information for that topic can be obtained. This enhanced topic information can then be input into a linear regression layer to predict the peak value of that topic.

[0087] As an example, to predict the peak value for each topic, the enhanced topic information for each topic can be used. Input a linear regression layer to obtain the corresponding topic. Predicted peak Specifically, it can be as follows:

[0088]

[0089] Then, the mean squared error (MSE) loss function is obtained using the prediction peak for each topic, as follows:

[0090]

[0091] in, Indicate topic The predicted peak, Indicate topic The true peak value It is the weight matrix of the regression layer. This represents the bias term of the regression layer. This represents the total number of topics in the heterogeneous graph.

[0092] Then, the parameters of the LLM and the linear regression layer can be adjusted by minimizing the loss function to obtain a trained LLM and linear regression layer.

[0093] Finally, the large language model after instruction tuning is input with customized prediction instructions, i.e., the preset prompt, and the aligned topic information as key input information to obtain enhanced topic information.

[0094] It should be noted that the linear prediction layer and the linear regression layer can also be collectively referred to as a linear layer, which is not limited by the present disclosure.

[0095] Return Figure 2 In step S205, the enhanced topic information is input into the linear regression layer to predict the peak value of the target topic.

[0096] As an example, the enhanced topic information of the target topic is input into the linear regression layer to obtain the predicted peak value of the corresponding topic , which can be specifically as follows:

[0097]

[0098] wherein, is the weight matrix of the linear regression layer, represents the bias term of the linear regression layer, represents the predicted peak value of the target topic.

[0099] In order to better understand the present disclosure, the following will take a short video as an example, and the system will be described in combination with Figure 3 and Figure 4 .

[0100] Figure 3 A flowchart of a short video topic peak prediction method is shown, as shown in Figure 3 , the prediction method comprises the following steps:

[0101] Step S301: Extracting the initial features of the multi-modal short video. Specifically, equally spaced frames are sampled for each short video, and the visual features of the single-frame image are extracted using the ViT model, and the average value of the features of all the sampled frames is calculated to obtain the aggregated visual features (i.e., the visual information of the video); the text features of each short video title, description, content, etc. are extracted using the BERT model, and the non-text features (such as user information, etc.) of each short video are randomly initialized.

[0102] ​​Step S302: Construct a heterogeneous graph for short video propagation (including video, user, and topic nodes), and extract topic information of each topic from the heterogeneous graph through a heterogeneous graph neural network (HGNN). Specifically, the HGNN adopts a 2-layer HetSANN model to process heterogeneous relationships, learn deep node representations, and extract topic information of the topic, and then superimposes a linear prediction layer to predict the peak value of the topic, and optimizes the prediction task with an MSE loss function, and then the optimized HGNN can be used to extract the topic information of the required topic.

[0103] Step S303: Align the features of the heterogeneous graph-large language model. Specifically, a graph adapter is designed to bridge the representation spaces of the HGNN and the LLM. That is, the topic information of the target topic output by the HGNN is linearly mapped to information suitable for the large language model, so that its dimension is compatible with the input of the LLM.

[0104] Step S304: Construct an instruction-guided prompt word and embed the aligned topic information into the prompt word, and integrate the semantic instructions and the adapted graph features (i.e., the aligned topic information). Specifically, the aligned topic information is embedded into the prompt graph as a graph structure identifier, and the input embedding of the large language model is formed by splicing.

[0105] Step S305: Efficiently fine-tune the parameters of the large language model by combining the PEFT technology, and use the fine-tuned large language model to predict the peak value of the topic. Specifically, lightweight fine-tuning can introduce a low-rank matrix using the LoRA technology, update only a small number of parameters of the LLM to reduce the risk of overfitting, and then implement peak regression to generate the final prediction result by applying the LLM output to a linear regression layer.

[0106] Step S306: Output the peak value of the multi-platform topic based on the fine-tuned large language model. Specifically, for each platform topic, the input can be cross-platform heterogeneous data (such as including video, text, user interaction, etc.), and the output is the peak prediction value of each topic (such as the number of plays, comments, and forwards, etc.).

[0107] Figure 4 A system architecture used by a short video topic peak prediction method is shown in FIG. 1. Figure 4

[0108] First, a heterogeneous graph containing video, user, and topic nodes is constructed to fuse multi-modal features, wherein the video node integrates visual and text features, the user node integrates social attributes, and the topic node is associated with cross-platform propagation paths. Then, a heterogeneous graph neural network is used to process heterogeneous relationships in layers: the first layer learns local node interactions, and the second layer captures global topic evolution, and then the topic information of topic t1 is extracted through the heterogeneous graph neural network, i.e., the topic graph representation in Figure 4

[0109] ​​Secondly, the topic graph representation output by the heterogeneous graph neural network is linearly mapped to a low-dimensional vector by a graph adaptor, key features of the graph structure (such as node centrality) are retained, and the input space of a large language model (LLM) is aligned. Then, a task-specific prompt is constructed, and the adapted topic graph representation is embedded into the prompt as a special token (GraphToken). The special token can be, for example, Figure 4 The graph structure identifier in the above embodiment is [Graph_Adaptor], which guides the LLM to combine semantic instructions with graph structure information to generate enhanced topic information, i.e. Figure 4 the topic LLM representation in the above embodiment. It should be noted that, during the LMM training process, the LoRA technology can be used to freeze most parameters of the LLM, and only the low-rank matrix and the regression layer are fine-tuned, so as to reduce the risk of overfitting and adapt to the peak regression task.

[0110] Thirdly, based on the topic LLM representation, subsequent downstream tasks such as topic peak prediction are performed.

[0111] In summary, the above embodiment provides an end-to-end large language model based on propagation of heterogeneous graphs for short video topic peak prediction. The method constructs an end-to-end two-stage learning framework, including a heterogeneous graph representation learning stage and a large model semantic enhancement stage, i.e. through heterogeneous graph structure modeling and semantic instruction fine-tuning, cross-platform prediction of short video topic peaks is realized. In the above embodiment, for cross-platform propagation data that is not explicitly aligned, the hierarchical learning mechanism of the heterogeneous graph neural network (HGNN) can be used to capture the dynamic relationship between video, user and topic nodes, form a structured node representation, and a graph adaptor (Graph Adaptor) is designed to map the heterogeneous graph representation to the language model space. Through the instruction prompt (Prompt), the graph features are embedded into the natural language task description as special tokens, and combined with LoRA parameter efficient fine-tuning, the modal specificity is eliminated and cross-platform peak regression prediction is realized, which significantly reduces the prediction error (MAE average reduction of 8.84%); moreover, the present disclosure opens up a new way for the fusion of graph learning and large language model understanding of complex short video topic dynamics.

[0112] In order to verify the feasibility of the present disclosure, the present disclosure also uses the public TopicVid short video trend topic dataset, which covers 7701 hot topics, 58539 users and 96936 short videos from multiple mainstream video platforms, and collects multi-modal information such as video content and text. The dataset records the daily views, comments and forwarding quantities of each video within two weeks after publication, and defines the peak of each topic as the sum of the peaks of all short videos under it.

[0113] Table 1 is a comparison of the method of the present disclosure with other methods on the TopicVid dataset, using three evaluation indicators: mean absolute error (MAE), normalized mean squared error (nMSE), and mean absolute percentage error (MAPE). As can be seen from Table 1, the performance of the present disclosure on the dataset is significantly better than other methods. Specifically, the method of the present disclosure consistently outperforms traditional graph neural network (GNN) models, and compared with the best GNN model (HetSANN), the method of the present disclosure improves by an average of 8.84% in MAE, 16.54% in nMSE, and 0.41% in MAPE, which shows significant improvements in MAE and nMSE indicators, proving the effectiveness of large language model instruction fine-tuning. The improvement of RGCN over GCN, HAN, and GAT shows that using heterogeneous information in the propagation network can improve the performance of the SVTPP task, i.e., when only using topic text information, large language models (LLMs) can achieve basic topic prediction performance; but by incorporating graph structure information and combining with LLMs, the method of the present disclosure achieves better results, and there are significant differences between different topic peak types.

[0114] Table 1 Evaluation results of short video topic heat peak prediction

[0115]

[0116] Figure 5 is a block diagram of a video topic peak prediction device according to an embodiment of the present disclosure, as shown in Figure 5 The device includes an acquisition unit 50, an alignment unit 52, an embedding unit 54, an enhancement unit 56, and a prediction unit 58.

[0117] The acquisition unit 50 is configured to acquire topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph contains topic information of multiple videos; the alignment unit 52 is configured to align the topic information of the target topic with an input of a large language model, to obtain aligned topic information of the target topic; the embedding unit 54 is configured to embed the aligned topic information into a preset prompt for the large language model, to obtain input information of the large language model; the enhancement unit 56 is configured to input the input information into the large language model, to obtain enhanced topic information of the target topic, wherein the enhanced topic information contains the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; and the prediction unit 58 is configured to input the enhanced topic information into a linear regression layer, to predict a peak value of the target topic.

[0118] According to an embodiment of the present disclosure, the device further comprises a training unit configured to train the large language model by: processing the initial large language model by using the parameter-efficient fine-tuning method to obtain a fine-tuned large language model, wherein the number of adjustable parameters of the fine-tuned large language model is less than the number of adjustable parameters of the initial large language model; training the fine-tuned large language model and the linear regression layer based on topic information of multiple topics obtained from the heterogeneous graph to obtain a trained large language model.

[0119] Optionally, the training unit is further configured to align the topic information of the multiple topics with the input of the large language model respectively to obtain the aligned topic information of the multiple topics respectively; for each topic in the multiple topics, the following processing is performed: embedding the aligned topic information of the current topic into a preset prompt to obtain estimated input information of the fine-tuned large language model; inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; inputting the enhanced estimated topic information into the initial linear regression layer to predict a peak value of the current topic; adjusting the parameters of the fine-tuned large language model and the initial linear regression layer based on the loss of the predicted peak value and the real peak value of each topic to obtain the trained large language model and the linear regression layer.

[0120] Optionally, the obtaining unit 50 is further configured to obtain visual text information, user information, and topic information of multiple videos, wherein the visual text information comprises visual information and text information; construct a heterogeneous graph with videos, users, and topics as nodes based on the visual text information, the user information, and the topic information of the multiple videos; obtain topic information of a target topic, visual text information related to the target topic, and user information related to the target topic from the heterogeneous graph, and take the obtained topic information of the target topic, the visual text information related to the target topic, and the user information related to the target topic as the final topic information of the target topic.

[0121] Optionally, the visual text information, the user information, and the topic information of each video in the multiple videos are obtained by the following methods respectively: performing equal-interval frame sampling on a current video to obtain multiple frames of images; inputting the multiple frames of images into a video feature extraction model to extract visual information of each frame of image; performing average aggregation processing on the visual information of each frame of image to obtain visual information of the current video; inputting information of a text modality of the current video into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; initializing publisher information of the current video to obtain user information of the current video; inputting a topic of the text modality of the current video into the text feature extraction model to obtain topic information of the current video.

[0122] Optionally, the alignment unit 52 is further configured to input the topic information of the target topic into a graph adapter to obtain the aligned topic information of the target topic.

[0123] Optionally, the obtaining unit 50 is further configured to input the heterogeneous graph into a pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, wherein the training unit is further configured to train the heterogeneous graph neural network by: inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain the topic information of each topic; for each topic, inputting the topic information of the topic into an initial linear prediction layer to predict a peak value of the topic; and adjusting parameters of the initial heterogeneous graph neural network and the initial linear prediction layer based on a loss of the predicted peak value and a real peak value of each topic to obtain the trained heterogeneous graph neural network and the linear prediction layer.

[0124] According to an embodiment of the present disclosure, there is provided a computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one computing device, cause the at least one computing device to perform the method of predicting a video topic peak value according to any one of the above embodiments.

[0125] According to an embodiment of the present disclosure, there is provided a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the method of predicting a video topic peak value according to any one of the above embodiments.

[0126] According to an embodiment of the present disclosure, there is provided a computer program product comprising computer instructions that, when executed by a processor, implement the method of predicting a video topic peak value according to any one of the above embodiments.

[0127] Although some embodiments of the present disclosure have been shown and described, it should be understood by those skilled in the art that modifications can be made to these embodiments without departing from the principles and spirit of the present disclosure, which are defined by the following claims and their equivalents.

Claims

1. A method for predicting peak video topics, characterized in that, include: Obtain visual text information, user information, and topic information from multiple videos, wherein the visual text information includes visual information and text information; Based on the visual text information, user information, and topic information of the multiple videos, a heterogeneous graph is constructed with videos, users, and topics as nodes. Obtain topic information, visual text information and user information related to the target topic from the heterogeneous graph, and use the obtained topic information, visual text information and user information related to the target topic as the final topic information of the target topic. Align the final topic information of the target topic with the input of the large language model to obtain the aligned topic information of the target topic; The aligned topic information is embedded into a preset prompt for the large language model to obtain the input information of the large language model; The input information is input into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information includes the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; The enhanced topic information is input into a linear regression layer to predict the peak value of the target topic; The large language model is trained in the following manner: An efficient parameter fine-tuning method is used to process the initial large language model to obtain a fine-tuned large language model, wherein the number of adjustable parameters in the fine-tuned large language model is less than the number of adjustable parameters in the initial large language model. Based on topic information obtained from the heterogeneous graph, the fine-tuned large language model and the linear regression layer are trained to obtain the trained large language model.

2. The prediction method as described in claim 1, characterized in that, The step of training the fine-tuned large language model and the linear regression layer based on topic information obtained from the heterogeneous graph to obtain the trained large language model includes: Align the topic information of the multiple topics with the input of the large language model to obtain the aligned topic information of each of the multiple topics. For each of the multiple topics, perform the following processing: The aligned topic information of the current topic is embedded into the preset prompt to obtain the predicted input information of the fine-tuned large language model; The predicted input information is input into the fine-tuned large language model to obtain the enhanced predicted topic information for the current topic. The enhanced predicted topic information is input into the initial linear regression layer to predict the peak value of the current topic; Based on the loss of the predicted peak and the actual peak for each topic, the parameters of the fine-tuned large language model and the initial linear regression layer are adjusted to obtain the trained large language model and linear regression layer.

3. The prediction method as described in claim 1, characterized in that, The visual text information, user information, and topic information of each of the multiple videos are obtained in the following ways: The current video is sampled at equal intervals to obtain multiple frames of images; The multi-frame images are input into a video feature extraction model to extract the visual information of each frame. The visual information of each frame is averaged and aggregated to obtain the visual information of the current video. The text modality information of the current video is input into the text feature extraction model to obtain the text information of the current video, wherein the text modality information is the information of the topic of the current video after removing the text modality. Initialize the publisher information of the current video to obtain the user information of the current video; The topic of the current video's text modality is input into the text feature extraction model to obtain the topic information of the current video.

4. The prediction method as described in claim 1, characterized in that, The step of aligning the topic information of the target topic with the input of the large language model to obtain the aligned topic information of the target topic includes: Input the topic information of the target topic into the graph adapter to obtain the aligned topic information of the target topic.

5. The prediction method as described in claim 1, characterized in that, The process of obtaining topic information for the target topic from the heterogeneous graph includes: The heterogeneous graph is input into a pre-trained heterogeneous graph neural network to obtain topic information for the target topic. The heterogeneous graph neural network is trained in the following manner: The heterogeneous graph is input into the initial heterogeneous graph neural network to obtain topic information for each topic; For each topic, the topic information is input into the initial linear prediction layer to predict the peak value of the topic; Based on the loss of the predicted peak and the actual peak for each topic, the parameters of the initial heterogeneous graph neural network and the initial linear prediction layer are adjusted to obtain the trained heterogeneous graph neural network and the linear prediction layer.

6. A device for predicting peak video topics, characterized in that, include: The acquisition unit is configured to acquire visual text information, user information, and topic information from multiple videos, wherein the visual text information includes visual information and text information; construct a heterogeneous graph with videos, users, and topics as nodes based on the visual text information, user information, and topic information of the multiple videos; acquire topic information of a target topic, visual text information related to the target topic, and user information from the heterogeneous graph, and use the acquired topic information of the target topic, visual text information related to the target topic, and user information as the final topic information of the target topic; The alignment unit is configured to align the final topic information of the target topic with the input of the large language model to obtain the aligned topic information of the target topic. The embedding unit is configured to embed the aligned topic information into a preset prompt for a large language model to obtain the input information of the large language model; An enhancement unit is configured to input the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information includes topic information of the target topic and prior knowledge in the large language model related to the topic information of the target topic. The prediction unit is configured to input the enhanced topic information into a linear regression layer to predict the peak value of the target topic; The large language model is trained as follows: the initial large language model is processed using a parameter-efficient fine-tuning method to obtain a fine-tuned large language model, wherein the number of adjustable parameters in the fine-tuned large language model is less than the number of adjustable parameters in the initial large language model; based on topic information obtained from the heterogeneous graph, the fine-tuned large language model and the linear regression layer are trained to obtain the trained large language model.

7. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the video topic peak prediction method as described in any one of claims 1 to 5.

8. A system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the video topic peak prediction method as described in any one of claims 1 to 5.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method for predicting video topic peaks as described in any one of claims 1 to 5.

Images