Method and device for predicting peak value of video topic

Abstract

The invention discloses a video topic peak value prediction method and device, and the method comprises the steps: obtaining the topic information of a target topic from a heterogeneous graph which comprises the topic information of a plurality of videos; aligning the topic information of the target topic with the input of the large language model to obtain the aligned topic information of the target topic; embedding the aligned topic information into a preset prompt for the large language model to obtain input information of the large language model; the input information is input into the large language model, enhanced topic information of the target topic is obtained, and the enhanced topic information comprises topic information of the target topic and priori knowledge related to the topic information of the target topic in the large language model; and inputting the enhanced topic information into a linear regression layer, and predicting the peak value of the target topic.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of multimedia technology, and more particularly, to a method and apparatus for predicting a video topic peak. BACKGROUND

[0002] With the increasing popularity of video platforms, a large number of hot events and phenomena emerge every day. Users participate in popular topic discussions on the platform by publishing, browsing, and commenting on videos, thereby generating a large amount of social interaction data and multi-modal data (such as images, videos, and text). These data interweave with each other and collectively build a complex heterogeneous information network, providing multiple channels for the spread of various types of content. Current research mainly focuses on the fields of recommendation systems, content classification, and video heat prediction. Traditional prediction methods usually focus on the analysis of fine-grained features of a single video to predict its heat peak in the next few hours, however, such isolated content analysis is significantly disturbed by noise and is difficult to capture the macro-level spread patterns. In contrast, studying from the topic level (rather than the single video level) can more effectively reveal the potential patterns of collective user emotions and emerging cultural trends, thereby providing deeper insights into understanding the dynamic evolution of content and user behavior. Therefore, predicting the heat peak of a video topic over a long period of time becomes an important research challenge.

[0003] Current research is dedicated to constructing a video propagation graph to learn representations and make heat predictions, however, such methods mostly rely on learning representations of topic nodes on a heterogeneous propagation graph and directly predicting topic peaks based on the representations, but this fails to effectively mine and utilize deep semantic information contained in multi-modal data itself, resulting in inaccurate prediction results. SUMMARY

[0004] Embodiments of the present disclosure provide a method and apparatus for predicting a video topic peak, which can effectively solve the problem of inaccurate topic peak prediction in the prior art.

[0005] In one general aspect, a method for predicting a video topic peak is provided, including: obtaining topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph contains topic information of a plurality of videos; aligning the topic information of the target topic with an input of a large language model to obtain aligned topic information of the target topic; embedding the aligned topic information into a preset prompt for the large language model to obtain input information of the large language model; inputting the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information contains the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; and inputting the enhanced topic information into a linear regression layer to predict a peak value of the target topic.

[0006] Optionally, the large language model is trained by: processing the initial large language model by using a parameter efficient fine-tuning method to obtain a fine-tuned large language model, wherein a number of adjustable parameters of the fine-tuned large language model is less than a number of adjustable parameters of the initial large language model; and training the fine-tuned large language model and a linear regression layer based on topic information of multiple topics obtained from the heterogeneous graph to obtain a trained large language model.

[0007] Optionally, the fine-tuned large language model and the linear regression layer are trained based on topic information of multiple topics obtained from the heterogeneous graph to obtain a trained large language model, including: aligning the topic information of the multiple topics with inputs of the large language model respectively to obtain respective aligned topic information of the multiple topics; and for each topic of the multiple topics, performing the following processing: embedding the aligned topic information of the current topic into a preset prompt to obtain estimated input information of the fine-tuned large language model; inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; inputting the enhanced estimated topic information into the initial linear regression layer to predict a peak value of the current topic; and adjusting parameters of the fine-tuned large language model and the initial linear regression layer based on a loss of the predicted peak value and a real peak value of each topic to obtain the trained large language model and the linear regression layer.

[0008] Optionally, before obtaining the topic information of the target topic from the heterogeneous graph, the method further includes: obtaining visual text information, user information, and topic information of multiple videos, wherein the visual text information includes visual information and text information; and constructing a heterogeneous graph with videos, users, and topics as nodes based on the visual text information, the user information, and the topic information of the multiple videos; and wherein obtaining the topic information of the target topic from the heterogeneous graph includes: obtaining the topic information of the target topic, visual text information related to the target topic, and user information related to the target topic from the heterogeneous graph, and taking the obtained topic information of the target topic, the visual text information related to the target topic, and the user information related to the target topic as final topic information of the target topic.

[0009] Optionally, the visual text information, the user information, and the topic information of each video of the multiple videos are obtained by: performing equal-interval frame sampling on a current video to obtain multiple frames of images; inputting the multiple frames of images into a video feature extraction model to extract visual information of each frame of image; performing average aggregation processing on the visual information of each frame of image to obtain visual information of the current video; inputting information of a text modality of the current video into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; initializing publisher information of the current video to obtain user information of the current video; and inputting a topic of the text modality of the current video into the text feature extraction model to obtain topic information of the current video.

[0010] Optionally, the topic information of the target topic is aligned with an input of a large language model to obtain aligned topic information of the target topic.

[0011] Optionally, the topic information of the target topic is obtained from the heterogeneous graph, including: inputting the heterogeneous graph into a pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, wherein the heterogeneous graph neural network is trained by: inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain topic information of each topic; for each topic, inputting the topic information of the topic into an initial linear prediction layer to predict a peak value of the topic; based on a loss of the predicted peak value and a real peak value of each topic, adjusting parameters of the initial heterogeneous graph neural network and the initial linear prediction layer to obtain the trained heterogeneous graph neural network and the linear prediction layer. In another general aspect, there is provided a device for predicting a peak value of a video topic, including: an obtaining unit configured to obtain topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph contains topic information of a plurality of videos; an aligning unit configured to align the topic information of the target topic with an input of a large language model to obtain aligned topic information of the target topic; an embedding unit configured to embed the aligned topic information into a preset prompt for the large language model to obtain input information of the large language model; an enhancing unit configured to input the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information contains the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; and a predicting unit configured to input the enhanced topic information into a linear regression layer to predict a peak value of the target topic.

[0012] Optionally, the device further includes a training unit configured to train the large language model by: processing an initial large language model using a parameter-efficient fine-tuning method to obtain a fine-tuned large language model, wherein a number of adjustable parameters of the fine-tuned large language model is less than a number of adjustable parameters of the initial large language model; and training the fine-tuned large language model and the linear regression layer based on the topic information of the plurality of topics obtained from the heterogeneous graph to obtain a trained large language model.

[0013] Optionally, the training unit is further configured to align the topic information of the plurality of topics with inputs of the large language model respectively to obtain respective aligned topic information of the plurality of topics; for each topic in the plurality of topics, perform the following processing: embedding the aligned topic information of the current topic into the preset prompt to obtain estimated input information of the fine-tuned large language model; inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; inputting the enhanced estimated topic information into the initial linear regression layer to predict a peak value of the current topic; adjusting parameters of the fine-tuned large language model and the initial linear regression layer based on a loss of the predicted peak value and a real peak value of each topic to obtain a trained large language model and linear regression layer.

[0014] Optionally, the obtaining unit is further configured to obtain visual text information, user information and topic information of a plurality of videos, wherein the visual text information comprises visual information and text information; construct a heterogeneous graph with videos, users and topics as nodes based on the visual text information, the user information and the topic information of the plurality of videos; obtain topic information of a target topic, visual text information and user information related to the target topic from the heterogeneous graph, and take the obtained topic information of the target topic, the visual text information and the user information related to the target topic as final topic information of the target topic.

[0015] Optionally, the visual text information, the user information and the topic information of each video in the plurality of videos are obtained by the following manner respectively: performing equal-interval frame sampling on a current video to obtain a plurality of images; inputting the plurality of images into a video feature extraction model to extract visual information of each image; performing average aggregation processing on the visual information of each image to obtain visual information of the current video; inputting information of a text modality of the current video into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; initializing publisher information of the current video to obtain user information of the current video; inputting a topic of the text modality of the current video into the text feature extraction model to obtain topic information of the current video.

[0016] Optionally, the alignment unit is further configured to input the topic information of the target topic into a graph adapter to obtain aligned topic information of the target topic.

[0017] Optionally, the obtaining unit is further configured to input the heterogeneous graph into the pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, and the training unit is further configured to train the heterogeneous graph neural network by: inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain the topic information of each topic; inputting the topic information of each topic into an initial linear prediction layer to predict the peak value of the topic; and adjusting parameters of the initial heterogeneous graph neural network and the initial linear prediction layer based on a loss of the predicted peak value and the real peak value of each topic to obtain the trained heterogeneous graph neural network and the linear prediction layer. In another general aspect, a computer-readable storage medium storing instructions is provided, wherein the instructions, when executed by at least one computing device, cause the at least one computing device to perform any of the above-described methods for predicting a video topic peak value.

[0018] In another general aspect, a system including at least one computing device and at least one storage device storing instructions is provided, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform any of the above-described methods for predicting a video topic peak value.

[0019] In another general aspect, a computer program product including computer instructions that, when executed by a processor, implement any of the above-described methods for predicting a video topic peak value.

[0020] The method and device for predicting a video topic peak value according to the embodiments of the present disclosure embed topic information of a target topic obtained from a heterogeneous graph into a preset prompt of a large language model, so that the large language model can be guided to enhance the topic information of the target topic based on the embedded topic information and semantic instructions in the preset prompt using prior knowledge of the large language model, and then predict a topic peak value based on the enhanced topic information. Thus, the present disclosure combines a heterogeneous graph with a large language model, uses the large language model to obtain external knowledge to assist in predicting a topic peak value, and improves the accuracy of predicting a topic peak value. Therefore, the present disclosure can effectively solve the problem of inaccurate prediction of a topic peak value in the prior art.

[0021] Additional aspects and / or advantages of the general inventive concept will be set forth in part in the description that follows, and in part will be obvious from the description, or can be learned by practice of the general inventive concept. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects and features of embodiments of the present disclosure will become more apparent from the following description made with reference to the accompanying drawings, in which: Figure 1 is a schematic diagram illustrating a short video topic peak prediction (SVTPP) according to an embodiment of the present disclosure; Figure 2 is a flowchart illustrating a prediction method of a frequency topic peak value of an embodiment of the present disclosure; Figure 3 is a flowchart illustrating a short video topic peak prediction method of an embodiment of the present disclosure; Figure 4 is a schematic diagram illustrating a system architecture employed by a short video topic peak prediction method of an embodiment of the present disclosure; Figure 5 is a block diagram illustrating a prediction device of a video topic peak value of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following detailed description is provided to help the reader obtain a thorough understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be clear to those skilled in the art after understanding the disclosure provided herein. For example, the order of the operations described herein is merely an example, and is not limited to those set forth herein, but can be changed as will be clear to those skilled in the art after understanding the disclosure provided herein, except for operations that must occur in a specific order. Also, the description of features known in the art can be omitted for the sake of clarity and conciseness.

[0024] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided so as to merely demonstrate some of the many possible ways of implementing the methods, devices, and / or systems described herein, which will be clear to those skilled in the art after understanding the disclosure provided herein.

[0025] As used herein, the term "and / or" includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0026] Although terms such as "first", "second", and "third" can be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another component, assembly, region, layer, or part. Thus, a first component, a first assembly, a first region, a first layer, or a first part referred to in the examples described herein can also be called a second component, a second assembly, a second region, a second layer, or a second part without departing from the teachings of the examples.

[0027] In the description, when an element (such as a layer, a region, or a substrate) is referred to as being "on" another element, "connected to" or "coupled to" another element, it can be directly on, directly connected to, or directly coupled to the other element, or one or more other elements can be interposed therebetween. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element, there are no other elements interposed therebetween.

[0028] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including" and "has," "having" as used herein, specify the presence of stated features, numbers, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, components, elements, and / or combinations thereof.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs when read in light of the present disclosure. Unless otherwise explicitly defined herein, the terms (such as terms commonly used in general dictionaries) should be interpreted as having meanings consistent with their meanings in the context of the relevant art and in the disclosure, and should not be interpreted ideally or too formally.

[0030] In addition, in the description of examples, when it is considered that a detailed description of a well-known related structure or function will cause a blurred explanation of the disclosure, such a detailed description will be omitted.

[0031] For video topic heat peak prediction, especially short video topic heat peak prediction (abbreviated as SVTPP), as shown in Figure 1 A heterogeneous graph at the topic level is generally constructed, and the cumulative heat peak of each topic in a specific time period is predicted, such as the topic peak can be defined as the sum of the heat peaks of all associated videos under the topic, but the prediction of the topic peak in the related art is not ideal.

[0032] To solve the above problems, the present disclosure proposes a large language model enhancement solution, that is, a video topic peak prediction method based on a heterogeneous graph large language model. Specifically, first, topic information of a target topic is extracted from the constructed heterogeneous graph, such as the topic information of the target topic extracted from the constructed heterogeneous graph by a heterogeneous graph neural network (HGNN). Second, align the topic information of the target topic with the input of the large language model, such as using a graph adapter (Graph Adapter) to adapt and map the topic information of the target topic learned by the HGNN to the space of the large language model. Third, embed the aligned topic information into the preset prompt for the large language model to obtain the input information of the large language model. Then, input the input information into the large language model to guide the large language model to generate optimized topic information containing rich semantic and structural information, which is used to support the regression prediction of the final topic peak. The present disclosure fully integrates heterogeneous graphs and multi-modal semantic information, effectively solves the shortcomings of existing methods in mining deep semantic information and modeling complex topic dynamics, and significantly improves the accuracy of long-term heat peak prediction at the video topic level. In addition, to improve the model training efficiency and reduce the parameter update cost, the large language model of the present disclosure uses LoRA technology for fine-tuning.

[0033] It should be noted that the above-mentioned video can be a short video or a long video, and the present disclosure does not limit it. The above-mentioned topic can be any topic, such as "daily record", "walking baby holy place", etc., and the present disclosure does not limit it. The above-mentioned topic peak can be topic play quantity, topic forwarding quantity, topic comment quantity, etc., and the present disclosure does not limit it.

[0034] The prediction method and device for video topic peak of the present disclosure will be described in detail below in combination with the drawings.

[0035] The present disclosure proposes a prediction method for video topic peak, Figure 2 is a flowchart showing the prediction method for video topic peak of the embodiment of the present disclosure. Referring to Figure 2 , the prediction method for video topic peak includes the following steps: In step S201, topic information of a target topic is obtained from a heterogeneous graph, wherein the heterogeneous graph contains topic information of multiple videos.

[0036] As an example, the above-mentioned heterogeneous graph can be constructed based on user information and topic information, or can be constructed based on visual text information and topic information, and the present disclosure does not limit it.

[0037] According to an embodiment of the present disclosure, before obtaining topic information of a target topic from a heterogeneous graph, visual text information, user information and topic information of a plurality of videos can be obtained, wherein the visual text information comprises visual information and text information; based on the visual text information, the user information and the topic information of the plurality of videos, a heterogeneous graph taking videos, users and topics as nodes is constructed; wherein obtaining the topic information of the target topic from the heterogeneous graph can comprise: obtaining the topic information of the target topic, visual text information related to the target topic and user information related to the target topic from the heterogeneous graph, and taking the obtained topic information of the target topic, the visual text information related to the target topic and the user information related to the target topic as the final topic information of the target topic. Through the present embodiment, the present disclosure constructs a heterogeneous graph taking videos, users and topics as nodes based on the visual text information, the user information and the topic information of the plurality of videos, thereby unifying cross-platform propagation dynamics and multi-modal data into a time-series heterogeneous graph, and solving the problem of insufficient modeling of long-term propagation patterns at the topic level in traditional methods.

[0038] As an example, after obtaining the visual text information, the user information and the topic information of the plurality of videos, a heterogeneous graph containing three types of nodes, i.e. video nodes, user nodes and topic nodes, can be constructed based on the visual text information, the user information and the topic information of the plurality of videos, so as to fuse multi-modal data, wherein the video nodes integrate visual information and text information, the user nodes integrate social attributes (i.e. integrate user information), and the topic nodes are associated with cross-platform propagation paths (i.e. integrate topic information), so that cross-platform propagation dynamics and multi-modal content can be unified into a time-series heterogeneous graph, and the problem of insufficient modeling of long-term propagation patterns at the topic level in traditional methods is solved.

[0039] It should be noted that the present disclosure is not limited to visual text information, user information and topic information, and other information can also be obtained to construct a heterogeneous graph, which is not limited by the present disclosure.

[0040] According to an embodiment of the present disclosure, the visual text information, the user information and the topic information of each video in the plurality of videos can be obtained in the following manner: equally spaced frame sampling is performed on a current video to obtain a plurality of images; the plurality of images are input into a video feature extraction model to extract visual information of each image; average aggregation processing is performed on the visual information of each image to obtain visual information of the current video; information of a text modality of the current video is input into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; the publisher information of the current video is initialized to obtain user information of the current video; and a topic of the text modality of the current video is input into the text feature extraction model to obtain topic information of the current video. Through the present embodiment, the visual text information, the user information and the topic information of each video can be obtained conveniently and quickly.

[0041] Specifically, the video feature extraction model and the text feature extraction model can be pre-trained good models. For example, the video feature extraction model can be a pre-trained Vision Transformer (ViT), and the text feature extraction model can be a pre-trained Bert-base-chinese1 (BERT).

[0042] As an example, for the information of the video modality of each video, visual features can be extracted from the frame sequence of the video. For example, for the information of the video modality of the video The equally spaced frame sampling strategy can be used to extract frames from , so as to obtain the sampled frame set of the video , where , and is the number of sampled frames. It should be noted that the sampling interval can be a fixed interval and 8 frames are uniformly sampled for each video, but the present disclosure is not limited thereto. After sampling, a video feature extraction model (Vision Transformer, abbreviated as ViT) can be applied to process each sampled frame to extract its frame-level feature vector, i.e., the visual information of each frame image. After obtaining the visual information of each sampled frame , the original feature of the video used to construct the heterogeneous graph can be obtained by the following formula:

[0043] wherein represents the frame-level feature vector extracted from the i-th frame by applying the ViT model.

[0044] As an example, for the information of the text modality of each video, such as the information of the fields of the content, title and description of the video (except the information of the topic field). For example, for the information of the text modality of the video A text feature extraction model (Bert-base-chinese1, abbreviated as BERT) can be applied to extract the original feature of the video used to construct the heterogeneous graph from these fields:

[0045] Herein, represents the feature vector extracted from the embedding splicing result of the fields of the content, title and description, i.e., the text information of the video.​​​

[0046] As an example, for the features that cannot be directly extracted from the text content (e.g., the publisher of the video), it can be randomly initialized. For example, the publisher information of the video can be initialized to obtain the user information of the video, and the present disclosure is not limited thereto.

[0047] As an example, for the information of the topic field in the information of the text modality of each video, a BERT model can be applied to extract the topic information of the video from the information of the topic field (i.e., the topic of the video), and the present disclosure is not limited thereto.

[0048] According to an embodiment of the present disclosure, obtaining the topic information of the target topic from the heterogeneous graph can include: inputting the heterogeneous graph into a pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, wherein the heterogeneous graph neural network can be trained by: inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain the topic information of each topic; for each topic, inputting the topic information of the topic into an initial linear prediction layer to predict the peak value of the topic; and based on the loss of the predicted peak value and the real peak value of each topic, adjusting the parameters of the initial heterogeneous graph neural network and the initial linear prediction layer to obtain the trained heterogeneous graph neural network and the linear prediction layer. Through the present embodiment, in the training process of the heterogeneous graph neural network (HGNN), the heterogeneous graph that fuses the multi-modal data of multiple videos is utilized, so that the trained HGNN can capture the relationship between various video components and the associated topics, thereby utilizing the trained HGNN to extract the topic information of the required topic, which can improve the accuracy and richness of the extracted topic information.

[0049] As an example, the above-mentioned heterogeneous graph neural network (HGNN) can process the heterogeneous relationship in layers, such as the HetSANN model with 2 layers, wherein the first layer learns the local node interaction, and the second layer captures the global topic evolution, so as to learn a deeper representation of each node in the heterogeneous graph, and further extract the topic information, and the present disclosure is not limited thereto.

[0050] It should be noted that the HGNN can also assign weights to different edge types through an attention mechanism to strengthen the perception ability of the key propagation path, and the present disclosure is not limited thereto.

[0051] As an example, assuming that the heterogeneous graph and the original features used to construct the heterogeneous graph (including the above-mentioned extracted features), wherein, represents a node, represents an edge, represents the features of the node, represents the features of the edge, and the HGNN model can be represented as:

[0052] wherein, represents the topic information obtained by the HGNN model from the heterogeneous graph.

[0053] The above obtained is input into the linear prediction layer to predict the peak value of the topic

[0054] Then, the mean square error (MSE) loss function can be obtained by using the predicted peak value of each topic, as follows:

[0055] wherein, represents the predicted peak value of the topic represents the true peak value of the topic and respectively represent the weight matrix and the bias term that can be learned by the prediction layer, represents the total number of topics in the heterogeneous graph.

[0056] Further, by minimizing the above loss function, the parameters of the heterogeneous graph neural network and the linear prediction layer can be adjusted to obtain a trained heterogeneous graph neural network and linear prediction layer.

[0057] As an example, after the heterogeneous graph neural network is trained, the heterogeneous graph is input into the trained heterogeneous graph neural network, and the topic information of each topic in the heterogeneous graph can be obtained, and then the topic information of the target topic is selected, and the obtained topic information can be used in the subsequent stage.

[0058] Returning to Figure 2 In step S202, the topic information of the target topic is aligned with the input of the large language model to obtain the aligned topic information of the target topic.

[0059] According to an embodiment of the present disclosure, the topic information of the target topic can be aligned with the input of the large language model to obtain the aligned topic information of the target topic in the following manner: the topic information of the target topic is input into the graph adapter to obtain the aligned topic information of the target topic. Through this embodiment, the graph adapter is used to bridge the graph knowledge and the large language model, realize the bridging of heterogeneous features to semantic space, and project the output of the graph neural network as input to the large language model, realize the cooperation of semantic understanding and structural reasoning.

[0060] As an example, the topic information of the target topic can be obtained by inputting the output of the HGNN model into the linear prediction layer. ​​​​Further, a graph adapter is inputted to align the output of the HGNN with the input requirement of a large language model (LLM). Specifically, the graph adapter can be defined as a linear mapping as follows:

[0061] wherein, represents the aligned topic information, represents a weight matrix, represents a bias term.

[0062] In step S203, the aligned topic information is embedded into a preset prompt for the large language model to obtain input information of the large language model.

[0063] Specifically, a prompt can be designed in advance, and the preset prompt can include a special graph token and a task description. The structure of the preset prompt is designed to include a video and its context information, wherein the position of the graph token is replaced by the aligned topic information , that is, the aligned topic information is embedded in the preset prompt. In this way, the input embedding of the large language model (i.e., the above input information) can be constructed by splicing the task description and the graph token as follows:

[0064] In step S204, the input information is inputted into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information includes topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model.

[0065] As an example, after obtaining the above input information , it can be inputted into the LLM, and the LLM can obtain the enhanced topic information by combining its own prior knowledge, that is, the enhanced topic information output by the LLM can be obtained as follows:

[0066] According to an embodiment of the present disclosure, the large language model can be trained by processing the initial large language model by using a parameter efficient fine-tuning method to obtain a fine-tuned large language model, wherein the number of adjustable parameters of the fine-tuned large language model is less than the number of adjustable parameters of the initial large language model; and training the fine-tuned large language model and the linear regression layer based on topic information of multiple topics obtained from the heterogeneous graph to obtain a trained large language model. Through this embodiment, the parameter efficient fine-tuning method (such as LoRA) is used to freeze most of the parameters of the large language model, and only the low-rank matrix and the linear regression layer are fine-tuned, which reduces the risk of overfitting while adapting to the peak regression task.

[0067] As an example, before training the LLM, a parameter efficient fine-tuning (PEFT) method, in particular LoRA, can be used to process the large language model to alleviate the overfitting problem commonly encountered by large language models. Specifically, the LoRA technique introduces a low-rank matrix in the large language model, i.e., freezes most of the parameters of the large language model, and only allows efficient adaptation with fewer trainable parameters, thereby preserving previous pre-training knowledge and improving generalization performance.

[0068] It should be noted that the training process of the large language model is similar to that of the HGNN model, and mean square error (MSE) loss can also be used to adjust the parameters of the large language model and the linear regression layer, which is not limited by the present disclosure. Moreover, the present disclosure combines the graph adapter with the LoRA-based PEFT to achieve efficient fine-tuning while reducing the risk of overfitting while maintaining performance.

[0069] According to an embodiment of the present disclosure, the training of the fine-tuned large language model and the linear regression layer based on the topic information of the multiple topics obtained from the heterogeneous graph can include: aligning the topic information of the multiple topics with the input of the large language model respectively to obtain the aligned topic information of each of the multiple topics; for each topic in the multiple topics, performing the following processing: embedding the aligned topic information of the current topic into a preset prompt to obtain estimated input information of the fine-tuned large language model; inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; inputting the enhanced estimated topic information into the initial linear regression layer to predict the peak value of the current topic; and adjusting the parameters of the fine-tuned large language model and the initial linear regression layer based on the loss of the predicted peak value and the true peak value of each topic to obtain the trained large language model and the linear regression layer. Through this embodiment, the large language model can be conveniently and accurately trained.

[0070] As an example, after obtaining topic information of multiple topics obtained from the heterogeneous graph, the topic information of each topic can be aligned with the input of the large language model, and then for each topic, the aligned topic information of the topic is embedded into the preset prompt to obtain corresponding input information. After inputting the input information into the LLM, the enhanced topic information of the topic can be obtained, so that the enhanced topic information can be input into the linear regression layer to predict the peak value of the topic.

[0071] As an example, in order to predict the peak value of each topic, the enhanced topic information of each topic can be input into the linear regression layer.

[0072] Then, the mean square error (MSE) loss function is obtained by using the predicted peak value of each topic, as follows:

[0073] Wherein, represents the predicted peak value of the topic, represents the true peak value of the topic, is the weight matrix of the regression layer, is the bias term of the regression layer, represents the total number of topics in the heterogeneous graph. Then, the parameters of the LLM and the linear regression layer can be adjusted by minimizing the above loss function to obtain the trained LLM and linear regression layer.

[0074] Finally, the customized prediction instructions (Prediction Instructions) are input into the large language model after instruction tuning, i.e. the above-mentioned preset prompt, and the aligned topic information is used as the key input information to obtain the enhanced topic information.

[0075] It should be noted that the linear prediction layer and the linear regression layer can also be collectively referred to as a linear layer, and the present disclosure does not limit this.

[0076] Returning to

[0077] In step S205, the enhanced topic information is input into the linear regression layer to predict the peak value of the target topic. Figure 2 As an example, the enhanced topic information of the target topic is input into the linear regression layer.

[0078] ​​​​​input linear regression layer, to obtain a corresponding topic predicted peak value of the topic Specifically, the following can be used:

[0079] wherein, is a weight matrix of the linear regression layer, represents a bias term of the linear regression layer, represents a predicted peak value of the target topic.

[0080] In order to better understand the present disclosure, the following takes a short video as an example, and a systematic description is made in combination with Figure 3 and Figure 4 .

[0081] Figure 3 A flowchart of a short video topic peak prediction method is shown in FIG. 1, and the prediction method includes the following steps: Figure 3 Step S301: Extracting multi-modal initial features of the short video. Specifically, equally spaced frame sampling is performed on each short video, and a ViT model is used to extract visual features of the single-frame image, and the feature average value of all the sampled frames is calculated to obtain the aggregated visual features (i.e., the visual information of the video described above); the text features of each short video are extracted using a BERT model, and the non-text features (such as user information, etc.) of each short video are randomly initialized. Step S302: Constructing a heterogeneous graph of short video propagation (containing video, user, topic nodes), and extracting topic information of each topic from the heterogeneous graph through a heterogeneous graph neural network (HGNN). Specifically, the HGNN adopts a 2-layer HetSANN model to process the heterogeneous relationship, learn the deep representation of the nodes, and extract the topic information of the topic, and then superimposes a linear prediction layer to predict the peak value of the topic, and uses the MSE loss function to optimize the prediction task, and then the HGNN can be used to extract the topic information of the required topic.

[0082] Step S303: Aligning the features of the heterogeneous graph-large language model. Specifically, a graph adapter is designed to bridge the representation space of the HGNN and the LLM. That is, the topic information of the target topic output by the HGNN is linearly mapped to the information suitable for the large language model, so that the dimension is compatible with the input of the LLM.

[0083] Step S304: Constructing instruction-guided prompt words and embedding the aligned topic information into the prompt words, and integrating the semantic instructions and the adapted graph features (i.e., the aligned topic information). Specifically, the aligned topic information is embedded into the prompt graph as a graph structure identifier, and spliced to form the input embedding of the large language model.

[0084]

[0085] ​Step S305: Efficient fine-tuning of the parameters of the large language model in combination with the PEFT technology, and prediction of the peak value of the topic by the fine-tuned large language model. Specifically, the lightweight fine-tuning can introduce a low-rank matrix by using the LoRA technology, update only a small number of parameters of the LLM to reduce the risk of overfitting, and then realize peak regression to generate the final prediction result by applying the LLM output to a linear regression layer.

[0086] Step S306: Peak output of the multi-platform topic based on the fine-tuned large language model. Specifically, for the topic of each platform, the input can be cross-platform heterogeneous data (such as including video, text, user interaction, etc.), and the output is the peak prediction value of each topic (such as the number of plays, the number of comments, the number of forwards, etc.).

[0087] Figure 4 A system architecture used by a short video topic peak prediction method is shown as follows: Figure 4 First, a heterogeneous graph containing video, user, and topic nodes is constructed to fuse multi-modal features, wherein the video node integrates visual and text features, the user node integrates social attributes, and the topic node is associated with cross-platform transmission paths. Then, the heterogeneous graph neural network is used to process the heterogeneous relationship in layers: the first layer learns the local node interaction, and the second layer captures the global topic evolution, and then the topic information of topic t1 is extracted by the heterogeneous graph neural network, i.e., the topic graph representation in Figure 4

[0088] Secondly, the graph adapter (Graph Adaptor) linearly maps the topic graph representation output by the heterogeneous graph neural network into a low-dimensional vector, retains the key features of the graph structure (such as node centrality), and aligns with the input space of the large language model (LLM). Then, a task-specific prompt word (Prompt) is constructed, and the adapted topic graph representation is embedded into the prompt as a special token (GraphToken), such as the graph structure identifier: [Graph_Adaptor] in Figure 4 Figure 4 guides the LLM to generate enhanced topic information combining semantic instructions and graph structure information, i.e., the topic LLM representation in. It should be noted that during the LMM training process, the LoRA technology can be used to freeze most of the parameters of the LLM, and only the low-rank matrix and the regression layer are fine-tuned, which reduces the risk of overfitting while adapting the peak regression task.

[0089] Thirdly, based on the topic LLM representation, subsequent downstream tasks such as prediction of the peak value of the topic are performed.

[0090] ​​In summary, the above-mentioned embodiment provides an end-to-end short video topic peak prediction method based on a propagation heterogeneous graph large language model. The method constructs an end-to-end two-stage learning framework, including a heterogeneous graph representation learning stage and a large model semantic enhancement stage, that is, through heterogeneous graph structure modeling and semantic instruction fine-tuning, cross-platform prediction of short video topic peaks is realized. In the embodiment, for unaligned cross-platform propagation data, the hierarchical learning mechanism of the heterogeneous graph neural network (HGNN) can be used to capture the dynamic relationship between the video, user and topic nodes, form a structured node representation, and design a graph adapter (Graph Adaptor) to map the heterogeneous graph representation to the language model space. Through the instruction prompt (Prompt), the graph features are embedded as special markers into the natural language task description, and combined with LoRA parameter efficient fine-tuning, the modal specificity is eliminated and cross-platform peak regression prediction is realized, which significantly reduces the prediction error (MAE average reduction of 8.84%); moreover, the present disclosure opens up a new way for integrating graph learning and large language model understanding of complex short video topic dynamics.

[0091] In order to verify the feasibility of the present disclosure, the present disclosure also uses the disclosed TopicVid short video trend topic dataset, which covers 7701 hot topics, 58539 users and 96936 short videos, and collects multi-modal information such as video content and text. The dataset records the daily views, comments and forwarding quantities and other heat indicators within two weeks after each video is published, and defines the peak of each topic as the sum of the peaks of all short videos under it.

[0092] Table 1 is a comparison of the method of the present disclosure and other methods on the TopicVid dataset, using three evaluation indicators: mean absolute error (MAE), normalized mean square error (nMSE) and mean absolute percentage error (MAPE). As can be seen from Table 1, the performance of the present disclosure on the dataset is significantly better than that of other methods. Specifically, the method of the present disclosure consistently outperforms the traditional graph neural network (GNN) model, and compared with the best GNN model (HetSANN), the method of the present disclosure improves the MAE by an average of 8.84%, the nMSE by 16.54%, and the MAPE by 0.41%, which shows the significant improvement of MAE and nMSE indicators, proving the effectiveness of large language model instruction fine-tuning. The improvement of RGCN over GCN, HAN and GAT shows that using heterogeneous information in the propagation network can improve the performance of the SVTPP task, that is, when only using topic text information, the large language model (LLMs) can achieve the basic topic prediction performance; but by integrating graph structure information and combining with LLMs, the method of the present disclosure obtains better results, and there are significant differences between different topic peak types.

[0093] Table 1 Evaluation results of short video topic popularity peak prediction

[0094] Figure 5 is a block diagram illustrating a device for predicting a video topic peak according to an embodiment of the present disclosure. Figure 5 As shown, the apparatus includes an acquisition unit 50 , an alignment unit 52 , an embedding unit 54 , an enhancement unit 56 and a prediction unit 58 .

[0095] The acquisition unit 50 is configured to acquire topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph contains topic information of multiple videos; the alignment unit 52 is configured to align the topic information of the target topic with the input of the large language model to obtain the aligned topic information of the target topic; the embedding unit 54 is configured to embed the aligned topic information into a preset prompt for the large language model to obtain input information of the large language model; the enhancement unit 56 is configured to input the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information includes the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; the prediction unit 58 is configured to input the enhanced topic information into a linear regression layer to predict the peak value of the target topic.

[0096] According to an embodiment of the present disclosure, the above-mentioned device also includes a training unit, which is configured to train the large language model in the following manner: using an efficient parameter fine-tuning method to process the initial large language model to obtain a fine-tuned large language model, wherein the number of adjustable parameters of the fine-tuned large language model is less than the number of adjustable parameters of the initial large language model; based on obtaining topic information of multiple topics from the heterogeneous graph, the fine-tuned large language model and the linear regression layer are trained to obtain a trained large language model.

[0097] Optionally, the training unit is further configured to align the topic information of multiple topics with the input of the large language model respectively to obtain the aligned topic information of the multiple topics; for each of the multiple topics, perform the following processing: embed the aligned topic information of the current topic into a preset prompt to obtain the estimated input information of the fine-tuned large language model; input the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; input the enhanced estimated topic information into the initial linear regression layer to predict the peak of the current topic; based on the loss of the predicted peak and the true peak of each topic, adjust the parameters of the fine-tuned large language model and the initial linear regression layer to obtain the trained large language model and linear regression layer.

[0098] Optionally, the obtaining unit 50 is further configured to obtain visual text information, user information and topic information of a plurality of videos, wherein the visual text information comprises visual information and text information; construct a heterogeneous graph taking videos, users and topics as nodes based on the visual text information, the user information and the topic information of the plurality of videos; obtain topic information of a target topic, visual text information related to the target topic and user information related to the target topic from the heterogeneous graph, and take the obtained topic information of the target topic, the visual text information related to the target topic and the user information related to the target topic as final topic information of the target topic.

[0099] Optionally, the visual text information, the user information and the topic information of each video in the plurality of videos are obtained in the following manner: equally spaced frame sampling is performed on a current video to obtain a plurality of images; the plurality of images are input into a video feature extraction model to extract visual information of each image; average aggregation processing is performed on the visual information of each image to obtain visual information of the current video; information of a text modality of the current video is input into a text feature extraction model to obtain text information of the current video, wherein the information of the text modality is information of a topic of the text modality of the current video; the publisher information of the current video is initialized to obtain user information of the current video; a topic of the text modality of the current video is input into the text feature extraction model to obtain topic information of the current video.

[0100] Optionally, the aligning unit 52 is further configured to input the topic information of the target topic into a graph adapter to obtain aligned topic information of the target topic.

[0101] Optionally, the obtaining unit 50 is further configured to input the heterogeneous graph into a pre-trained heterogeneous graph neural network to obtain the topic information of the target topic, wherein the training unit is further configured to train the heterogeneous graph neural network in the following manner: input the heterogeneous graph into an initial heterogeneous graph neural network to obtain topic information of each topic; for each topic, input the topic information of the topic into an initial linear prediction layer to predict a peak value of the topic; based on a loss of the predicted peak value and a real peak value of each topic, adjust parameters of the initial heterogeneous graph neural network and the initial linear prediction layer to obtain a trained heterogeneous graph neural network and a linear prediction layer.

[0102] According to an embodiment of the present disclosure, a computer readable storage medium storing instructions is provided, wherein when the instructions are run by at least one computing device, the at least one computing device is caused to perform the method for predicting a video topic peak value according to any one of the above embodiments.

[0103] According to an embodiment of the present disclosure, there is provided a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the method of predicting video topic peaks as in any of the above embodiments.

[0104] According to an embodiment of the present disclosure, there is provided a computer program product comprising computer instructions which, when executed by a processor, implement the method of predicting video topic peaks as in any of the above embodiments.

[0105] While certain embodiments of the present disclosure have been shown and described, it is understood that modifications will occur to those skilled in the art, without departing from the spirit and scope of the present disclosure as defined by the following claims and their equivalents.

Claims

1. A method for predicting a video topic peak, characterized in that: include: Acquiring topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph includes topic information of multiple videos; Aligning the topic information of the target topic with the input of the large language model to obtain aligned topic information of the target topic; Embedding the aligned topic information into a preset prompt for a large language model to obtain input information of the large language model; Inputting the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information includes the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; The enhanced topic information is input into a linear regression layer to predict the peak value of the target topic.

2. The prediction method according to claim 1, wherein: The large language model is trained in the following way: Processing the initial large language model using an efficient parameter fine-tuning method to obtain a fine-tuned large language model, wherein the number of adjustable parameters of the fine-tuned large language model is less than the number of adjustable parameters of the initial large language model; Based on topic information of multiple topics obtained from the heterogeneous graph, the fine-tuned large language model and the linear regression layer are trained to obtain a trained large language model.

3. The prediction method according to claim 2, wherein: The method of acquiring topic information of multiple topics from the heterogeneous graph and training the fine-tuned large language model and the linear regression layer to obtain a trained large language model includes: Aligning the topic information of the multiple topics with the input of the large language model respectively to obtain aligned topic information of the multiple topics; For each of the plurality of topics, perform the following processing: Embedding the aligned topic information of the current topic into the preset prompt to obtain the estimated input information of the fine-tuned large language model; Inputting the estimated input information into the fine-tuned large language model to obtain enhanced estimated topic information of the current topic; Inputting the enhanced estimated topic information into an initial linear regression layer to predict the peak value of the current topic; Based on the loss of the predicted peak value and the true peak value of each topic, the parameters of the fine-tuned large language model and the initial linear regression layer are adjusted to obtain the trained large language model and linear regression layer.

4. The prediction method according to claim 1, wherein: Before obtaining the topic information of the target topic from the heterogeneous graph, it also includes: Acquire visual text information, user information, and topic information of the plurality of videos, wherein the visual text information includes visual information and text information; Based on the visual text information, user information and topic information of the plurality of videos, a heterogeneous graph is constructed with videos, users and topics as nodes; The step of obtaining the topic information of the target topic from the heterogeneous graph includes: The topic information of the target topic, visual text information and user information related to the target topic are obtained from the heterogeneous graph, and the obtained topic information of the target topic, visual text information and user information related to the target topic are used as the final topic information of the target topic.

5. The prediction method according to claim 4, wherein: The visual text information, user information and topic information of each video in the plurality of videos are obtained respectively in the following manners: Perform equal-interval frame sampling on the current video to obtain multiple frame images; Inputting the multiple frames of images into a video feature extraction model to extract visual information of each frame of image; Perform average aggregation processing on the visual information of each frame image to obtain the visual information of the current video; Inputting the text modality information of the current video into a text feature extraction model to obtain text information of the current video, wherein the text modality information is information from which the topic of the text modality of the current video is removed; Initialize the publisher information of the current video to obtain the user information of the current video; The topic of the text modality of the current video is input into the text feature extraction model to obtain the topic information of the current video.

6. The prediction method according to claim 1, wherein: The step of aligning the topic information of the target topic with the input of the large language model to obtain the aligned topic information of the target topic includes: The topic information of the target topic is input into a graph adapter to obtain the aligned topic information of the target topic.

7. The prediction method according to claim 1, wherein: The acquiring of topic information of the target topic from the heterogeneous graph includes: The heterogeneous graph is input into a pre-trained heterogeneous graph neural network to obtain topic information of the target topic, wherein the heterogeneous graph neural network is trained in the following manner: Inputting the heterogeneous graph into an initial heterogeneous graph neural network to obtain topic information of each topic; For each topic, input the topic information of the topic into the initial linear prediction layer to predict the peak value of the topic; Based on the loss of the predicted peak value and the true peak value of each topic, the parameters of the initial heterogeneous graph neural network and the initial linear prediction layer are adjusted to obtain the trained heterogeneous graph neural network and linear prediction layer.

8. A device for predicting video topic peaks, characterized in that: include: an acquiring unit configured to acquire topic information of a target topic from a heterogeneous graph, wherein the heterogeneous graph includes topic information of a plurality of videos; an alignment unit configured to align the topic information of the target topic with the input of the large language model to obtain the aligned topic information of the target topic; an embedding unit configured to embed the aligned topic information into a preset prompt for a large language model to obtain input information of the large language model; an enhancement unit configured to input the input information into the large language model to obtain enhanced topic information of the target topic, wherein the enhanced topic information includes the topic information of the target topic and prior knowledge related to the topic information of the target topic in the large language model; The prediction unit is configured to input the enhanced topic information into a linear regression layer to predict the peak value of the target topic.

9. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the method for predicting a video topic peak according to any one of claims 1 to 7.

10. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to perform the method for predicting a video topic peak according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method for predicting a video topic peak value according to any one of claims 1 to 7 is implemented.

Images