Video frame prediction method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a video frame prediction method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: carrying out image feature extraction on existing video frames in a video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is the combination of the image features of the corresponding existing video frame under a plurality of different scales; for each group of image feature combinations in the image feature combination sequence, obtaining prompt word semantic features of prompt words associated with the image feature combinations, and fusing the prompt word semantic features and the image feature combinations to obtain multi-modal fusion features corresponding to the image feature combinations; the prompt word is used for describing video content of the video frame sequence; and predicting the next video frame of the existing video frame according to each group of multi-modal fusion features. By adopting the method, the prediction effect of the video frame can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video frame prediction method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In recent years, with the development of artificial intelligence technology, time series prediction has been increasingly used in discrete data, such as the prediction of trends in stock prices, temperature, sales, and key points of robot movements.

[0003] However, there are still many challenges in the field of computer vision, especially in video frame prediction in video frame sequences. For example, it is difficult for traditional technologies to model the spatiotemporal relationship between images in a video frame sequence, making it difficult for traditional technologies to achieve high-precision video frame prediction.

[0004] Therefore, there is a problem of low video frame prediction accuracy in traditional technologies. Summary of the invention

[0005] Based on this, it is necessary to provide a video frame prediction method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the accuracy of video frame prediction in order to address the above technical problems.

[0006] In a first aspect, the present application provides a video frame prediction method, the method comprising:

[0007] Extracting image features from existing video frames in a video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is a combination of image features of the corresponding existing video frames at multiple different scales;

[0008] For each group of the image feature combinations in the image feature combination sequence, obtaining a prompt word semantic feature of a prompt word associated with the image feature combination, fusing the prompt word semantic feature with the image feature combination, and obtaining a multimodal fusion feature corresponding to the image feature combination; the prompt word is used to describe the video content of the video frame sequence;

[0009] The next video frame of the existing video frame is predicted according to each group of the multimodal fusion features.

[0010] In one embodiment, predicting the next video frame of the existing video frame according to each group of the multimodal fusion features includes:

[0011] Inputting each group of the multimodal fusion features into a pre-trained large language model to obtain video temporal features of the video frame sequence;

[0012] The next video frame of the existing video frame is predicted according to the video timing feature.

[0013] In one embodiment, inputting each group of the multimodal fusion features into a pre-trained large language model to obtain the video timing features of the video frame sequence includes:

[0014] Inputting each group of multimodal fusion features into a pre-trained large language model to obtain a model output feature vector;

[0015] At least part of the feature vectors associated with the prompt word in the feature vector output by the model is eliminated to obtain a video timing feature vector; the video timing feature vector is used to characterize the video timing features of the video frame sequence.

[0016] In one embodiment, predicting the next video frame of the existing video frame according to the video timing feature includes:

[0017] Inputting the video temporal features into a pre-trained video frame generation model to obtain an image frame prediction result; wherein the pre-trained video frame generation model is a generator in a pre-trained generative adversarial network;

[0018] The predicted video frame in the image frame prediction result is obtained as the next video frame of the existing video frame.

[0019] In one embodiment, the fusing the prompt word semantic feature and the image feature combination to obtain a multimodal fusion feature corresponding to the image feature combination includes:

[0020] Re-encoding the image feature combination to obtain re-encoded image features; wherein the data format of the re-encoded image features matches the data format of the semantic features of the prompt word;

[0021] The semantic features of the prompt word and the re-encoded image features are spliced ​​to obtain a multimodal fusion feature corresponding to the image feature combination.

[0022] In one embodiment, the acquiring of the prompt word semantic features of the prompt word associated with the image feature combination includes:

[0023] Obtaining a prompt word text of a prompt word associated with the image feature combination;

[0024] Inputting the prompt word text into a pre-trained large language model, and extracting semantic features of the prompt word text through the pre-trained large language model;

[0025] The semantic feature of the prompt word text is obtained as the prompt word semantic feature.

[0026] In a second aspect, the present application further provides a video frame prediction device, the device comprising:

[0027] An extraction module is used to extract image features from existing video frames in a video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is a combination of image features of the corresponding existing video frames at multiple different scales;

[0028] a fusion module, configured to obtain, for each group of the image feature combinations in the image feature combination sequence, a prompt word semantic feature of a prompt word associated with the image feature combination, fuse the prompt word semantic feature with the image feature combination, and obtain a multimodal fusion feature corresponding to the image feature combination; the prompt word is used to describe the video content of the video frame sequence;

[0029] A prediction module is used to predict the next video frame of the existing video frame according to each group of multimodal fusion features.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0032] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0033] The above-mentioned video frame prediction method, device, computer equipment, computer-readable storage medium and computer program product obtain an image feature combination sequence by extracting image features of existing video frames in a video frame sequence, wherein the image feature combinations in the image feature combination sequence are combinations of image features of corresponding existing video frames at multiple different scales; for each group of image feature combinations in the image feature combination sequence, the prompt word semantic features of the prompt words associated with the image feature combination are obtained, and the prompt word semantic features and the image feature combination are fused to obtain the multimodal fusion features corresponding to the image feature combination; the prompt words are used to describe the video content of the video frame sequence; based on each group of multimodal fusion features, the next video frame of the existing video frame is predicted; in this way, by extracting image features of existing video frames in the video frame sequence at multiple different scales, it is possible to implement multi-scale feature extraction of the video frame sequence at the image level, better capture subtle changes between image frames, and thus improve the accuracy of video frame prediction; at the same time, by obtaining The prompt word semantic features of the prompt word associated with the image feature combination are taken, the prompt word semantic features and the image feature combination are fused, and the multimodal fusion features corresponding to the image feature combination are obtained for predicting the next video frame of the existing video frame. This can effectively integrate multimodal data, and enhance the comprehensive processing capability of the video frame prediction model for different information sources through collaborative processing and fusion of multiple types of data. In addition, the above technical solution effectively establishes a time series prediction framework that can model across time and space, so that the time series data in the video sequence can be processed and understood more efficiently, and the video frame prediction model can capture data over a long time span and a large spatial range. The model can achieve higher robustness in different application scenarios, so that it can cope with complex environments and data changes and accurately predict image frame changes in video sequences, and can be applied to fields such as video surveillance, autonomous driving, smart homes, and medical image analysis. It has important practical application value, theoretical research value, and broad market prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A diagram showing an application environment of a video frame prediction method in an embodiment;

[0036] Figure 2 is a schematic flow chart of a video frame prediction method in an embodiment;

[0037] Figure 3 A model framework diagram of a video frame prediction method in one embodiment;

[0038] Figure 4 is a flow chart of a video frame prediction method in another embodiment;

[0039] Figure 5 is a structural block diagram of a video frame prediction device in one embodiment;

[0040] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The video frame prediction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. In a specific implementation, the terminal 102 can obtain a video resource from the server 104, and the video resource includes a video frame sequence; then, the terminal 102 can extract image features from the existing video frames in the video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is a combination of image features of the corresponding existing video frames at multiple different scales; the terminal 102 can obtain the prompt word semantic features of the prompt word associated with the image feature combination for each group of image feature combinations in the image feature combination sequence, fuse the prompt word semantic features and the image feature combination, and obtain the multimodal fusion features corresponding to the image feature combination; the prompt word is used to describe the video content of the video frame sequence; the terminal 102 can predict the next video frame of the existing video frame according to each group of multimodal fusion features. Among them, the terminal 102 can be, but not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0043] In an exemplary embodiment, Figure 2 As shown, a video frame prediction method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, including the following steps S202 to S206. Among them:

[0044] Step S202: extract image features from existing video frames in the video frame sequence to obtain an image feature combination sequence.

[0045] The image feature combination in the image feature combination sequence is a combination of image features of the corresponding existing video frames at multiple different scales. The image feature combination can also be named as an image feature patch.

[0046] In a specific implementation, in the time domain, the terminal can input the image of each existing video frame in the video frame sequence into a pre-trained image feature extraction model, and extract the image features of each existing video frame at multiple different scales through the image feature extraction model in an overlapping manner to obtain the image feature combination corresponding to each existing video frame; then, the terminal can obtain an image feature combination sequence based on the image feature combination corresponding to each existing video frame.

[0047] In practical applications, the above-mentioned pre-trained image feature extraction model can be a shifted window transformer model for computer vision tasks. When the image feature extraction model processes an image of any existing video frame, the image feature extraction model can divide the image of the existing video frame into small windows in a hierarchical manner, and apply a self-attention mechanism within these windows, while shifting the windows at different levels, thereby effectively processing high-resolution existing video frames, reducing computational complexity, and improving the performance of the video frame prediction model.

[0048] In practical applications, the terminal can re-encode the image feature combination to obtain an image feature combination whose encoding format matches the natural language processing field. In this article, the feature vector of the image feature combination can be represented by S.

[0049] Step S204, for each group of image feature combinations in the image feature combination sequence, obtaining the prompt word semantic features of the prompt words associated with the image feature combination, fusing the prompt word semantic features and the image feature combination to obtain a multimodal fusion feature corresponding to the image feature combination.

[0050] The prompt word is used to describe the video content of the video frame sequence. In practical applications, the prompt word can be a natural language used to describe the video content, such as "a child is playing football on the lawn".

[0051] In a specific implementation, in the field of natural language processing, a pre-trained large-scale language model can be pre-selected and frozen. For each image feature combination in the image feature combination sequence, the terminal can obtain the prompt word associated with the image feature combination, and input the prompt word associated with the image feature combination into the pre-trained large-scale language model to obtain the semantic features of the prompt word. In this article, the feature vector of the semantic features of the prompt word can be represented by N. The feature vector N contains the semantic information of the prompt word.

[0052] Then, the terminal can fuse the semantic features of the prompt word and the image feature combination to obtain the multimodal fusion features corresponding to the image feature combination. Specifically, the terminal can concatenate the feature vector S with the feature vector N to obtain a concatenated feature vector P; the feature vector P is used to represent the multimodal fusion features corresponding to the image feature combination.

[0053] Step S206: predicting the next video frame of the existing video frame according to each group of multimodal fusion features.

[0054] In a specific implementation, the terminal can input the sequence composed of each group of multimodal fusion features into a pre-trained large language model, and use the pre-trained large language model to mine the feature change characteristics of each group of multimodal fusion features, and output the video timing features for predicting the next video frame. The terminal can input the video timing features into the pre-trained generative artificial intelligence, and use the generative artificial intelligence to predict the next video frame of the existing video frame.

[0055] Continuing with the previous example, the terminal inputs the concatenated feature vector P into the frozen pre-trained large-scale language model. Then, the terminal can intercept the feature vector output by the pre-trained large-scale language model to completely separate it from the prompt word and obtain a feature vector F related only to the timing features. The feature vector F is used to represent the video timing features used to predict the next video frame.

[0056] Then, the terminal selects a suitable generative adversarial network, takes the feature vector F as input, and passes it to the generative adversarial network. The generator and discriminator of the generative adversarial network compete with each other to obtain an image frame prediction result that meets the preset quality requirements. The predicted video frame in the image frame prediction result is used to output the next video frame of the existing video frame.

[0057] In the above-mentioned video frame prediction method, by extracting image features of existing video frames in a video frame sequence, an image feature combination sequence is obtained, and the image feature combinations in the image feature combination sequence are combinations of image features of corresponding existing video frames at multiple different scales; for each group of image feature combinations in the image feature combination sequence, the prompt word semantic features of the prompt words associated with the image feature combination are obtained, and the prompt word semantic features and the image feature combination are fused to obtain the multimodal fusion features corresponding to the image feature combination; the prompt words are used to describe the video content of the video frame sequence; according to each group of multimodal fusion features, the next video frame of the existing video frame is predicted; in this way, by extracting image features of existing video frames in a video frame sequence at multiple different scales, multi-scale feature extraction of the video frame sequence can be achieved at the image level, so as to better capture subtle changes between image frames, thereby improving the accuracy of video frame prediction; at the same time, by obtaining the prompt words associated with the image feature combination The semantic features of the prompt words are combined with the image feature combination to obtain the multimodal fusion features corresponding to the image feature combination, which are used to predict the next video frame of the existing video frame. This can effectively integrate multimodal data, and enhance the comprehensive processing ability of the video frame prediction model for different information sources through the collaborative processing and fusion of various types of data. In addition, the above-mentioned technical solution effectively establishes a time series prediction framework that can model across time and space, so that the time series data in the video sequence can be processed and understood more efficiently, and the video frame prediction model can capture data over a long time span and a large spatial range. The model can achieve higher robustness in different application scenarios, so that it can cope with complex environments and data changes and accurately predict image frame changes in video sequences, and can be applied to fields such as video surveillance, autonomous driving, smart homes, and medical image analysis. It has important practical application value, theoretical research value and broad market prospects.

[0058] In an exemplary embodiment, predicting the next video frame of an existing video frame based on each group of multimodal fusion features includes: inputting each group of multimodal fusion features into a pre-trained large language model to obtain video timing features of a video frame sequence; and predicting the next video frame of an existing video frame based on the video timing features.

[0059] In a specific implementation, when the terminal predicts the next video frame of an existing video frame based on each group of multimodal fusion features, the terminal can input each group of multimodal fusion features into a pre-trained large language model to obtain the video timing features of the video frame sequence. Specifically, the terminal can input the sequence composed of each group of multimodal fusion features into a pre-trained large language model, and use the pre-trained large language model to mine the feature change characteristics of each group of multimodal fusion features, and output the video timing features for predicting the next video frame.

[0060] Then, the terminal can predict the next video frame of the existing video frame based on the video timing feature. Specifically, the terminal can input the video timing feature into the pre-trained generative artificial intelligence, and predict the next video frame of the existing video frame through the generative artificial intelligence.

[0061] The technical solution of this embodiment obtains the video timing features of the video frame sequence by inputting each group of multimodal fusion features into a pre-trained large language model, and predicts the next video frame of the existing video frame based on the video timing features. This enables the video frame prediction model to more efficiently process and understand the timing data in the video sequence, thereby improving the video frame prediction model's ability to capture data over a long time span and a large spatial range.

[0062] In an exemplary embodiment, each group of multimodal fusion features is input into a pre-trained large language model to obtain video timing features of a video frame sequence, including: inputting each group of multimodal fusion features into a pre-trained large language model to obtain a model output feature vector; eliminating at least part of the feature vectors associated with the prompt word in the model output feature vector to obtain a video timing feature vector; the video timing feature vector is used to characterize the video timing features of the video frame sequence.

[0063] In a specific implementation, the terminal inputs each group of multimodal fusion features into a pre-trained large language model. In the process of obtaining the video timing features of the video frame sequence, the terminal can input the sequence composed of each group of multimodal fusion features into the pre-trained large language model, and mine the feature change characteristics between each group of multimodal fusion features through the pre-trained large language model to obtain the model output feature vector; then, the terminal can eliminate at least part of the feature vectors associated with the prompt word in the model output feature vector to obtain the video timing feature vector; the video timing feature vector is used to characterize the video timing features of the video frame sequence.

[0064] The technical solution of this embodiment, by inputting each group of multimodal fusion features into a pre-trained large language model to obtain a model output feature vector, and by eliminating at least part of the feature vectors associated with the prompt word in the model output feature vector, it is possible to achieve complete separation from the prompt word through a interception operation, and obtain a timing feature vector only related to the timing feature as the video timing feature of the video frame sequence.

[0065] In an exemplary embodiment, predicting the next video frame of an existing video frame based on video timing features includes: inputting the video timing features into a pre-trained video frame generation model to obtain an image frame prediction result; obtaining the predicted video frame in the image frame prediction result as the next video frame of the existing video frame.

[0066] Among them, the pre-trained video frame generation model is a generator in a pre-trained generative adversarial network.

[0067] Among them, the Generative Adversarial Network (GAN) is a deep learning model composed of two neural networks: the generator and the discriminator. In the process of training the GAN, the generator can be responsible for generating new predicted video frames, while the discriminator evaluates the similarity between the generated predicted video frames and the real video frames. Based on the similarity, the model parameters of the generator and the discriminator are continuously adjusted, and finally the generator can generate predicted video frames that are closer and closer to the real video frames.

[0068] In a specific implementation, when the terminal predicts the next video frame of an existing video frame based on the video timing features, the terminal can input the video timing features into a pre-trained video frame generation model. The pre-trained video frame generation model is a generator in a generative adversarial network. The generator is trained adversarially with the discriminator and can effectively convert the video timing features into an image frame prediction result with a quality equivalent to that of the training data, so that the terminal can use the predicted video frame in the image frame prediction result as the next video frame of the existing video frame.

[0069] In an exemplary embodiment, the semantic features of the prompt word and the image feature combination are fused to obtain the multimodal fusion features corresponding to the image feature combination, including: re-encoding the image feature combination to obtain the re-encoded image features; splicing the semantic features of the prompt word and the re-encoded image features to obtain the multimodal fusion features corresponding to the image feature combination.

[0070] Recoding may refer to converting data into another form or format.

[0071] The data format of the re-encoded image features matches the data format of the semantic features of the prompt words.

[0072] In a specific implementation, when the terminal fuses the semantic features of the prompt word and the image feature combination to obtain the multimodal fusion features corresponding to the image feature combination, the terminal can re-encode the image feature combination, that is, convert the data format of the image feature combination from the initial data format to the target data format, and the target data format matches the data format of the semantic features of the prompt word. In other words, the target data format can refer to the corresponding encoding format in the field of natural language processing (NLP).

[0073] After the terminal re-encodes the image feature combination into the above-mentioned target data format, the terminal obtains the re-encoded image features; the terminal can then fuse the re-encoded image features with the prompt word semantic features; specifically, the terminal can splice the prompt word semantic features and the re-encoded image features to obtain the spliced ​​features; then, the terminal obtains the spliced ​​features as the multimodal fusion features corresponding to the image feature combination.

[0074] The technical solution of this embodiment re-encodes the image feature combination to obtain the re-encoded image features, and splices the prompt word semantic features and the re-encoded image features to obtain the multimodal fusion features corresponding to the image feature combination, thereby achieving a good fusion of the features of the two modalities, namely, image features and semantic features, realizing collaborative processing and fusion of various types of data, and enhancing the model's comprehensive processing capabilities for different information sources.

[0075] In an exemplary embodiment, obtaining a prompt word semantic feature of a prompt word associated with an image feature combination includes: obtaining a prompt word text of the prompt word associated with the image feature combination; inputting the prompt word text into a pre-trained large language model, and extracting the semantic feature of the prompt word text through the pre-trained large language model; and obtaining the semantic feature of the prompt word text as the prompt word semantic feature.

[0076] In a specific implementation, in the process of acquiring the prompt word semantic features of the prompt word associated with the image feature combination, the terminal may acquire the prompt word text (Input Text) of the prompt word associated with the image feature combination.

[0077] The prompt word text may refer to a text that describes the video content in natural language.

[0078] Then, the terminal can input the prompt word text into a pre-trained large language model; the pre-trained large language model is pre-configured with a semantic feature extraction function, and the large language model extracts the semantic features of the prompt word text through the semantic extraction function and outputs a semantic feature vector (Output Token Embeddings); the terminal can use the semantic feature vector output by the large language model as the prompt word semantic feature of the prompt word associated with the image feature combination.

[0079] The technical solution of this embodiment obtains the prompt word text of the prompt word associated with the image feature combination, and inputs the prompt word text into a pre-trained large language model, so that based on the semantic feature extraction function of the large language model, the context information in the natural language in the prompt word text can be considered to achieve a better understanding of the semantics in the prompt word text, and effectively extract the prompt word semantic features that can characterize the characteristics of the prompt word.

[0080] In order to facilitate the understanding of those skilled in the art, Figure 3 As shown, a model framework diagram of a video frame prediction method is provided; wherein the terminal can obtain a video image, which includes an existing video frame in a video frame sequence; the terminal can input the existing video frame in the video frame sequence into a shift window transformer model to obtain an image feature combination; then, the terminal re-encodes the image feature combination to obtain a re-encoded image feature; wherein the data format of the re-encoded image feature matches the data format of the semantic feature of the prompt word. The terminal can also input the prompt word text of the prompt word associated with the image feature combination into a frozen pre-trained large language model to obtain the semantic feature of the prompt word;

[0081] Then, the terminal merges the semantic features of the prompt word and the re-encoded image features to obtain a multimodal fusion feature corresponding to the image feature combination. The terminal inputs the multimodal fusion feature into a pre-trained large language model to obtain a model output feature vector. The terminal can remove at least part of the feature vectors associated with the prompt word in the model output feature vector to obtain a video timing feature vector; the video timing feature vector is used to characterize the video timing features of the video frame sequence. The terminal can input the video timing features into a pre-trained video frame generation model, and use the predicted video frame in the image frame prediction result as the next video frame of the existing video frame; wherein the pre-trained video frame generation model is a generator in a pre-trained generative adversarial network.

[0082] In another embodiment, Figure 4 As shown, a video frame prediction method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0083] Step S402, extracting image features from existing video frames in the video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is a combination of image features of corresponding existing video frames at multiple different scales.

[0084] Step S404: for each image feature combination in the image feature combination sequence, obtain the prompt word text of the prompt word associated with the image feature combination; the prompt word is used to describe the video content of the video frame sequence.

[0085] Step S406: input the prompt word text into a pre-trained large language model, and extract the semantic features of the prompt word through the pre-trained large language model.

[0086] Step S408, re-encoding the image feature combination to obtain re-encoded image features; wherein the data format of the re-encoded image features matches the data format of the semantic features of the prompt words.

[0087] Step S410, concatenating the semantic features of the prompt word and the re-encoded image features to obtain a multimodal fusion feature corresponding to the image feature combination.

[0088] Step S412: input each group of multimodal fusion features into the pre-trained large language model to obtain a model output feature vector.

[0089] Step S414, removing at least part of the feature vectors associated with the prompt word in the feature vector output by the model, to obtain a video timing feature vector; the video timing feature vector is used to characterize the video timing features of the video frame sequence.

[0090] Step S416, inputting the video timing features into a pre-trained video frame generation model, and using the predicted video frame in the image frame prediction result as the next video frame of the existing video frame; wherein the pre-trained video frame generation model is a generator in a pre-trained generative adversarial network.

[0091] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a video frame prediction method mentioned above, which will not be repeated here.

[0092] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0093] Based on the same inventive concept, the embodiment of the present application also provides a video frame prediction device for implementing the video frame prediction method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more video frame prediction device embodiments provided below can refer to the limitations of the video frame prediction method above, and will not be repeated here.

[0094] In an exemplary embodiment, Figure 5 As shown, a video frame prediction device is provided, comprising:

[0095] An extraction module 510 is used to extract image features from existing video frames in a video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is a combination of image features of the corresponding existing video frames at multiple different scales;

[0096] A fusion module 520 is used to obtain, for each group of the image feature combinations in the image feature combination sequence, a prompt word semantic feature of a prompt word associated with the image feature combination, and fuse the prompt word semantic feature with the image feature combination to obtain a multimodal fusion feature corresponding to the image feature combination; the prompt word is used to describe the video content of the video frame sequence;

[0097] The prediction module 530 is used to predict the next video frame of the existing video frame according to each group of the multimodal fusion features.

[0098] In one embodiment, the prediction module 530 is specifically used to input each group of the multimodal fusion features into a pre-trained large language model to obtain the video timing features of the video frame sequence; and predict the next video frame of the existing video frame based on the video timing features.

[0099] In one embodiment, the prediction module 530 is specifically used to input each group of the multimodal fusion features into a pre-trained large language model to obtain a model output feature vector; eliminate at least part of the feature vectors associated with the prompt word in the model output feature vector to obtain a video timing feature vector; the video timing feature vector is used to characterize the video timing features of the video frame sequence.

[0100] In one embodiment, the prediction module 530 is specifically used to input the video timing features into a pre-trained video frame generation model to obtain an image frame prediction result; wherein the pre-trained video frame generation model is a generator in a pre-trained generative adversarial network; and the predicted video frame in the image frame prediction result is obtained as the next video frame of the existing video frame.

[0101] In one embodiment, the fusion module 520 is specifically used to re-encode the image feature combination to obtain re-encoded image features; wherein the data format of the re-encoded image features matches the data format of the prompt word semantic features; the prompt word semantic features and the re-encoded image features are spliced ​​to obtain a multimodal fusion feature corresponding to the image feature combination.

[0102] In one of the embodiments, the fusion module 520 is specifically used to obtain the prompt word text of the prompt word associated with the image feature combination; input the prompt word text into a pre-trained large language model, and extract the semantic features of the prompt word text through the pre-trained large language model; and obtain the semantic features of the prompt word text as the prompt word semantic features.

[0103] Each module in the above video frame prediction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0104] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a video frame prediction method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0105] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned video frame prediction method. The steps of the video frame prediction method here may be the steps of the video frame prediction method in each of the above-mentioned embodiments.

[0107] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned video frame prediction method. The steps of the video frame prediction method here can be the steps of the video frame prediction method in each of the above-mentioned embodiments.

[0108] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-mentioned video frame prediction method. The steps of the video frame prediction method here may be the steps of the video frame prediction method in each of the above-mentioned embodiments.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0110] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magneto resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0112] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video frame prediction method, characterized in that: The method comprises: Extracting image features from existing video frames in a video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is a combination of image features of the corresponding existing video frames at multiple different scales; For each group of the image feature combinations in the image feature combination sequence, obtaining a prompt word semantic feature of a prompt word associated with the image feature combination, fusing the prompt word semantic feature with the image feature combination, and obtaining a multimodal fusion feature corresponding to the image feature combination; the prompt word is used to describe the video content of the video frame sequence; The next video frame of the existing video frame is predicted according to each group of the multimodal fusion features.

2. The method according to claim 1, characterized in that The predicting the next video frame of the existing video frame according to each group of the multimodal fusion features includes: Inputting each group of the multimodal fusion features into a pre-trained large language model to obtain video temporal features of the video frame sequence; The next video frame of the existing video frame is predicted according to the video timing feature.

3. The method according to claim 2, characterized in that The step of inputting each group of the multimodal fusion features into a pre-trained large language model to obtain the video temporal features of the video frame sequence includes: Inputting each group of multimodal fusion features into a pre-trained large language model to obtain a model output feature vector; At least part of the feature vectors associated with the prompt word in the feature vector output by the model is eliminated to obtain a video timing feature vector; the video timing feature vector is used to characterize the video timing features of the video frame sequence.

4. The method according to claim 2, characterized in that: The predicting the next video frame of the existing video frame according to the video timing feature includes: Inputting the video temporal features into a pre-trained video frame generation model to obtain an image frame prediction result; wherein the pre-trained video frame generation model is a generator in a pre-trained generative adversarial network; The predicted video frame in the image frame prediction result is obtained as the next video frame of the existing video frame.

5. The method according to claim 1, characterized in that The fusing the semantic feature of the prompt word with the image feature combination to obtain a multimodal fusion feature corresponding to the image feature combination includes: Re-encoding the image feature combination to obtain re-encoded image features; wherein the data format of the re-encoded image features matches the data format of the semantic features of the prompt word; The semantic features of the prompt word and the re-encoded image features are spliced ​​to obtain a multimodal fusion feature corresponding to the image feature combination.

6. The method according to claim 1, characterized in that The step of obtaining the prompt word semantic features of the prompt word associated with the image feature combination includes: Obtaining a prompt word text of a prompt word associated with the image feature combination; Inputting the prompt word text into a pre-trained large language model, and extracting semantic features of the prompt word text through the pre-trained large language model; The semantic feature of the prompt word text is obtained as the prompt word semantic feature.

7. A video frame prediction device, characterized in that: The device comprises: An extraction module is used to extract image features from existing video frames in a video frame sequence to obtain an image feature combination sequence; the image feature combination in the image feature combination sequence is a combination of image features of the corresponding existing video frames at multiple different scales; a fusion module, configured to obtain, for each group of the image feature combinations in the image feature combination sequence, a prompt word semantic feature of a prompt word associated with the image feature combination, fuse the prompt word semantic feature with the image feature combination, and obtain a multimodal fusion feature corresponding to the image feature combination; the prompt word is used to describe the video content of the video frame sequence; A prediction module is used to predict the next video frame of the existing video frame according to each group of multimodal fusion features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images