Real-time and efficient video prediction method and system based on key points

Abstract

This invention relates to the field of deep learning technology and discloses a real-time and efficient video prediction method and system based on keypoints. The method includes: extracting keypoint sets of each consecutive video frame through a keypoint extraction network; the keypoint extraction network includes a video frame encoder and a coordinate generator; transforming each keypoint set into a one-dimensional vector, adjusting it to a fixed length through a transformation matrix, and adding position embedding based on trigonometric functions; inputting the obtained position-embedded vectors into a Transformer encoder for parallel prediction, and outputting a keypoint set for future time moments; decoding the sequence of keypoint sets for future time moments into a sequence of video frames for future time moments through a video frame processing network with a feature generator and a video frame decoder; this invention improves prediction speed by sacrificing a very small amount of accuracy.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, specifically to a real-time and efficient video prediction method and system based on key points. Background Technology

[0002] Video data, as a time-varying stream of images, contains richer information and content than other media can express. It can intuitively, vividly, realistically, and efficiently represent the real world. Predicting video content can help people better understand the future and prevent unknown dangers. Due to the richer and more complex nature of video content, video prediction is more difficult than other time series prediction techniques, attracting increasing attention from scholars. Video prediction involves inferring subsequent video sequences from known video frame sequences. Current methods mostly use encoders to encode the features of video frames, then use recurrent neural networks to encode inter-frame temporal features, and finally generate predicted video frames. To pursue high accuracy in predicted frames, current methods extract a large number of multi-dimensional video frame features, which contain a lot of redundant background information, ultimately leading to a significant waste of time and memory in both training and testing.

[0003] The existing problems with video prediction technologies can be summarized as follows:

[0004] 1) Current methods overemphasize prediction accuracy, neglecting the equally important prediction time. In sudden dangerous situations, such as a speeding car, drivers only have 2.3 to 3 seconds to react. Excessive prediction time can lead to serious consequences. Current methods only achieve 80 to 100 fps, unable to predict video frames several seconds in advance. In fact, for consecutive video frame sequences, most of the background information is similar. If only the important information in the video frames can be extracted for prediction, the time spent on video prediction can be greatly reduced, as can the memory consumed during prediction.

[0005] 2) Current prediction methods generally use recurrent neural networks, such as RNNs. These networks often lose initial information during the transmission of information in the hidden layers, leading to a decrease in prediction accuracy. Therefore, how to utilize all the information in the time domain is also a challenge for video prediction.

[0006] Overall, the technology for high-speed video prediction while ensuring high accuracy still needs further exploration and improvement. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a real-time, efficient video prediction method and system based on key points.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a real-time and efficient video prediction method based on key points, comprising:

[0010] The keypoint extraction network extracts keypoint sets from each consecutive video frame. The keypoint extraction network includes a video frame encoder and a coordinate generator. The video frame encoder encodes the video frame into a feature matrix through a multi-layer structure, and the coordinate generator converts the feature matrix into a keypoint set by converting the number of channels to the number of keypoints. Each keypoint set includes multiple keypoints.

[0011] Each keypoint set is transformed into a one-dimensional vector, adjusted to a fixed length using a transformation matrix, and a position embedding based on trigonometric functions is added. The resulting position-embedded vectors are then input into a Transformer encoder for parallel prediction, outputting keypoint sets for future time moments.

[0012] A video frame processing network with a feature generator and a video frame decoder decodes a sequence of keypoints for future moments into a sequence of video frames for future moments. The feature generator restores each keypoint set for future moments into a feature heatmap, and the video frame decoder performs skip connections between the feature matrices output from each layer of the video frame encoder and the feature heatmap to reconstruct the video frames for future moments.

[0013] In one embodiment, the video frame encoder encodes video frames into a feature matrix through a multi-layer structure, specifically including:

[0014] Adjust the video frames to a uniform size and sort them in chronological order;

[0015] The video frame encoder processes video frames through an n-layer convolutional network. Encode, define For video frame encoder The Layered convolutional networks, yes Output feature matrix:

[0016] ;

[0017] .

[0018] In one embodiment, the coordinate generator converts the feature matrix into a keypoint set by converting the number of channels to the number of keypoints, specifically including: a coordinate generator. The feature matrix output from the last layer of the video frame encoder is processed through a fully connected layer. Convert to feature tensor :

[0019] ;

[0020] Let i be the i-th key point in the key point set. , This represents the total number of key points corresponding to each video frame. for Width and height coordinates for , characteristic intensity , , They are respectively Height and width, for The index value in the width direction, for The index value in the height direction, through It can locate a unique feature tensor.

[0021] In one embodiment, the i-th key point The width direction coordinates, height direction coordinates, and feature intensity are calculated as follows:

[0022] ; ;

[0023] ; ;

[0024] ;

[0025] express Weight values ​​for the width direction coordinate. express Weight values ​​for the height coordinate; All are of length The vector is composed of values ​​sampled uniformly from -1 to 1.

[0026] In one embodiment, the step of transforming each keypoint set into a one-dimensional vector, adjusting it to a fixed length using a transformation matrix, and adding a position embedding based on trigonometric functions specifically includes:

[0027] set up It is a single current video frame A keypoint set consisting of keypoints is then:

[0028] ;

[0029] For the first One key point, For the first The width and height coordinates and feature intensity of each key point; Transform into a one-dimensional vector :

[0030] ;

[0031] Will Adjust to a fixed length: ; It is a fixed-length vector. It is a transformation matrix;

[0032] right Add trigonometric function-based position embedding :

[0033] ;

[0034] This indicates tensor splicing. This represents the total number of frames in the current video. Let be the fixed-length vector corresponding to the video frame at time t; This is the vector after position embedding.

[0035] In one embodiment, the calculation method for the trigonometric function-based position embedding is as follows:

[0036] Concatenate the one-dimensional vectors of N current video frames in chronological order to form a keypoint sequence;

[0037]

[0038] The keypoint sequence represents the order of the video frames corresponding to each keypoint, and i represents a fixed-length vector. The index of the i-th element, Indicates the position index is Position embedding value; A fixed-length vector The length.

[0039] In one embodiment, the step of inputting the obtained position-embedded vector into a Transformer encoder for parallel prediction and outputting a set of keypoints for future time moments specifically includes:

[0040] Transformer encoder Output One predicted value:

[0041] ;

[0042] The vector after position embedding. Indicates the number to be predicted The predicted value corresponding to the video frame at a given time;

[0043] Using the transformation matrix Reconstruct the predicted value vector Output a vector consisting of the keypoints at future time points. :

[0044] .

[0045] In one embodiment, the feature generator reconstructs a feature heatmap from the set of key points at future time moments, specifically including:

[0046] A single keypoint set in the future is set as , ; for The i-th key point in for Width direction coordinates, height direction coordinates, and feature intensity;

[0047] Using a two-dimensional Gaussian distribution to reconstruct the feature heatmap, The horizontal vector composed of the width coordinates of all key points in the middle Convert to a horizontal one-dimensional Gaussian vector ,Will The vertical vector composed of the height and direction coordinates of all key points in the middle Convert to a vertical one-dimensional Gaussian vector :

[0048] ;

[0049] ;

[0050] and They are and Expectations Respectively , The variance; and Multiplication yields a two-dimensional Gaussian distribution. :

[0051] ;

[0052] Final calculation and Hadamaji obtained Corresponding feature heatmap :

[0053] ;

[0054] This indicates the calculation of the Hadamard product.

[0055] In one embodiment, the video frame decoder performs skip connections between the feature matrices and feature heatmaps output from each layer of the video frame encoder to reconstruct video frames at future moments, specifically including:

[0056] Both the video frame decoder and the video frame encoder have an n-layer structure; the n-layer structure of the video frame decoder is an n-layer deconvolutional network.

[0057] set up Represents the first video frame decoder Layered deconvolutional networks, express The process of reconstructing future video frames from the output includes:

[0058] ;

[0059] ;

[0060] This indicates tensor splicing. For the feature heatmap, The feature matrix is ​​the output of the nth layer of the video frame encoder, and the output of the last deconvolutional layer of the video frame decoder. That is, the reconstructed video frame .

[0061] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0062] Compared with the prior art, the beneficial technical effects of the present invention are:

[0063] This invention designs a video frame encoder to obtain high-dimensional feature representations of video frames; a coordinate generator converts the high-dimensional feature matrix encoded by the video frame encoder into keypoints; a feature generator restores the keypoints into feature heatmaps; and a video frame decoder converts the feature heatmaps back into video frames. The method proposed in this invention achieves a prediction speed improvement of nearly 3 times compared to the best existing methods, while sacrificing a minimal amount of accuracy. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart illustrating a real-time, efficient video prediction method based on key points, provided in an embodiment of the present invention;

[0066] Figure 2 A schematic diagram illustrating a key point extraction method provided in an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the overall architecture of the video prediction model provided in an embodiment of the present invention. Detailed Implementation

[0068] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0069] like Figure 1 As shown, this invention provides a real-time and efficient video prediction method based on key points, comprising the following steps:

[0070] S1, extract key point sets of each consecutive video frame through a key point extraction network; the key point extraction network includes a video frame encoder and a coordinate generator; wherein, the video frame encoder encodes the video frame into a feature matrix through a multi-layer structure, and the coordinate generator converts the feature matrix into a key point set by converting the number of channels into the number of key points, and each key point set includes multiple key points.

[0071] S2 transforms each keypoint set into a one-dimensional vector, adjusts it to a fixed length using a transformation matrix, and adds a position embedding based on trigonometric functions. The resulting position-embedded vector is then input into the Transformer encoder for parallel prediction, outputting the keypoint set for future time moments.

[0072] S3 uses a video frame processing network with a feature generator and a video frame decoder to decode the sequence of keypoints in future moments into a sequence of video frames in future moments; the feature generator restores each keypoint set in future moments into a feature heatmap; and the video frame decoder performs skip connections between the feature matrices output by each layer of the video frame encoder and the feature heatmap to reconstruct the video frames in future moments.

[0073] Figure 3 The overall architecture of the video prediction model of this invention is shown. Video frames are fed into a keypoint extraction network in temporal order to obtain multiple keypoint sets. These sets are then further integrated through deformation and input into a Transformer-based encoder to predict keypoint sets for future moments. Finally, the video frames for future moments are decoded through skip connections. For ease of understanding, a detailed technical description of each part of this invention is provided below.

[0074] You can use OpenCV software to acquire video frames at a certain frequency and then uniformly adjust these video frames to a size of 64×64 or 128×128.

[0075] In a series of video frames, key points are those pixels that are prone to dynamic changes, while the background remains unchanged. If key points can be extracted for prediction and then combined with the background, the computational load of video prediction will be greatly reduced. To explain the principle of key point extraction, the principle will be explained below.

[0076] set up It refers to two frames in a video frame sequence that are close in time. Called the source video frame, This is called the target video frame. Within each source video frame... The set of key points is represented as Each point is a keypoint in the form of a triple, which includes the width coordinate, height coordinate, and feature intensity of the keypoint.

[0077] Suppose there is a function It can extract the key point set from the source video frame. Predict the keypoint set of the target video frame using predictor T. There is a function It can be obtained through target video frames The set of K key points and source video frames Background feature reconstruction The formula is as follows:

[0078] ;

[0079] ;

[0080] .

[0081] like Figure 2 As shown, This represents the reconstructed video frames. Therefore, we only need to minimize the target video frames. and reconstructed frames The differences between them, key point set That can represent and The difference. This invention uses a method based on Reconstruction loss To measure and Pixel differences:

[0082] ;

[0083] It is the F2 norm.

[0084] pass Unsupervised learning can be used for training and Because this invention only requires The pixel values ​​do not need to be labeled separately.

[0085] like Figure 2 As shown, the function A video frame encoder with an n-layer convolutional network (CNN) can be used. A coordinate generator (CG) is implemented. (Video frame encoder) Used to encode deep features of video frames, the coordinate generator (CG) converts the generated feature matrix into coordinate triplets (coordinates and feature intensities at those coordinates). Function It can be done through a feature generator A video frame decoder with an n-layer deconvolutional network Achieve, overall yes The opposite process. Feature generator It converts coordinate triplets into feature heatmaps; video frame decoder The structure is similar to that of a video frame encoder. Similarly, it is used to reconstruct video frames from feature heatmaps.

[0086] The following provides a detailed description of the video frame encoder, coordinate generator, feature generator, video frame decoder, and predictor for predicting key points of the present invention.

[0087] In one embodiment, the video frame encoder in step S1 encodes the video frame into a feature matrix through a multi-layer structure, specifically including:

[0088] Adjust the video frames to a uniform size and sort them in chronological order;

[0089] The video frame encoder processes video frames through an n-layer convolutional network. Encode, define For video frame encoder The Layered convolutional networks, yes Output feature matrix:

[0090] ;

[0091] .

[0092] In a preferred embodiment, n=6, meaning the video frame encoder uses a six-layer convolutional network.

[0093] In one embodiment, the coordinate generator in step S1 converts the feature matrix into a keypoint set by converting the number of channels to the number of keypoints, specifically including: a coordinate generator. The feature matrix output from the last layer of the video frame encoder is processed through a fully connected layer. Convert to key point form : ; For the i-th key point, , This represents the total number of key points corresponding to each video frame. for Width and height coordinates for The characteristic intensity.

[0094] In one embodiment, the i-th key point The width direction coordinates, height direction coordinates, and feature intensity are calculated as follows:

[0095] ; ;

[0096] ; ;

[0097] ;

[0098] express Weight values ​​for the width direction coordinate. express Weight values ​​for the height coordinate; They are respectively The height and width.

[0099] This invention uses a fully connected layer to integrate the feature matrix. Switch to This is done in the hope of making each layer By compressing it into point coordinates, we can obtain... Each coordinate. The transformed feature matrix. It can be rewritten as ,in, , , , respectively represent The three dimensions allow for the location of a unique feature tensor value; the overall feature space is... . It is a length of The vector consists of values ​​sampled uniformly from -1 to 1 (e.g., if...). ,So, By doing so, in the characteristic matrix An additional axis has been added in the dimension. Reconstructing video frames also requires feature intensities at these coordinates, which this invention utilizes... It is represented by the average value across dimensions.

[0100] like Figure 3 As shown, this invention transforms the original video prediction problem from predicting video frames to predicting key points by extracting key points. It is assumed that key points have already been extracted. Before the moment A continuous set of key points Based on these key point sets, it is necessary to predict the future. Key point set ; Figure 3 In the corresponding embodiment, N=t. The predictor does not use a traditional recurrent neural network (RNN) because for long sequence prediction, networks like RNNs often lose previous information, leading to a decrease in accuracy. Therefore, the predictor in this invention uses a Transformer encoder structure, which incorporates a self-attention mechanism to fully utilize global information and improve accuracy. Furthermore, the parallel computing unique to Transformer can accelerate the prediction speed, which is also the key to real-time prediction in this invention.

[0101] In one embodiment, step S2, which involves transforming each keypoint set into a one-dimensional vector, adjusting it to a fixed length using a transformation matrix, and adding a position embedding based on trigonometric functions, specifically includes:

[0102] set up It is a single current video frame A keypoint set consisting of keypoints is then:

[0103] ;

[0104] For the first One key point, For the first The width and height coordinates and feature intensity of each key point; Transform into a one-dimensional vector :

[0105] ;

[0106] Will Adjust to a fixed length: ; It is a fixed-length vector. It is a transformation matrix; It is a vector of length 3K. Since the dimension of the Transformer encoder parameters is related to the length of the input vector, it is necessary to... from Adjust to fixed length In some embodiments, Usually taken .

[0107] right Add trigonometric function-based position embedding :

[0108] ;

[0109] This indicates tensor splicing. This represents the total number of frames in the current video. Let be the fixed-length vector corresponding to the video frame at time t; This is the vector after position embedding.

[0110] In one embodiment, the calculation method for the trigonometric function-based position embedding is as follows:

[0111] Concatenate the one-dimensional vectors of N current video frames in chronological order to form a keypoint sequence;

[0112]

[0113] This represents the order of the video frames corresponding to each keypoint in the keypoint sequence, where i represents the current fixed-length vector. The i-th element; A fixed-length vector The length.

[0114] Figure 3 In the corresponding embodiment, N = t. For example... Figure 3 As shown, only the calculation of... Extracted key point set vector With the predicted key point set vector Between The loss can be used to train the predictor in an unsupervised manner:

[0115] .

[0116] To predict losses, It is the F2 norm.

[0117] Feature generator It is a coordinate generator The inversion process is used to convert key points into feature heatmaps:

[0118] .

[0119] In one embodiment, the feature generator in step S3 reconstructs the keypoint sets at future time points into feature heatmaps, specifically including:

[0120] A single keypoint set in the future is set as , ; for The i-th key point in for Width direction coordinates, height direction coordinates, and feature intensity;

[0121] Using a two-dimensional Gaussian distribution to reconstruct the feature heatmap, The horizontal vector composed of the width coordinates of all key points in the middle Convert to a horizontal one-dimensional Gaussian vector ,Will The vertical vector composed of the height and direction coordinates of all key points in the middle Convert to a vertical one-dimensional Gaussian vector :

[0122] ;

[0123] ;

[0124] and They are and The expectation; will and Multiplication yields a two-dimensional Gaussian distribution. :

[0125] ;

[0126] Final calculation and Hadamaji obtained Corresponding feature heatmap :

[0127] .

[0128] This indicates the calculation of the Hadamard product.

[0129] In one embodiment, the video frame decoder performs skip connections between the feature matrices and feature heatmaps output from each layer of the video frame encoder to reconstruct video frames at future moments, specifically including:

[0130] Both the video frame decoder and the video frame encoder have an n-layer structure; the n-layer structure of the video frame decoder is an n-layer deconvolutional network.

[0131] set up Represents the first video frame decoder Layered deconvolutional networks, express The process of reconstructing future video frames from the output includes:

[0132] ;

[0133] ;

[0134] For the feature heatmap, The feature matrix is ​​the output of the nth layer of the video frame encoder, and the output of the last deconvolutional layer of the video frame decoder. That is, the reconstructed video frame .

[0135] Through this "skip connection" method, background information is stored in the output of each layer of the encoder, and the final output of the encoder... This allows for more dynamic key point information in the image, resulting in more accurate key point extraction.

[0136] For video frame decoders The predicted video frames are then used to generate the predicted video at a certain frequency using OpenCV software.

[0137] To demonstrate the effectiveness of this invention, the following experiments were conducted for verification.

[0138] Experiments were conducted on two video datasets (KTH and Human3.6). The KTH dataset was selected for demonstration. Structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) were used as accuracy metrics, while time to generate 20 frames, frames per second (FPS), memory usage, and total floating-point operations (FLOPs) were used as metrics for real-time performance and efficiency. Ablation experiments were also designed to verify the effectiveness of the designed module.

[0139] The experiment is divided into the following parts:

[0140] (1) The accuracy, real-time performance and efficiency of the method of the present invention (TKN) on the KTH dataset are compared with those of other methods (PredRNN model, PredRNNv2 model, PhyDNet model, SLAMP model, E3D-LSTM model, Grid-Keypoint model, Struct-VRNN model). The results are shown in Table 1.

[0141] Table 1: Comparison of accuracy, real-time performance, and efficiency of different methods on the KTH dataset.

[0142]

[0143] As can be seen, compared with the most accurate method to date, this invention improves the prediction speed by nearly 3 times with a very small amount of precision, reaching a generation rate of 233 frames per second, and achieving the ability to predict in real time. This is based on the fact that the computational cost (FLOPs) of this invention is much smaller than that of other models. At the same time, the method of this invention (TKN) consumes less memory resources during prediction, laying the foundation for industrial application.

[0144] (2) Compared with the key point extraction module and predictor that also use the key point method, the time and total number of floating point operations (FLOPs) for generating 20 frames are shown in Table 2.

[0145] Table 2: Comparison of keypoint-based methods

[0146]

[0147] Table 2 compares the Struct-VRNN model and the Grid-Keypoint model, which also use the keypoint method. Due to their similar structures, this invention compares their keypoint extraction modules and predictors. It can be seen that the method of this invention (TKN) is much lower in terms of time and computational cost than these two methods. In addition, combined with Table 1, the accuracy of this invention is also better than the existing methods, which demonstrates the superiority of the method of this invention.

[0148] (3) Comparison of the predictor with different recurrent networks.

[0149] Table 3: Comparison of Transformer encoder and other recurrent networks in the predictor

[0150]

[0151] The present invention conducted ablation experiments, as shown in Table 3. The Transformer encoder in the predictor was compared with other classic recurrent networks. Since the Transformer uses global information, it is superior to other networks in terms of accuracy. Moreover, due to the parallel computing capability of the Transformer encoder, it is also superior to other networks in terms of computing speed. Therefore, the Transformer encoder achieves accelerated computing while ensuring high-precision prediction.

[0152] The keypoint-based video frame generation method in this invention encodes video frames to obtain key points that represent key information within the frames, significantly reducing the amount of features required for prediction. This is the foundation for the real-time prediction capability of this invention. Furthermore, the video frame decoder can reconstruct the video frames while maintaining high accuracy.

[0153] This invention extracts keypoint features with extremely small feature quantities and feeds them into a predictor based on a Transformer encoder that can utilize global features and perform parallel computation, thus ensuring high prediction accuracy while achieving high-speed prediction.

[0154] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0155] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0156] In one embodiment, a computer system is provided, which may be a server. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in the methods described above. The network interface communicates with external terminals via a network connection. The computer program is executed by the processor to implement the methods described above.

[0157] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0159] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A real-time and efficient video prediction method based on key points, characterized in that, include: A keypoint extraction network extracts keypoint sets from consecutive video frames. This network includes a video frame encoder and a coordinate generator. The video frame encoder encodes the video frames into a feature matrix using a multi-layer structure. The coordinate generator converts the feature matrix into a keypoint set by converting the number of channels to the number of keypoints. Each keypoint set includes multiple keypoints. Specifically, the coordinate generator... The feature matrix output from the last layer of the video frame encoder is processed through a fully connected layer. Convert to feature tensor : ; Let i be the i-th key point in the key point set. , This represents the total number of key points corresponding to each video frame. for Width and height coordinates for , characteristic intensity , , They are respectively Height and width, for The index value in the width direction, for The index value in the height direction, through It can locate a unique feature tensor; i-th key point The width direction coordinates, height direction coordinates, and feature intensity are calculated as follows: ; ; ; ; ; express Weight values ​​for the width direction coordinate. express Weight values ​​for the height coordinate; All are of length The vector is composed of values ​​sampled uniformly from -1 to 1; Each keypoint set is transformed into a one-dimensional vector, adjusted to a fixed length using a transformation matrix, and a position embedding based on trigonometric functions is added. The resulting position-embedded vectors are then input into a Transformer encoder for parallel prediction, outputting keypoint sets for future time moments. A video frame processing network with a feature generator and a video frame decoder decodes a sequence of keypoints for future moments into a sequence of video frames for future moments. The feature generator restores each keypoint set for future moments into a feature heatmap, and the video frame decoder performs skip connections between the feature matrices output from each layer of the video frame encoder and the feature heatmap to reconstruct the video frames for future moments.

2. The real-time and efficient video prediction method based on key points according to claim 1, characterized in that, The video frame encoder encodes video frames into a feature matrix through a multi-layer structure, specifically including: Adjust the video frames to a uniform size and sort them in chronological order; The video frame encoder processes video frames through an n-layer convolutional network. Encode, define For video frame encoder The Layered convolutional networks, yes Output feature matrix: ； 。 3. The real-time and efficient video prediction method based on key points according to claim 1, characterized in that, The process of transforming each keypoint set into a one-dimensional vector, adjusting it to a fixed length using a transformation matrix, and adding a position embedding based on trigonometric functions specifically includes: set up It is a single current video frame A keypoint set consisting of keypoints is then: ； For the first One key point, For the first The width and height coordinates and feature intensity of each key point; Transform into a one-dimensional vector : ； Will Adjust to a fixed length: ; It is a fixed-length vector. It is a transformation matrix; right Add trigonometric function-based position embedding : ； This indicates tensor splicing. This represents the total number of frames in the current video. Let be the fixed-length vector corresponding to the video frame at time t; This is the vector after position embedding.

4. The real-time and efficient video prediction method based on key points according to claim 3, characterized in that, The calculation method based on trigonometric function-based position embedding is as follows: Concatenate the one-dimensional vectors of N current video frames in chronological order to form a keypoint sequence; The keypoint sequence represents the order of the video frames corresponding to each keypoint, and i represents a fixed-length vector. The index of the i-th element, Indicates the position index is Position embedding value; A fixed-length vector The length.

5. The real-time and efficient video prediction method based on key points according to claim 1, characterized in that, The step of inputting the obtained position-embedded vector into the Transformer encoder for parallel prediction and outputting a set of key points for future time moments specifically includes: Transformer encoder Output One predicted value: ； The vector after position embedding. Indicates the number to be predicted The predicted value corresponding to the video frame at a given time; Using the transformation matrix Reconstruct the predicted value vector Output a vector consisting of the keypoints at future time points. : 。 6. The real-time and efficient video prediction method based on key points according to claim 1, characterized in that, The feature generator reconstructs a feature heatmap from the set of key points at future time moments, specifically including: A single keypoint set in the future is set as , ; for The i-th key point in for Width direction coordinates, height direction coordinates, and feature intensity; Using a two-dimensional Gaussian distribution to reconstruct the feature heatmap, The horizontal vector composed of the width coordinates of all key points in the middle Convert to a horizontal one-dimensional Gaussian vector ,Will The vertical vector composed of the height and direction coordinates of all key points in the middle Convert to a vertical one-dimensional Gaussian vector : ； ； and They are and Expectations Respectively , The variance; and Multiplication yields a two-dimensional Gaussian distribution. : ； Final calculation and Hadamaji obtained Corresponding feature heatmap : ； This indicates the calculation of the Hadamard product.

7. The real-time and efficient video prediction method based on key points according to claim 1, characterized in that, The video frame decoder performs skip connections between the feature matrices and feature heatmaps output from each layer of the video frame encoder to reconstruct video frames at future moments, specifically including: Both the video frame decoder and the video frame encoder have an n-layer structure; the n-layer structure of the video frame decoder is an n-layer deconvolutional network. set up Represents the first video frame decoder Layered deconvolutional networks, express The process of reconstructing future video frames from the output includes: ； ； This indicates tensor splicing. For the feature heatmap, The feature matrix is ​​the output of the nth layer of the video frame encoder, and the output of the last deconvolutional layer of the video frame decoder. That is, the reconstructed video frame .

8. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images