Video frame interpolation positioning method and system based on graph convolution multi-level inter-frame correlation

Abstract

The invention provides a video frame interpolation positioning method and system based on graph convolution multi-level inter-frame correlation, relates to the technical field of computer vision, and aims to solve the problems that an existing video frame interpolation detection technology does not consider unnatural traces of forged video frames, processing of time continuity features is insufficient, and video tampering detection precision is low. Comprising the following steps: acquiring three continuous video frames, respectively inputting the three continuous video frames into three node embedding modules, extracting intra-frame features, and acquiring inter-frame relationship representation through an edge embedding module; the method comprises the following steps: constructing a graph structure by taking intra-frame features as nodes and taking relation expressions as edges, updating node states through a graph neural network feature transmission module, obtaining global feature graphs of each frame, splicing the global feature graphs of each frame, inputting the spliced global feature graphs into a depth feature extraction and classification module, and performing feature extraction and classification to obtain a video frame interpolation position. According to the method, the tampering features can be better extracted, and the tampered video frame position can be more accurately positioned.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular to a video frame interpolation positioning method and system based on graph convolution multi-level inter-frame correlation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, the rapid development of deep video frame generation has made it possible to freely convert video frame rates, especially the video interpolation technology based on the diffusion model, which can generate highly realistic video content. Video frame interpolation technology is a video tampering method used to generate intermediate frames in a video sequence, with the purpose of increasing the frame rate of the video, smoothing motion or enhancing visual effects. For example, a realistic video can be generated based on two static pictures, making the content in the pictures look very natural and indistinguishable from real shots.

[0004] Existing video interpolation techniques are generally divided into flow-based and kernel-based methods, which are more stable when dealing with complex motions and when dealing with illumination changes and brightness mutations. They can generate highly realistic video frames that are indistinguishable from real frames by the human eye, and have made great progress in the fields of video production and entertainment.

[0005] However, video frame interpolation technology may also be used for improper purposes and there is a risk of abuse. Video content is prone to trust crisis, and the generated video may be used to create false news, slander or influence public opinion, posing a threat to social security, information security and legal order. Therefore, the development of effective frame interpolation detection technology is of great significance to maintaining social stability, information security and legal justice. In the process of technological development, there is an adversarial balance between generation technology and detection technology. With the advancement of generation technology, detection technology also needs to be continuously updated and improved to maintain this balance and ensure the healthy development of technology. Although many researchers have conducted extensive research in this field, there are still many difficulties in video tampering forensics that have not been overcome.

[0006] In addition, in the process of generating forged videos, forgers generally focus on the generation effect of a single video frame, and do not consider the coherence and naturalness between continuous videos well, which makes the forged video content appear overall unnatural. The unnatural traces on the continuous video frames can be used as the basis for identifying forged videos. Most of the existing methods focus on extracting the spatial features of video frames, and do not consider the unnatural traces of forged video frames. They do not consider that the insertion of video frames will destroy the consistency of the edge density of continuous frames in the time domain. That is, the processing of temporal continuity features is not sufficient, the accuracy of video tampering detection is low, and the position of the tampered video frame cannot be located more accurately. Summary of the Invention

[0007] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a video frame interpolation and localization method and system based on graph convolutional multi-level inter-frame correlation. By combining the node module of dilated convolution and position attention, the depth feature extraction module and the graph neural network, it can effectively identify whether a video frame has been illegally inserted and tampered with, improving the accuracy of tampered video frame localization.

[0008] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of the present invention provides a video frame interpolation and localization method based on graph convolutional multi-level inter-frame correlation, including:

[0010] Construct a video frame interpolation dataset and obtain three consecutive video frames;

[0011] Input the three consecutive video frames into three node embedding modules respectively to extract intra-frame features;

[0012] Obtain an inter-frame relationship representation through an edge embedding module;

[0013] Use the intra-frame features as the nodes of the graph structure and the inter-frame relationship representation as the edges of the graph structure to construct a graph structure;

[0014] Based on the graph structure, update the node states through the graph neural network feature transfer module to obtain a global feature map for each frame;

[0015] Concatenate the global feature maps of each frame and input them into the depth feature extraction and classification module for feature extraction and classification to obtain the position of video frame interpolation.

[0016] As an implementation, constructing a video frame interpolation dataset specifically means using the AdaCof and IFRNet video frame insertion technologies to perform frame insertion and tampering processing on the video dataset to obtain a video frame interpolation dataset.

[0017] As an implementation, inputting the three consecutive video frames into three node embedding modules respectively to extract intra-frame features, where the node embedding module includes a dilated convolution module, a spatial pyramid pooling module, and a position attention module. The specific process is as follows:

[0018] In each node embedding module, input the video frame into the dilated convolution module to extract multi-scale context information;

[0019] Based on the multi-scale context information, extract enhanced multi-scale features through the spatial pyramid pooling module;

[0020] Input the enhanced multi-scale features into the position attention module to identify the key features, i.e., obtain the intra-frame features.

[0021] As an implementation, based on the graph structure, update the node states through the graph neural network feature transfer module. The specific process is as follows:

[0022] Based on the graph structure, obtain the neighbor information and self-loop information of each node;

[0023] Update the state of each node through the convolutional gated recurrent unit so that each node contains the global features, i.e., obtain the global feature map of each frame.

[0024] As an implementation, splice the global feature maps of each frame and input them into the deep feature extraction and classification module for feature extraction and classification. Among them, the deep feature extraction and classification module includes a deep feature extraction module and a binary classification module. The specific process is as follows:

[0025] Splice the global feature maps of each frame to obtain the spliced feature map;

[0026] Input the spliced feature map into the deep feature extraction module to extract the deep feature map;

[0027] Input the deep feature map into the binary classification module for classification to obtain the class probability;

[0028] Determine the video frame interpolation according to the class probability, i.e., obtain the position of the video frame interpolation.

[0029] As an implementation, input the spliced feature map into the deep feature extraction module to extract the deep feature map. Among them, the depth extraction module includes a pointwise convolutional layer and two depthwise separable convolutional layers. The specific process is as follows:

[0030] Through the pointwise convolutional block, mix the features at each position of the spliced feature map pointwise to obtain the aggregated temporal features;

[0031] Extract features through two depthwise separable convolutional blocks to obtain the deep feature map.

[0032] As an implementation, the calculation formula of the deep feature map is:

[0033]

[0034] where W s is a depthwise separable convolutional layer with a size of 3×3, and W p is a pointwise convolutional layer with a size of 3×3, represents the updated node state.

[0035] The second aspect of the present invention provides a video frame interpolation and positioning system based on graph convolutional multi-level inter-frame correlation, including:

[0036] A data acquisition module for constructing a video frame interpolation data set and acquiring three consecutive video frames;

[0037] A graph structure construction module for respectively inputting three consecutive video frames into three node embedding modules to extract intra-frame features;

[0038] Obtaining an inter-frame relationship representation through an edge embedding module;

[0039] Taking the intra-frame features as the nodes of the graph structure and the inter-frame relationship representation as the edges of the graph structure to construct a graph structure;

[0040] A video frame global feature extraction module for updating the node states based on the graph structure through a graph neural network feature transfer module to obtain a global feature map for each frame;

[0041] A video frame interpolation and positioning module for splicing the global feature maps of each frame and inputting them into a deep feature extraction and classification module for feature extraction and classification to obtain the position of video frame interpolation.

[0042] The third aspect of the present invention provides a computer device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the steps in the method described in the first aspect of the present invention are implemented.

[0043] The fourth aspect of the present invention aims to provide a computer-readable storage medium with a program stored thereon. When the program is executed by a processor, the steps in the method described in the first aspect of the present invention are implemented.

[0044] The above one or more technical solutions have the following beneficial effects:

[0045] In this embodiment, a video frame interpolation data set is constructed by using a variety of efficient video frame insertion techniques, providing rich and diverse video frame interpolation data, providing more comprehensive and realistic data for video frame interpolation and positioning, and making the model video frame interpolation and positioning more effective and accurate.

[0046] In this embodiment, through the node embedding module and edge embedding module based on dilated convolution and position attention, the intra-frame features are sequentially extracted as node representations. These features can often pay more attention to the motion regions. Then, a fully connected graph, i.e., a graph structure, is constructed based on attention graph convolution, effectively representing the frame features as nodes and the relationships between each pair of frames as edges. During the edge embedding process, the connections between nodes are captured. The edges in this graph are the pairwise interactions of their connected nodes; the information of neighbor nodes is aggregated onto themselves through message passing to capture richer and higher-order relationships between video frames and within video frames. Thus, the model can better understand the video content and extract the tampering features of the motion regions more effectively.

[0047] In this embodiment, the deep feature extraction and classification module is combined with the graph neural network to obtain an optimized graph neural network, further optimizing the inter-frame relationships, extracting effective depth features, capturing multi-directional features while keeping the number of parameters low, and fully considering the situation that the insertion of video frames will destroy the consistency of the edge density in the time domain of consecutive frames, and being able to more accurately locate the positions of the tampered video frames.

[0048] This embodiment not only improves the accuracy of video tampering detection, but also provides a powerful tool for the field of computer vision through innovative network architectures and feature extraction technologies to ensure the authenticity and integrity of video content.

[0049] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0051] Figure 1 It is a schematic diagram of the framework of the video frame interpolation and localization method based on graph convolution multi-level inter-frame correlation for the first embodiment of the present invention;

[0052] Figure 2 It is a schematic diagram of the edge embedding and message passing process between adjacent nodes for the first embodiment of the present invention;

[0053] Figure 3 It is the effect of the video frame interpolation dataset for the first embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further explanations of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0055] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0056] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0057] Embodiment 1

[0058] This embodiment discloses a video frame interpolation and localization method based on graph convolutional multi-level inter-frame correlation.

[0059] To more clearly illustrate this embodiment, the implementation process of video frame interpolation and localization based on graph convolutional multi-level inter-frame correlation can be specifically described as follows:

[0060] A video frame interpolation and localization method based on graph convolutional multi-level inter-frame correlation includes:

[0061] S1. Construct a video frame interpolation data set and obtain three consecutive video frames;

[0062] S2. Input the three consecutive video frames into three node embedding modules respectively to extract intra-frame features;

[0063] Obtain an inter-frame relationship representation through an edge embedding module;

[0064] Use the intra-frame features as the nodes of the graph structure and the inter-frame relationship representation as the edges of the graph structure to construct a graph structure;

[0065] S3. Based on the graph structure, update the node states through a graph neural network feature transfer module to obtain a global feature map for each frame;

[0066] S4. Concatenate the global feature maps for each frame and input them into a deep feature extraction and classification module for feature extraction and classification to obtain the position of video frame interpolation.

[0067] As Figure 1 shown, in step S1, construct a video frame interpolation data set and obtain three consecutive video frames.

[0068] S1-1. Construct a video frame interpolation data set.

[0069] In this embodiment, its own data set is created by using a variety of efficient video frame insertion techniques.

[0070] Specifically, use the AdaCof and IFRNet video frame insertion techniques to perform frame insertion tampering on the video data set to obtain a video frame interpolation data set.

[0071] A total of 26,633 videos with an original frame rate of 25 frames per second (fps) from UCF101 and Vimeo90K were selected, and training and test data were constructed in a ratio of 7:1. There were a total of 186,431 frames (26,633 videos × 7 frames / video = 186,431 frames). Two representative DVFI methods, AdaCoF and IFRNet, were used. Seven consecutive frames were taken as a group. As Figure 3 shown, error maps were used to measure the quality of the generated frames. The first row was the original video frames, the second row was the video frames after video frame insertion operation, and the third row was the error between the first two rows of video frames. The interpolated frame positions were marked with red boxes. Whether the interpolated frames were at the first frame, middle frames, or the last frame, the model in this embodiment could accurately locate the positions of the interpolated frames.

[0072] A total of 35,660 groups of frames were used in the training set, and 3,760 groups of frames were tested in the test set. This rich and diverse dataset provided strong support for the research, made it possible to comprehensively evaluate the performance of the DVFI forensics task, and a large number of experiments were conducted on this dataset. The experimental results showed that compared with the existing video frame insertion forensics methods, the algorithm of this embodiment could effectively achieve better detection accuracy.

[0073] S1-2. Obtain three consecutive video frames.

[0074] Three consecutive video frames can be obtained each time according to the time from the constructed video frame interpolation dataset. Or, three consecutive video frames to be detected can be obtained according to actual needs.

[0075] Randomly crop the three consecutive video frames into 240×320 blocks.

[0076] As Figure 1 shown, in step S2, the three consecutive video frames were respectively input into three node embedding modules to extract intra-frame features;

[0077] Through the edge embedding module, the inter-frame relationship representation was obtained;

[0078] The intra-frame features were used as the nodes of the graph structure, and the inter-frame relationship representation was used as the edges of the graph structure to construct the graph structure.

[0079] S2-1. Input the three consecutive video frames into three node embedding modules respectively to extract intra-frame features.

[0080] In this embodiment, the node embedding module included a dilated convolution module, a spatial pyramid pooling module, and a position attention module to extract intra-frame features. The specific process was as follows:

[0081] (1) In each node embedding module, the video frame was input into the dilated convolution module to extract multi-scale context information.

[0082] The dilated convolution module includes four layers: Layer1, Layer2, Layer3, and Layer4, and each layer consists of a bottleneck block based on the residual structure.

[0083] Among them, the bottleneck block based on the residual structure mainly includes a convolutional layer, a normalization layer, and the bottleneck block, and the bottleneck block includes three groups of convolutional layers and normalization layers.

[0084] The residual structure includes a convolutional layer, a normalization layer, a max pooling layer, and a residual block.

[0085] In each node embedding module, the video frames are sequentially input into Layer1, Layer2, Layer3, and Layer4 to extract multi-scale context information.

[0086] Through dilated convolution, without increasing the number of parameters and computational complexity, the model's ability to capture large-range context information is improved.

[0087] (2) Based on the multi-scale context information, through the spatial pyramid pooling module, enhanced multi-scale features are extracted.

[0088] Through the Atrous Spatial Pyramid Pooling (ASPP) module, which combines atrous convolution and spatial pyramid pooling, convolutional kernels with different dilation rates are used to capture features at different scales, and by extracting features at different scales, the model's perception ability of multi-scale information is enhanced, and effective features are extracted as node representations.

[0089] (3) The enhanced multi-scale features are input into the position attention module to identify the key features, that is, the intra-frame features are obtained.

[0090] Calculate the attention weights between different positions of the enhanced multi-scale features to identify the key features in the video, that is, the intra-frame features. The formula is:

[0091]

[0092] Among them, represents the embedding representation of the node, that is, the intra-frame feature, which is a three-dimensional feature tensor with dimensions of W×H×C. β is a learnable parameter, and W x is a weight matrix, and γ reshape represents reshaping and permuting the tensor, and · represents the convolution operation.

[0093] Capture the spatial relationship within the input feature map through the position attention module, enabling the model to pay more attention to the moving regions in the video where tampering traces are most likely to exist. These moving regions often produce artifacts and blurs during the depth video frame interpolation process, and then extract effective features within the frame as node representations.

[0094] S2-2. Obtain the inter-frame relationship representation through the edge embedding module.

[0095] In the previous step, the initial node embedding of the given node v i is achieved. In the attention-based edge embedding module, obtain the relationship representation between frames v i and v j , and the formula is:

[0096]

[0097] where, represents the relationship between v i and v j in the k-th iteration, W c represents the weight matrix with dimensions C×C, represents the weight matrix with dimensions (W×H)×C.

[0098] In the above process, an attention mechanism is adopted to reveal the interaction and connection between vertices v i and v j . The specific calculation process is as Figure 2 shown. This mechanism can capture and understand the complex mutual relationships between nodes in two directions, enabling the model to consider the entire feature map when calculating attention, pay more attention to important regions or features, and at the same time enhance the ability to capture key information. It reflects the similarity between the node features of v i and v j .

[0099] Meanwhile, calculate the response of a certain position by paying attention to all positions within the same node embedding, and capture the internal relationship of node v i through a series of convolution-based attention mechanisms. The formula is:

[0100]

[0101] where, represents the connection of node v i to itself in the k-th iteration, γ is a learnable parameter, and W y represents a weight matrix.

[0102] S2-3. Take the intra-frame features as the nodes of the graph structure, and represent the inter-frame relationships as the edges of the graph structure to construct the graph structure.

[0103] Represent the embedded intra-frame features as nodes and the embedded inter-frame relationships as edges to establish a graph structure. The expression is: G = (V, E), where V is the set of nodes and E is the set of edges.

[0104] As Figure 1 、 Figure 2 shown, in step S3, based on the graph structure, through the graph neural network feature transfer module, update the node states to obtain the global feature map for each frame.

[0105] S3-1. Based on the graph structure, obtain the neighbor information and self-loop information of each node.

[0106] (1) Aggregate the neighbor information of each node onto the node itself as a supplement to the node features.

[0107] In the graph neural network feature transfer module, transfer the message m j,i from node v j to v i , and perform message aggregation. The formula is:

[0108]

[0109] where softmax(·) represents normalizing each row of the input. Each row of is a weighted combination of each row of , where the weights come from the corresponding columns of is each position of weighted sum with the column weights of is reshaped back into a three-dimensional tensor of size W×H×C.

[0110] (2) Consider the self-loop message of the node as its own message, set the message iteration times to 10, and collect the neighbor information and self-loop information of each node onto the current node. The self-loop message formula is:

[0111]

[0112] where N i represents the neighbor nodes of node i.

[0113] S3-2. Update the state of each node through a convolutional gated recurrent unit. Each node contains global features, that is, obtain the global feature map for each frame.

[0114] Three convolutional gated recurrent units (ConvGRUs) are respectively used to update the states of three nodes. During the update process, the feature receptive field of each node will gradually expand, and finally each node will contain global features. The previous state of the node Combine the neighbor information and self-loop information of this node obtained in the previous step Perform node update to obtain a new node state

[0115] Since ConvGRU is good at dealing with scenarios that require simultaneous processing of spatial and temporal information, and it combines the characteristics of convolutional neural networks and gated recurrent units, ConvGRU is used to update the state of each node. The updated node state The formula is:

[0116]

[0117] By message passing, the information of neighbor nodes is aggregated onto itself, capturing richer and higher-order relationships between and within video frames. Thus, the model can better understand the video content and extract the tampering features of the motion area more effectively. By updating the node state, the feature receptive field of each node will gradually expand, and finally each node will contain global features, providing a data basis for video frame interpolation positioning.

[0118] Such as Figure 1 As shown, in step S4, the global feature maps of each frame are concatenated and input into the deep feature extraction and classification module for feature extraction and classification to obtain the position of video frame interpolation.

[0119] Among them, the deep feature extraction and classification module includes a deep feature extraction module and a binary classification module. The feature extraction and classification process is as follows:

[0120] (1) Concatenate the global feature maps of each frame to obtain a concatenated feature map.

[0121] After k iterations of message passing, the updated node representation is obtained Fuse the feature spaces contained in all graph nodes.

[0122] (2) Input the concatenated feature map into the deep feature extraction module to extract the deep feature map.

[0123] Among them, the depth extraction module includes a pointwise convolutional layer and two depthwise separable convolutional layers. The process of extracting the deep feature map is as follows:

[0124] 1) Through the pointwise convolutional block, mix the features at each position of the concatenated feature map pointwise to obtain the aggregated temporal features.

[0125] With a stride of 2, a padding of 1, and a kernel size of 3×3, the feature maps in the channel direction are combined through pointwise convolution, the features at each position are pointwise mixed, and the temporal features are aggregated.

[0126] 2) Feature extraction is performed through two depthwise separable convolution blocks to obtain a depth feature map.

[0127] Then, through two depthwise separable convolution blocks, depthwise separable convolution is used to learn the temporal features between frames. The point convolution, stride, and padding are 3×3, 1, and 0 respectively.

[0128] Depthwise separable convolution is a combination of depthwise convolution and pointwise convolution. The formula for obtaining the depth feature map is:

[0129]

[0130] After the above steps, the model can capture multi-directional features while keeping the number of parameters low, maintain the resolution of the spatial dimension, and there is no pooling operation in this module, which can effectively avoid the removal of interpolation traces.

[0131] (3) The depth feature map is input into a binary classification module for classification to obtain class probabilities;

[0132] Based on the class probabilities, it is determined as video frame interpolation, that is, the position of the video frame interpolation is obtained.

[0133] 1) In the binary classification module, the obtained depth feature map F is successively passed through average pooling, flattening, and Softmax operations to classify the data into video frame interpolation (false frame) and true frame, and the probability distributions of the two types of frames are obtained, denoted as f i,0 and f i,1 .

[0134] 2) The cross-entropy loss is used as the loss function, and the model is optimized with the goal of minimizing the loss function.

[0135] Make the predicted probability distribution of the model as close as possible to the true probability distribution. The formula is:

[0136]

[0137] where N is the batch size, y i is the true label of the i-th sample (taking values of 0 or 1), f i,0 is the probability that the i-th sample predicted by the model belongs to the "false frame" class, and f i,1 is the probability that the i-th sample predicted by the model belongs to the "true frame" class.

[0138] 3) Based on the class probabilities, it is determined as video frame interpolation (false frame), that is, the position of the video frame interpolation is obtained.

[0139] After the above steps, according to the probability of the "false frame" category predicted by the model, the video frame interpolation can be obtained, the position of the video frame interpolation can be located, and the position of the interpolated frame can be realized.

[0140] In this embodiment, a video frame interpolation dataset is constructed by using a variety of efficient video frame insertion techniques, providing rich and diverse video frame interpolation data, making the model video frame interpolation positioning more effective and accurate; through the node embedding module and edge embedding module based on dilated convolution and position attention, the intra-frame features are extracted in sequence as node representations, and these features can often pay more attention to the motion area, and then a fully connected graph, that is, a graph structure, is constructed based on the attention graph convolution, effectively representing the frame features as nodes and representing the relationship between each pair of frames as edges. During the edge embedding process, the connections between nodes are captured; the information of neighbor nodes and self-loop information are aggregated onto itself through message passing to capture richer and higher-order relationships between video frames and within video frames, better extracting the tampering features of the motion area; the depth feature extraction and classification module is combined with the graph neural network to obtain an optimized graph neural network, further optimizing the inter-frame relationship, extracting effective depth features, and performing accurate classification. While keeping the number of parameters low, it captures multi-directional features, fully considering the situation that the insertion of video frames will destroy the consistency of the edge density in the time domain of consecutive frames, and can more accurately locate the position of the tampered video frame.

[0141] Embodiment 2

[0142] The purpose of this embodiment is to provide a video frame interpolation positioning system based on multi-level inter-frame correlation of graph convolution, including:

[0143] A data acquisition module, used to construct a video frame interpolation dataset and obtain three consecutive video frames;

[0144] A graph structure construction module, used to input three consecutive video frames into three node embedding modules respectively to extract intra-frame features;

[0145] Through the edge embedding module, obtain the inter-frame relationship representation;

[0146] Use the intra-frame features as the nodes of the graph structure and the inter-frame relationship representation as the edges of the graph structure to construct the graph structure;

[0147] A video frame global feature extraction module, used to update the node state based on the graph structure through the graph neural network feature transfer module to obtain the global feature map of each frame;

[0148] A video frame interpolation positioning module, used to splice the global feature map of each frame and input it into the depth feature extraction and classification module for feature extraction and classification to obtain the position of the video frame interpolation.

[0149] A video frame interpolation and positioning system based on graph convolutional multi-level inter-frame correlation realizes the method steps in Embodiment 1.

[0150] Embodiment 3

[0151] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0152] Embodiment 4

[0153] The purpose of this embodiment is to provide a computer-readable storage medium.

[0154] A computer-readable storage medium stores a computer program, and when the program is executed by a processor, the steps of the above method are executed.

[0155] Embodiment 5

[0156] The purpose of this embodiment is to provide a computer program product containing instructions, which when running on a computer, enables the computer to execute the methods and functions involved in any one of the above embodiments.

[0157] The steps involved in the devices in the above embodiments correspond to those in Method Embodiment 1. For specific implementation manners, refer to the relevant description part of Embodiment 1. The term "computer-readable storage medium" should be understood to include a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0158] Those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. The present invention is not limited to any specific combination of hardware and software.

[0159] Although the specific implementation manners of the present invention are described above in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A video frame interpolation and localization method based on graph convolutional multi-level inter-frame correlation, characterized in that Including: Construct a video frame interpolation dataset and obtain three consecutive video frames; Input the three consecutive video frames into three node embedding modules respectively to extract intra-frame features; Through the edge embedding module, obtain the representation of inter-frame relationships; Use the intra-frame features as the nodes of the graph structure and the representation of inter-frame relationships as the edges of the graph structure to construct the graph structure; Based on the graph structure, through the graph neural network feature transfer module, update the node states to obtain the global feature map for each frame; Concatenate the global feature maps for each frame and input them into the deep feature extraction and classification module for feature extraction and classification to obtain the position of video frame interpolation.

2. The video frame interpolation and localization method based on graph convolutional multi-level inter-frame correlation according to claim 1, characterized in that Construct a video frame interpolation dataset. Specifically, use the AdaCof and IFRNet video frame insertion techniques to perform frame insertion tampering on the video dataset to obtain the video frame interpolation dataset.

3. The video frame interpolation and localization method based on graph convolutional multi-level inter-frame correlation according to claim 1, wherein, Input the three consecutive video frames into three node embedding modules respectively to extract intra-frame features. Among them, the node embedding module includes a dilated convolution module, a spatial pyramid pooling module, and a position attention module. The specific process is as follows: In each node embedding module, input the video frame into the dilated convolution module to extract multi-scale context information; Based on the multi-scale context information, through the spatial pyramid pooling module, extract enhanced multi-scale features; Input the enhanced multi-scale features into the position attention module to identify the key features, that is, obtain the intra-frame features.

4. The video frame interpolation and positioning method based on graph convolution multi-level inter-frame correlation according to claim 1, wherein Based on the graph structure, through the graph neural network feature transfer module, update the node states. The specific process is as follows: Based on the graph structure, obtain the neighbor information and self-loop information of each node; Update the state of each node through a convolutional gated recurrent unit so that each node contains global features, that is, obtain the global feature map for each frame.

5. The video frame interpolation and localization method based on graph convolution multi-level inter-frame correlation according to claim 1, characterized in that, Concatenate the global feature maps for each frame and input them into the deep feature extraction and classification module for feature extraction and classification. Among them, the deep feature extraction and classification module includes a deep feature extraction module and a binary classification module. The specific process is as follows: Concatenate the global feature maps for each frame to obtain the concatenated feature map; Input the concatenated feature map into the deep feature extraction module to extract the deep feature map; Input the deep feature map into the binary classification module for classification to obtain the class probability; According to the class probability, determine it as video frame interpolation, that is, obtain the position of video frame interpolation.

6. The video frame interpolation and localization method based on graph convolution multi-level inter-frame correlation according to claim 5, characterized in that, Input the concatenated feature map into the deep feature extraction module to extract the deep feature map. Among them, the deep extraction module includes a pointwise convolutional layer and two depthwise separable convolutional layers. The specific process is as follows: Through the pointwise convolutional block, mix the features at each position of the concatenated feature map pointwise to obtain the aggregated temporal features; Perform feature extraction through two depthwise separable convolutional blocks to obtain the deep feature map.

7. The video frame interpolation and positioning method based on graph convolution multi-level inter-frame correlation according to claim 1, characterized in that The calculation formula of the deep feature map is: Among them, W s is a separable convolutional layer of size 3×3, and W p is a pointwise convolutional layer of size 3×3, represents the updated node state.

8. A video frame interpolation and localization system based on graph convolution multi-level inter-frame correlation, characterized in that, Including: A data acquisition module for constructing a video frame interpolation dataset and obtaining three consecutive video frames; A graph structure construction module for inputting the three consecutive video frames into three node embedding modules respectively to extract intra-frame features; Through the edge embedding module, obtain the representation of inter-frame relationships; Use the intra-frame features as the nodes of the graph structure and the representation of inter-frame relationships as the edges of the graph structure to construct the graph structure; The video frame global feature extraction module is used to update the node states based on the graph structure through the graph neural network feature transfer module to obtain the global feature map of each frame; The video frame interpolation positioning module is used to splice the global feature maps of each frame and input them into the deep feature extraction and classification module for feature extraction and classification to obtain the position of the video frame interpolation.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-7 above.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it executes the steps of the method described in any one of claims 1-7 above.

Images