Video interpolation method, device, electronic equipment and medium

Abstract

The present invention discloses a video interpolation method, device, electronic device, and medium, belonging to the field of video processing. The method comprises the following steps: obtaining a video to be interpolated and its missing segments, constructing an image dataset using the video to be interpolated and its missing segments as a set of data; obtaining positional information of the first frame of the missing segment in the video to be interpolated, and locating the interpolation range of the missing segment; training an interpolation frame model using the image dataset, and during the model training process, interpolating the video to be interpolated based on the positional information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment; and using the trained interpolation frame model for video interpolation. The present invention is suitable for interpolating missing frames in videos or repairing damaged video segments, and has important application value in fields such as restoration of old films, improving the quality of surveillance videos, and dynamic medical imaging.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video interpolation method, device, electronic equipment and medium. Background Art

[0002] Video interpolation aims to restore the integrity and smoothness of a video by algorithmically repairing or generating missing or damaged frames. This technology is of great significance in the field of video processing. For example, restoring old films and low-quality videos can significantly improve the overall quality and viewing experience. In video surveillance systems, video interpolation technology can fill in lost frames caused by network bandwidth limitations or equipment failures, ensuring the integrity and continuity of surveillance videos. In the field of medical imaging, video interpolation can help generate more intermediate frames, improving the temporal resolution of dynamic images, allowing doctors to more accurately observe changes in lesions and enhance diagnostic accuracy. This is particularly important in dynamic medical imaging such as ultrasound scans.

[0003] Currently, mainstream interpolation models are primarily based on recurrent neural network (RNN) architectures. Due to the temporal and spatial complexity of video data, these models need to capture dependencies in both temporal and spatial dimensions. In recent years, researchers have proposed a variety of hybrid models combining convolutional neural networks (CNNs) and RNNs, such as ConvLSTM and E3D-LSTM, to better model spatiotemporal features in videos. However, traditional convolution operations primarily focus on local spatial dependencies. While the receptive field can be expanded by stacking convolutional layers or increasing the kernel size, the actual effective receptive field is often far smaller than the theoretical value, which limits the model's video interpolation performance. To address this issue, the Video Vision Transformer (ViViT) was proposed. It models global spatiotemporal information through a self-attention mechanism, temporal encoding, and positional encoding. However, the Transformer architecture lacks the inherent inductive bias of CNNs, resulting in weak generalization performance and typically requiring large amounts of training data to achieve optimal results. Therefore, there is a need for a video frame interpolation method that requires minimal training data, is fast, and produces high-quality images. Summary of the Invention

[0004] The object of the present invention is to provide a video interpolation method, device, electronic equipment and medium to address the deficiencies of the prior art.

[0005] The object of the present invention is achieved through the following technical solutions: A video interpolation method comprising:

[0006] Obtain the video to be interpolated and its missing segments, and construct an image dataset using the video to be interpolated and its missing segments as a set of data;

[0007] Obtain the position information of the first frame of the missing segment in the video to be interpolated, and locate the interpolation range of the missing segment;

[0008] Using the image dataset to train an interpolation frame model, during the model training process, interpolating the video to be interpolated based on position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment;

[0009] The trained interpolation frame model is used for video interpolation.

[0010] Furthermore, an image dataset is constructed using the video to be interpolated and its missing segments as a set of data, including:

[0011] Maintain a one-to-one correspondence between the video to be interpolated and its missing segments;

[0012] The paired video data is decomposed frame by frame according to the time series, converted into a continuous image frame sequence, and saved as an image file;

[0013] The image is denoised, then cropped to the same resolution and the image data is enhanced.

[0014] Furthermore, obtaining the position information of the first frame of the missing segment in the video to be interpolated and locating the interpolation range of the missing segment includes:

[0015] Obtain the first frame of the missing segment and use it as a reference frame. Calculate the structural similarity index between the reference frame and each frame in the sequence of video image frames to be interpolated to determine the frame with the highest similarity. Use the frame immediately following the reference frame as the interpolation starting point, i.e., the position of the first frame of the missing segment in the video to be interpolated.

[0016] The interpolation range of the missing segment is determined based on the interpolation starting point and the length of the missing segment.

[0017] Furthermore, the interpolation frame model is a bidirectional gated recurrent unit.

[0018] The bidirectional gated recurrent unit includes a forward gated recurrent unit, a backward gated recurrent unit and a dual-stream feature fusion layer; the dual-stream feature fusion layer fuses the image features of the bidirectional information stream using a summing and averaging operation;

[0019] The forward gated recurrent unit and the backward gated recurrent unit both include image block encoding, a first multi-scale spatiotemporal feature gated recurrent unit, image block fusion, a second multi-scale spatiotemporal feature gated recurrent unit, image block expansion and image reconstruction layers, and are sequentially connected;

[0020] The image block encoding, image block fusion and image block expansion are used for image vectorization processing, feature dimensionality reduction aggregation and feature upsampling respectively; the image reconstruction layer uses a two-dimensional deconvolution operation to restore the spatial resolution and detail information of the image;

[0021] The first multi-scale spatiotemporal feature gated recurrent unit and the second multi-scale spatiotemporal feature gated recurrent unit both include a gated recurrent unit.

[0022] Furthermore, the gated recurrent unit nests a linear projection module and at least one Transformer module including a hierarchical multi-scale window mechanism; the input of the gated recurrent unit and the hidden state of the previous moment are spliced ​​as the input of the linear projection module, and the output of the linear projection module is used as the input of the Transformer module including the hierarchical multi-scale window mechanism; the output of the Transformer module including the hierarchical multi-scale window mechanism is the output of the gated recurrent unit after passing through the key equation, that is, the output of the multi-scale spatiotemporal feature gated recurrent unit; the key equation is:

[0023] Update Gate

[0024] Reset Gate

[0025] Candidate hidden states

[0026] Hidden State

[0027] in, Represents the Sigmoid activation function; Represents the image frame input at the current moment; Indicates the hidden state at the previous moment; is the hyperbolic tangent function; 、 、 、 、 and are all weight parameters of the model; 、 and are the biases of the model.

[0028] Furthermore, if the gated recurrent unit is nested with multiple Transformer modules containing a hierarchical multi-scale window mechanism, the Transformer modules containing a hierarchical multi-scale window mechanism are connected in series.

[0029] Furthermore, the method further includes: executing only the forward gated recurrent unit to predict the next image frame.

[0030] The present invention also provides a video interpolation device based on a bidirectional gated recurrent unit, comprising:

[0031] A data set construction module is used to obtain the video to be interpolated and its missing segments, and construct an image data set using the video to be interpolated and its missing segments as a set of data;

[0032] The interpolation position and range acquisition module is used to obtain the position information of the first frame of the missing segment in the video to be interpolated and locate the interpolation range of the missing segment;

[0033] a model training module, configured to train an interpolation frame model using the image dataset, and, during the model training process, interpolate the video to be interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment;

[0034] The interpolation module is used to input the missing video to be processed into the trained interpolation frame model for interpolation and output the complete video.

[0035] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned video interpolation method based on a bidirectional gated recurrent unit.

[0036] The present invention also provides a computer-readable storage medium storing a computer program, wherein the program, when executed by a processor, implements the above-mentioned video interpolation method based on a bidirectional gated recurrent unit.

[0037] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0038] 1. The bidirectional structure simultaneously models both historical and future information about image frames, comprehensively capturing temporal dependencies in video sequences and improving the accuracy and coherence of interpolated frames. During training, the loss function is constrained by integrating forward and backward prediction results, fully leveraging the bidirectional information flow to further enhance the model's interpolation accuracy and robustness.

[0039] 2. The multi-scale spatiotemporal feature gated recurrent unit, which combines a gated recurrent unit with a Transformer with a hierarchical multi-scale window mechanism, can efficiently extract local details and global semantic features of images in small sample data sets, thereby achieving high-quality interpolation of missing video segments.

[0040] 3. The method of the present invention can only execute the forward gated recurrent unit to predict the next image frame, which is suitable for scenarios such as autonomous driving systems understanding the surrounding dynamic environment and reducing game rendering delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic flow chart of a video interpolation method based on a bidirectional gated recurrent unit is provided in an embodiment of the present invention.

[0043] Figure 2 A schematic diagram of the structure of a bidirectional gated recurrent unit provided in an embodiment of the present invention.

[0044] Figure 3 A schematic diagram of the structure of a forward / backward gated recurrent unit provided in an embodiment of the present invention.

[0045] Figure 4 A schematic diagram of the structure of a multi-scale spatiotemporal feature gated recurrent unit provided in an embodiment of the present invention, wherein activation function a is a Sigmoid activation function and activation function b is a hyperbolic tangent function.

[0046] Figure 5 The present invention provides a method corresponding to Figure 1 Schematic diagram of the structure of the electronic equipment. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0048] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0050] Figure 1 The flowchart of a video interpolation method based on a bidirectional gated recurrent unit of the present invention includes the following steps:

[0051] S1, establish image dataset;

[0052] Specifically, the video to be interpolated and its missing segments must maintain a strict one-to-one correspondence to ensure data accuracy and integrity. The paired video data is decomposed frame by frame according to the time series, converted into a continuous sequence of image frames, and saved as an image file. A Gaussian filter is used to reduce noise on the image, and the high-quality images are then cropped to the same resolution. The image data is then enhanced using common data augmentation methods used in deep learning.

[0053] S2, locating the interpolation range of the missing segment by obtaining the position information of the first frame of the missing segment in the video to be interpolated;

[0054] Specifically, the first frame of the missing segment is obtained and used as a reference frame. The Structural Similarity Index (SSIM) is calculated between this reference frame and each frame in the video image sequence to be interpolated, locating the frame with the highest similarity. The frame immediately following this frame is used as the interpolation starting point, and the subsequent frame sequence becomes the interpolation range for the missing segment. The interpolation length remains the same as the length of the missing segment's image frame sequence.

[0055] S3, constructs a bidirectional gated recurrent unit;

[0056] like Figure 2 As shown, the bidirectional gated recurrent unit includes: a forward gated recurrent unit, a backward gated recurrent unit and a dual-stream feature fusion layer;

[0057] Figure 3 This is a schematic diagram of the forward / backward gated recurrent unit structure, which includes the following structure: image block encoding, two multi-scale spatiotemporal feature gated recurrent units, image block fusion, image block expansion, and image reconstruction layer; wherein, one multi-scale spatiotemporal feature gated recurrent unit is set between image block encoding and image block fusion, and the other is set between image block fusion and image block expansion.

[0058] Image block encoding, image block fusion, and image block expansion are the core technologies used in the general Transformer architecture for image vectorization, feature dimensionality reduction and aggregation, and feature upsampling, respectively. The image reconstruction layer uses a two-dimensional deconvolution operation to restore the spatial resolution and detail information of the image.

[0059] The dual-stream feature fusion layer uses an addition operation to fuse the image features of the bidirectional information stream.

[0060] like Figure 4 As shown, the multi-scale spatiotemporal feature gated recurrent unit includes: a gated recurrent unit, a linear projection module and a Transformer including a hierarchical multi-scale window mechanism.

[0061] The multi-scale spatiotemporal feature gated recurrent unit is an innovative architecture improved on the basis of the gated recurrent unit, that is, the gated recurrent unit is embedded in a linear projection module and a Transformer containing a hierarchical multi-scale window mechanism, wherein the Transformer containing a hierarchical multi-scale window mechanism can be embedded in one or more, and if multiple are embedded, they are connected in series, that is, the output of the first Transformer containing a hierarchical multi-scale window mechanism is the input of the second Transformer containing a hierarchical multi-scale window mechanism, the output of the second Transformer containing a hierarchical multi-scale window mechanism is the input of the third Transformer containing a hierarchical multi-scale window mechanism, and so on; the hierarchical multi-scale window mechanism can capture local detail features and global semantic information at the same time, thereby significantly enhancing the model's ability to model long-distance dependencies. In the calculation process of each time step, the gated recurrent unit is mainly composed of the following four core components: (1) Update gate , which is used to control the update degree of the hidden state at the current moment and decide whether to integrate new information into the hidden state; (2) reset gate , used to control the influence of the previous hidden state and decide whether to forget historical information; (3) candidate hidden state , based on the current input and the result of the reset gate, generate new memory content as a potential state update candidate; (4) hidden state , through the update gate, the hidden state of the previous moment and the candidate hidden state are weightedly fused to determine the final state output, thereby realizing the dynamic update and retention of information. Figure 3 As shown in Figure 2, the key equation of the multi-scale spatiotemporal feature gated recurrent unit is:

[0062]

[0063]

[0064]

[0065]

[0066] in, Represents the Sigmoid activation function; Represents the image frame input at the current moment; Indicates the hidden state at the previous moment; is the hyperbolic tangent function, and the result is compressed to [-1,1]; 、 、 、 、 and are all weight parameters of the model; 、 and Both are model biases; is the Hadamard product operator.

[0067] The hierarchical multi-scale window mechanism decomposes the original global self-attention mechanism operation into multiple local window calculations of different scales, and performs feature extraction at different resolutions through a hierarchical processing approach.

[0068] S4, model evaluation uses mean square error (MSE), peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as evaluation indicators to comprehensively measure the quality of the generated image frames and their similarity with the missing segments.

[0069] Example 1: Missing segment interpolation method based on ovarian cancer ultrasound images;

[0070] S1, establish image dataset;

[0071] Specifically, a high-resolution ultrasound device suitable for abdominal or pelvic scanning is used to scan the lesion site of an ovarian cancer patient. Ultrasound images are acquired by receiving and processing echoes containing characteristic information about the lesion site. Radiologists annotate video clips containing the tumor site. The acquired ultrasound and video clips of the tumor site are formatted in DICOM format. The acquired ultrasound image serves as the video to be interpolated, while the video clip of the tumor site serves as the missing segment. A one-to-one correspondence between the video to be interpolated and the missing segment is maintained. The video to be interpolated and its missing segment are decomposed frame by frame in time series into a sequence of image frames. The images are denoised using a Gaussian filter, cropped to a resolution of 320 x 240, and then enhanced using a common deep learning data augmentation method involving rotation, flipping, and scaling. In this example, the ovarian cancer ultrasound image dataset used contains 68 ultrasound examination results. Each video to be interpolated consists of 50 frames, with the marked missing segment being 10 frames long. To ensure the training and evaluation effects of the model, the dataset is divided into training set, validation set and test set in a ratio of 6:2:2, which are used for model training, hyperparameter tuning and performance evaluation respectively.

[0072] S2, locating the interpolation range of the missing segment by obtaining the position information of the first frame of the missing segment in the video to be interpolated;

[0073] Specifically, for each set of ultrasound data, the first frame of the missing segment is extracted as the reference frame. The structural similarity index between the reference frame and each frame in the video sequence to be interpolated is calculated. The frame with the highest similarity to the reference frame is located, and the index position of its next frame is recorded. During model training, this index is used as the starting point for interpolation of the missing segment, and the missing content is predicted and generated according to a fixed interpolation length (10 frames).

[0074] S3, constructs a bidirectional gated recurrent unit;

[0075] Specifically, each forward gated recurrent unit and backward gated recurrent unit are connected to two multi-scale spatiotemporal feature gated recurrent units before and after the image block fusion operation to enhance the model's ability to extract local and global features.

[0076] Each multi-scale spatiotemporal feature gated recurrent unit integrates two sets of Transformer modules with a hierarchical multi-scale window mechanism (this embodiment 1 integrates two sets of Transformer modules with a hierarchical multi-scale window mechanism, but the number of such modules is not limited; one, three, or more sets can also be integrated). Therefore, each bidirectional gated recurrent unit contains a total of eight sets of Transformer modules with a hierarchical multi-scale window mechanism, embedded in the forward and backward gated recurrent units respectively. During model training, the batch size is set to 2, the Adam optimizer is used, the initial learning rate is set to 0.0001, and the mean squared error loss function is selected as the optimization objective to minimize the pixel-level difference between the generated interpolated frames and the true frames.

[0077] S4, the model evaluation uses mean square error (MSE), peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as evaluation indicators, see Table 1 for details.

[0078] Table 1: Evaluation indicators

[0079]

[0080] In this embodiment 1, when the performance of the model on both the training set and the validation set reaches convergence, the complete network model structure and its corresponding hyperparameter configuration are saved. When the neural network algorithm is called in an actual application, the saved model file can be loaded and the input data can be directly predicted based on the stored hyperparameters, thereby ensuring the consistency of the model and the reliability of the prediction results.

[0081] This specification also provides a video interpolation device based on a bidirectional gated recurrent unit, comprising:

[0082] A data set construction module is used to obtain the video to be interpolated and its missing segments, and construct an image data set using the video to be interpolated and its missing segments as a set of data;

[0083] The interpolation position and range acquisition module is used to obtain the position information of the first frame of the missing segment in the video to be interpolated and locate the interpolation range of the missing segment;

[0084] a model training module, configured to train an interpolation frame model using the image dataset, and, during the model training process, interpolate the video to be interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment;

[0085] The interpolation module is used to input the missing video to be processed into the trained interpolation frame model for interpolation and output the complete video.

[0086] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment. You can refer to the content of the above method embodiment and will not repeat it here.

[0087] This manual also provides Figure 5 The one shown corresponds to Figure 1 Schematic diagram of the electronic equipment. Figure 5 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The video interpolation method.

[0088] This specification also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the video interpolation method based on a bidirectional gated recurrent unit is implemented.

[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, apparatuses, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0093] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A video interpolation method based on a bidirectional gated recurrent unit, characterized in that: include: Obtain the video to be interpolated and its missing segments, and construct an image dataset using the video to be interpolated and its missing segments as a set of data; Obtain the position information of the first frame of the missing segment in the video to be interpolated, and locate the interpolation range of the missing segment; The interpolation frame model is trained using the image dataset. During the model training process, the video to be interpolated is interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment; including: Obtain the first frame of the missing segment and use it as a reference frame. Calculate the structural similarity index between the reference frame and each frame in the sequence of video image frames to be interpolated to determine the frame with the highest similarity. Use the frame immediately following the reference frame as the interpolation starting point, i.e., the position of the first frame of the missing segment in the video to be interpolated. Determine the interpolation range of the missing segment based on the interpolation starting point and the length of the missing segment; The interpolation frame model is a bidirectional gated recurrent unit; the bidirectional gated recurrent unit includes a forward gated recurrent unit, a backward gated recurrent unit and a dual-stream feature fusion layer; the dual-stream feature fusion layer uses a summing and averaging operation to fuse the image features of the bidirectional information stream; the first multi-scale spatiotemporal feature gated recurrent unit and the second multi-scale spatiotemporal feature gated recurrent unit both include a gated recurrent unit; the gated recurrent unit nests a linear projection module and at least one Transformer module including a hierarchical multi-scale window mechanism; the input of the gated recurrent unit and the hidden state of the previous moment are spliced ​​as the input of the linear projection module, and the output of the linear projection module is used as the input of the Transformer module including the hierarchical multi-scale window mechanism; the output of the Transformer module including the hierarchical multi-scale window mechanism is the output of the gated recurrent unit after passing through the key equation; The trained interpolation frame model is used for video interpolation.

2. The video interpolation method based on a bidirectional gated recurrent unit according to claim 1, characterized in that: An image dataset is constructed using the video to be interpolated and its missing segments as a set of data, including: Maintain a one-to-one correspondence between the video to be interpolated and its missing segments; The paired video data is decomposed frame by frame according to the time series, converted into a continuous image frame sequence, and saved as an image file; The image is denoised, then cropped to the same resolution and the image data is enhanced.

3. The video interpolation method based on a bidirectional gated recurrent unit according to claim 1, characterized in that: The forward gated recurrent unit and the backward gated recurrent unit both include image block encoding, a first multi-scale spatiotemporal feature gated recurrent unit, image block fusion, a second multi-scale spatiotemporal feature gated recurrent unit, image block expansion and image reconstruction layers, and are sequentially connected; The image block encoding, image block fusion and image block expansion are respectively used for image vectorization processing, feature dimensionality reduction aggregation and feature upsampling; the image reconstruction layer adopts a two-dimensional deconvolution operation to restore the spatial resolution and detail information of the image.

4. The video interpolation method based on a bidirectional gated recurrent unit according to claim 3, characterized in that: The key equation is: Update Gate ; Reset Gate ; Candidate hidden states ; Hidden State ; in, Represents the Sigmoid activation function; Represents the image frame input at the current moment; Indicates the hidden state at the previous moment; is the hyperbolic tangent function; 、 、 、 、 and are all weight parameters of the model; 、 and are the biases of the model.

5. The video interpolation method based on bidirectional gated recurrent unit according to claim 4, characterized in that: If the gated recurrent unit nests multiple Transformer modules containing the hierarchical multi-scale window mechanism, the Transformer modules containing the hierarchical multi-scale window mechanism are connected in series.

6. A video interpolation method based on a bidirectional gated recurrent unit according to any one of claims 3 to 5, characterized in that: Also includes: Only the forward gated recurrent unit is executed to predict the next image frame.

7. A video interpolation device based on a bidirectional gated recurrent unit, characterized in that: include: A data set construction module is used to obtain the video to be interpolated and its missing segments, and construct an image data set using the video to be interpolated and its missing segments as a set of data; The interpolation position and range acquisition module is used to obtain the position information of the first frame of the missing segment in the video to be interpolated and locate the interpolation range of the missing segment; it includes: Obtain the first frame of the missing segment and use it as a reference frame. Calculate the structural similarity index between the reference frame and each frame in the sequence of video image frames to be interpolated to determine the frame with the highest similarity. Use the frame immediately following the reference frame as the interpolation starting point, i.e., the position of the first frame of the missing segment in the video to be interpolated. Determine the interpolation range of the missing segment based on the interpolation starting point and the length of the missing segment; a model training module, configured to train an interpolation frame model using the image dataset, and, during the model training process, interpolate the video to be interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment; The interpolation frame model is a bidirectional gated recurrent unit; the bidirectional gated recurrent unit includes a forward gated recurrent unit, a backward gated recurrent unit and a dual-stream feature fusion layer; the dual-stream feature fusion layer uses a summing and averaging operation to fuse the image features of the bidirectional information stream; the first multi-scale spatiotemporal feature gated recurrent unit and the second multi-scale spatiotemporal feature gated recurrent unit both include a gated recurrent unit; the gated recurrent unit nests a linear projection module and at least one Transformer module including a hierarchical multi-scale window mechanism; the input of the gated recurrent unit and the hidden state of the previous moment are spliced ​​as the input of the linear projection module, and the output of the linear projection module is used as the input of the Transformer module including the hierarchical multi-scale window mechanism; the output of the Transformer module including the hierarchical multi-scale window mechanism is the output of the gated recurrent unit after passing through the key equation; The interpolation module is used to input the missing video to be processed into the trained interpolation frame model for interpolation and output the complete video.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the video interpolation method based on a bidirectional gated recurrent unit according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a video interpolation method based on a bidirectional gated recurrent unit as described in any one of claims 1 to 6 is implemented.

Images