Intermediate frame image generation method, model training method and device

Abstract

The embodiments of this specification provide an intermediate frame image generation method, an intermediate frame image reconstruction model training method and device, wherein the intermediate frame image generation method includes: obtaining a reference image of a target image to be inserted, wherein the reference image contains texture information missing from the target image; determining the optical flow of the target image, and generating an initial intermediate frame image of the target image according to the optical flow; inputting the reference image, the initial intermediate frame image, the optical flow and the target image into a pre-trained intermediate frame image reconstruction model for feature fusion, and obtaining a target intermediate frame image of the target image. Through the embodiments of this specification, it can be ensured that the generated intermediate frame image is relatively natural.

Description

Technical Field

[0001] This document relates to the field of computer vision technology, and in particular to a method for generating an intermediate frame image, and a training method and device for an intermediate frame image reconstruction model. Background Art

[0002] With the development of virtual digital human technology and the rise of the concept of the metaverse, people have an increasing demand for smooth, natural, and high-quality virtual digital humans. For example, in some business scenarios, it may involve splicing and transitioning video materials of virtual digital humans. Based on this, how to achieve splicing and transition to improve the smoothness between video materials and avoid problems such as "flashing" has gradually become a key issue of concern.

[0003] In the related art, interpolation technology is usually used to realize the splicing and transition of the video materials of the virtual digital human. Among them, the interpolation (Video Frame Interpolation, VFI) technology, also known as the frame rate conversion (Frame Rate Conversion) technology, is to add one or more frames (intermediate frame images) in every two consecutive frames of the original video to shorten the display time between frames, thereby improving the smoothness of the video and achieving a better visual sensory effect. However, in the related art, the generated intermediate frame images are not natural, which easily causes the interpolated video to look unsmooth. Summary of the invention

[0004] The purpose of one or more embodiments of the present specification is to provide a method for generating an intermediate frame image and a method for training an intermediate frame image reconstruction model to solve the problem that the intermediate frame images generated by the intermediate frame image generation method provided by the prior art are unnatural.

[0005] To solve the above technical problems, one or more embodiments of this specification are implemented as follows:

[0006] One or more embodiments of this specification provide a method for generating an intermediate frame image, including:

[0007] Obtain a reference image of the target image to be interpolated, where the reference image contains texture information missing from the target image;

[0008] Determine the optical flow of the target image, and generate an initial intermediate frame image of the target image according to the optical flow;

[0009] The reference image, the initial intermediate frame image, the optical flow and the target image are input into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image.

[0010] It can be seen that in the intermediate frame image generation method provided in the embodiment of the present specification, during the process of generating the intermediate frame image, the reference image, the initial intermediate frame image, the optical flow and the target image can be input into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image, wherein the reference image contains the texture information that is missing in the target image. In this way, during the fusion process, the missing texture information of the target image can be supplemented based on the reference image, thereby avoiding the problem of unnatural intermediate frame images generated based on the target image due to the missing texture of the target image.

[0011] One or more embodiments of this specification provide a method for training an intermediate frame image reconstruction model, including:

[0012] Obtaining an image sample and a reference image sample, wherein the reference image sample contains texture information missing from the image sample;

[0013] Acquire a first optical flow of the image sample, and generate a first intermediate frame image of the image sample according to the first optical flow;

[0014] The reconstruction model to be trained is trained by using the image samples, the reference image samples, the first intermediate stream, and the first intermediate frame images to obtain the intermediate frame image reconstruction model.

[0015] It can be seen that the training method of the intermediate frame image reconstruction model provided in the embodiment of the present specification can, when training the intermediate frame image reconstruction model, use image samples, the first intermediate stream, the first intermediate frame image and the reference image samples as the input of the reconstruction model to be trained to train the model to be trained. In this way, the reconstruction model to be trained can complement the missing texture information of the image samples according to the reference image samples during the training process, thereby avoiding the problem of unnatural intermediate frame images output by the intermediate frame image reconstruction model obtained by training according to the image samples.

[0016] One or more embodiments of the present specification provide an intermediate frame image generation device, including:

[0017] An acquisition module is used to acquire a reference image of a target image to be inserted, wherein the reference image contains texture information missing from the target image;

[0018] A determination module, used to determine the optical flow of the target image and generate an initial intermediate frame image of the target image according to the optical flow;

[0019] The fusion module is used to input the reference image, the initial intermediate frame image, the optical flow and the target image into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image.

[0020] One or more embodiments of the present specification provide an electronic device, including:

[0021] processor; and,

[0022] A memory arranged to store computer executable instructions, wherein the executable instructions are configured to be executed by the processor, wherein the executable instructions include steps for executing the above-mentioned intermediate frame image generation method, or the executable instructions include steps for executing the above-mentioned intermediate frame image reconstruction network training method.

[0023] One or more embodiments of the present specification provide a storage medium for storing computer-executable instructions, wherein the executable instructions enable a computer to execute the above-mentioned intermediate frame image generation method, or the executable instructions enable a computer to execute the above-mentioned intermediate frame image reconstruction network training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A schematic diagram of a scene of a method for generating an intermediate frame image provided in one or more embodiments of this specification;

[0026] Figure 2 A schematic flow chart of a method for generating an intermediate frame image provided in one or more embodiments of this specification;

[0027] Figure 3 A schematic diagram of the structure of an intermediate flow estimation network provided for one or more embodiments of this specification;

[0028] Figure 4 A flowchart of a method for training an intermediate frame image reconstruction model provided in one or more embodiments of this specification;

[0029] Figure 5 A schematic diagram of the structure of an intermediate frame image reconstruction model provided by one or more embodiments of this specification;

[0030] Figure 6 A flowchart of another intermediate frame image generation method provided for one or more embodiments of this specification;

[0031] Figure 7 A flowchart of a method for training an intermediate frame image reconstruction model provided in one or more embodiments of this specification;

[0032] Figure 8 A schematic diagram of the module composition of an intermediate frame image generating device provided in one or more embodiments of this specification;

[0033] Fig. 9 A schematic diagram of the module composition of a training device for an intermediate frame image reconstruction network provided in one or more embodiments of this specification;

[0034] Fig.10 A schematic diagram of the structure of an electronic device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0036] The terms "first", "second", etc. in this specification are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, in this specification, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0037] At present, there are two main ways to generate intermediate frame images: Method 1, the flow-based video interpolation method, usually estimates the bidirectional optical flow between two adjacent frames in the video first, then obtains the approximate intermediate flow through linear combination based on the bidirectional optical flow, and finally determines the intermediate frame image based on the approximate intermediate flow. Method 2, the real-time intermediate flow estimation algorithm directly estimates the intermediate flow from coarse to fine through the intermediate flow model IFNet; then, the input frame is distorted according to the estimated intermediate flow, and the distorted input frame and intermediate flow are fused and improved using a convolutional neural network (CNN) to obtain the intermediate frame image.

[0038] However, in the above method 1, since the approximate intermediate flow is obtained by linear combination based on the bidirectional optical flow, and in actual applications, the linear combination based on the bidirectional optical flow often cannot directly simulate occlusion and complex nonlinear motion from the video, when the method 1 is adopted, it is easy to cause artifacts / distortion in the motion boundary of the intermediate frame image. In the above method 2, although the artifact / distortion problem of the intermediate frame image can be solved, when estimating the intermediate flow, this method usually forcibly supplements some texture information out of thin air for the video frame with texture missing due to occlusion as the missing texture information, which makes the generated intermediate frame image unnatural.

[0039] To solve the above technical problems, an embodiment of the present specification provides a method for generating an intermediate frame image. First, a reference image of a target image to be inserted is obtained, wherein the reference image contains texture information that is missing from the target image. Then, the optical flow of the target image is determined, and an initial intermediate frame image of the target image is generated based on the optical flow. Finally, the reference image, the initial intermediate frame image, the optical flow, and the target image are input into a pre-trained intermediate frame image reconstruction model for feature fusion to obtain a target intermediate frame image of the target image.

[0040] It can be seen that in the intermediate frame image generation method provided in the embodiment of the present specification, during the process of generating the intermediate frame image, the reference image, the initial intermediate frame image, the optical flow and the target image can be input into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image, wherein the reference image contains the texture information that is missing in the target image. In this way, during the fusion process, the missing texture information of the target image can be supplemented based on the reference image, thereby avoiding the problem of unnatural intermediate frame images generated based on the target image due to the missing texture of the target image.

[0041] Figure 1 A schematic diagram of a scene of a method for generating an intermediate frame image provided in one or more embodiments of this specification, such as Figure 1 As shown, the scenario includes: an intermediate frame image generation device, database 1 and database 2. The intermediate frame image generation device can be deployed in a terminal device, which can be a mobile device such as a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc. The terminal device can be a server, such as an independent server, or a server cluster composed of multiple servers. Database 1 is used to store the target image to be inserted; database 2 is used to store the reference image of the target image to be inserted; in some embodiments, the target image to be inserted and the reference image can also be stored in the same database.

[0042] The intermediate frame image generation method provided in one or more embodiments of this specification can be applied to any scene that requires interpolation, such as film and television post-processing scenes, game animation generation scenes, and scenes for clarifying blurred frames in videos, etc. Moreover, the method can be applied to products in these scenes, such as film and television post-processing systems, game animation generation systems, and systems for clarifying blurred frames in videos.

[0043] Among them, the film and television post-processing system can improve the video frame rate displayed by the device through the intermediate frame image generation method provided in the embodiment of this specification when the film and television screen has frame drops, freezes or blurs, so that the user can feel that the video is clearer and smoother; or, through the intermediate frame image generation method, assist in slowing down the wonderful shots of the video during the video production and editing process. Or, through the intermediate frame image generation method, the intermediate frame image of the video is compressed.

[0044] The game animation generation system can interpolate unstable image frames through the intermediate frame image generation method when the game screen fluency and smoothness are not good, so as to improve the fluency and smoothness of the game playback screen.

[0045] Based on the above application scenario architecture, an embodiment of the present application provides a method for generating an intermediate frame image. Figure 2 A flowchart of a frame insertion method provided for one or more embodiments of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0046] Step 202: Obtain a reference image of the target image to be inserted, wherein the reference image contains texture information that is missing from the target image.

[0047] The target image to be inserted can be understood as at least two images before and after the position where the insertion is required. The target image to be inserted can be an image captured by a camera component, or an image drawn by drawing software, video production software, etc.

[0048] For example, in a film and television post-processing scenario, assuming that in order to improve the smoothness of video A and achieve a better visual effect, it is necessary to insert the 7th frame video image and the 8th frame video image in video A. At this time, the 7th frame video image and the 8th frame video image in video A can be called the target images to be inserted.

[0049] Alternatively, in a game animation generation scenario, assuming that in order to improve the smoothness of the game playback screen, it is necessary to add an intermediate frame image between every two consecutive game screen frames to shorten the display time between frames of the game screen, thereby improving the smoothness of the game playback screen, then at this time, every two consecutive game screen frames can be called target images to be inserted. For example, assuming that the game video includes a total of 6 frames, the 1st and 2nd frames, the 2nd and 3rd frames, the 3rd and 4th frames, the 4th and 5th frames, and the 5th and 6th frames in the game video can be respectively called target images to be inserted.

[0050] The reference image can be an image of the video / game where the target image is located, or an image of another video / game that contains the same scene information as the target image, as long as it contains the texture information missing from the target image. For example, using the above example, assuming that the 7th and 8th frames in video A need to be interpolated, the reference image can be an image in video A, or an image of video B that contains the same scene information as the 7th and 8th frames in video A, as long as the image contains the texture information missing from the 7th and 8th frames.

[0051] In the embodiments of the present specification, a reference image can be obtained by an image acquisition device or an image sensor, etc. Specifically, the target image to be inserted can be first obtained by the image acquisition device or the image sensor, and the scene information contained in the target image and the video to which the target image belongs can be determined according to the obtained target image; then, an image containing the texture information missing from the target image can be obtained from the video to which the target image belongs by the image acquisition device or the image sensor as a reference image.

[0052] Alternatively, if it is determined that other image frames in the video to which the target image belongs do not contain the texture information that is missing from the target image, an image that has the same scene information as the target image and contains the texture information that is missing from the target image can be obtained from other videos through an image acquisition device or an image sensor based on the determined scene information as a reference image.

[0053] Alternatively, in the embodiments of the present specification, the target image may be first acquired; then, the missing texture information in the target image may be supplemented to obtain the supplemented target image, and the supplemented target image may be determined as the reference image. When supplementing the missing texture information in the target image, for example, the missing texture information in the target image may be generated first, and then, based on the generated texture information and the target image, a splicing process may be performed in the image channel dimension to fuse the feature information of the two, thereby achieving the purpose of enhancing the overall supplement effect.

[0054] In some embodiments, the target image to be inserted may not be determined temporarily, or it may be difficult to determine the missing texture information of the target image. At this time, in order to make up for the missing texture information of the target image to the greatest extent, when selecting a reference image, you can choose an image with the same scene information as the target image and richer texture information as much as possible based on the scene information contained in the target image.

[0055] In one or more embodiments, when the number of reference images obtained is small, it may not be possible to completely include all the texture information missing from the target image, thereby failing to achieve the expectation of supplementing the missing texture of the target image. When the number of reference images is large, although the texture information missing from the target image can be included to a large extent, since the reference images need to be input into the intermediate frame image reconstruction model for processing later, if the number of reference images is too large, the input data dimension of the intermediate frame image reconstruction model will be too large, thereby causing the intermediate frame image reconstruction model to run slower. In order to solve this problem, when obtaining reference images, this specification only needs to ensure that all reference images can cover all the texture information missing from the target image. For example, assuming that the texture information missing from the target image can be obtained in one reference image, the reference image can use only this one image.

[0056] Step 204: determine the optical flow of the target image, and generate an initial intermediate frame image of the target image according to the optical flow.

[0057] The optical flow of the target image is used to describe the movement or spatial position correspondence of each grayscale pixel from the previous frame to the next frame where the interpolation position is required in the target image.

[0058] The initial intermediate frame image of the target image can be understood as a rough, preliminary intermediate frame image obtained based on the optical flow and the target image; in this scheme, it can also be understood as an intermediate result of the target intermediate frame image, which needs to be further processed to obtain the target intermediate frame image.

[0059] In the embodiments of the present specification, when determining the optical flow of the target image, the target image to be inserted can be first obtained; then, the target image is input into the intermediate flow estimation network for intermediate flow estimation, thereby obtaining the optical flow of the target image.

[0060] like Figure 3 As shown in FIG. 1 , a schematic diagram of the network structure of an intermediate flow estimation network (IFEN) provided by an example of this specification is shown. Specifically, the IFEN may include three layers of intermediate flow estimation blocks (block0 to block2), wherein each block is composed of a resolution parameter K i, a convolutional layer (Conv) with a stride of 2, 6 residual blocks (Resblock) and a transposed convolutional layer (Transpose Conv) with a stride of 2.

[0061] The following, combined Figure 3 The network structure shown in the figure describes in detail the specific implementation method of using IFEN to determine the optical flow of the target image in the embodiments of this specification.

[0062] For example, assuming that the target image to be inserted includes video frame I0 and video frame I1, video frame I0 and video frame I1 can be input into IFEN. After video frames I0 and I1 enter IFEN, they successively enter the three-layer intermediate flow estimation block from block0 to block2 (block0 to block2), and are processed by the convolution layer, the residual block and the transposed convolution layer, so as to obtain the first reverse optical flow F corresponding to video frame I0. t ->0, and the second reverse optical flow F corresponding to the video frame I1 t ->1, which is the optical flow of the target image.

[0063] After obtaining the intermediate flow, we can use the first reverse optical flow F in the intermediate flow to t ->0 and the second reverse optical flow F t ->1 backward warping is performed on video frame I0 and video frame I1 respectively to obtain a backward warping result; then, the backward warping result, the optical flow and the target image are input into the initial intermediate frame image construction model to obtain an initial intermediate frame image of the target image.

[0064] Among them, the initial intermediate frame image construction model can be trained by taking the image samples to be interpolated, the optical flow of the image samples, the backward distortion processing results of the optical flow of the image samples, and the multi-scale features of the image samples to be interpolated as the input of the initial intermediate frame image construction model to be trained, and taking the initial intermediate frame image as the output result of the initial intermediate frame image construction model to be trained.

[0065] In an optional embodiment, the initial intermediate frame image construction model can be a network model with information fusion and intermediate frame image generation functions. In the present application, the initial intermediate frame image construction model can be a codec network with a network structure similar to that of a U-Net network, for example, a FusionNet model, wherein the encoder part of the FusionNet model includes four convolution blocks, each convolution block consists of two 3*3 convolution layers, and the decoder part can consist of four transposed convolution layers.

[0066] In the embodiments of the present specification, considering that the initial intermediate frame image construction model is pre-trained based on the image samples to be interpolated, the optical flow of the image samples, the backward warping processing results of the optical flow of the image samples, and the multi-scale features of the image samples to be interpolated, therefore, in the specific implementation process of obtaining the initial intermediate frame image of the target image through the initial intermediate frame image construction model according to the backward warping processing results, the optical flow and the target image, it is also necessary to obtain the multi-scale features of the target image, and then input the backward warping processing results, the optical flow, the target image and the multi-scale features of the target image into the initial intermediate frame image construction model to obtain the initial intermediate frame image of the target image.

[0067] The multi-scale features of the target image can be understood as pyramid features of the target image at multiple scales, which can be extracted by a context extraction network, wherein the context extraction network can be, for example, a ContextNet network. Generally, the scale of the multi-scale features of the target image can be determined according to the number of convolution kernels contained in the extraction network and the number of downsampling times.

[0068] For example, assuming that the resolution of the target image is 1920*1088, for an extraction network with a downsampling number of 1 and a number of convolution kernels of 16, the scale of the target image can be (1*960*544*16); for an extraction network with a downsampling number of 2 and a number of convolution kernels of 32, the scale of the target image can be (1*480*272*32); for an extraction network with a downsampling number of 3 and a number of convolution kernels of 64, the scale of the target image can be (1*240*136*64); for an extraction network with a downsampling number of 4 and a number of convolution kernels of 128, the scale of the target image can be (1*120*68*128).

[0069] In the embodiments of this specification, the multi-scale features of the target image can be extracted by a context extraction network. For example, taking the context extraction network as a ContextNet network, when extracting the multi-scale features of the target image, the video frame and the intermediate flow corresponding to the video frame can be input into the ContextNet network in pairs. Using the above example, the video frame I0 and the first reverse optical flow F t ->0 as a pair of feature information, the video frame I1 and the second reverse optical flow F t ->1 is input into the ContextNet network as a pair of feature information to obtain a multi-scale feature corresponding to the target image, wherein the multi-scale feature can be a pyramid feature.

[0070] Step 206 , input the reference image, the initial intermediate frame image, the optical flow and the target image into a pre-trained intermediate frame image reconstruction model for feature fusion to obtain a target intermediate frame image of the target image.

[0071] The intermediate frame image reconstruction model is used to reconstruct the target intermediate frame image based on the initial intermediate frame image. The intermediate frame image reconstruction model to be trained can be obtained by taking the image sample, the reference image sample corresponding to the image sample, the optical flow of the image sample, and the initial intermediate frame image as the input of the intermediate frame image reconstruction model to be trained, and taking the target intermediate frame image as the output result of the intermediate frame image reconstruction model to be trained. The reference image sample corresponding to the image sample contains the texture information missing from the image sample.

[0072] In the embodiments of this specification, considering that the intermediate frame image reconstruction model is mainly used to perform feature fusion based on relevant information such as the target image to be inserted and the reference image to achieve target intermediate frame image reconstruction, a network model with information fusion and intermediate frame image reconstruction functions can also be selected as the intermediate frame image reconstruction model. Among them, the network model with information fusion and intermediate frame image reconstruction functions can be, for example, a FusionNet model, which is a codec network with a network structure similar to that of a U-Net network, wherein the encoder part includes four convolution blocks, each of which is composed of two 3*3 convolution layers, and the decoder part can be composed of four transposed convolution layers.

[0073] It should be noted that the intermediate frame image reconstruction model cited in the embodiments of this specification is a FusionNet model, which is only an exemplary description and does not impose any limitation on this specification.

[0074] In the embodiments of the present specification, taking the intermediate frame image reconstruction model as a FusionNet model as an example, after the reference image, target image, optical flow, initial intermediate frame image and other information are input into the intermediate frame image reconstruction model, the first convolution block in the encoder of the FusionNet model begins to process the above input information, and sends the first output result obtained by processing to the second convolution block. After receiving the first output result, the second convolution block will process the first output result, and send the second output result obtained after processing to the third convolution block, and so on, until the fourth convolution block is processed, the reconstructed target intermediate frame image can be obtained.

[0075] It can be seen that, using the intermediate frame image generation method provided in the embodiments of this specification, in the process of generating the intermediate frame image, the reference image, the initial intermediate frame image, the optical flow and the target image can be input into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image, wherein the reference image contains the texture information that is missing in the target image. In this way, in the fusion process, the missing texture information of the target image can be supplemented based on the reference image, thereby avoiding the problem of unnatural intermediate frame images generated based on the target image due to the missing texture of the target image.

[0076] In one or more embodiments of the present specification, before inputting the reference image, the initial intermediate frame image, the optical flow and the target image into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image, the intermediate frame reconstruction model may be pre-trained, such as Figure 4 As shown, the process may specifically include the following steps 402 to 406.

[0077] Step 402: Acquire an image sample and a reference image sample, wherein the reference image sample contains texture information that is missing from the image sample.

[0078] The image samples include at least two images before and after the position where interpolation is required. The image samples may be images captured by a camera assembly, or images drawn by drawing software, video production software, and the like.

[0079] Optionally, considering that when generating an intermediate frame image, it is usually necessary to perform a convolution operation on the image samples to be interpolated, and the convolution operation will make the image smaller and smaller, and cause some boundary information of the image to be lost, resulting in less information on the corners and boundaries of the image. Therefore, in order to avoid this problem, in one or more embodiments, a padding operation can be performed on the image samples to be interpolated, wherein the padding operation is to fill the image with "0" before the convolution operation, and can also be understood as adding a grid around the image block of the image sample to be interpolated, to ensure that the size of the image sample to be interpolated does not change after the convolution, and that the edge data of the image sample to be interpolated can also be used, thereby better expanding the edge features of the entire image. For example, assuming that the resolution of an image sample to be interpolated is 1920*1080, the height can be padded to 1088, so that the resolution of the image sample to be interpolated is 1920*1088.

[0080] It should be noted that the above-mentioned example of padding the height to 1088 is only an exemplary description of the embodiment of this specification and does not impose any limitation on this specification.

[0081] The reference image sample may also be an image captured by a camera component, or an image drawn by drawing software, video production software, etc.

[0082] In the embodiments of this specification, every two consecutive video frames can be obtained from any one or more videos as image samples through an image acquisition device or an image sensor. Correspondingly, the reference sample can be obtained from the video where the pattern sample is located, or from other videos, as long as the acquired reference sample includes the texture information missing from the image sample. Alternatively, in an optional implementation, a target image can also be obtained; then, the missing texture information in the target image is complemented to obtain a complemented target image, and the complemented target image is determined as a reference image.

[0083] Step 404: Acquire a first optical flow of the image sample, and generate a first intermediate frame image of the image sample according to the first optical flow.

[0084] In the embodiments of this specification, the specific implementation of obtaining the first optical flow of the image sample is similar to the specific implementation of determining the optical flow of the target image in the above embodiments. The specific implementation of generating the first intermediate frame image of the image sample according to the first optical flow is similar to the specific implementation of generating the initial intermediate frame image of the target image according to the optical flow in the above embodiments. Therefore, the specific method for implementing step 404 can refer to the relevant content in step 204 of the above embodiments. To avoid redundancy, it will not be repeated here.

[0085] Step 406 , training the reconstruction model to be trained by using the image samples, the reference image samples, the first optical flow, and the first intermediate frame image to obtain an intermediate frame image reconstruction model.

[0086] In this specification, considering that the intermediate frame image reconstruction model is mainly used to achieve intermediate frame image reconstruction based on the fusion of relevant information such as the target image to be inserted and the reference image, during the training process, a network model with information fusion and reconstruction functions can be selected as the reconstruction model to be trained.

[0087] Among them, the network model with information fusion and reconstruction functions can be, for example, a FusionNet model. The FusionNet model is a codec network with a network structure similar to that of the U-Net network, wherein the encoder part includes four convolution blocks, each convolution block consists of two 3*3 convolution layers, and the decoder part can consist of four transposed convolution layers.

[0088] In the following, taking the FusionNet model as an example, the reconstruction model to be trained is described in detail for the intermediate frame image reconstruction model obtained by training in the embodiments of this specification.

[0089] In an embodiment of the present specification, an image sample, a reference image sample, a first optical flow, and a first intermediate frame image can be input as input information into an intermediate frame image reconstruction model. Then, the first convolution block in the encoder of the intermediate frame image reconstruction model starts to process the above input information, and sends the first output result obtained by processing to the second convolution block. After receiving the first output result, the second convolution block will process the first output result and send the second output result obtained after processing to the third convolution block. After receiving the second output result, the third convolution block will process the second output result and send the third output result obtained after processing to the fourth convolution block. After receiving the third output result, the fourth convolution block will process the third output result, thereby training the intermediate frame image reconstruction model.

[0090] After obtaining the intermediate frame image reconstruction model, it can be further determined whether the quality of the target intermediate frame image generated based on the intermediate frame image reconstruction model meets the preset requirements, for example, whether the clarity of the target intermediate frame image generated based on the intermediate frame image reconstruction model reaches the preset clarity threshold, and whether the generated target intermediate frame image is similar to the real intermediate frame image. If it is determined that the quality of the generated target intermediate frame image does not meet the preset requirements, the reconstruction loss can be calculated based on the real intermediate frame image and the target intermediate frame image obtained by the model, and then the FusionNet model is continuously optimized and trained based on the reconstruction loss until the target intermediate frame image output by the FusionNet model meets the preset requirements, for example, the reconstructed target intermediate frame image is the same or similar to the real intermediate frame image, and the training is stopped, and the optimized and trained FusionNet model at this time is used as the intermediate frame image reconstruction model.

[0091] In one or more embodiments of this specification, considering the limitations of the FusionNet model itself, when using this model, it is usually only possible to reconstruct a natural target intermediate frame image with clearer details for a target image with lower resolution. If this model is used to reconstruct a target intermediate frame image for some high-resolution high-definition target images, some details of the reconstructed target intermediate frame image may be blurred.

[0092] In order to solve the above technical problems, in the embodiments of this specification, an improved FusionNet model is provided as an intermediate frame image reconstruction model. For the convenience of description, this improved FusionNet model is referred to as the FusionNet HD model below. Figure 5As shown, the FusionNet HD model includes a global generator network, a first local enhancer network and a second local enhancer network, wherein the network structure of the global generator network is similar to that of the U-Net network, and can be composed of a convolution front end, a transposed convolution (also called) back end, and a series of residual blocks.

[0093] Among them, the global generator network is used to extract the input information of the intermediate frame image reconstruction model to perform global feature information extraction; the first local enhancer network is used to perform detail feature extraction on the input information of the intermediate frame image reconstruction model to extract the detail feature information of the input information; the second local enhancer network is used to fuse the output results of the global generator network and the first local enhancer network to obtain fused feature information, and perform detail feature extraction on the fused feature information; the second local enhancer network and the second local enhancer network have the same network structure.

[0094] In one or more embodiments of the present specification, considering the detail feature information extracted by the first local enhancer network and the second local enhancer network, a suitable receptive field is usually required. Therefore, downsampling operations can be added in the first local enhancer network and the second local enhancer network.

[0095] It should be noted that if the number of downsampling is too many, the receptive field of the extracted detail feature information may be enlarged, and it may not be able to focus on local details; moreover, the downsampling operation itself may also lose detail information. In order to ensure that the extracted detail feature information has a suitable receptive field and that the detail information is not lost too much, in the embodiments of this specification, the number of downsampling in the first local enhancer network and the second local enhancer network can be set to 1. In this way, it can avoid the problem of too many downsampling times, which leads to an excessive expansion of the feature receptive field and the inability to focus on local details, and it can also ensure that the detail feature information has a suitable receptive field.

[0096] In the following, the method for generating intermediate frame images provided by the present application is described in detail by taking the intermediate frame image reconstruction model as the FusionNet HD model as an example. Figure 6 The figure is a flow chart of an intermediate frame image generation method provided by the present application.

[0097] In the embodiment of this specification, the target image and the reference image to be inserted can be obtained respectively. The specific implementation of obtaining the target image and the reference image can refer to the relevant description of step 202, which will not be repeated here.

[0098] For the acquired target image (assuming the target image is I0 and I1), it can be input into the intermediate flow estimation model for intermediate flow estimation to obtain the first reverse optical flow F corresponding to I0 t->0, and the second reverse optical flow F corresponding to I1 t ->1, as the intermediate stream corresponding to the target image. The specific implementation of estimating the intermediate stream by the intermediate stream estimation model can refer to the relevant content of the above step 204, which will not be repeated here.

[0099] After obtaining the intermediate stream, the intermediate stream and the target images I0 and I1 can be subjected to backward warping to obtain a backward warping result; then, the backward warping result, the optical flow and the target image are input into the initial intermediate frame image construction model to obtain the initial intermediate frame image of the target image.

[0100] Finally, the reference image, target image, optical flow, and initial intermediate frame image are input into the FusionNet HD model to obtain the target intermediate frame image.

[0101] In one or more embodiments, a reference image, a target image, an optical flow, and an initial intermediate frame image are input into a FusionNet HD model to obtain a target intermediate frame image, which specifically includes the following steps (1) to (3):

[0102] (1) Input the reference image, target image, optical flow, and initial intermediate frame image into the global generator network to obtain the first global feature information.

[0103] In one or more embodiments, considering that the global generator network is mainly used to extract global features, a larger feature receptive field is usually required for the image information to be extracted. In order to solve this problem, the embodiments of this specification may downsample the reference image, target image, optical flow, and initial intermediate frame image into the global generator network multiple times to increase the feature receptive field before obtaining the first global feature information.

[0104] It should be noted that although increasing the number of downsampling times can increase the receptive field, as the number of downsampling times continues to increase, the amount of data obtained will also become larger and larger. Subsequently, all of these data usually need to be input into the global generator network. This will cause the input parameter amount of the global generator network to be too large, thereby affecting the speed at which the global generator network extracts global features. Based on this, in an optional implementation, the number of downsampling samplings can be, for example, 1, and the downsampling sampling multiple can be, for example, 2. In this way, on the one hand, due to the downsampling, the receptive field can be increased; on the other hand, since the number of downsampling samplings is only 1, the amount of data obtained after downsampling will not be very large, which can reduce the model parameter amount to a certain extent and improve the feature extraction speed of the global generator network.

[0105] (2) Input the reference image, the target image, the optical flow, and the initial intermediate frame image into the first local enhancer network to obtain the first detail feature information.

[0106] In the embodiments of the present specification, considering that the number of downsampling in the first local enhancer network is usually 1, and that when the first local enhancer network extracts detail feature information, feature fusion is usually required, and during the feature fusion process, feature dimension matching needs to be ensured. Therefore, for a target image with multi-scale features, only the first feature vector corresponding to the first scale in the multi-scale features can be selected and input into the first local enhancer network.

[0107] (3) The first global feature information and the first detail feature information are input into the second local enhancer network for fusion processing to obtain fused feature information, and the fused feature information is extracted by the second local enhancer network to obtain a target intermediate frame image.

[0108] The FusionNet HD model provided in this specification is adopted, and a first local enhancer and a second local enhancer are added, wherein the first local enhancer network can perform local detail feature extraction processing on the input information of the intermediate frame image reconstruction model, and the second local enhancer network can supplement the detail texture in the global feature information according to the local detail features extracted by the first local enhancer network. In this way, the high-resolution global generator network can learn local detail features, and the low-resolution first local enhancer and the second local enhancer can learn global continuity, so that the reconstructed target intermediate frame image can take into account the authenticity of both local fine features and global features, thereby improving the problem of blurred details that may be caused when the target intermediate frame image is reconstructed using FusionNet.

[0109] On the other hand, when the FusionNet HD model provided in this specification is used, since the residual block in the global generator network can guide the target intermediate frame image output by the FusionNet HD model to be consistent with the real intermediate frame image, the FusionNet HD model can obtain texture information from the reference image to complete the missing texture information of the target image, so that the reconstructed target intermediate frame image can be guaranteed to be more natural.

[0110] The following describes in detail the intermediate frame image generation method provided in this specification in combination with actual scenarios.

[0111] For example, suppose there are two recorded video segments of people, A and B. Although the actions of the people in the end frame of video segment A and the start frame of video segment B are roughly similar, directly splicing video segments A and B will give people an abrupt visual experience. At this time, in order to solve the problem of abrupt visual experience, an intermediate frame image can be generated based on video segments A and B, and then interpolation processing is performed based on the generated intermediate frame image.

[0112] It is assumed that the image resolutions of the person video segment A and the person video segment B are both 1920*1088, and the shape is 1*1920*1088*3.

[0113] In one or more embodiments, it is usually necessary to perform a convolution operation on the interpolated image when generating an intermediate frame, and the convolution operation will make the image smaller and smaller, and cause certain boundary information of the image to be lost, resulting in less information on the corners and boundaries of the image. Therefore, in order to avoid this problem, in one or more embodiments, a padding operation may be performed on the interpolated image, wherein the padding operation is to fill the image with "0" before convolution, which can also be understood as adding a grid around the image block to ensure that the size of the image does not change after convolution, and that the edge data of the image can also be used, thereby better expanding the edge features of the entire image. For example, assuming that the height of a video image resolution is 1080, the height can be padded to 1088.

[0114] Then, the end frame I0 of video band A and the start frame I1 of video B are input into IFNet as input frames to obtain an F network with a shape of 1*1920*1088*2. t ->0, and then use the linear motion assumption approximation method to get the shape of F as 1*1920*1088*2 t ->1.

[0115] Afterwards, according to the two optical flows F t ->0, F t ->1 and the input frame image, perform backward warping on the input frames I0 and I1 respectively, and obtain the backward warped The shape is 1*1920*1088*3.

[0116] Next, the obtained optical flow F t ->0, original frame I0, optical flow F t->1. The original frame I1 is input into ContextNet in pairs, and two aligned pyramid features C0 and C1 are obtained respectively, each containing 4 scales: [(1*960*544*16), (1*480*272*32), (1*240*136*48), (1*120*68)].

[0117] Finally, the two optical flows F t ->0, F t ->1. Two input frames I0, I1, two aligned pyramid features, after backward warping The initial intermediate frame image is input to build the model, and the initial intermediate frame image of the input frame is obtained.

[0118] After processing the character video segment A and the character video segment B to be inserted, a reference image can be further obtained. Considering that the video segment A and the video segment B to be inserted are both character videos, when selecting a reference image, N (N is an integer greater than 1, such as N = 4) pictures of different character postures can be selected from the material video library as reference images. Assuming that the number of reference images is 4, since the reference image is an RGB image containing three channels, the shape of the reference image can be 1*1920*1088*12.

[0119] Finally, the original frames I0, I1, optical flow F t ->0, F t ->1, wrapping results The initial intermediate frame image and the reference image are input into FusionNetHD, and the target intermediate frame image is finally output.

[0120] An embodiment of the present application also provides a method for training an inter-frame reconstruction model used in the above-mentioned intermediate frame image generation method. Figure 7 A flowchart of a method for training an intermediate frame image reconstruction model provided in one or more embodiments of the present application is shown in FIG. Figure 7 As shown, the method comprises the following steps:

[0121] Step 702: Acquire an image sample and a reference image sample, wherein the reference image sample contains texture information that is missing from the image sample.

[0122] Step 704: Acquire a first optical flow of the image sample, and generate a first intermediate frame image of the image sample according to the first optical flow.

[0123] Step 706: train the reconstruction model to be trained by using the image samples, the reference image samples, the first optical flow and the first intermediate frame image to obtain an intermediate frame image reconstruction model.

[0124] Among them, the intermediate frame image reconstruction model includes a global generator network, a first local enhancer network and a second local enhancer network, wherein the global generator network is used to extract global feature information of the input information of the intermediate frame image reconstruction model; the first local enhancer network is used to extract detail feature information of the input information of the intermediate frame image reconstruction model; the second local enhancer network is used to fuse the output results of the global generator network and the first local enhancer network to obtain fused feature information, and perform detail feature extraction on the fused feature information; the second local enhancer network and the second local enhancer network have the same network structure.

[0125] The detailed implementation of steps 702 to 706 in the training method of the intermediate frame image reconstruction model is the same as the specific implementation of steps 402 to 406 in the above-mentioned intermediate frame image generation method. Please refer to the relevant content of steps 402 to 406, which will not be repeated here.

[0126] It can be seen that the training method of the intermediate frame image reconstruction model provided in the embodiments of this specification is

[0127] When training the intermediate frame image reconstruction model, the image samples, the first intermediate stream, the first intermediate frame image and the reference image samples can be used as the input of the reconstruction model to be trained. In this way, the reconstruction model to be trained can complement the missing texture information of the image samples according to the reference image samples during the training process, thereby avoiding the problem of unnatural intermediate frame images output by the intermediate frame image reconstruction model obtained by training according to the image samples.

[0128] In addition, Figure 2 Corresponding to the intermediate frame image generating method shown, the embodiment of the present application also provides an intermediate frame image generating device. Figure 8 A schematic structural diagram of an intermediate frame image generating device provided in one embodiment of the present application, the device comprising:

[0129] An acquisition module 801 is used to acquire a reference image of a target image to be inserted, wherein the reference image contains texture information missing from the target image;

[0130] A determination module 802 is used to determine the optical flow of the target image and generate an initial intermediate frame image of the target image according to the optical flow;

[0131] The fusion module 803 is used to input the reference image, the initial intermediate frame image, the optical flow and the target image into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain a target intermediate frame image of the target image.

[0132] Optionally, the determination module 802 is specifically configured to:

[0133] Performing a backward warping process on the target image according to the optical flow to obtain a backward warping process result;

[0134] The back-warping processing result, the optical flow and the target image are input into the initial intermediate frame image construction model to obtain the initial intermediate frame image of the target image.

[0135] Optionally, the determination module 802 is specifically configured to:

[0136] Get the target image to be inserted;

[0137] The target image is input into the intermediate flow estimation network for intermediate flow estimation to obtain the optical flow of the target image.

[0138] Optionally, the intermediate frame image reconstruction model includes a global generator network, a first local enhancer network, and a second local enhancer network;

[0139] The global generator network is used to extract global features from the reference image, the target image, the optical flow, and the initial intermediate frame image to obtain first global feature information;

[0140] The first local enhancer network is used to extract detail features from the reference image, the target image, the optical flow, and the initial intermediate frame image to obtain first detail feature information;

[0141] The second local enhancer network is used to fuse the first global feature information and the first detail feature information to obtain fused feature information, and to extract detail features from the fused feature information to obtain a target intermediate frame image.

[0142] Optionally, the acquisition module 801 is specifically used for:

[0143] Determine the scene information contained in the target image to be inserted;

[0144] According to the scene information, an image matching the scene information and containing the texture information missing from the target image is selected as a reference image;

[0145] or,

[0146] Get the target image;

[0147] The missing texture information in the target image is completed to obtain a completed target image, and the completed target image is determined as a reference image.

[0148] It can be seen that, when using the intermediate frame image generation device provided in the embodiment of the present specification, in the process of generating the intermediate frame image, the reference image, the initial intermediate frame image, the optical flow and the target image can be input into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image, wherein the reference image contains the texture information that is missing in the target image. In this way, in the fusion process, the missing texture information of the target image can be supplemented based on the reference image, thereby avoiding the problem of unnatural intermediate frame images generated based on the target image due to the missing texture of the target image.

[0149] In addition, Figure 7 Corresponding to the training of the intermediate frame image reconstruction model shown, an embodiment of the present application also provides a training device for the intermediate frame image reconstruction model. Fig. 9 A schematic diagram of the structure of a training device for an intermediate frame image reconstruction model provided in one embodiment of the present application, the device comprising:

[0150] The sample acquisition module 901 is used to acquire image samples and reference image samples, wherein the reference image samples contain texture information that is missing from the image samples;

[0151] An information acquisition module 902 is used to acquire a first optical flow of an image sample, and generate a first intermediate frame image of the image sample according to the first optical flow;

[0152] The model training module 903 is used to train the reconstruction model to be trained through image samples, reference image samples, the first optical flow and the first intermediate frame image to obtain the intermediate frame image reconstruction model.

[0153] It can be seen that the training device of the intermediate frame image reconstruction model provided in the embodiment of the present specification can, when training the intermediate frame image reconstruction model, use the image samples, the first intermediate stream, the first intermediate frame image and the reference image samples as the input of the reconstruction model to be trained to train the model to be trained. In this way, the reconstruction model to be trained can complement the missing texture information of the image samples according to the reference image samples during the training process, thereby avoiding the problem of unnatural intermediate frame images output by the intermediate frame image reconstruction model obtained by training according to the image samples.

[0154] Optionally, the information acquisition module 902 is specifically used to:

[0155] Performing a backward warping process on the image sample according to the first optical flow to obtain a first backward warping process result;

[0156] The first backward warping result, the first optical flow and the image sample are input into the initial intermediate frame image construction model to obtain a first intermediate frame image of the image sample.

[0157] Corresponding to the intermediate frame image generation method and the intermediate frame image reconstruction model training method provided in the above embodiments, based on the same technical concept, the embodiments of this specification also provide an electronic device, which is used to execute the above intermediate frame image generation method and / or the intermediate frame image reconstruction model training method. Fig.10 A schematic diagram of the hardware structure of an electronic device to implement each embodiment of this specification is provided. Fig.10 The electronic device 100 shown includes but is not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a user input unit 106, an interface unit 107, a memory 108, a processor 109, and a power supply 110. Those skilled in the art will appreciate that Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0158] The processor 109 is used to obtain a reference image of the target image to be inserted, where the reference image contains texture information missing from the target image;

[0159] Determine the optical flow of the target image, and generate an initial intermediate frame image of the target image according to the optical flow;

[0160] The reference image, the initial intermediate frame image, the optical flow and the target image are input into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image.

[0161] Alternatively, the processor 109 may be configured to:

[0162] Obtaining an image sample and a reference image sample, wherein the reference image sample contains texture information missing from the image sample;

[0163] Acquire a first optical flow of the image sample, and generate a first intermediate frame image of the image sample according to the first optical flow;

[0164] The reconstruction model to be trained is trained by using the image samples, the reference image samples, the first intermediate stream, and the first intermediate frame images to obtain the intermediate frame image reconstruction model.

[0165] It should be noted that the electronic device 100 provided in the embodiment of this specification can implement each process implemented by the electronic device in the above-mentioned electronic method embodiment, and to avoid repetition, it will not be described again here.

[0166] It should be understood that in the embodiments of this specification, the RF unit 101 can be used to receive and send signals during the process of sending and receiving information or making calls. Specifically, after receiving the downlink data from the upstream device, it is sent to the processor 109 for processing; in addition, the uplink data is sent to the upstream device. Generally, the RF unit 101 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the RF unit 101 can also communicate with the network and other devices through a wireless communication system.

[0167] The electronic device provides users with wireless broadband Internet access through the network module 102, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0168] The audio output unit 103 can convert the audio data received by the RF unit 101 or the network module 102 or stored in the memory 109 into an audio signal and output it as sound. Moreover, the audio output unit 103 can also provide audio output related to a specific function performed by the mobile terminal 100 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 103 includes a speaker, a buzzer, a receiver, etc.

[0169] The input unit 104 is used to receive audio or video signals. The input unit 104 may include a graphics processor (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 106. The image frame processed by the graphics processor 1041 can be stored in the memory 109 (or other storage medium) or sent via the radio frequency unit 101 or the network module 102. The microphone 1042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 101 in the case of a telephone call mode.

[0170] The interface unit 107 is an interface for connecting an external device to the electronic device 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 107 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 100 or may be used to transmit data between the electronic device 100 and an external device.

[0171] The memory 101 can be used to store software programs and various data. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 108 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0172] The processor 109 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 101, and calling data stored in the memory 108, so as to monitor the electronic device as a whole. The processor 109 may include one or more processing units; preferably, the processor 109 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 109.

[0173] The electronic device 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 110 may be logically connected to the processor 109 via a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0174] In addition, the electronic device 100 includes some functional modules not shown, which will not be described in detail here.

[0175] Preferably, an embodiment of the present specification also provides an electronic device, including a processor 109, a memory 108, and a computer program stored in the memory 108 and executable on the processor 109. When the computer program is executed by the processor 109, each process of the above-mentioned electronic method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0176] Further, based on the above method, one or more embodiments of this specification also provide a storage medium for storing computer executable instruction information. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instruction information stored in the storage medium is executed by the processor, the following process can be implemented:

[0177] Obtain a reference image of the target image to be interpolated, where the reference image contains texture information missing from the target image;

[0178] Determine the optical flow of the target image, and generate an initial intermediate frame image of the target image according to the optical flow;

[0179] The reference image, the initial intermediate frame image, the optical flow and the target image are input into the pre-trained intermediate frame image reconstruction model for feature fusion to obtain the target intermediate frame image of the target image.

[0180] Alternatively, when the computer executable instruction information stored in the storage medium is executed by the processor, the following process can be implemented:

[0181] Obtain image samples and reference image samples, wherein the reference image samples contain texture information that is missing from the image samples; obtain a first optical flow of the image samples, and generate a first intermediate frame image of the image samples based on the first optical flow; train a reconstruction model to be trained using the image samples, the reference image samples, the first intermediate flow, and the first intermediate frame images to obtain an intermediate frame image reconstruction model.

[0182] It should be noted that the embodiment of the storage medium in this specification and the embodiment of the training method of the speaker segmentation model in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding training method of the speaker segmentation model mentioned above, and the repeated parts will not be repeated.

[0183] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0184] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0185] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0186] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0187] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0188] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0192] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0193] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0194] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0195] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0196] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0197] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0198] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A method for generating an intermediate frame image, characterized in that: include: Acquire a reference image of the target image to be inserted, wherein the reference image contains all texture information missing from the target image; the reference image includes an image with the same scene information as that contained in the target image, or the reference image includes an image of a video or game in which the target image is located; Determine the optical flow of the target image, and generate an initial intermediate frame image of the target image according to the optical flow; The reference image, the initial intermediate frame image, the optical flow and the target image are input into a pre-trained intermediate frame image reconstruction model for feature fusion to obtain a target intermediate frame image of the target image.

2. The method according to claim 1, characterized in that Generating an initial intermediate frame image of the target image according to the optical flow includes: Performing a backward warping process on the target image according to the optical flow to obtain a backward warping process result; The backward warping result, the optical flow and the target image are input into an initial intermediate frame image construction model to obtain the initial intermediate frame image of the target image.

3. The method according to claim 1, characterized in that Determining the optical flow of the target image includes: Acquire the target image to be inserted into a frame; The target image is input into an intermediate flow estimation network for intermediate flow estimation to obtain the optical flow of the target image.

4. The method according to claim 1, characterized in that: The intermediate frame image reconstruction model includes a global generator network, a first local enhancer network and a second local enhancer network; The global generator network is used to perform global feature extraction on the reference image, the target image, the optical flow, and the initial intermediate frame image to obtain first global feature information; The first local enhancer network is used to extract detail features from the reference image, the target image, the optical flow, and the initial intermediate frame image to obtain first detail feature information; The second local enhancer network is used to fuse the first global feature information and the first detail feature information to obtain fused feature information, and perform detail feature extraction on the fused feature information to obtain the target intermediate frame image.

5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining a reference image of a target image to be inserted into a frame comprises: Determining scene information contained in the target image to be inserted; Selecting, according to the scene information, an image that matches the scene information and contains texture information missing from the target image as the reference image; or, Acquire the target image; The missing texture information in the target image is complemented to obtain a complemented target image, and the complemented target image is determined as the reference image.

6. The method according to claim 1, characterized in that The intermediate frame image reconstruction model is trained by the following method: Acquire an image sample and a reference image sample, wherein the reference image sample contains texture information missing from the image sample; Acquire a first optical flow of the image sample, and generate a first intermediate frame image of the image sample according to the first optical flow; The reconstruction model to be trained is trained by using the image samples, the reference image samples, the first optical flow and the first intermediate frame image to obtain the intermediate frame image reconstruction model.

7. The method according to claim 6, characterized in that The step of generating a first intermediate frame image of the image sample according to the first optical flow comprises: Performing a backward warping process on the image sample according to the first optical flow to obtain a first backward warping process result; The first backward warping result, the first optical flow, and the image sample are input into an initial intermediate frame image construction model to obtain the first intermediate frame image of the image sample.

8. An intermediate frame image generating device, characterized in that: include: An acquisition module is used to acquire a reference image of a target image to be inserted, wherein the reference image contains all texture information missing from the target image; the reference image includes an image with the same scene information as that contained in the target image, or the reference image includes an image of a video or game in which the target image is located; A determination module, used to determine the optical flow of the target image and generate an initial intermediate frame image of the target image according to the optical flow; A fusion module is used to input the reference image, the initial intermediate frame image, the optical flow and the target image into a pre-trained intermediate frame image reconstruction model for feature fusion to obtain a target intermediate frame image of the target image.

9. An electronic device, comprising: processor; as well as, A memory arranged to store computer executable instructions, the executable instructions being configured to be executed by the processor, the executable instructions comprising instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is used to store computer-executable instructions, and the executable instructions enable a computer to execute the method according to any one of claims 1 to 7.

Images