Method and device for generating intermediate frame, computer equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a method and device for generating an intermediate frame, computer equipment and a storage medium, and the method comprises the steps: carrying out the frequency domain decomposition of a front image frame and a rear image frame, and determining a plurality of front frequency domain graphs of different scales and a plurality of rear frequency domain graphs of different scales; performing feature extraction on each front frequency domain image by using a preset image generation model to determine a front feature image corresponding to the scale of each front frequency domain image, and performing feature extraction on each rear frequency domain image to determine a rear feature image corresponding to the scale of each rear frequency domain image; generating a fusion feature map corresponding to each scale; and generating a middle frame between the front image frame and the rear image frame according to the fusion feature map corresponding to the maximum scale. According to the invention, feature information such as phase information and amplitude information of each scale can be fully utilized, and the precision of the generated intermediate frame can be ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device, computer equipment and storage medium for generating an intermediate frame. Background Art

[0002] Video interpolation refers to adding one or more frames between every two frames of the original video. Through video interpolation, the display time between each frame can be shortened, and the illusion caused by the persistence of human vision can be corrected, thereby improving the smoothness of the video. In addition, in the current large-scale model training work, a large amount of pictures need to be collected as training data. The collection and cleaning of these data often requires a lot of manpower and time costs. Using video interpolation can also reduce costs and provide data several times the original data capacity in a short period of time.

[0003] The neural network video interpolation algorithm based on the optical flow method defines the interpolation algorithm as a sub-problem of the optical flow method and generates intermediate frames based on the optical flow information. This method spends a lot of energy on how to generate the optical flow information and easily ignores the characteristics of the image itself, resulting in low image generation accuracy. Summary of the invention

[0004] In view of this, the present invention provides a method, apparatus, computer device and storage medium for generating an intermediate frame to solve the problem of low precision of image interpolation.

[0005] In a first aspect, the present invention provides a method for generating an intermediate frame, comprising:

[0006] Acquire a front image frame and a rear image frame;

[0007] Performing frequency domain decomposition on the front image frame to determine a plurality of front frequency domain images of different scales; performing frequency domain decomposition on the rear image frame to determine a plurality of rear frequency domain images of different scales;

[0008] Using a preset image generation model, feature extraction is performed on each of the front frequency domain images to respectively determine a front feature image corresponding to the scale of each of the front frequency domain images, and feature extraction is performed on each of the rear frequency domain images to respectively determine a rear feature image corresponding to the scale of each of the rear frequency domain images;

[0009] Generate a fused feature map corresponding to each scale; each fused feature map except the fused feature map of the first scale is generated according to the front feature map and the rear feature map of the corresponding scale, and the fused feature map corresponding to a scale smaller than the corresponding scale;

[0010] An intermediate frame between the front image frame and the rear image frame is generated according to the fusion feature map corresponding to the maximum scale.

[0011] In some optional implementations, generating fusion feature maps corresponding to each scale includes:

[0012] Generate a fused feature map of the first scale according to the front feature map and the back feature map of the first scale;

[0013] A fused feature map of the i-th scale is generated according to the front feature map, the back feature map of the i-th scale, and the fused feature map corresponding to a scale smaller than the i-th scale; wherein i is a positive integer greater than 1, used to represent the index of the scale, and the smaller i is, the smaller the scale is.

[0014] In some optional implementations, generating a fused feature map of the i-th scale according to the front feature map, the rear feature map of the i-th scale, and the fused feature map corresponding to a scale smaller than the i-th scale includes:

[0015] Perform feature fusion on the front feature map and the back feature map of the i-th scale, as well as the front frequency domain map and the back frequency domain map of the i-th scale to generate an intermediate feature map of the i-th scale;

[0016] On the basis of the intermediate feature map of the i-th scale, the fused feature map of the i-1-th scale is fused to generate the fused feature map of the i-th scale.

[0017] In some optional implementations, the step of fusing the fused feature map of the i-1th scale on the basis of the intermediate feature map of the i-th scale to generate the fused feature map of the i-th scale includes:

[0018] Amplify the fused feature map of the i-1th scale to generate an amplified feature map corresponding to the i-th scale;

[0019] Feature fusion is performed on the intermediate feature map and the enlarged feature map of the i-th scale to generate a fused feature map of the i-th scale.

[0020] In some optional embodiments, the image generation model includes: a front feature extraction unit, a rear feature extraction unit and a fusion unit; the number of the front feature extraction unit, the rear feature extraction unit and the fusion unit is n, and n is the number of scales of frequency domain decomposition;

[0021] The i-th front feature extraction unit is used to extract features from the i-th scale front frequency domain image to determine the i-th scale front feature image;

[0022] The i-th post-feature extraction unit is used to extract features from the i-th scale post-frequency domain image to determine the i-th scale post-feature image;

[0023] The i-th fusion unit is used to generate a fused feature map of the i-th scale according to the fused feature map of the i-1-th scale, the front feature map of the i-th scale, and the back feature map of the i-th scale.

[0024] In some optional embodiments, the front feature extraction unit and the rear feature extraction unit both include a feature extraction subunit;

[0025] The feature extraction subunit is used to perform feature extraction on the input data of the feature extraction subunit; the corresponding feature extraction unit is used to combine the feature extraction result and the input data to generate output data of the corresponding feature extraction unit.

[0026] In some optional embodiments, the fusion unit includes: a first fusion subunit and a second fusion subunit;

[0027] The first fusion subunit of the i-th fusion unit is used to perform feature fusion on the front feature map and the rear feature map of the i-th scale, and the front frequency domain map and the rear frequency domain map of the i-th scale to generate an intermediate feature map of the i-th scale;

[0028] The second fusion subunit of the i-th fusion unit is used to enlarge the fused feature map of the i-1-th scale to generate an enlarged feature map corresponding to the i-th scale; perform feature fusion on the intermediate feature map and the enlarged feature map of the i-th scale to generate a fused feature map of the i-th scale; wherein the fused feature map of the i-1-th scale is generated by the i-1-th fusion unit.

[0029] In a second aspect, the present invention provides a device for generating an intermediate frame, comprising:

[0030] An acquisition module, used for acquiring a front image frame and a rear image frame;

[0031] A decomposition module, configured to perform frequency domain decomposition on the front image frame to determine a plurality of front frequency domain images of different scales; and perform frequency domain decomposition on the rear image frame to determine a plurality of rear frequency domain images of different scales;

[0032] A processing module, used to use a preset image generation model to perform feature extraction on each of the front frequency domain images to determine a front feature image corresponding to the scale of each of the front frequency domain images, and to perform feature extraction on each of the rear frequency domain images to determine a rear feature image corresponding to the scale of each of the rear frequency domain images; generate a fused feature image corresponding to each scale; each fused feature image except the fused feature image of the first scale is generated based on the front feature image, the rear feature image, and the fused feature image corresponding to a scale smaller than the corresponding scale of the corresponding scale;

[0033] A generation module is used to generate an intermediate frame between the front image frame and the rear image frame according to the fusion feature map corresponding to the maximum scale.

[0034] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for generating intermediate frames according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for generating an intermediate frame according to the first aspect or any corresponding embodiment thereof.

[0036] The present invention performs frequency domain decomposition on the original two images in the frequency domain, decomposing the images into frequency domain graphs of multiple scales; and, based on the image generation model, the frequency domain graphs of each scale after decomposition are gradually fused in order from small to large scales to form fusion feature graphs of each scale, so that the fusion feature graph of the largest scale can be used to generate an intermediate frame. This solution can make full use of feature information such as phase information and amplitude information of each scale, and can ensure the accuracy of the generated intermediate frame. In addition, the overall image generation model is an encoder architecture, that is, a decoder may not be provided, and the efficiency of image generation can also be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a schematic flow chart of a method for generating an intermediate frame according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a process of generating an intermediate frame according to an embodiment of the present invention;

[0040] Figure 3 is a flowchart of another method for generating an intermediate frame according to an embodiment of the present invention;

[0041] Figure 4 is a structural schematic diagram of an image generation model according to an embodiment of the present invention;

[0042] Figure 5is a structural schematic diagram of a feature extraction unit according to an embodiment of the present invention;

[0043] Figure 6 is a structural schematic diagram of an i-th fusion unit according to an embodiment of the present invention;

[0044] Figure 7 is a structural block diagram of an apparatus for generating an intermediate frame according to an embodiment of the present invention;

[0045] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] Implicit motion estimation focuses on the changes of fixed pixels in the image in consecutive frames. The simplest case of such change information is brightness change, which is easy to obtain but not easy to process later. Phase-based motion estimation methods (also called phase methods for short) attempt to decompose the image into different frequency components, and then focus on the phase change of each point on a certain frequency component.

[0048] The implementation of the phase-based video interpolation algorithm introduces the image pyramid method to preprocess the image, and integrates the image pyramid extraction information into the feature extraction process as supplementary information for the interpolation algorithm. Specifically, the phase information can be integrated into the neural network interpolation algorithm based on the convolutional neural network (CNN) to achieve the prediction of the intermediate frame. However, these interpolation algorithms have low utilization of phase information. The image restoration accuracy is low.

[0049] The method for generating an intermediate frame provided in this embodiment decomposes two images in the frequency domain, and based on the image generation model, gradually fuses the images of each scale after decomposition from a small scale to form a large-scale image, thereby generating an intermediate frame between the two images; this method can make full use of the phase information of each scale and can ensure the image restoration accuracy.

[0050] According to an embodiment of the present invention, a method embodiment for generating an intermediate frame is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] In this embodiment, a method for generating an intermediate frame is provided, which can be applied to a mobile terminal, a server, etc. Figure 1 is a flowchart of a method for generating an intermediate frame according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps.

[0052] Step S101, obtaining a front image frame and a rear image frame.

[0053] In this embodiment, two frames of images, namely a front image frame and a rear image frame, can be taken from the video to generate an intermediate frame between the front image frame and the rear image frame, wherein the front image frame and the rear image frame are generally two consecutive frames of images.

[0054] Step S102, performing frequency domain decomposition on the front image frame to determine a plurality of front frequency domain images of different scales; performing frequency domain decomposition on the rear image frame to determine a plurality of rear frequency domain images of different scales.

[0055] In this embodiment, in the frequency domain, the acquired front image frame and rear image frame are respectively subjected to frequency domain decomposition to generate a plurality of frequency domain graphs of different scales, namely, the front frequency domain graph and the rear frequency domain graph. The frequency domain decomposition of the front image frame and the rear image frame is respectively performed based on the same frequency domain decomposition method, and the scale of the decomposed front frequency domain graph corresponds to the scale of the rear frequency domain graph.

[0056] For example, image Fourier transform can be used to perform frequency domain decomposition on the front image frame and the back image frame respectively; generally, in order to extract the phase information in the image, decomposition can be performed based on the Complex Steerable Pyramid (CSP); the Complex Steerable Pyramid is a multi-scale, multi-directional decomposition technology that can decompose to obtain multi-scale information, namely, the frequency domain map.

[0057] Moreover, the complex steerable pyramid can obtain information in multiple directions, such as directions of 0°, 45°, 90°, etc.; in this embodiment, the front frequency domain graph and the rear frequency domain graph can specifically include feature information in multiple directions.

[0058] Specifically, the complex steerable pyramid decomposition process requires setting two key parameters: the pyramid inter-layer scaling factor A and the number of frequency band directions n. For example, n=4, that is, each front frequency domain image and rear frequency domain image contains feature information in four directions. In addition, A=2 can be set, that is, the frequency domain is decomposed by octave, and a pyramid layer with a resolution reduced by 0.5 times can be obtained in the spatial domain. In addition, the frequency band can be further subdivided, and multiple sub-octaves can be divided within each octave. For example, That is, each octave contains 2 sub-octaves. Dividing the sub-octaves into finer ones can enhance the connection between the pyramid layers and increase the support range of the equivalent filter in the spatial domain, so that larger motion can be processed.

[0059] Step S103, using a preset image generation model, feature extraction is performed on each front frequency domain image to respectively determine a front feature image corresponding to the scale of each front frequency domain image, and feature extraction is performed on each rear frequency domain image to respectively determine a rear feature image corresponding to the scale of each rear frequency domain image.

[0060] In this embodiment, a model for generating intermediate frames, namely, an image generation model, is pre-set. The image generation model has a feature extraction function.

[0061] Specifically, the image generation model can extract features from each front frequency domain image of different scales, thereby generating multiple feature maps, namely, front feature maps; and each front feature map also has a corresponding scale, namely, multiple front feature maps of different scales can be generated, and the scale of each front feature map corresponds to the scale of the corresponding front frequency domain map. Similarly, based on the image generation model, feature extraction is performed on each rear frequency domain image, and multiple front feature maps of different scales can also be generated, and the scale of each rear feature map corresponds to the scale of the corresponding rear frequency domain image.

[0062] Among them, the feature extraction process can be mainly used to extract the phase information of the image, that is, the front feature map and the rear feature map respectively contain phase information at different scales; or, the feature extraction process can also be used to extract the amplitude information of the image. Combining the amplitude information and the phase information is conducive to generating an intermediate frame with higher algorithm accuracy.

[0063] Figure 2 FIG. 2 shows a schematic diagram of the process of generating an intermediate frame. Figure 2 As shown, the front image frame and the rear image frame are respectively decomposed into frequency domain images of three scales (i.e., n=3), and, in order from small to large scale, the three front frequency domain images are respectively marked as P1, P2, and P3, and the three rear frequency domain images are respectively marked as Q1, Q2, and Q3.

[0064] Step S104, generating a fused feature map corresponding to each scale; each fused feature map except the fused feature map of the first scale is generated based on the front feature map and the back feature map of the corresponding scale, and the fused feature map corresponding to a scale smaller than the corresponding scale.

[0065] In this embodiment, the image generation model also has a feature fusion function.

[0066] Specifically, each front feature map and back feature map corresponds to a corresponding scale, and the scale of a front feature map is the same as that of a back feature map. Figure 2 As shown in FIG. 1 , the scale of the front frequency domain image P1 is the same as that of the rear frequency domain image Q1, the scale of the front frequency domain image P2 is the same as that of the rear frequency domain image Q2, and the scale of the front frequency domain image P3 is the same as that of the rear frequency domain image Q3. Correspondingly, the feature maps generated based on the frequency domain images of each scale also correspond to a certain scale.

[0067] Based on the front feature map and the back feature map of the same scale, feature fusion can be performed; and when performing feature fusion, a smaller scale fusion feature map can also be referred to. Figure 1 Generally, it is a fusion feature map of an adjacent and smaller scale. Among them, the fusion feature map of the smallest scale (i.e., the fusion feature map of the first scale) can be directly generated by performing feature fusion based on the previous feature map and the subsequent feature map of the smallest scale.

[0068] like Figure 2 As shown in the figure, the front frequency domain map P1 and the back frequency domain map Q1 correspond to the minimum scale, and the feature map generated by the front frequency domain map P1 and the back frequency domain map Q1 can be directly fused to generate the fused feature map M1 of the minimum scale; for the feature fusion of the next scale, the feature map generated by the front frequency domain map P2 and the back frequency domain map Q2 of this scale, and the fused feature map M1 of the previous scale can be combined to generate the fused feature map of this scale, i.e., M2. Similarly, for the next scale, i.e., the maximum scale, the feature map generated by the front frequency domain map P3 and the back frequency domain map Q3 based on the maximum scale and the fused feature map M2 of the previous scale can be combined to comprehensively generate the fused feature map M3 of the maximum scale.

[0069] Step S105: generating an intermediate frame between the front image frame and the rear image frame according to the fusion feature map corresponding to the maximum scale.

[0070] In this embodiment, the fused feature map corresponding to the maximum scale is fused with relevant information of the front feature map and the rear feature map of the maximum scale, so the features of the intermediate frame between the two frames of images can be determined, and the intermediate frame located between the front image frame and the rear image frame is generated based on the fused feature map corresponding to the maximum scale; Figure 2As shown, an intermediate frame F can be generated. In addition, the fused feature maps of multiple scales are obtained in sequence based on the previous fused feature map, so that the feature information of multiple scales can be combined to generate an intermediate frame with higher accuracy.

[0071] The intermediate frame can be reconstructed by pyramid collapsing. For example, the reconstructed intermediate frame is: in, Indicates the intermediate frame, represents the processed frequency component, G ω,θ represents the corresponding transfer function.

[0072] It can be understood that the image generation model can also integrate the frequency domain decomposition function and the reconstruction function of the intermediate frame based on the feature map. For example, the image generation model also integrates the complex steerable pyramid decomposition algorithm and the pyramid collapse reconstruction algorithm, so that the image generation model can be used to directly generate the intermediate frame between two image frames.

[0073] The method for generating an intermediate frame provided in this embodiment performs frequency domain decomposition on the original two images in the frequency domain, decomposing the images into frequency domain graphs of multiple scales; and, based on the image generation model, the frequency domain graphs of each scale after decomposition are gradually fused in order from small to large scales to form fusion feature graphs of each scale, so that the intermediate frame can be generated using the fusion feature graph of the largest scale. This method can make full use of feature information such as phase information and amplitude information of each scale, and can ensure the accuracy of the generated intermediate frame. In addition, the overall image generation model is an encoder architecture, that is, a decoder may not be provided, and the efficiency of image generation can also be guaranteed.

[0074] In this embodiment, another method for generating an intermediate frame is provided, which can be applied to a mobile terminal, a server, etc. Figure 3 is a flowchart of a method for generating an intermediate frame according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps.

[0075] Step S301, obtaining a front image frame and a rear image frame.

[0076] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0077] Step S302, performing frequency domain decomposition on the front image frame to determine a plurality of front frequency domain images of different scales; performing frequency domain decomposition on the rear image frame to determine a plurality of rear frequency domain images of different scales.

[0078] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0079] Step S303, using a preset image generation model, feature extraction is performed on each front frequency domain image to respectively determine a front feature image corresponding to the scale of each front frequency domain image, and feature extraction is performed on each rear frequency domain image to respectively determine a rear feature image corresponding to the scale of each rear frequency domain image.

[0080] For details, please see Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0081] In some optional embodiments, the image generation model includes a feature extraction unit with a feature extraction function. Specifically, the image generation model includes: a front feature extraction unit, a rear feature extraction unit and a fusion unit; the number of the front feature extraction unit, the rear feature extraction unit and the fusion unit is n, and n is the number of scales of frequency domain decomposition.

[0082] The i-th front feature extraction unit is used to extract features from the i-th scale front frequency domain image to determine the i-th scale front feature image.

[0083] The i-th post-feature extraction unit is used to extract features from the i-th scale post-frequency domain image to determine the i-th scale post-feature image.

[0084] The i-th fusion unit is used to generate a fused feature map of the i-th scale according to the fused feature map of the i-1-th scale, the front feature map of the i-th scale, and the back feature map of the i-th scale.

[0085] In this embodiment, i is an index of the scale, and the smaller i is, the smaller the scale is.

[0086] by Figure 2 Taking the process shown in FIG. 1 as an example, the number of scales of frequency domain decomposition n=3, so the image generation model can include 3 front feature extraction units, 3 rear feature extraction units and 3 fusion units. An implementation method of the image generation model can be found in Figure 4 shown.

[0087] like Figure 4 As shown, the first front feature extraction unit 501 is used to perform feature extraction on the front frequency domain image P1 of the minimum scale to generate the front feature image a1 of the minimum scale; the second front feature extraction unit 502 is used to perform feature extraction on the front frequency domain image P2 of the intermediate scale to generate the front feature image a2 of the intermediate scale; the third front feature extraction unit 503 is used to perform feature extraction on the front frequency domain image P3 of the maximum scale to generate the front feature image a3 of the maximum scale.

[0088] Similarly, the first post-feature extraction unit 511 is used to perform feature extraction on the minimum-scale post-frequency domain image Q1 to generate the minimum-scale post-feature image b1; the second post-feature extraction unit 522 is used to perform feature extraction on the intermediate-scale post-frequency domain image Q2 to generate the intermediate-scale post-feature image b2; the third post-feature extraction unit 533 is used to perform feature extraction on the maximum-scale post-frequency domain image Q3 to generate the maximum-scale post-feature image b3.

[0089] In this embodiment, the scales corresponding to the index i are arranged in ascending order, that is, the first scale corresponds to the smallest scale, the second scale corresponds to the middle scale, and the third scale corresponds to the largest scale.

[0090] For the first fusion unit 521, since there is no fused feature map with a smaller scale, that is, the fused feature map of the i-1th scale does not exist, the first fusion unit 521 can only generate a fused feature map of the first scale, that is, the fused feature map M1 of the minimum scale, for the front feature map of the first scale and the back feature map of the first scale.

[0091] For the second fusion unit 522, on the basis of the fusion feature map of the first scale (i.e., M1), the front feature map a2 of the second scale and the back feature map b2 of the second scale can be combined to comprehensively generate the fusion feature map of the second scale, i.e., the fusion feature map M2 of the intermediate scale. Similarly, for the third fusion unit 523, on the basis of the fusion feature map of the second scale (i.e., M2), the front feature map a3 of the third scale and the back feature map b3 of the third scale can be combined to comprehensively generate the fusion feature map of the third scale, i.e., the fusion feature map M3 of the maximum scale.

[0092] In this embodiment, each feature extraction unit (including the front feature extraction unit and the rear feature extraction unit) of the image generation model is mainly used for feature extraction, and each fusion unit is used to realize decoding of each scale layer by layer. The image generation model has only one decoder architecture.

[0093] Optionally, both the front feature extraction unit and the rear feature extraction unit include a feature extraction subunit; the feature extraction subunit is used to perform feature extraction on the input data of the feature extraction subunit; and the corresponding feature extraction unit is used to combine the feature extraction result and the input data to generate output data of the corresponding feature extraction unit.

[0094] In this embodiment, for each feature extraction unit such as the front feature extraction unit and the back feature extraction unit, a residual structure can be adopted to avoid problems such as gradient explosion during the training process. Figure 5 A structural schematic diagram of a feature extraction unit is shown. It can be understood that the feature extraction unit can be a front feature extraction unit or a rear feature extraction unit.

[0095] Among them, the input data of the feature extraction subunit is the corresponding frequency domain diagram; after the feature extraction subunit extracts the features of the input data, the corresponding feature extraction results can be obtained, and then the original input data and the feature extraction results obtained after processing are combined to comprehensively generate the output data of the feature extraction unit.

[0096] For example, Figure 4 As shown in the first front feature extraction unit 503, the input data of its feature extraction subunit is the front frequency domain image of the first scale, and the output data of the feature extraction unit is the front feature image a1 of the first scale.

[0097] In this embodiment, the feature extraction subunit can be implemented based on CNN (convolutional neural network) and LSTM (long short-term memory network); to improve the effect, this embodiment adopts transformer architecture to construct the feature extraction subunit.

[0098] Step S304, generating a fused feature map corresponding to each scale; each fused feature map except the fused feature map of the first scale is generated according to the front feature map and the rear feature map of the corresponding scale, and the fused feature map corresponding to a scale smaller than the corresponding scale.

[0099] In this embodiment, the process of generating the fused feature map of the minimum scale is slightly different from that of generating the fused feature maps of other scales. Among them, the fused feature map of the first scale is the fused feature map of the minimum scale, which is generated based on the front feature map and the rear feature map of the first scale. The fused feature map of the i-th scale other than the fused feature map of the first scale is generated based on the front feature map, the rear feature map of the i-th scale, and the fused feature map corresponding to the scale smaller than the i-th scale; wherein i is a positive integer greater than 1, which is used to represent the index of the scale, and the smaller i is, the smaller the scale is.

[0100] Specifically, the above step S304 "generating fusion feature maps corresponding to each scale" may include steps S3041 to S3042.

[0101] Step S3041, generating a fused feature map of the first scale according to the front feature map and the rear feature map of the first scale.

[0102] In this embodiment, Pi represents the front frequency domain map of the i-th scale, Qi represents the rear frequency domain map of the i-th scale, Mi represents the fused feature map of the i-th scale, where i is a positive integer and its value can be 1; and, ai represents the front feature map of the i-th scale, bi represents the rear feature map of the i-th scale, and ci represents the intermediate feature map of the i-th scale. Based on the front feature map ai of the i-th scale and the rear feature map bi of the i-th scale, the intermediate feature map ci of the i-th scale can be generated.

[0103] Among them, since there is no scale smaller than the minimum scale, the intermediate feature map of the minimum scale can be directly used as the fused feature map of the minimum scale, that is, the intermediate feature map c1 of the first scale is used as the fused feature map M1 of the first scale. That is, the intermediate feature map c1 of the first scale generated based on the front feature map a1 of the first scale and the back feature map b1 of the first scale can be directly used as the fused feature map of the first scale.

[0104] In some optional implementations, the process of generating the intermediate feature map may specifically include: performing feature fusion on the front feature map and the rear feature map of the i-th scale, and the front frequency domain map and the rear frequency domain map of the i-th scale to generate the intermediate feature map of the i-th scale.

[0105] In this embodiment, when generating the intermediate feature map ci of the i-th scale, in addition to using the front feature map ai and the back feature map bi of the i-th scale, the original front frequency domain map Pi and the back frequency domain map Qi of the i-th scale are also used as input data. By combining the original data and the data after feature extraction processing, the features corresponding to the intermediate frames between the front frequency domain map and the back frequency domain map can be more accurately identified, which is conducive to generating a more accurate intermediate feature map.

[0106] Step S3042, generating a fused feature map of the i-th scale according to the front feature map, the back feature map of the i-th scale, and the fused feature map corresponding to a scale smaller than the i-th scale; wherein i is a positive integer greater than 1, used to represent the index of the scale, and the smaller i is, the smaller the scale is.

[0107] In this embodiment, for scales other than the first scale (i.e., the minimum scale), that is, when i is 2, 3, etc., when generating a fused feature map of the i-th scale, on the basis of the intermediate feature map of the i-th scale, a fused feature map corresponding to a scale smaller than the i-th scale is further fused, that is, a fused feature map of the i-1-th scale, and finally a fused feature map of the i-th scale is generated.

[0108] For example, the front feature map and the back feature map of the i-th scale, and the front frequency domain map and the back frequency domain map of the i-th scale can be feature fused to generate an intermediate feature map of the i-th scale; then, based on the intermediate feature map of the i-th scale, the fused feature map of the i-1-th scale is fused to generate a fused feature map of the i-th scale.

[0109] Optionally, the above step of “on the basis of the intermediate feature map of the i-th scale, fusing the fused feature map of the i-1th scale to generate a fused feature map of the i-th scale” may specifically include: enlarging the fused feature map of the i-1th scale to generate an enlarged feature map corresponding to the i-th scale; performing feature fusion on the intermediate feature map and the enlarged feature map of the i-th scale to generate a fused feature map of the i-th scale.

[0110] In this embodiment, since each fused feature map corresponds to a certain size, it cannot be directly fused with the intermediate feature maps of different scales; in order to be able to fuse the fused feature map of a smaller scale, the fused feature map of the smaller scale (i.e., the fused feature map of the i-1th scale) is first enlarged to generate a feature map of a larger scale, i.e., the enlarged feature map of the i-th scale, and then the enlarged feature map and the intermediate feature map of the same scale can be fused, that is, the intermediate feature map of the i-th scale and the enlarged feature map of the i-th scale are feature fused to generate the fused feature map Mi of the i-th scale.

[0111] In some optional embodiments, the fusion unit may include: a first fusion subunit and a second fusion subunit. Figure 6 A structural schematic diagram of the i-th fusion unit is shown.

[0112] like Figure 6 As shown, the first fusion subunit of the i-th fusion unit is used to perform feature fusion on the front feature map ai and the back feature map bi of the i-th scale, and the front frequency domain map Pi and the back frequency domain map Qi of the i-th scale to generate the intermediate feature map ci of the i-th scale. For example, the above step S3041 can be implemented based on the first fusion subunit.

[0113] The second fusion subunit of the i-th fusion unit is used to enlarge the fused feature map Mi-1 of the i-1th scale to generate an enlarged feature map di corresponding to the i-th scale; perform feature fusion on the intermediate feature map ci and the enlarged feature map di of the i-th scale to generate a fused feature map Mi of the i-th scale; wherein the fused feature map of the i-1th scale is generated by the i-1th fusion unit. For example, the above step S3042 can be implemented based on the second fusion subunit.

[0114] In this embodiment, Figure 6As shown, for the front frequency domain map Pi and the back frequency domain map Qi of the i-th scale, based on the corresponding feature extraction units, the corresponding feature maps, namely the front feature map ai and the back feature map bi, can be generated; and, combined with the original front frequency domain map Pi and the back frequency domain map Qi, the intermediate feature map ci of the i-th scale can be obtained.

[0115] The second fusion subunit includes a unit capable of implementing an amplification process, and the amplification process may specifically be an upsampling process. After the i-1th fusion unit generates a fused feature map of the i-1th scale, the second fusion subunit of the i-th fusion unit may amplify the fused feature map of the i-1th scale to generate a feature map corresponding to the i-th scale, i.e., the amplified feature map di, and then feature fusion may be performed on the intermediate feature map ci and the amplified feature map di of the same scale, and finally the fused feature map Mi of the i-th scale is generated.

[0116] It can be understood that since there is no scale smaller than the minimum scale, the first fusion unit 521 does not need to be fused with fusion feature maps of other scales, so the first fusion unit 521 may not be provided with a second fusion sub-unit.

[0117] In this embodiment, the partial structure of the first fusion subunit and the second fusion subunit that implements the amplification processing function can be implemented based on CNN, LSTM, etc.; to improve the effect, this embodiment adopts the transformer architecture to construct the first fusion subunit and the second fusion subunit.

[0118] In this embodiment, the weight parameters in the front feature extraction unit, the rear feature extraction unit, the fusion unit, etc. can be obtained through model training.

[0119] Among them, the loss function used when training the image generation model is as follows:

[0120]

[0121] in, L phase represents the loss function, ω represents the center frequency, θ represents the frequency direction, is the phase information difference under the corresponding parameters, is the expected value, is the actual value predicted by the model, and atan2 is the four-quadrant inverse tangent function.

[0122] Step S305: generating an intermediate frame between the front image frame and the rear image frame according to the fusion feature map corresponding to the maximum scale.

[0123] For details, please see Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0124] The method for generating intermediate frames provided in this embodiment can make full use of feature information such as phase information and amplitude information at various scales to generate intermediate frames, and can ensure the accuracy of the generated intermediate frames. First, intermediate feature maps of various scales are generated, and then, based on the intermediate feature maps, fused feature maps of smaller scales can be fused to generate fused feature maps of various scales in a recursive manner, and then, based on the fused feature maps of the largest scale, the required intermediate frames can be obtained more accurately. In addition, the various encoding units of the image generation model form an encoder architecture, that is, a decoder may not be provided, which can ensure the efficiency of image generation.

[0125] In this embodiment, a device for generating an intermediate frame is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0126] This embodiment provides a device for generating an intermediate frame, such as Figure 7 As shown, including:

[0127] An acquisition module 701 is used to acquire a front image frame and a rear image frame;

[0128] The decomposition module 702 is used to perform frequency domain decomposition on the front image frame to determine a plurality of front frequency domain images of different scales; perform frequency domain decomposition on the rear image frame to determine a plurality of rear frequency domain images of different scales;

[0129] The processing module 703 is used to use a preset image generation model to perform feature extraction on each of the front frequency domain images to determine a front feature image corresponding to the scale of each of the front frequency domain images, and to perform feature extraction on each of the rear frequency domain images to determine a rear feature image corresponding to the scale of each of the rear frequency domain images; generate a fused feature image corresponding to each scale; each fused feature image except the fused feature image of the first scale is generated based on the front feature image and the rear feature image of the corresponding scale, and the fused feature image corresponding to a scale smaller than the corresponding scale;

[0130] The generating module 704 is used to generate an intermediate frame between the front image frame and the rear image frame according to the fusion feature map corresponding to the maximum scale.

[0131] In some optional implementations, the processing module 703 generates fusion feature maps corresponding to each scale including:

[0132] Generate a fused feature map of the first scale according to the front feature map and the back feature map of the first scale;

[0133] A fused feature map of the i-th scale is generated according to the front feature map, the back feature map of the i-th scale, and the fused feature map corresponding to a scale smaller than the i-th scale; wherein i is a positive integer greater than 1, used to represent the index of the scale, and the smaller i is, the smaller the scale is.

[0134] In some optional implementations, the processing module 703 generates a fused feature map of the i-th scale according to the front feature map, the rear feature map of the i-th scale, and the fused feature map corresponding to a scale smaller than the i-th scale, including:

[0135] Perform feature fusion on the front feature map and the back feature map of the i-th scale, as well as the front frequency domain map and the back frequency domain map of the i-th scale to generate an intermediate feature map of the i-th scale;

[0136] On the basis of the intermediate feature map of the i-th scale, the fused feature map of the i-1-th scale is fused to generate the fused feature map of the i-th scale.

[0137] In some optional implementations, the processing module 703 fuses the fused feature map of the i-1th scale based on the intermediate feature map of the i-th scale to generate the fused feature map of the i-th scale, including:

[0138] Amplify the fused feature map of the i-1th scale to generate an amplified feature map corresponding to the i-th scale;

[0139] Feature fusion is performed on the intermediate feature map and the enlarged feature map of the i-th scale to generate a fused feature map of the i-th scale.

[0140] In some optional embodiments, the image generation model includes: a front feature extraction unit, a rear feature extraction unit and a fusion unit; the number of the front feature extraction unit, the rear feature extraction unit and the fusion unit is n, and n is the number of scales of frequency domain decomposition;

[0141] The i-th front feature extraction unit is used to extract features from the i-th scale front frequency domain image to determine the i-th scale front feature image;

[0142] The i-th post-feature extraction unit is used to extract features from the i-th scale post-frequency domain image to determine the i-th scale post-feature image;

[0143] The i-th fusion unit is used to generate a fused feature map of the i-th scale according to the fused feature map of the i-1-th scale, the front feature map of the i-th scale, and the back feature map of the i-th scale.

[0144] In some optional embodiments, the front feature extraction unit and the rear feature extraction unit both include a feature extraction subunit;

[0145] The feature extraction subunit is used to perform feature extraction on the input data of the feature extraction subunit; the corresponding feature extraction unit is used to combine the feature extraction result and the input data to generate output data of the corresponding feature extraction unit.

[0146] In some optional embodiments, the fusion unit includes: a first fusion subunit and a second fusion subunit;

[0147] The first fusion subunit of the i-th fusion unit is used to perform feature fusion on the front feature map and the rear feature map of the i-th scale, and the front frequency domain map and the rear frequency domain map of the i-th scale to generate an intermediate feature map of the i-th scale;

[0148] The second fusion subunit of the i-th fusion unit is used to enlarge the fused feature map of the i-1-th scale to generate an enlarged feature map corresponding to the i-th scale; perform feature fusion on the intermediate feature map and the enlarged feature map of the i-th scale to generate a fused feature map of the i-th scale; wherein the fused feature map of the i-1-th scale is generated by the i-1-th fusion unit.

[0149] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0150] The device for generating intermediate frames in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, including a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0151] The embodiment of the present invention also provides a computer device having the above Figure 7 The device for generating intermediate frames is shown.

[0152] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0153] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0154] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0155] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0157] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0158] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0159] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0160] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0161] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations should all be included in the protection scope of the present invention.

Claims

1. A method for generating an intermediate frame, characterized in that: The method comprises: Acquire a front image frame and a rear image frame; Performing frequency domain decomposition on the front image frame to determine a plurality of front frequency domain images of different scales; performing frequency domain decomposition on the rear image frame to determine a plurality of rear frequency domain images of different scales; Using a preset image generation model, feature extraction is performed on each of the front frequency domain images to respectively determine a front feature image corresponding to the scale of each of the front frequency domain images, and feature extraction is performed on each of the rear frequency domain images to respectively determine a rear feature image corresponding to the scale of each of the rear frequency domain images; Generate a fused feature map corresponding to each scale; each fused feature map except the fused feature map of the first scale is generated according to the front feature map and the rear feature map of the corresponding scale, and the fused feature map corresponding to a scale smaller than the corresponding scale; An intermediate frame between the front image frame and the rear image frame is generated according to the fusion feature map corresponding to the maximum scale.

2. The method according to claim 1, characterized in that Generating fusion feature maps corresponding to each scale includes: Generate a fused feature map of the first scale according to the front feature map and the back feature map of the first scale; A fused feature map of the i-th scale is generated according to the front feature map, the back feature map of the i-th scale, and the fused feature map corresponding to a scale smaller than the i-th scale; wherein i is a positive integer greater than 1, used to represent the index of the scale, and the smaller i is, the smaller the scale is.

3. The method according to claim 2, characterized in that The generating of the fused feature map of the i-th scale according to the front feature map, the rear feature map of the i-th scale, and the fused feature map corresponding to a scale smaller than the i-th scale includes: Perform feature fusion on the front feature map and the back feature map of the i-th scale, as well as the front frequency domain map and the back frequency domain map of the i-th scale to generate an intermediate feature map of the i-th scale; On the basis of the intermediate feature map of the i-th scale, the fused feature map of the i-1-th scale is fused to generate the fused feature map of the i-th scale.

4. The method according to claim 3, characterized in that The step of fusing the fused feature map of the i-1th scale on the basis of the intermediate feature map of the i-th scale to generate the fused feature map of the i-th scale includes: Amplify the fused feature map of the i-1th scale to generate an amplified feature map corresponding to the i-th scale; Feature fusion is performed on the intermediate feature map and the enlarged feature map of the i-th scale to generate a fused feature map of the i-th scale.

5. The method according to claim 2, characterized in that: The image generation model comprises: a front feature extraction unit, a rear feature extraction unit and a fusion unit; the number of the front feature extraction unit, the rear feature extraction unit and the fusion unit is n, and n is the number of scales of frequency domain decomposition; The i-th front feature extraction unit is used to extract features from the i-th scale front frequency domain image to determine the i-th scale front feature image; The i-th post-feature extraction unit is used to extract features from the i-th scale post-frequency domain image to determine the i-th scale post-feature image; The i-th fusion unit is used to generate a fused feature map of the i-th scale according to the fused feature map of the i-1-th scale, the front feature map of the i-th scale, and the back feature map of the i-th scale.

6. The method according to claim 5, characterized in that The front feature extraction unit and the rear feature extraction unit both include a feature extraction subunit; The feature extraction subunit is used to perform feature extraction on the input data of the feature extraction subunit; the corresponding feature extraction unit is used to combine the feature extraction result and the input data to generate output data of the corresponding feature extraction unit.

7. The method according to claim 5, characterized in that The fusion unit comprises: a first fusion subunit and a second fusion subunit; The first fusion subunit of the i-th fusion unit is used to perform feature fusion on the front feature map and the rear feature map of the i-th scale, and the front frequency domain map and the rear frequency domain map of the i-th scale to generate an intermediate feature map of the i-th scale; The second fusion subunit of the i-th fusion unit is used to enlarge the fused feature map of the i-1-th scale to generate an enlarged feature map corresponding to the i-th scale; perform feature fusion on the intermediate feature map and the enlarged feature map of the i-th scale to generate a fused feature map of the i-th scale; wherein the fused feature map of the i-1-th scale is generated by the i-1-th fusion unit.

8. A device for generating an intermediate frame, characterized in that: The device comprises: An acquisition module, used for acquiring a front image frame and a rear image frame; A decomposition module, configured to perform frequency domain decomposition on the front image frame to determine a plurality of front frequency domain images of different scales; and perform frequency domain decomposition on the rear image frame to determine a plurality of rear frequency domain images of different scales; A processing module, used to use a preset image generation model to perform feature extraction on each of the front frequency domain images to determine a front feature image corresponding to the scale of each of the front frequency domain images, and to perform feature extraction on each of the rear frequency domain images to determine a rear feature image corresponding to the scale of each of the rear frequency domain images; generate a fused feature image corresponding to each scale; each fused feature image except the fused feature image of the first scale is generated based on the front feature image, the rear feature image, and the fused feature image corresponding to a scale smaller than the corresponding scale of the corresponding scale; A generation module is used to generate an intermediate frame between the front image frame and the rear image frame according to the fusion feature map corresponding to the maximum scale.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for generating an intermediate frame according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for generating an intermediate frame according to any one of claims 1 to 7.

Images