Video frame insertion method, electronic equipment, chip system and readable storage medium

Abstract

The invention provides a video frame insertion method, electronic equipment, a chip system and a readable storage medium, the method is applied to the electronic equipment, and the method comprises the following steps: obtaining a first image and a second image; inputting the first image and the second image into a first module, and extracting pixel features through the first module to obtain a first feature map; inputting the first feature map into a second module, segmenting the first feature map into a first sub-feature map and a second sub-feature map through the second module, determining difference data between the first sub-feature map and the second sub-feature map, and predicting according to the difference data to obtain a third image, the difference data is used for representing the motion condition of pixels in the time sequence from the first image to the second image. Therefore, through the frame insertion model which does not comprise non-convolution operations such as pooling operation and interpolation up-sampling operation, the intermediate image between the two images is obtained, and the time for outputting the intermediate image by the frame insertion model can be shortened.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video frame insertion method, an electronic device, a chip system and a readable storage medium. Background Art

[0002] Video interpolation is the process of predicting and inserting one or more intermediate frames between the previous and next frames of the original video to shorten the display time span between frames and thus obtain a video with a higher frame rate.

[0003] Currently, video interpolation can be achieved through neural networks. Specifically, two frames of images in the video are input into a neural network consisting of a convolutional layer, a pooling layer, and an upsampling (interpolation type upsampling) layer, and the intermediate frame image between the two frames is output.

[0004] However, in the above implementation, the non-convolution operations of the pooling layer and the upsampling (interpolation upsampling) layer take a long time to execute on the neural-network processing unit (NPU) of the electronic device, which makes it take a long time to obtain the intermediate frame image through two frames of images each time, so that the picture is not updated in time when the video is played, resulting in low smoothness of the video playback. Summary of the invention

[0005] The present application provides a video interpolation method, an electronic device, a chip system and a readable storage medium, which can shorten the output time of intermediate frame images in an interpolation model as much as possible and improve the smoothness of video playback.

[0006] In a first aspect, the present application provides a video frame insertion method, which is applied to an electronic device, wherein the electronic device includes a frame insertion model, the frame insertion model includes a first module and a second module, the first module includes a convolution layer, and the second module includes a first segmentation module and a first calculation module; the method includes: A first image and a second image are obtained from a target video, wherein the first image is a frame in the target video that is earlier than the second image in a time series; the first image and the second image are input into a first module to obtain a first feature map, wherein the first module is used to extract pixel features of the first image and the second image, wherein the first feature map includes pixel features of the first image and pixel features of the second image; the first feature map is input into a second module to obtain a third image, wherein a first segmentation module in the second module is used to segment the first feature map into a first sub-feature map and a second sub-feature map, wherein a first calculation module is used to determine difference data between the first sub-feature map and the second sub-feature map, and predict a third image based on the difference data, wherein the difference data is used to represent the movement of pixels from the first image to the second image in a time series, and wherein the third image is an intermediate image to be inserted between the first image and the second image.

[0007] In the above method, when performing interpolation processing, the first image and the second image in the target video can be input into the first module, and the pixel features can be extracted through the aforementioned convolution layer to obtain a first feature map, and the first feature map can be input into the second module. The first segmentation module in the second module segments the first feature map to obtain a first sub-feature map and a second sub-feature map. Then, the first calculation module in the second module can determine the difference data between the first sub-feature map and the second sub-feature map through subtraction calculation. Since the difference data is used to represent the movement of pixels of the first image and the second image in the time series, the first calculation module can obtain a third image for insertion between the first image and the second image according to the difference data. In this way, the electronic device can obtain the third image not through an interpolation model including non-convolution operations such as pooling operations and interpolation upsampling operations, but after obtaining the first feature map, the first feature map is segmented into two sub-feature maps, and the difference data obtained by the subtraction operation between the two sub-feature maps is used to predict the third image for insertion between the first image and the second image, thereby shortening the duration of the interpolation model outputting the third image. When the video is played, the picture can be updated in time, and the video playback smoothness of the target video can be further guaranteed.

[0008] In combination with the first aspect, in some implementations of the first aspect, inputting the first image and the second image into the first module to obtain the first feature map includes: The first image and the second image are spliced ​​to obtain a spliced ​​image; the spliced ​​image is input into the first module to obtain a first feature map.

[0009] In the above method, before the first image and the second image are input into the first module, the electronic device splices the first image and the second image to obtain a spliced ​​image. After that, the first module can directly process the spliced ​​image, and in the intermediate process, the second module can directly splice the first difference data, the second difference data and the spliced ​​image, without the need to splice the first image and the second image before executing the step of splicing the first difference data, the second difference data and the spliced ​​image, and the fourth module can directly splice the first transformation feature map, the second transformation feature map and the spliced ​​image, without the need to splice the first image and the second image again before executing the step of splicing the first transformation feature map, the second transformation feature map and the spliced ​​image, thereby saving the splicing time of the first image and the second image and shortening the time for the interpolation model to output the third image.

[0010] In combination with the first aspect, in some implementations of the first aspect, the interpolation model further includes a third module and a fourth module, the third module includes a convolution layer, and the fourth module includes a second segmentation module and a second calculation module; the first feature map is input into the second module to obtain a third image, including: The first feature map and the spliced ​​image are input into the second module to obtain the second feature map. The first calculation module is used to determine the first difference data and the second difference data between the first sub-feature map and the second sub-feature map, and the first difference data, the second difference data and the spliced ​​image are spliced. The first difference data is used to represent the movement of pixels from the first image to the second image in the time series, and the second difference data is used to represent the movement of pixels from the second image to the first image in the time series. The second feature map is input into the third module to obtain the third feature map, and the third module is used to extract features from the second feature map. The third feature map, the first difference data, the second difference data, the first image and the second image are input into the fourth module to obtain the third image. The second segmentation module in the fourth module is used to segment the third feature map into a third sub-feature map and a fourth sub-feature map, the second calculation module is used to fuse the first difference data and the third sub-feature map to obtain a first fused feature map, fuse the second difference data and the fourth sub-feature map to obtain a second fused feature map, distort the first image according to the first fused feature map to obtain a first transformed feature map, distort the second image according to the second fused feature map to obtain a second transformed feature map, and predict the third image according to the first transformed feature map and the second transformed feature map, the accuracy of the first transformed feature map and the second transformed feature map for predicting the third image is higher than the accuracy of the first difference data and the second difference data for predicting the third image.

[0011] In the above method, after obtaining the first feature, in addition to the first feature map, the electronic device can also input the spliced ​​image into the second module, and after obtaining the third feature map, in addition to the third feature map, the electronic device can also input the spliced ​​image into the fourth module, so that the middle layer of the interpolation model can also learn the original image information (referring to the spliced ​​image), avoiding the interpolation model from losing important feature information and ensuring the performance of the model.

[0012] Furthermore, after inputting the first feature map and the spliced ​​image into the second module to obtain the second feature map, the electronic device can also input the second feature map into the third module, perform feature extraction on the second feature map to obtain the third feature map, and input the third feature map, the first difference data, the second difference data, the first image and the second image into the fourth module to obtain the first transformed feature map and the second transformed feature map, the first transformed feature map is equivalent to the optimized first difference data, and the second transformed feature map is equivalent to the optimized second difference data, so that the accuracy of the first transformed feature map and the second transformed feature map for predicting the third image can be higher than the accuracy of the first difference data and the second difference data for predicting the third image.

[0013] In combination with the first aspect, in some implementations of the first aspect, the interpolation model further includes a fifth module and a sixth module, the fifth module includes a convolution layer, and the sixth module includes a transformation module, a third segmentation module, and a third calculation module; the third feature map, the first difference data, and the second difference data are input into the fourth module to obtain a third image, including: The third feature map, the first difference data, the second difference data, the first image, the second image and the spliced ​​image are input into the fourth module to obtain a fourth feature map, which is obtained by splicing the first transformation feature map, the second transformation feature map and the spliced ​​image; the fourth feature map is input into the fifth module to obtain a fifth feature map, and the fifth module is used to extract features from the fourth feature map; the fifth feature map, the first transformation feature map and the second transformation feature map are input into the sixth module to obtain a third image, the transformation module in the sixth module is used to normalize each feature data in the fifth feature map to obtain a sixth feature map, the third segmentation module is used to segment the sixth feature map into a fifth sub-feature map and a sixth sub-feature map, and the third calculation module is used to generate a third image including target information according to the fifth sub-feature map, the sixth sub-feature map, the first transformation feature map and the second transformation feature map, the target information includes light and shadow information and / or occlusion information, and the target information included in the third image is the target information included in the first image and the second image.

[0014] In the above method, after obtaining the fifth feature map, the electronic device can also input the fifth feature map, the first transformation feature map and the second transformation feature map into the sixth module, so that through the processing of the third segmentation module and the third calculation module, the obtained third image has the light and shadow information and / or occlusion information included in the first image and the second image. In this way, it can be ensured that there will be no abrupt changes when the subsequent first image, third image, and second image are played in sequence, which can avoid the visual jump during playback and ensure the continuity during playback.

[0015] In combination with the first aspect, in some implementations of the first aspect, the interpolation model further includes a seventh module and a first activation function, the seventh module includes a convolution layer, and the fifth feature map, the first transformed feature map, and the second transformed feature map are input into a sixth module to obtain a third image, including: The fifth feature map, the first transformed feature map and the second transformed feature map are input into the sixth module to obtain the seventh feature map; the seventh feature map and the spliced ​​image are spliced ​​to obtain the eighth feature map; the eighth feature map is input into the seventh module to obtain the ninth feature map, and the seventh module is used to extract features from the eighth feature map; the seventh feature map and the ninth feature map are fused to obtain the third fused feature map; the third fused feature map is input into the first activation function to obtain the third image, and the first activation function is used to convert the third fused feature map into an image in which the pixel values ​​are kept within a preset range.

[0016] In the above method, the seventh feature map and the ninth feature map are fused to form a residual connection to retain as many image detail features as possible including texture and clear edges, thereby facilitating the generation of a third image with more complete feature information.

[0017] Furthermore, through the processing of the first activation function, the pixel values ​​of the third image output by the electronic device are maintained within a preset range, so that the electronic device can normally recognize the third image and normally display the third image.

[0018] In combination with the first aspect, in some implementations of the first aspect, the first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer; inputting the spliced ​​image into the first module to obtain a first feature map includes: The spliced ​​image is input into the first sublayer to obtain a first intermediate feature map. The first convolution layer in the first sublayer is used to extract pixel features of the spliced ​​image to obtain a second intermediate feature map. The second activation function layer is used to perform a nonlinear transformation on the second intermediate feature map to obtain the first intermediate feature map. In the process of extracting pixel features of the spliced ​​image by the first convolution layer, the convolution step size used is related to the target output duration of the third image. The convolution step size is an even number greater than or equal to 2, and the size of the first intermediate feature map is related to the convolution step size. The first intermediate feature map is input into the second sublayer to obtain a first feature map. The second convolution layer in the second sublayer is used to adjust the size of the first intermediate feature map to the same size as the spliced ​​image.

[0019] In the above method, the convolution step size is adjustable. The smaller the convolution step size, the larger the overall calculation amount of the first module, the longer the overall execution time of the first module, and the longer the output time of the third image. The larger the convolution step size, the smaller the overall calculation amount of the first module, the shorter the overall execution time of the first module, and the shorter the output time of the third image. By setting the size of the convolution step size to a value related to the target output time of the third image, the output time of the third image can be guaranteed.

[0020] In combination with the first aspect, in some implementations of the first aspect, the first module further includes a padding layer and a truncation layer, and inputs the spliced ​​image into the first sublayer to obtain a first intermediate feature map, including: The stitched image is input into the padding layer, and the padding layer is used to determine whether the length and width of the stitched image are odd numbers respectively; if the length and / or width of the stitched image is an odd number, the pixels of the stitched image are first expanded to obtain an updated stitched image, and the first expansion is used to make the length and width of the stitched image both even numbers; the updated stitched image is input into the first sublayer to obtain a first intermediate feature map; the first intermediate feature map is input into the second sublayer to obtain a first feature map, including: the first intermediate feature map is input into the second sublayer to obtain a second intermediate feature map, and the second convolution layer of the second sublayer is used to adjust the size of the first intermediate feature map to the same size as the updated stitched image; the second intermediate feature map is input into the truncation layer to obtain the first feature map, and the truncation layer is used to adjust the size of the second intermediate feature map to the same size as the stitched image.

[0021] In the above method, considering that when the length and / or width of the stitched image is an odd number, the stitched image cannot be processed by the convolution layer corresponding to the even convolution step size, when the length and / or width of the stitched image is an odd number, the length and width of the stitched image can be made even numbers by padding with 0 to obtain an updated stitched image, and then input into the first convolution layer for processing. The size of the first intermediate feature map obtained after processing by the first convolution layer and the second activation function layer is a smaller size. Based on this, the second convolution layer of the second sublayer can also adjust the size of the first intermediate feature map to the same as the updated stitched image by padding with 0 to obtain the second intermediate feature map, and then the second intermediate feature map is cropped by the cropping layer to obtain the first feature map. In this way, the size of the first feature map is the same as the size of the stitched image, which can avoid the situation where the third image obtained by the first feature map has different sizes from the first image and the second image.

[0022] In combination with the first aspect, in certain implementations of the first aspect, the first image and the second image are two adjacent frames of images in the target video, or the first image and the second image are two frames of images corresponding to each other in the target video with an interval of N frames, where N is a positive integer greater than or equal to 1.

[0023] For example, the target video includes 9 frames of images in total, the first image is the first frame of the 9 frames of images, and the second image is the second frame of the 9 frames of images.

[0024] For example, the target video includes 9 frames of images in total, the second frame, the fourth frame, the sixth frame, and the eighth frame are extracted from the target video, and the remaining images include the first frame, the third frame, the fifth frame, the seventh frame, and the ninth frame. The value of N is 1, the first image is the first frame of the 9 frames, and the second image is the third frame of the 9 frames.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, a process of generating an interpolation model includes: A plurality of training image groups are obtained, wherein each of the plurality of training image groups includes three images that are continuous in a time series, and the second image in each training image group is a labeled intermediate image corresponding to each training image group; the first image and the third image in each training image group are input into a first sample module of a neural network to obtain a sample feature map, wherein the first sample module is used to extract pixel features of the first image and the third image through a convolutional layer included in the first sample module, and the sample feature map includes pixel features of the first image and pixel features of the third image; the sample feature map is input into a second sample module of the neural network to obtain an estimated intermediate image corresponding to each training image group, wherein the second sample module includes a first sample segmentation module and a first sample calculation module, wherein the first sample segmentation module is used to segment the sample feature map into a first sub-sample feature map and a second sub-sample feature map, and the first sample calculation module is used to determine sample difference data between the first sub-sample feature map and the second sub-sample feature map, and determine an estimated intermediate image according to the sample difference data, wherein the sample difference data is used to represent the movement of pixels from the first image to the third image in a time series; the neural network is trained according to the difference between the labeled intermediate image and the estimated intermediate image to obtain an interpolation model.

[0026] The generation process of the interpolation model in the present application can be performed by different devices from the video interpolation method in the embodiment of the present application. For example, the generation process of the interpolation model is performed by a server, and the video interpolation method in the embodiment of the present application is performed by an electronic device.

[0027] In the above method, during the training process, the intermediate image can be predicted based on the first image and the third image in multiple groups of three consecutive images, and then based on the difference between the predicted intermediate image and the second image in the three consecutive images, that is, the loss between the predicted intermediate image and the second image in the three consecutive images, the neural network can update the model parameters of the neural network according to the aforementioned loss through the back propagation method, thereby obtaining the interpolation model.

[0028] In a second aspect, the present application provides a video frame interpolation device, which includes a module for executing the method in the first aspect and any possible implementation manner of the first aspect.

[0029] In a third aspect, the present application provides an electronic device, comprising: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method in the first aspect and any possible implementation of the first aspect.

[0030] In a fourth aspect, the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the method in the first aspect and any possible implementation of the first aspect.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes the method in the first aspect and any possible implementation of the first aspect.

[0032] In a sixth aspect, the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the method in the first aspect and any possible implementation manner of the first aspect.

[0033] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a video or motion effect insertion provided in an embodiment of the present application; Figure 2 A schematic diagram of the classification of various layers in an interpolation model provided by the prior art; Figure 3 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application; Figure 4 A schematic diagram of a software architecture of an electronic device provided in one embodiment of the present application; Figure 5 A schematic diagram of a flow chart of a video frame insertion method provided in one embodiment of the present application; Figure 6 A schematic diagram of the structure of a first module provided in one embodiment of the present application; Figure 7 A schematic diagram of a flow chart of a video frame insertion method provided in one embodiment of the present application; Figure 8 A schematic diagram of the structure of a second module provided in one embodiment of the present application; Fig. 9 A schematic diagram of the structure of a fourth module provided in one embodiment of the present application; Fig.10 A schematic diagram of the structure of a sixth module provided in one embodiment of the present application; Fig.11 A schematic diagram of the structure of a frame insertion model provided in one embodiment of the present application. DETAILED DESCRIPTION

[0035] In this application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the first conversion relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c alone can represent: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0036] To facilitate understanding, some of the examples given are provided for reference to the description of concepts related to the embodiments of the present application.

[0037] 1. Convolutional layer The convolution layer can include many convolution operators, which are also called kernels. Their role in image processing is equivalent to a filter that extracts specific information from the input image. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed horizontally on the input image one pixel after another (or two pixels after two pixels... depending on the value of the stride), thereby completing the work of extracting specific features from the image.

[0038] 2. Loss Function In the process of training neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function. They are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0039] 3. Back Propagation Algorithm Neural networks can use the error back propagation (BP) algorithm to correct the size of the initial model parameters during the training process, so that the reconstruction error loss of the model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the initial model parameters, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal model parameters, such as the weight matrix.

[0040] 4. Pixel Value The pixel value of an image can be a red-green-blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For grayscale images, the pixel value can be a grayscale value.

[0041] Video interpolation is the process of predicting and inserting one or more intermediate frames between the previous and next frames of the original video to shorten the display time span between frames and thus obtain a video with a higher frame rate.

[0042] In actual application scenarios, interpolation technology can be applied to scenes such as videos and motion effects (i.e. dynamic images). Figure 1 As shown in the figure, if the original video or animation is 9 frames, there may be two situations when the interpolation technology is applied: Case 1: The 9 frames of images rendered by the graphics processing unit (GPU) in the original video or animation can be reduced to 5 frames. For the 5 frames, the images between every two images can be interpolated using interpolation technology. The interpolation operation is performed on the NPU, which can reduce the power consumption of generating intermediate images. Figure 1 As shown in (a) in .

[0043] Case 2: You can directly insert frames into the original video or animation, that is, insert a frame between two adjacent frames, so that the original 9 frames become 17 frames, improving the smoothness of the video or animation. Figure 1 As shown in (b) in .

[0044] Currently, interpolation technology can be implemented through traditional interpolation algorithms or through neural networks. In the implementation method of obtaining intermediate frame images through an interpolation model based on a neural network, two frames of images in a video can be input into an interpolation model including a convolution layer, a pooling layer, and an upsampling layer (such as nearest neighbor interpolation and / or bilinear interpolation), and an intermediate frame image between the two frames of images can be output.

[0045] Each time the interpolation process is performed, two images can be input into the interpolation model, and after being processed by the convolution layer, the pooling layer and the interpolation layer, an intermediate frame image is obtained, so that the intermediate frame image can be inserted between the aforementioned two images.

[0046] Combination Figure 2 It can be seen that in the implementation method of obtaining the intermediate frame image based on the neural network interpolation model, the interpolation model includes a convolution layer, a pooling layer and an interpolation layer, so that the processing of the interpolation model includes many non-convolution operations, such as pooling operations, resizing operations, interpolation operations, etc. Non-convolution operations have large parameters and calculations, and the execution time on the NPU may be close to or exceed the execution time of a convolution layer. In other words, the execution time of non-convolution operations on the NPU is relatively long, which will make it take a long time to obtain the intermediate frame image through two frames of images. When the video is played, the picture is not updated in time, resulting in low playback smoothness of the video picture.

[0047] In view of the above problems, the present application can provide a video interpolation method, an electronic device, a chip system, a computer-readable storage medium and a computer program product, which can cancel non-convolution operations, so that the interpolation model includes a first module and a second module, the first module includes a convolution layer, and the second module includes a segmentation module and a calculation module. When performing interpolation processing, the two images in the target video can be input into the first module, and the pixel features are extracted through the aforementioned convolution layer to obtain a feature map, and the feature map is input into the second module. The aforementioned feature map is segmented into two sub-feature maps corresponding to the two images respectively through the segmentation module, and the difference between the two sub-feature maps is determined through the calculation module, so as to determine the movement of pixels of the two images in the time series according to the aforementioned difference. Through the processing of the second module, an intermediate image for insertion between the two images can be output. In this way, an interpolation model that does not include non-convolution operations such as pooling operations and interpolation upsampling operations can be used to obtain an intermediate image between the two images, which can shorten the time for the interpolation model to output the intermediate image, and further ensure the smoothness of the target video during playback.

[0048] Among them, the above-mentioned electronic device can be an electronic device with neural network processor hardware and corresponding software support.

[0049] The electronic devices may include mobile phones, tablet computers, vehicle-mounted devices, laptop computers, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart cars, smart TVs, robots and other devices.

[0050] It should be noted that, in some possible implementations, the electronic device may also be referred to as a terminal device (station), a user equipment (UE), etc., which is not limited in the embodiments of the present application.

[0051] For ease of explanation, Figure 3 In the description, the electronic device 100 is taken as a mobile phone as an example.

[0052] like Figure 3 As shown, in some embodiments, the electronic device 100 may include a processor 101, a communication module 102, a display screen 103, a camera 104, a sensor 105, an internal memory 106, a USB interface 107, an external memory interface 108, a charging management module 109, a power management module 110, and a battery 111, etc.

[0053] The processor 101 may include one or more processing units, for example, the processor 101 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video stream codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or may be integrated in one or more processors 101.

[0054] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0055] The processor 101 may also be provided with a memory for storing instructions and data.

[0056] A neural network processor is a hardware component specifically designed to accelerate artificial intelligence (AI) and machine learning tasks. Neural network processors are mainly used to handle calculations related to neural networks, such as image recognition, speech recognition, natural language processing, image interpolation processing, etc.

[0057] The communication module 102 may include antenna 1 and antenna 2, a mobile communication module, and / or a wireless communication module.

[0058] Optionally, the electronic device 100 may further include peripheral devices, such as a mouse, buttons, indicator lights, a keyboard, a speaker, a microphone, etc.

[0059] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100 .

[0060] In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine some components, or separate some components, or arrange the components differently. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0061] Please refer to Figure 4 , is a schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application. The video frame insertion method provided in an embodiment of the present application is applied to Figure 3 When the electronic device 100 is shown, the software in the electronic device 100 can be divided into the following Figure 4 The illustrated application layer 201 , application framework layer (framework, FWK) 202 , hardware abstraction layer (hardware abstraction layer, HAL) 203 and driver layer 204 .

[0062] A plurality of applications may be installed in the application layer 201 .

[0063] The application framework layer 202 provides a set of basic functions and services for the application layer 201 to call and use.

[0064] The hardware abstraction layer 203 is a software located between the operating system kernel and the hardware circuit, and is usually used to abstract the hardware to achieve the interaction between the operating system and the hardware circuit at the logic layer.

[0065] The driver layer 204 may be installed with multiple drivers for driving the hardware.

[0066] It should be noted that the application layer 201, the application framework layer 202, the hardware abstraction layer 203 and the driver layer 204 may also include other contents, which are not specifically limited here.

[0067] Among them, the embodiment of the present application involves an interpolation model, which is obtained by training a neural network through an interpolation model generation process. The method provided by the present application is described below from the model training side and the model application side of the interpolation model: The interpolation model generation process provided in the embodiment of the present application involves computer vision processing, and can be specifically applied to data training, machine learning, deep learning and other data processing methods to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on the training data (such as the multiple training image groups described later in this application), and finally obtain a trained interpolation model; and the video interpolation method provided in the embodiment of the present application can use the above-mentioned trained interpolation model to input input data (such as the two images in the target video in this application) into the trained interpolation model to obtain output data (such as the intermediate image in this application). It should be noted that the interpolation model generation process and video interpolation method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0068] Based on the above description, the training process of the frame interpolation model involved in the video frame interpolation method of the embodiment of the present application is first described in detail below.

[0069] Specifically, the system architecture involved in the generation process of the interpolation model includes a first sample module and a second sample module. The generation process of the interpolation model includes: Step 11: Acquire multiple training image groups, each of the multiple training image groups includes three images that are continuous in time series, and the second image in each training image group is a labeled intermediate image corresponding to each training image group.

[0070] Step 12: Input the first image and the third image in each training image group into the first sample module of the neural network to obtain a sample feature map. The first sample module is used to extract pixel features of the first image and the third image through the convolutional layer included in the first sample module. The sample feature map includes pixel features of the first image and pixel features of the third image.

[0071] Step 13, input the sample feature map into the second sample module of the neural network to obtain an estimated intermediate image corresponding to each training image group, the second sample module includes a first sample segmentation module and a first sample calculation module, the first sample segmentation module is used to segment the sample feature map into a first sub-sample feature map and a second sub-sample feature map, the first sample calculation module is used to determine the sample difference data between the first sub-sample feature map and the second sub-sample feature map, and determine the estimated intermediate image according to the sample difference data, the sample difference data is used to represent the movement of pixels in a time series from the first image to the third image.

[0072] Step 14: According to the difference between the labeled intermediate image and the estimated intermediate image, the neural network is trained to obtain an interpolation model.

[0073] Combined with the above description, during the training process, an estimated intermediate image can be obtained based on the first image and the third image of multiple groups of three consecutive images, and then based on the difference between the estimated intermediate image and the second image of the three consecutive images, that is, the loss between the predicted intermediate image and the second image of the three consecutive images, the neural network can update the model parameters of the neural network according to the aforementioned loss through the back propagation method, thereby obtaining an interpolation model.

[0074] Based on this, the obtained interpolation model can predict an intermediate image that can be inserted between the two images in the target video.

[0075] It should be noted that the neural network used in the generation process of the interpolation model of the embodiment of the present application is generally the same as the structure of the interpolation model used in the video interpolation method of the embodiment of the present application. The structure of the model is not specifically described here. For details, please refer to the description below.

[0076] In addition, the generation process of the frame insertion model in the present application and the video frame insertion method in the embodiment of the present application can be performed by the same device, for example, both use Figure 3 The electronic device 100 shown is executed; the generation process of the interpolation model in the present application can be executed by different devices from the video interpolation method of the embodiment of the present application. For example, the generation process of the interpolation model is executed by a training device (such as the training device can be a server).

[0077] Based on the above description, the video interpolation method in the interpolation model application stage of an embodiment of the present application is described in detail below in combination with the accompanying drawings and application scenarios.

[0078] The video frame insertion method provided in the embodiment of the present application is applied to an electronic device, wherein the electronic device includes a frame insertion model, the frame insertion model includes a first module and a second module, the first module includes a convolution layer, and the second module includes a first segmentation module and a first calculation module. The aforementioned electronic device may be Figure 3 An electronic device 100 is shown.

[0079] See also Figure 5 , Figure 5 A flow chart of a video frame insertion method provided in an embodiment of the present application is shown.

[0080] like Figure 5 As shown, the video frame insertion method provided by the present application may include: S301. Acquire a first image and a second image from a target video, where the first image is a frame in the target video that is earlier than the second image in time sequence.

[0081] There are usually two situations for the first image and the second image.

[0082] In some embodiments, the first image and the second image are two adjacent frames of images in the target video.

[0083] Typically, an electronic device can interpolate frames between two adjacent frames in a target video, that is, insert a frame of image between every two adjacent frames in the target video, so that the number of frames of the target video can be increased, thereby obtaining an updated target video. When the electronic device plays the updated target video, the updated target video has a higher playback smoothness than the original target video. The embodiments of the present application are suitable for scenarios where the playback smoothness of the target video needs to be improved.

[0084] For example, the target video includes 9 frames of images in total, the first image is the first frame of the 9 frames of images, and the second image is the second frame of the 9 frames of images.

[0085] In some other embodiments, the first image and the second image are two frames of images corresponding to each other at an interval of N frames in the target video, where N is a positive integer greater than or equal to 1.

[0086] Typically, an electronic device can extract frames from a target video every N frames, and then interpolate the remaining two frames corresponding to each N frame interval to obtain the same number of images as the original target video. Since the interpolation process is performed on the NPU, the generation of a smaller number of intermediate images can reduce the power consumption of the electronic device. The embodiments of the present application are suitable for scenarios where the power consumption of electronic devices needs to be reduced.

[0087] For example, the target video includes 9 frames of images in total, the second frame, the fourth frame, the sixth frame, and the eighth frame are extracted from the target video, and the remaining images include the first frame, the third frame, the fifth frame, the seventh frame, and the ninth frame. The value of N is 1, the first image is the first frame of the 9 frames, and the second image is the third frame of the 9 frames.

[0088] S302: Input the first image and the second image into the first module to obtain a first feature map. The first module is used to extract pixel features of the first image and the second image through a convolution layer. The first feature map includes pixel features of the first image and pixel features of the second image.

[0089] Because the first module includes a convolution layer, after the electronic device inputs the first image and the second image into the first module, the electronic device can extract pixel features of the first image and the second image through the convolution layer to obtain a first feature map, which includes the pixel features of the first image and the pixel features of the second image.

[0090] In some embodiments, the first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer; inputting the spliced ​​image into the first module to obtain a first feature map includes: The spliced ​​image is input into the first sublayer to obtain a first intermediate feature map. The first convolution layer in the first sublayer is used to extract pixel features of the spliced ​​image to obtain a second intermediate feature map. The second activation function layer is used to perform a nonlinear transformation on the second intermediate feature map to obtain the first intermediate feature map. In the process of extracting pixel features of the spliced ​​image by the first convolution layer, the convolution step size used is related to the target output duration of the third image. The convolution step size is an even number greater than or equal to 2, and the size of the first intermediate feature map is related to the convolution step size. The first intermediate feature map is input into the second sublayer to obtain a first feature map. The second convolution layer in the second sublayer is used to adjust the size of the first intermediate feature map to the same size as the spliced ​​image.

[0091] It should be understood that the convolution step size (stride) is adjustable. The smaller the convolution step size, the greater the overall computational amount of the first module, the longer the overall execution time of the first module, and the longer the output time of the third image. The larger the convolution step size, the smaller the overall computational amount of the first module, the shorter the overall execution time of the first module, and the shorter the output time of the third image.

[0092] Based on this, by setting the size of the convolution step to a value related to the target output duration of the third image, the output duration of the third image can be guaranteed.

[0093] The convolution step size can be determined according to actual conditions; for example, the convolution step size can be set to 2.

[0094] In addition, in order to further reduce the overall computational complexity of the first module, the convolution step size is set to an even number greater than or equal to 2. Considering that when the length and / or width of the stitched image is an odd number, the stitched image (that is, the first image and the second image) cannot be processed by the convolution layer corresponding to the even convolution step size, the stitched image can be expanded first and then further processed by the first convolution layer.

[0095] Specifically, the convolution step size is an even number greater than or equal to 2. The first module also includes a padding layer and a truncation layer. The spliced ​​image is input into the first sublayer to obtain a first intermediate feature map, including: The stitched image is input into the padding layer, and the padding layer is used to determine whether the length and width of the stitched image are odd numbers; if the length and / or width of the stitched image is odd, a first expansion is performed on the pixels of the stitched image to obtain an updated stitched image, and the first expansion is used to make the length and width of the stitched image both even numbers; the updated stitched image is input into the first sublayer to obtain a first intermediate feature map.

[0096] Furthermore, the first intermediate feature map is input into the second sub-layer to obtain a first feature map, including: The first intermediate feature map is input into the second sub-layer to obtain a second intermediate feature map, and the second convolution layer (also called deconvolution layer) of the second sub-layer is used to adjust the size of the first intermediate feature map to the same size as the updated spliced ​​image; the second intermediate feature map is input into the truncation layer to obtain the first feature map, and the truncation layer is used to adjust the size of the second intermediate feature map to the same size as the spliced ​​image.

[0097] Among them, Figure 6 The first module with a convolution step of 2 shown in the figure includes a padding layer, a first sublayer (convolution layer and activation function layer, the number of the first sublayer is P, where P is 2), a second sublayer (a convolution layer) and a truncation layer. If a convolution layer with a convolution step of 2 is used, when the length and width of the input of the subsequent layer become 1 / 2 of the original, the amount of calculation of the subsequent convolution layer can be reduced to 1 / 4. When the length and width of the original input are odd numbers, the size of the output may be smaller than the original output. Therefore, a padding layer is added before the convolution layer with a convolution step of 2 to fill the size of the image input to the current layer with zeros to make it an even number. Finally, instead of using an interpolation-type upsampling layer, a transposed convolution operation is used. For the final output, the image corresponding to the original odd width and height is truncated. In this way, a stacking method of pure convolution layers can be achieved.

[0098] Among them, if a convolution layer with a convolution step size of 2 is used, in order to maintain the performance of the network, the existing scheme will first use a pooling layer to reduce the input size of the subsequent convolution layer, and then add an additional convolution layer for preprocessing or post-processing, and require a subsequent additional interpolation-type upsampling layer operation to maintain the output image and the original input image have the same size, which may increase a lot of computational overhead and delay; and the embodiment of the present application here replaces the use of an interpolation-type upsampling layer, and replaces the pooling layer with a convolution layer with an even step size, and then uses a transposed convolution operation, so that the output image and the original input image have the same size, in this way, it can be further ensured that the first module is a pure convolution module, and the use of a pure convolution module as a feature extraction unit can reduce the overall parameter amount and computational amount of the model.

[0099] For example, if the dimension of the current input spliced ​​image i is [b, c1, n, m], and the length of the spliced ​​image i is n, where n is an odd number, it is necessary to pad after the rightmost column of the spliced ​​image i. The length after padding is n+1, and then it is sent to the convolution layer with a convolution step of 2 for convolution processing. Finally, the dimension of the output feature map after the transposition operation is [b, c2,n+1, m], and then it is clipped through the truncation layer to [:, :, 0:n-1, :] as the final output of the first module.

[0100] Based on the above description, Figure 6 For the first module with a convolution step size of 2, the output time of the first feature map of the first module with a convolution step size of 2 is shorter than that of the first module with a convolution step size of 1.

[0101] Combination Figure 6 The structure of the first module shown and the above description can be used to expand the spliced ​​image by padding with 0 when the length and / or width of the spliced ​​image is an odd number, that is, padding, so that the length and width of the spliced ​​image are both even numbers, to obtain an updated spliced ​​image, and then input it into the first convolution layer (the convolution step size can be 2) for processing. The size of the first intermediate feature map obtained after processing by the first convolution layer and the second activation function layer is a smaller size. Based on this, the second convolution layer of the second sublayer adjusts the size of the first intermediate feature map to the same as the updated spliced ​​image through a transposed convolution operation (ConvTranspose) with a convolution step size of 2 to obtain a second intermediate feature map. Then, the second intermediate feature map is cropped by the cropping layer to obtain the first feature map. In this way, the size of the first feature map is the same as the size of the spliced ​​image, which can avoid the situation where the third image obtained by the first feature map has different sizes from the first image and the second image.

[0102] The number of the first sub-layers is P, the value of P is related to the target output duration of the third image, and each first sub-layer is used to extract pixel features. The present application does not specifically limit the value of P.

[0103] When the value of P is 2, the specific structure of the first module can be found in Figure 6 From the structure shown, it can be seen that the first module includes a padding layer, two first sub-layers (first convolution layer + second activation function layer), a second sub-layer (second convolution layer) and a truncation layer, and each first sub-layer includes a convolution layer and an activation function layer.

[0104] It should be understood that the larger the value of P, the larger the overall computational amount of the first module, the longer the overall execution time of the first module, and the longer the output time of the third image; the smaller the value of P, the smaller the overall computational amount of the first module, the shorter the overall execution time of the first module, and the shorter the output time of the third image.

[0105] Based on this, if you want to shorten the output time of the third image, you need to shorten the overall execution time of the first module. The value of P can be set to a smaller value. Specifically, the value of P can be set according to the target output time of the third image. The shorter the target output time of the third image, the smaller the set value of P, that is, the smaller the number of first sub-layers.

[0106] In some embodiments, the first convolutional layer includes M output channels, the second convolutional layer includes R output channels, the values ​​of M and R are related to the target output duration of the third image, the first intermediate feature map is a feature map of M dimensions, and the first feature map is a feature map of R dimensions.

[0107] In some embodiments, the second activation function layer is a rectified linear unit (ReLU) function.

[0108] In the embodiment of the present application, the calculation process of the ReLU function is simple. The use of the ReLU function can shorten the forward inference time of the interpolation model while ensuring the operating performance of the interpolation model, that is, shorten the time required from inputting the first image and the second image to outputting the third image.

[0109] Among them, the input channels of the first convolution layer in the first sublayer are determined according to the input image, for example, it can be three RGB channels, and the number of input channels of each other convolution layer is the same as the number of output channels of the previous convolution layer.

[0110] For example, the number of input channels of the first convolutional layer is 3, the number of output channels of the first convolutional layer is 1, and the number of input channels of the second convolutional layer is 1.

[0111] Among them, the output channels of each convolutional layer are adjustable. After the output channels of a single convolutional layer reach a certain threshold, at this time, the larger the number of output channels, the larger the amount of calculation that the convolutional layer needs to perform, and the smaller the number of output channels, the smaller the amount of calculation that the convolutional layer needs to perform. It can be seen that the size of the output channels can affect the amount of calculation that the convolutional layer needs to perform. This application does not specifically limit the number of output channels of each convolutional layer.

[0112] Moreover, after the output channel of a single convolutional layer reaches a certain threshold, at this time, the greater the amount of calculation that each convolutional layer needs to perform, the longer the overall execution time of the first module, and the longer the output time of the third image; the smaller the amount of calculation that each convolutional layer needs to perform, the shorter the overall execution time of the first module, and the longer the output time of the third image.

[0113] Based on this, if you want to shorten the output time of the third image, you need to shorten the overall execution time of the first module, and the number of output channels of the first convolutional layer and / or the second convolutional layer can be set to a smaller value. Specifically, the number of output channels of the first convolutional layer and / or the second convolutional layer can be set according to the target output time of the third image; the shorter the target output time of the third image, the smaller the value of the number of output channels of the first convolutional layer and / or the second convolutional layer is set within a certain interval.

[0114] It should be noted that the interpolation model runs on the neural network processor of the electronic device. Considering the performance of the neural network processor, the number of output channels of each convolutional layer will not be adjusted indefinitely, but only needs to be adjusted to the number of output channels that corresponds to the target output duration of the third image.

[0115] S303, input the first feature map into the second module to obtain a third image, the first segmentation module in the second module is used to segment the first feature map into a first sub-feature map and a second sub-feature map, the first calculation module is used to determine the difference data between the first sub-feature map and the second sub-feature map, the difference data is used to represent the movement of pixels from the first image to the second image in a time series, and the third image is an intermediate image to be inserted between the first image and the second image.

[0116] The second module includes a first segmentation module. After the electronic device inputs the first feature map into the second module, the first segmentation module in the second module can segment the first feature map into a first sub-feature map and a second sub-feature map, thereby preparing data for the electronic device to determine a third image that can be inserted between the first image and the second image based on the difference between the first sub-feature map and the second sub-feature map.

[0117] The second module also includes a first calculation module. After the first segmentation module segments the first feature map into a first sub-feature map and a second sub-feature map, the first calculation module can calculate the difference data between the first sub-feature map and the second sub-feature map. Since the difference data is used to represent the movement of pixels of the first image and the second image in a time series, the first calculation module can predict the third image based on the difference data.

[0118] In some embodiments, the difference data includes first difference data and second difference data, the first difference data is used to represent the movement of pixels from the first image to the second image in a time series, and the second difference data is used to represent the movement of pixels from the second image to the first image in a time series.

[0119] The video interpolation method of the present application, the interpolation model includes a first module and a second module, the first module includes a convolution layer, the second module includes a first segmentation module and a first calculation module, when performing interpolation processing, the first image and the second image in the target video can be input into the first module, the pixel features are extracted through the aforementioned convolution layer to obtain a first feature map, the first feature map is input into the second module, the first segmentation module in the second module segments the first feature map to obtain a first sub-feature map and a second sub-feature map, then, the first calculation module in the second module can determine the difference data between the first sub-feature map and the second sub-feature map through subtraction calculation, because the difference data is used to represent the movement of pixels of the first image and the second image in the time series, the first The calculation module can obtain a third image for insertion between the first image and the second image based on the difference data. In this way, the constructed first module is a pure convolution module. As an extraction unit, it can reduce the overall parameter amount and calculation amount of the model, and the electronic device can obtain the third image not through an interpolation model including non-convolution operations such as pooling operations and interpolation upsampling operations. Instead, after obtaining the first feature map, the first feature map is divided into two sub-feature maps, and the difference data obtained by the subtraction operation between the two sub-feature maps is used to predict the third image for insertion between the first image and the second image, thereby shortening the duration of the interpolation model to output the third image. When the video is played, the picture can be updated in time, and the video playback smoothness of the target video can be further guaranteed.

[0120] Furthermore, the convolution step size is adjustable. The smaller the convolution step size, the larger the overall computational load of the first module, the longer the overall execution time of the first module, and the longer the output time of the third image. The larger the convolution step size, the smaller the overall computational load of the first module, the shorter the overall execution time of the first module, and the shorter the output time of the third image. By setting the size of the convolution step size to a value related to the target output time of the third image, the output time of the third image can be guaranteed. Moreover, the convolution step size can be further set to an even number. Considering that when the length and / or width of the spliced ​​image is an odd number, the spliced ​​image cannot be processed by the convolution layer corresponding to the even convolution step size. When the length and / or width of the spliced ​​image is an odd number, the length and width of the spliced ​​image can be made even by first filling 0, and then processed by the convolution layer. Subsequently, the first feature map finally outputted can be made the same size as the spliced ​​image by filling 0 and truncating, so as to avoid the situation where the third image obtained by the first feature map has a different size from the first image and the second image.

[0121] Based on the above Figure 5 According to the description of the illustrated embodiment, the interpolation model further includes a third module, a fourth module, a fifth module, a sixth module, a seventh module and a first activation function.

[0122] The third module includes a convolution layer, the fourth module includes a second segmentation module and a second calculation module. The fifth module includes a convolution layer, the sixth module includes a transformation module, a third segmentation module and a third calculation module. The seventh module includes a convolution layer, and the first activation function is used to convert the feature map into an image whose pixel values ​​are kept within a preset range.

[0123] Next, combine Figure 7 , the video frame insertion method of the present application is introduced in detail.

[0124] See also Figure 7 , Figure 7 A flow chart of a video frame insertion method provided in an embodiment of the present application is shown.

[0125] like Figure 7 As shown, the video frame insertion method provided by the present application may include: S401. Acquire a first image and a second image from a target video, where the first image is a frame in the target video that is earlier than the second image in time sequence.

[0126] Among them, S401 and Figure 5 The implementation of S301 in the illustrated embodiment is similar and will not be described in detail here.

[0127] S402: Splice the first image and the second image to obtain a spliced ​​image.

[0128] The splicing of the first image and the second image is splicing in the channel direction, specifically, the first image and the second image are spliced ​​in a front-to-back stacking manner.

[0129] For example, the pixel point at position A of the first image is (3, 255, 255), and the pixel point at position A of the second image is (3, 255, 255). After the first image and the second image are spliced, the pixel point at position A of the first image is (3, 255, 255), and the pixel point at position A of the second image is (6, 255, 255).

[0130] S403: Input the stitched image into the first module to obtain a first feature map, where the first module is used to extract pixel features of the first image and the second image through a convolution layer, and the first feature map includes pixel features of the first image and pixel features of the second image.

[0131] Among them, S403 and Figure 5 The implementation of S302 in the illustrated embodiment is similar and will not be described in detail here.

[0132] S404. Input the first feature map and the stitched image into the second module to obtain the second feature map. The first segmentation module in the second module is used to segment the first feature map into a first sub-feature map and a second sub-feature map. The first calculation module is used to determine the first difference data and the second difference data between the first sub-feature map and the second sub-feature map, and to stitch the first difference data, the second difference data and the stitched image. The first difference data is used to represent the movement of pixels from the first image to the second image in a time series. The second difference data is used to represent the movement of pixels from the second image to the first image in a time series. The second feature map is a feature map obtained by stitching the first difference data, the second difference data and the stitched image.

[0133] Combination Figure 8 Specifically, after the electronic device inputs the first feature map and the spliced ​​image into the second module, the first segmentation module in the second module can segment the first feature map to obtain a first sub-feature map and a second sub-feature map. Taking the first feature map as a feature map of 4 dimensions as an example, Figure 8 It can be seen that the first segmentation module can segment the first feature map between the first two dimensions and the last two dimensions to obtain the first sub-feature map and the second sub-feature map ( Figure 8 In [:,:2,:,:] and [:,2:4,:,:], the two dimensions are separated by a comma).

[0134] Next, the first calculation module in the second module can subtract the first sub-feature map from the second sub-feature map to obtain the first difference data ( Figure 8 Flow_ 0 _ 0), subtract the constant 0 from the first difference data to obtain the second difference data ( Figure 8 Flow _ 0 _ 1), the first difference data and the second difference data are opposite difference data.

[0135] After obtaining the first difference data, the first calculation module subtracts the constant 0 from the first difference data to obtain the second difference data, which is equivalent to obtaining bidirectional flow information, that is, the movement of pixels from the first image to the second image in the time series, and the movement of pixels from the second image to the first image in the time series. Therefore, the second module can obtain the second feature map based on the bidirectional flow information. The second feature map carries the bidirectional flow information, which can ensure the accuracy of the subsequently obtained third image.

[0136] Finally, in some embodiments, the first calculation module in the second module concatenates the first difference data and the second difference data to obtain a second feature map.

[0137] In other embodiments, the first computing module splices the first difference data, the second difference data and the spliced ​​image to obtain a second feature map. Therefore, the spliced ​​second feature map includes the original input information. In this way, the subsequent intermediate layers of the interpolation model can learn the original input information, ensuring that the interpolation model does not lose important feature information, thereby improving the performance of the model.

[0138] The first calculation module subtracts the first sub-feature map from the second sub-feature map to obtain r _ 1, you can also add r _ 1 is multiplied by the constant C to obtain the first difference data ( Figure 8 Flow _ 0 _ 0).

[0139] The first calculation module will r _ 1 multiplied by the constant C can play a role in adjusting the contribution of the first difference data and the second difference data to each other.

[0140] It should be noted that S405-S412 are optional steps. When S405-S412 are not executed, after the first module outputs the first feature map, the electronic device can directly input the first feature map into the second module to obtain the third image.

[0141] S405: Input the second feature map into a third module to obtain a third feature map, and the third module is used to extract features from the second feature map.

[0142] The third module includes a convolutional layer, and the structure of the third module may be the same as or different from that of the first module.

[0143] When the structure of the third module is the same as that of the first module, if the first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer, the first module includes a first sublayer and a second sublayer; the third module includes a first sublayer and a second sublayer.

[0144] When the structure of the third module is different from that of the first module, if the first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer, the first module includes a first sublayer and a second sublayer; for example, the third module may include two first sublayers and one second sublayer; for example, the third module may include three first sublayers and one second sublayer.

[0145] This application does not specifically limit the structure of the third module.

[0146] The third module is used to extract features from the second feature map, and the third feature map obtained by inputting the second feature map into the third module includes the features of the second feature map.

[0147] S406. Input the third feature map, the first difference data, the second difference data, the first image, the second image and the spliced ​​image into the fourth module to obtain a fourth feature map. The second segmentation module in the fourth module is used to segment the third feature map into a third sub-feature map and a fourth sub-feature map. The second calculation module is used to fuse the first difference data and the third sub-feature map to obtain a first fused feature map, fuse the second difference data and the fourth sub-feature map to obtain a second fused feature map, distort the first image according to the first fused feature map to obtain a first transformed feature map, and distort the second image according to the second fused feature map to obtain a second transformed feature map. The fourth feature map is obtained by splicing the first transformed feature map, the second transformed feature map and the spliced ​​image. The accuracy of the first transformed feature map and the second transformed feature map for predicting the third image is higher than the accuracy of the first difference data and the second difference data for predicting the third image.

[0148] Combination Fig. 9 Specifically, after the electronic device inputs the third feature map, the first difference data, the second difference data and the spliced ​​image into the fourth module, the second segmentation module in the fourth module can segment the third feature map to obtain a third sub-feature map and a fourth sub-feature map. Taking the third feature map as a feature map of 4 dimensions as an example, Fig. 9It can be seen that the second segmentation module can segment the third feature map between the first two dimensions and the last two dimensions according to the channel direction to obtain the third sub-feature map and the fourth sub-feature map ( Fig. 9 in [:,:2,:,:] and [:,2:4,:,:]).

[0149] Next, the second calculation module in the fourth module may fuse (add) the first difference data and the third sub-feature map to obtain a first fused feature map, and fuse the second difference data and the fourth sub-feature map to obtain a second fused feature map.

[0150] Finally, in some embodiments, the second computing module may further concatenate the first fused feature map and the second fused feature map to obtain a fourth feature map.

[0151] In other embodiments, the second computing module may splice the first fused feature map, the second fused feature map and the spliced ​​image to obtain a fourth feature map. Therefore, the spliced ​​fourth feature map includes the original input information. In this way, the subsequent intermediate layers of the interpolation model can learn the original input information, ensuring that the interpolation model does not lose important feature information, thereby ensuring the performance of the model.

[0152] Optionally, the second calculation module may also perform a warp transformation on the first image according to the first fused feature map to obtain a first transformed feature map, and perform a warp transformation on the second image according to the second fused feature map to obtain a second transformed feature map.

[0153] Among them, the distortion transformation can be understood as an image transformation, which realizes the deformation of the image by changing the position of pixels in the image; the distortion transformation can enable the interpolation model to better learn the key features of the image such as the perspective transformation and posture transformation in the first image and the second image.

[0154] In some embodiments, the second calculation module may further concatenate the first transformed feature map and the second transformed feature map to obtain a fourth feature map.

[0155] In other embodiments, the second computing module may splice the first transformed feature map, the second transformed feature map, and the spliced ​​image to obtain a fourth feature map. Therefore, the spliced ​​fourth feature map includes the original input information. In this way, the subsequent intermediate layers of the interpolation model can learn the original input information, as well as key features such as perspective transformation and posture transformation in the first image and the second image, thereby ensuring that the interpolation model does not lose important feature information, thereby ensuring the performance of the model.

[0156] Based on this, the first transformation feature map is equivalent to the optimized first difference data, and the second transformation feature map is equivalent to the optimized second difference data. In this way, the accuracy of the first transformation feature map and the second transformation feature map for predicting the third image is higher than the accuracy of the first difference data and the second difference data for predicting the third image.

[0157] It should be noted that S407-S412 are optional steps. When S407-S412 are not executed, after the third module outputs the third feature map, the electronic device can directly input the third feature map, the first difference data and the second difference data into the fourth module to obtain the third image.

[0158] S407: Input the fourth feature map into the fifth module to obtain a fifth feature map, and the fifth module is used to extract features from the fourth feature map.

[0159] Among them, the fifth module includes a convolutional layer, and the structure of the fifth module can be the same as the first module and the third module, and can also be different from at least one of the first module and the third module.

[0160] When the structure of the fifth module is the same as the first module and the third module, if the first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer, the first module includes a first sublayer and a second sublayer; the third module includes a first sublayer and a second sublayer; and the fifth module includes a first sublayer and a second sublayer.

[0161] When the structure of the fifth module is different from that of the first module and the same as that of the third module, if the first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer, the first module includes one first sublayer and one second sublayer; for example, the third module includes two first sublayers and one second sublayer, and the fifth module includes two first sublayers and one second sublayer; for example, the third module includes three first sublayers and one second sublayer, and the fifth module may include three first sublayers and one second sublayer.

[0162] When the structure of the fifth module is different from that of the first module and the third module, if the first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer, the first module includes a first sublayer and a second sublayer; for example, the third module includes two first sublayers and one second sublayer, and the fifth module includes four first sublayers and one second sublayer.

[0163] This application does not specifically limit the structure of the fifth module.

[0164] The fifth module is used to extract features from the fourth feature map, and the fifth feature map obtained by inputting the fourth feature map into the fifth module includes the image features of the fourth feature map.

[0165] S408. The fifth feature map, the first transformed feature map, and the second transformed feature map are input into the sixth module to obtain a seventh feature map. The transformation module in the sixth module is used to normalize each feature data in the fifth feature map to obtain a sixth feature map. The third segmentation module is used to segment the sixth feature map into a fifth sub-feature map and a sixth sub-feature map. The third calculation module is used to generate a seventh feature map including target information according to the fifth sub-feature map, the sixth sub-feature map, the first transformed feature map, and the second transformed feature map. The target information includes light and shadow information and / or occlusion information. The target information included in the seventh feature map is the target information included in the first image and the second image.

[0166] Combination Fig.10 Specifically, after the electronic device inputs the fifth feature map, the first transformed feature map, and the second transformed feature map into the sixth module, the transformation module in the sixth module can normalize each feature data in the sixth feature map through a logical function (which can be called a sigmoid function), that is, each feature data in the sixth feature map is mapped to the [0,1] interval to obtain the sixth feature map.

[0167] Based on the above description, the transformation module normalizes each feature data in the sixth feature map through a logical function, and the obtained sixth feature map is a feature map including multiple weight coefficients, so that target information (light and shadow information and / or occlusion information) can be introduced into the seventh feature map.

[0168] Next, the third segmentation module in the sixth module can segment the sixth feature map to obtain a fifth sub-feature map and a sixth sub-feature map. Taking the sixth feature map as a feature map of two dimensions as an example, Fig.10 It can be seen that the third segmentation module can segment between the two dimensions of the sixth feature map to obtain the fifth sub-feature map and the sixth sub-feature map ( Fig.10 in [:,:1,:,:] and [:,1:2,:,:].

[0169] Next, the third calculation module in the sixth module can fuse (add) the fifth sub-feature map and the sixth sub-feature map to obtain a fourth fused feature map, the third calculation module can also multiply the fifth sub-feature map and the first transformed feature map to obtain a first enhanced feature map, the third calculation module can also multiply the sixth sub-feature map and the second transformed feature map to obtain a second enhanced feature map, fuse (add) the first enhanced feature map and the second enhanced feature map to obtain a fifth fused feature map, and divide the fifth fused feature map by the fourth fused feature map to obtain a seventh feature map.

[0170] Based on the above description, the sixth module can output a seventh feature map, and the output seventh feature map includes target information in the first image and the second image, and the target information includes light and shadow information and / or occlusion information.

[0171] Optionally, the third calculation module can also obtain a first product by multiplying the infinitesimal quantity C by the scale factor k, and add the first product to the fused feature map of the fifth sub-feature map and the sixth sub-feature map to obtain a fourth fused feature map, thereby avoiding the situation where the numerator is equal to 0 when dividing the fifth fused feature map by the fourth fused feature map, thereby preventing the interpolation model from running abnormally.

[0172] It should be noted that S409-S412 are optional steps. When S409-S412 are not executed, after the fifth module outputs the fifth feature map, the electronic device can directly input the fifth feature map, the first transformed feature map and the second transformed feature map into the sixth module to obtain the third image.

[0173] S409: splice the seventh feature map and the spliced ​​image to obtain an eighth feature map.

[0174] The stitching of the seventh feature map and the stitching image is stitching in the channel direction.

[0175] S410, inputting the eighth feature map into the seventh module to obtain a ninth feature map, and the seventh module is used to extract features from the eighth feature map.

[0176] Among them, the seventh module includes a convolution layer, and the structure of the seventh module can be the same as the first module, the third module and the fifth module, or different from at least one of the first module, the third module and the fifth module.

[0177] This application does not specifically limit the structure of the seventh module.

[0178] S411. Fuse the seventh feature map and the ninth feature map to obtain a third fused feature map.

[0179] S412. Input the third fused feature map into the first activation function to obtain a third image. The first activation function is used to convert the third fused feature map into an image in which pixel values ​​are kept within a preset range.

[0180] Among them, the preset range is [0, 255]; it can be understood that usually the RGB value corresponding to each pixel in the image is between [0, 255]. Therefore, the third fusion feature map needs to be converted so that the RGB value corresponding to each pixel is between [0, 255].

[0181] In the video interpolation method of the present application, after obtaining the first feature map, the electronic device can also input the spliced ​​image into the second module in addition to the first feature map, and after obtaining the third feature map, the electronic device can also input the spliced ​​image into the fourth module in addition to the third feature map, so that the middle layer of the interpolation model can also learn the original image information (referring to the spliced ​​image), avoid the interpolation model from losing important feature information, and ensure the performance of the model.

[0182] Furthermore, before inputting the first image and the second image into the first module, the electronic device splices the first image and the second image to obtain a spliced ​​image, after which the first module can directly process the spliced ​​image, and in the intermediate process, the second module can directly splice the first difference data, the second difference data and the spliced ​​image, without the need to splice the first image and the second image before executing the step of splicing the first difference data, the second difference data and the spliced ​​image, and the fourth module can directly splice the first transformation feature map, the second transformation feature map and the spliced ​​image, without the need to splice the first image and the second image again before executing the step of splicing the first transformation feature map, the second transformation feature map and the spliced ​​image, thereby saving the splicing time of the first image and the second image and shortening the time for the interpolation model to output the third image.

[0183] Furthermore, after inputting the first feature map and the spliced ​​image into the second module to obtain the second feature map, the electronic device can also input the second feature map into the third module, perform feature extraction on the second feature map to obtain the third feature map, input the third feature map, the first difference data, the second difference data, the first image, the second image and the spliced ​​image into the fourth module to obtain the first transformation feature map and the second transformation feature map, and splice the first transformation feature map, the second transformation feature map and the spliced ​​image to obtain the fourth feature map, the first transformation feature map is equivalent to the optimized first difference data, and the second transformation feature map is equivalent to the optimized second difference data. In this way, by correcting the first difference data and the second difference data, the accuracy of the first transformation feature map and the second transformation feature map for predicting the third image can be higher than the accuracy of the first difference data and the second difference data for predicting the third image.

[0184] In addition, after obtaining the fifth feature map, the electronic device can also input the fifth feature map, the first transformation feature map and the second transformation feature map into the sixth module, so that through the processing of the third segmentation module and the third calculation module, the seventh feature map can be obtained. The seventh feature map may include light and shadow information and / or occlusion information in the first image and the second image. In this way, it can be ensured that there will be no abrupt changes when the subsequent first image, the third image obtained by the seventh feature map, and the second image are played in sequence, which can avoid the visual jump during playback and ensure the continuity during playback.

[0185] In addition, after obtaining the ninth feature map, the electronic device can also fuse the seventh feature map with the ninth feature map to form a residual connection, so as to obtain a third fused feature map, so as to retain as many image detail features as possible including texture and clear edges to ensure the performance of the model.

[0186] based on Figure 7 In a specific embodiment, the following is assumed: 1. The electronic device is a mobile phone.

[0187] 2. The first module is a pure convolution stacking module 1, the second module is a redundant simplified post-processing module 1, the third module is a pure convolution stacking module 2, the fourth module is a redundant simplified post-processing module 2, the fifth module is a pure convolution stacking module 3, the sixth module is a redundant simplified post-processing module 3, the seventh module is a pure convolution stacking module 4, and the first activation function is a truncated activation function.

[0188] 3. The first image is input 1, the second image is input 2, and the spliced ​​image is input C.

[0189] Based on the above assumptions, if Fig.11 As shown, the mobile phone may perform the video frame insertion method of the embodiment of the present application, including the following steps: Step 21: Acquire a first image and a second image from a target video, where the first image is a frame in the target video that is earlier than the second image in time sequence.

[0190] Step 22: stitch the first image and the second image to obtain a stitched image.

[0191] Step 23: Input the spliced ​​image into the pure convolution stacking module 1 to obtain the first feature map.

[0192] Step 24: input the first feature map and the spliced ​​image into the redundancy simplification post-processing module 1 to obtain a second feature map.

[0193] Step 25: Input the second feature map into the pure convolution stacking module 2 to obtain the third feature map.

[0194] Step 26: input the third feature map, the first difference data, the second difference data, the first image, the second image and the spliced ​​image into the redundancy simplification post-processing module 2 to obtain a fourth feature map.

[0195] Step 27: Input the fourth feature map into the pure convolution stacking module 3 to obtain the fifth feature map.

[0196] Step 28: Input the fifth feature map, the first transformed feature map and the second transformed feature map into the redundancy simplification post-processing module 3 to obtain the seventh feature map (intermediate state output 1).

[0197] Step 29: splice the seventh feature map and the spliced ​​image to obtain an eighth feature map.

[0198] Step 30: Input the eighth feature map into the pure convolution stacking module 4 to obtain a ninth feature map.

[0199] Step 31: Fuse the seventh feature map and the ninth feature map to obtain a third fused feature map.

[0200] Step 32: Input the third fused feature map into the truncated activation function to obtain a third image.

[0201] Exemplarily, the present application provides a video frame interpolation device, which includes a module for executing the video frame interpolation method in the above embodiment.

[0202] Exemplarily, the present application provides an electronic device, including a processor; when the processor executes computer code or instructions in the memory, the electronic device executes the video interpolation method in the above embodiment.

[0203] Exemplarily, the present application provides an electronic device, comprising: a memory and a processor; the memory is coupled to the processor, and the memory is used to store program codes or instructions; the processor is used to call the program codes or instructions in the memory so that the electronic device executes the video interpolation method in the previous embodiment.

[0204] Exemplarily, the present application provides a chip system, which is applied to an electronic device including a memory, a display screen and a sensor; the chip system includes: one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive signals from the memory and send signals to the processor, and the signals include computer codes or instructions stored in the memory; when the electronic device (which may be an NPU in the electronic device) executes the computer code or instructions, the electronic device executes the video interpolation method in the previous embodiment.

[0205] Exemplarily, the present application provides a computer-readable storage medium, in which codes or instructions are stored. When the codes or instructions are executed on an electronic device, the electronic device implements the video interpolation method in the foregoing embodiment.

[0206] Exemplarily, the present application provides a computer program product, which, when executed on a computer, enables an electronic device to implement the video frame insertion method in the foregoing embodiment.

[0207] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer codes or instructions. When the computer program code or instructions are loaded and executed on the computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer code or instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0208] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A video frame insertion method, characterized in that: Applied to an electronic device, the electronic device includes an interpolation model, the interpolation model includes a first module and a second module, the first module includes a convolution layer, and the second module includes a first segmentation module and a first calculation module; the method includes: Acquire a first image and a second image from a target video, wherein the first image is a frame in the target video that is earlier than the second image in time sequence; Inputting the first image and the second image into the first module to obtain a first feature map, wherein the first module is used to extract pixel features from the first image and the second image, and the first feature map includes pixel features of the first image and pixel features of the second image; The first feature map is input into the second module to obtain a third image, the first segmentation module in the second module is used to segment the first feature map into a first sub-feature map and a second sub-feature map, the first calculation module is used to determine the difference data between the first sub-feature map and the second sub-feature map, and predict the third image according to the difference data, the difference data is used to represent the movement of pixels from the first image to the second image in a time series, and the third image is an intermediate image to be inserted between the first image and the second image.

2. The method according to claim 1, characterized in that The step of inputting the first image and the second image into the first module to obtain a first feature map includes: splicing the first image and the second image to obtain a spliced ​​image; The stitched image is input into the first module to obtain the first feature map.

3. The method according to claim 2, characterized in that The interpolation model further includes a third module and a fourth module, the third module includes a convolution layer, and the fourth module includes a second segmentation module and a second calculation module; the step of inputting the first feature map into the second module to obtain a third image includes: Inputting the first feature map and the stitched image into the second module to obtain a second feature map, the first calculation module is used to determine first difference data and second difference data between the first sub-feature map and the second sub-feature map, and stitching the first difference data, the second difference data and the stitched image, the first difference data is used to represent the movement of pixels from the first image to the second image in a time series, and the second difference data is used to represent the movement of pixels from the second image to the first image in a time series; Inputting the second feature map into the third module to obtain a third feature map, wherein the third module is used to extract features from the second feature map; The third feature map, the first difference data, the second difference data, the first image and the second image are input into the fourth module to obtain the third image, the second segmentation module in the fourth module is used to segment the third feature map into a third sub-feature map and a fourth sub-feature map, the second calculation module is used to fuse the first difference data and the third sub-feature map to obtain a first fused feature map, fuse the second difference data and the fourth sub-feature map to obtain a second fused feature map, warp the first image according to the first fused feature map to obtain a first transformed feature map, warp the second image according to the second fused feature map to obtain a second transformed feature map, and predict the third image according to the first transformed feature map and the second transformed feature map, the accuracy of the first transformed feature map and the second transformed feature map for predicting the third image is higher than the accuracy of the first difference data and the second difference data for predicting the third image.

4. The method according to claim 3, characterized in that The interpolation model further includes a fifth module and a sixth module, the fifth module includes a convolution layer, the sixth module includes a transformation module, a third segmentation module and a third calculation module; the third feature map, the first difference data and the second difference data are input into the fourth module to obtain the third image, including: Inputting the third feature map, the first difference data, the second difference data, the first image, the second image and the spliced ​​image into the fourth module to obtain a fourth feature map, where the fourth feature map is obtained by splicing the first transformed feature map, the second transformed feature map and the spliced ​​image; Inputting the fourth feature map into the fifth module to obtain a fifth feature map, wherein the fifth module is used to extract features from the fourth feature map; The fifth feature map, the first transformed feature map and the second transformed feature map are input into the sixth module to obtain the third image, the transformation module in the sixth module is used to normalize each feature data in the fifth feature map to obtain a sixth feature map, the third segmentation module is used to segment the sixth feature map into a fifth sub-feature map and a sixth sub-feature map, the third calculation module is used to generate the third image including target information according to the fifth sub-feature map, the sixth sub-feature map, the first transformed feature map and the second transformed feature map, the target information including light and shadow information and / or occlusion information, and the target information included in the third image is the target information included in the first image and the second image.

5. The method according to claim 4, characterized in that The interpolation model further includes a seventh module and a first activation function, the seventh module includes a convolution layer, and the fifth feature map, the first transformation feature map, and the second transformation feature map are input into the sixth module to obtain the third image, including: Inputting the fifth feature map, the first transformed feature map and the second transformed feature map into the sixth module to obtain a seventh feature map; Splicing the seventh feature map and the spliced ​​image to obtain an eighth feature map; Inputting the eighth feature map into the seventh module to obtain a ninth feature map, wherein the seventh module is used to extract features from the eighth feature map; Fusing the seventh feature map and the ninth feature map to obtain a third fused feature map; The third fused feature map is input into the first activation function to obtain the third image, and the first activation function is used to convert the third fused feature map into an image in which pixel values ​​are kept within a preset range.

6. The method according to claim 2, characterized in that The first module includes a first sublayer and a second sublayer, the first sublayer includes a first convolutional layer and a second activation function layer, and the second sublayer includes a second convolutional layer; the step of inputting the spliced ​​image into the first module to obtain the first feature map includes: Inputting the stitched image into the first sublayer to obtain a first intermediate feature map, the first convolution layer in the first sublayer is used to extract pixel features of the stitched image to obtain a second intermediate feature map, the second activation function layer is used to perform a nonlinear transformation on the second intermediate feature map to obtain the first intermediate feature map, and in the process of extracting pixel features of the stitched image by the first convolution layer, a convolution step length used is related to the target output duration of the third image, the convolution step length is an even number greater than or equal to 2, and a size of the first intermediate feature map is related to the convolution step length; The first intermediate feature map is input into the second sub-layer to obtain the first feature map, and the second convolution layer in the second sub-layer is used to adjust the size of the first intermediate feature map to the same size as the spliced ​​image.

7. The method according to claim 6, characterized in that The first module further includes a padding layer and a truncation layer, and the spliced ​​image is input into the first sub-layer to obtain a first intermediate feature map, including: Inputting the stitched image into the padding layer, and determining through the padding layer whether the length and width of the stitched image are odd numbers; If the length and / or width of the stitched image is an odd number, performing a first expansion on pixels of the stitched image to obtain an updated stitched image, wherein the first expansion is used to make the length and width of the stitched image both even numbers; Inputting the updated spliced ​​image into the first sublayer to obtain the first intermediate feature map; The step of inputting the first intermediate feature map into the second sub-layer to obtain the first feature map includes: Inputting the first intermediate feature map into the second sub-layer to obtain a second intermediate feature map, wherein the second convolutional layer of the second sub-layer is used to adjust the size of the first intermediate feature map to the same size as the updated spliced ​​image; The second intermediate feature map is input into the truncation layer to obtain the first feature map, and the truncation layer is used to adjust the size of the second intermediate feature map to the same size as the stitched image.

8. The method according to any one of claims 1 to 7, characterized in that The first image and the second image are two adjacent frames of images in the target video, or the first image and the second image are two frames of images corresponding to each other at an interval of N frames in the target video, where N is a positive integer greater than or equal to 1.

9. The method according to any one of claims 1 to 7, characterized in that The generation process of the interpolation model includes: Acquire a plurality of training image groups, each of the plurality of training image groups comprising three images that are continuous in time series, and the second image in each of the training image groups is a labeled intermediate image corresponding to each of the training image groups; Inputting the first image and the third image in each training image group into a first sample module of a neural network to obtain a sample feature map, wherein the first sample module is used to extract pixel features of the first image and the third image through a convolutional layer included in the first sample module, and the sample feature map includes pixel features of the first image and pixel features of the third image; Input the sample feature map into the second sample module of the neural network to obtain an estimated intermediate image corresponding to each training image group, wherein the second sample module includes a first sample segmentation module and a first sample calculation module, wherein the first sample segmentation module is used to segment the sample feature map into a first sub-sample feature map and a second sub-sample feature map, and the first sample calculation module is used to determine sample difference data between the first sub-sample feature map and the second sub-sample feature map, and determine an estimated intermediate image according to the sample difference data, wherein the sample difference data is used to represent the movement of pixels from the first image to the third image in a time series; The neural network is trained according to the difference between the labeled intermediate image and the estimated intermediate image to obtain the interpolation model.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1 to 9.

11. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when executed on an electronic device, enable the electronic device to perform the method as claimed in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 9.

Images