A video frame interpolation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a video frame interpolation method and device, electronic equipment and storage medium, comprising: performing key point detection of a preset type object and mask segmentation processing of the preset type object on a first video frame and a third video frame to obtain a first key point image, a third key point image, a first mask image and a third mask image; then performing frame interpolation prediction according to the first key point image, the third key point image, the first mask image and the third mask image to obtain an intermediate video frame. Since the key point image contains key point information, the intermediate video frame predicted based on the key point image can improve the disappearance of the limbs. Since the mask image contains the preset type object region, the intermediate video frame predicted based on the mask image has a smoother edge and reduces artifacts. The scheme of the present application can improve the appearance of artifacts or the disappearance of part of the limbs in the frame interpolation process for the human body.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of frame interpolation, and particularly relates to a video frame interpolation method and device, electronic equipment and storage medium. BACKGROUND

[0002] Video frame interpolation refers to adding one or more frames in each two frames of original video, shortening the display time between each frame, correcting the illusion formed by human visual persistence, and thus improving the fluency of the video. In the process of video frame interpolation by optical flow method, when the human body performs rapid and large motion, resulting in a large displacement, the video frame interpolation effect is often poor, and the frame interpolation effect of the human body may appear artifacts or partial body disappearance, thereby affecting the fluency of the overall video. SUMMARY

[0003] Therefore, the present application provides a video frame interpolation method and device, electronic equipment and storage medium, which are used to improve the artifacts or partial body disappearance in the frame interpolation process of the human body.

[0004] In the first aspect of the present application, a video frame interpolation method is provided, and the method comprises the following steps:

[0005] obtaining a first video frame and a third video frame, wherein the first video frame and the third video frame comprise a same preset type object;

[0006] performing key point detection of the preset type object on the first video frame and the third video frame to obtain a first key point image and a third key point image;

[0007] performing mask segmentation processing of the preset type object on the first video frame and the third video frame to obtain a first mask image and a third mask image;

[0008] performing frame interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

[0009] In the embodiments of the present application, key point detection of a preset type object is performed on the first video frame and the third video frame to obtain a first key point image and a third key point image; mask segmentation processing of the preset type object is performed on the first video frame and the third video frame to obtain a first mask image and a third mask image; then, according to the first key point image, the third key point image, the first mask image and the third mask image, interpolation prediction is performed to obtain an intermediate video frame. Since the first key point image and the third key point image contain key point information, the intermediate video frame predicted based on the key point image can improve the situation of limb disappearance. Since the first mask image and the third mask image contain the preset type object region, the intermediate video frame predicted based on the mask image has a smoother edge and reduces artifacts. In the present application, according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame, interpolation prediction is performed to obtain an intermediate video frame, which can improve the situation of artifacts or partial limb disappearance in the interpolation process for the human body.

[0010] In a possible embodiment, the interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame comprises:

[0011] performing feature extraction on the first key point image, the third key point image, the first video frame and the third video frame to obtain a first key point image feature, a third key point image feature, a first video frame feature and a third video frame feature;

[0012] performing optical flow estimation based on the first key point image feature, the third key point image feature, the first video frame feature and the third video frame feature to obtain a first stage prediction optical flow and a second stage prediction optical flow, the first stage prediction optical flow being used to describe the motion of the pixel points of the first video frame to the second video frame, and the second stage prediction optical flow being used to describe the motion of the pixel points of the third video frame to the second video frame; the second video frame being a video frame between the first video frame and the third video frame;

[0013] fusing the first stage prediction optical flow, the second stage prediction optical flow, the first video frame, the third video frame, the first mask image and the third mask image to obtain an intermediate video frame.

[0014] In the embodiments of the present application, in addition to feature extraction on the video frames, feature extraction is also performed on the key point images obtained through key point detection, and then optical flow estimation is performed based on the first key point image features, the third key point image features, the first video frame features and the third video frame features, so that the solved optical flow is more accurate, and then the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image and the third mask image are fused to obtain an intermediate video frame, effectively improving the situation that the limbs disappear in the frame insertion result when the human body moves rapidly.

[0015] In a possible embodiment, the feature extraction on the first key point image, the third key point image, the first video frame and the third video frame to obtain the first key point image features, the third key point image features, the first video frame features and the third video frame features comprises:

[0016] The first key point image and the third key point image are subjected to image channel dimension reduction processing to obtain a first key point single-channel image and a third key point single-channel image.

[0017] The first key point single-channel image and the third key point single-channel image are subjected to feature extraction to obtain first key point image features and third key point image features.

[0018] The first video frame and the third video frame are subjected to feature extraction to obtain first video frame features and third video frame features.

[0019] In the embodiments of the present application, after obtaining the first key point image and the third key point image, the two key point images are processed into single-channel images, and then feature extraction is performed on the single-channel images. The single-channel images have single color, small data volume, low memory occupation and fast image processing speed.

[0020] In a possible embodiment, the image channel dimension reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image comprises:

[0021] For each key point in the first key point image, the pixels in a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a first intermediate image; the pixel values of each pixel in the first intermediate image that is not subjected to single-channel Gaussian distribution processing are single-channeled into a first preset value to obtain a first key point single-channel image.

[0022] For each key point in the third key point image, pixels in a preset range centered on the key point are processed by single-channel Gaussian distribution to obtain a third intermediate image; pixel values of each pixel in the third intermediate image which is not processed by single-channel Gaussian distribution are single-channeled into a first preset value to obtain a third key point single-channel image.

[0023] In the embodiments of the present application, by processing pixels in a preset range centered on a key point by single-channel Gaussian distribution and single-channeling pixel values of the remaining pixels into a first preset value, the key point image is reduced in dimension to a single-channel image.

[0024] In a possible embodiment, the fusing the first stage predicted optical flow, the second stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame comprises:

[0025] The first stage predicted optical flow, the second stage predicted optical flow, the first video frame, and the third video frame are fused to obtain a fusion result;

[0026] Based on the first stage predicted optical flow, the second stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image, an edge refinement map is obtained;

[0027] The fusion result and the edge refinement map are fused to obtain an intermediate video frame.

[0028] In the embodiments of the present application, first, a fusion result is obtained based on optical flow and video frames, and then an edge refinement map is obtained based on optical flow, video frames, and mask images. The fusion result and the edge refinement map are fused to obtain an intermediate video frame. Since the edge refinement map can improve the phenomenon of burrs and unevenness of the edges of a preset type of object, the intermediate video frame obtained by fusing the fusion result and the edge refinement map can make the edges of a portrait more smooth and reduce artifacts.

[0029] In a possible embodiment, the fusing the first stage predicted optical flow, the second stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result comprises:

[0030] The first stage predicted optical flow and the first video frame are subjected to affine transformation to obtain a first transformation result;

[0031] The second stage predicted optical flow and the second video frame are subjected to affine transformation to obtain a second transformation result;

[0032] Based on the first transformation result and the second transformation result, a fusion result is obtained.

[0033] In the embodiments of the present application, the optical flow and the video frame are subjected to affine transformation, and the fusion result of the optical flow and the video frame is obtained based on the transformation results of the two sets of affine transformation, thereby maintaining the flatness and parallelism of the two-dimensional image.

[0034] In a possible embodiment, the obtaining of the edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image comprises:

[0035] A first weight value is set for a preset type object region in the first mask image and the third mask image.

[0036] A second weight value is set for a non-pre-set type object region in the first mask image and the third mask image.

[0037] The edge refinement map is obtained based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with the region weight value, and the third mask image with the region weight value.

[0038] In the embodiments of the present application, the mask image is added to the calculation of the edge refinement map as a weight map by setting the weight values for the preset type object region and the non-pre-set type object region of the mask image, and the edge refinement map with smooth edges is obtained.

[0039] In a second aspect, the present application provides a video frame interpolation device, which comprises:

[0040] An acquisition module is configured to acquire a first video frame and a third video frame, wherein the first video frame and the third video frame comprise a same preset type object.

[0041] A key point detection module is configured to perform key point detection of the preset type object on the first video frame and the third video frame, and obtain a first key point image and a third key point image.

[0042] A mask segmentation module is configured to perform mask segmentation processing of the preset type object on the first video frame and the third video frame, and obtain a first mask image and a third mask image.

[0043] An interpolation prediction module is configured to perform interpolation prediction based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame, and obtain an intermediate video frame.

[0044] In a possible embodiment, the interpolation prediction module comprises:

[0045] The feature extraction module is configured to perform feature extraction on the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features.

[0046] The optical flow solving module is configured to perform optical flow estimation based on the first key point image features, the third key point image features, the first video frame features, and the third video frame features to obtain first-stage predicted optical flow and second-stage predicted optical flow. The first-stage predicted optical flow is used to describe the motion of pixel points from the first video frame to the second video frame, and the second-stage predicted optical flow is used to describe the motion of pixel points from the third video frame to the second video frame. The second video frame is a video frame between the first video frame and the third video frame.

[0047] The intermediate video frame determining module is configured to fuse the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame.

[0048] In a possible embodiment, the feature extraction module includes:

[0049] The channel dimension reduction module is configured to perform image channel dimension reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image.

[0050] The single-channel image feature extraction module is configured to perform feature extraction on the first key point single-channel image and the third key point single-channel image to obtain first key point image features and third key point image features.

[0051] The video frame feature extraction module is configured to perform feature extraction on the first video frame and the third video frame to obtain first video frame features and third video frame features.

[0052] In a possible embodiment, the channel dimension reduction module is specifically configured to:

[0053] For each key point in the first key point image, perform single-channel Gaussian distribution processing on pixels within a preset range centered on the key point to obtain a first intermediate image. The pixel values of each pixel in the first intermediate image that is not subjected to single-channel Gaussian distribution processing are single-channeled to a first preset value to obtain a first key point single-channel image.

[0054] For each key point in the third key point image, pixels in a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a third intermediate image; pixel values of each pixel in the third intermediate image that is not subjected to single-channel Gaussian distribution processing are single-channeled into a first preset value to obtain a third key point single-channel image.

[0055] In a possible embodiment, the intermediate video frame determination module comprises:

[0056] a fusion module configured to fuse the first-stage prediction optical flow, the second-stage prediction optical flow, the first video frame, and the third video frame to obtain a fusion result;

[0057] an edge refinement map determination module configured to obtain an edge refinement map based on the first-stage prediction optical flow, the second-stage prediction optical flow, the first video frame, the third video frame, the first mask image, and the third mask image;

[0058] an intermediate video frame determination unit configured to fuse the fusion result and the edge refinement map to obtain an intermediate video frame.

[0059] In a possible embodiment, the fusion module comprises:

[0060] a first affine transformation module configured to perform affine transformation on the first-stage prediction optical flow and the first video frame to obtain a first transformation result;

[0061] a second affine transformation module configured to perform affine transformation on the second-stage prediction optical flow and the second video frame to obtain a second transformation result;

[0062] a fusion unit configured to obtain a fusion result based on the first transformation result and the second transformation result.

[0063] In a possible embodiment, the edge refinement map determination module comprises:

[0064] a first weight value setting module configured to set a first weight value for a preset type object region in the first mask image and the third mask image;

[0065] a second weight value setting module configured to set a second weight value for a non-pre-set type object region in the first mask image and the third mask image;

[0066] an edge refinement map determination unit configured to obtain an edge refinement map based on the first-stage prediction optical flow, the second-stage prediction optical flow, the first video frame, the third video frame, the first mask image with a region weight value, and the third mask image with a region weight value.

[0067] In a third aspect of the present application, an electronic device is provided, comprising:

[0068] one or more processors and a memory;

[0069] The memory is coupled to the one or more processors, and the memory is configured to store computer program codes including computer instructions, and the one or more processors are configured to invoke the computer instructions to cause the electronic device to perform the method steps of the first aspect.

[0070] In a fourth aspect of the present application, a computer readable storage medium is provided, comprising a computer program, which, when executed on an electronic device, causes the electronic device to perform the method steps of the first aspect.

[0071] In the embodiments of the present application, key point detection of the first video frame and the third video frame is performed on a preset type object to obtain a first key point image and a third key point image; mask segmentation processing of the first video frame and the third video frame is performed on the preset type object to obtain a first mask image and a third mask image; and then, interpolation prediction is performed according to the first key point image, the third key point image, the first mask image and the third mask image to obtain an intermediate video frame. Since the first key point image and the third key point image contain key point information, the intermediate video frame predicted based on the key point image can improve the situation of limb disappearance. Since the first mask image and the third mask image contain the preset type object region, the intermediate video frame predicted based on the mask image has a smoother edge and reduces artifacts. In the present application, interpolation prediction is performed according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame, which can improve the situation of artifacts or partial limb disappearance in the interpolation process for the human body. BRIEF DESCRIPTION OF DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0073] Figure 1 It is a structural schematic diagram of an electronic device;

[0074] Figure 2 It is a schematic diagram of the system architecture of an electronic device;

[0075] Figure 3A first flowchart of a video interpolation method provided by an embodiment of the present application;

[0076] Figure 4 A result diagram for mask segmentation processing;

[0077] Figure 5 A second flowchart of a video interpolation method provided by an embodiment of the present application;

[0078] Figure 6 A third flowchart of a video interpolation method provided by an embodiment of the present application;

[0079] Figure 7 A pixel value diagram for a single key point and its surrounding pixels;

[0080] Figure 8 A fourth flowchart of a video interpolation method provided by an embodiment of the present application;

[0081] Figure 9 A fifth flowchart of a video interpolation method provided by an embodiment of the present application;

[0082] Figure 10 A structure diagram of a video interpolation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to better understand the technical solutions of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0084] It should be clear that the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0085] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0086] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0087] Reference Figure 1 , Figure 1A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application is shown. The electronic device can be a mobile phone, a tablet computer, or other user terminal.

[0088] The electronic device includes a processor 110, a transceiver 120, and a display unit 170. The display unit 170 can include a display screen.

[0089] Optionally, the electronic device can further include a memory 130. The processor 110, the transceiver 120, and the memory 130 can communicate with each other through internal connection paths to transfer control and / or data signals. The memory 130 is configured to store a computer program, and the processor 110 is configured to call and run the computer program from the memory 130.

[0090] Optionally, the electronic device can further include an antenna 140 for transmitting wireless signals output by the transceiver 120.

[0091] The processor 110 described above can be combined with the memory 130 into one processing device, or more commonly, be independent components. The processor 110 is configured to execute program codes stored in the memory 130 to implement the above functions. In a specific implementation, the memory 130 can be integrated in the processor 110, or be independent of the processor 110.

[0092] In addition, in order to make the functions of the electronic device more perfect, the electronic device can further include one or more of an input unit 160, an audio circuit 180, a camera 190, and a sensor 101. The audio circuit can further include a speaker 182 and a microphone 184.

[0093] Optionally, the electronic device can further include a power supply 150 for providing power supply to various devices or circuits in the electronic device.

[0094] It can be understood that, Figure 1 The operations and / or functions of each module in the electronic device are respectively used to implement the corresponding processes in the method embodiments described below. For details, please refer to the description in the method embodiments below. To avoid repetition, the detailed description is appropriately omitted here.

[0095] It can be understood that, Figure 1The processor 110 in the illustrated electronic device can include one or more processing units. For example, the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors. The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or are recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0096] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0097] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation on the electronic device. In some other embodiments of the present application, the electronic device can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0098] It can be understood that, Figure 1 The power supply 150 is configured to supply power to the processor 110, the memory 130, the display unit 170, the camera 190, the input unit 160, the transceiver 120, and the like. The antenna 140 is configured to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 140 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0099] The transceiver 120 can provide a wireless communication solution applied to the electronic device, including wireless local area networks (WLAN) (such as a wireless fidelity (Wi-Fi) network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), and the like. The transceiver 120 can be one or more devices integrated with at least one communication processing module. The transceiver 120 receives electromagnetic waves via the antenna 140, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The transceiver 120 can also receive signals to be transmitted from the processor 110, perform frequency modulation, amplification, and convert the signals to electromagnetic wave radiation via the antenna 140.

[0100] In some embodiments, the antenna 140 and the transceiver 120 of the electronic device are coupled such that the electronic device can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0101] The electronic device implements a display function through a GPU, the display unit 170, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display unit 170 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0102] The display unit 170 is configured to display images, videos, and the like. The display unit 170 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the electronic device can include one or N display units 170, where N is a positive integer greater than 1.

[0103] The electronic device can implement a photographing function through an ISP, the camera 190, a video codec, a GPU, the display unit 170, and an application processor.

[0104] The ISP is configured to process data fed back by the camera 190. For example, when recording a video, the camera is turned on, light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image.

[0105] The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be disposed in the camera 190. The camera 190 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device can include one or N cameras 190, where N is a positive integer greater than 1. The digital signal processor is configured to process digital signals, in addition to processing digital image signals. For example, when the electronic device selects a frequency point, the digital signal processor is configured to perform Fourier transform on the frequency point energy, and the like.

[0106] Video codecs are used to compress or decompress digital video. An electronic device can support one or more video codecs. In this way, the electronic device can play or record video in a variety of encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and so on.

[0107] An NPU is a neural-network (NN) computing processor that rapidly processes input information by drawing on the structure of a biological neural network, such as the mode of transmission between neurons in the human brain, and can also constantly self-learn. Through an NPU, the electronic device can implement intelligent cognitive applications, such as image recognition, facial recognition, voice recognition, text understanding, and so on.

[0108] The memory 130 can be used to store computer-executable program code, including instructions. The memory 130 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, and so on), and the like. The data storage area can store data created during use of the electronic device (such as audio data, a phone book, and so on), and the like. In addition, the memory 130 can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and so on. The processor 110 executes various function applications and data processing of the electronic device by running instructions stored in the memory 130 and / or instructions stored in a memory disposed in the processor.

[0109] The electronic device can implement audio functions through the audio circuit 180, the speaker 182, the microphone 184, and an application processor, and so on. For example, music playing, voice recording, and so on.

[0110] The audio circuit 180 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio circuit 180 can also be used to encode and decode audio signals. In some embodiments, the audio circuit 180 can be disposed in the processor 110, or some function modules of the audio circuit 180 can be disposed in the processor 110.

[0111] The speaker 182, also known as a “loudspeaker”, is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or listen to a hands-free call through the speaker 182.

[0112] The microphone 184, also referred to as a "microphone", "microphone", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can input a sound signal into the microphone 184 by speaking close to the microphone 184. The electronic device can be provided with at least one microphone 184. In other embodiments, the electronic device can be provided with two microphones 184, in addition to collecting sound signals, noise reduction functions can also be achieved. In other embodiments, the electronic device can also be provided with three, four or more microphones 184, to achieve the functions of collecting sound signals, noise reduction, and identifying sound sources, and achieving directional recording functions, etc.

[0113] The video interpolation method in the embodiments of the present application can be implemented through the system architecture shown in Figure 2 Figure 2 The system architecture of the electronic device includes a kernel part, a framework layer part and an application layer part; the kernel part includes a driver layer and a real-time operating system, the driver layer includes a GPU (graphics processor), a display driver (specifically an LCD driver in the figure), a TP driver (touch screen driver), a key, etc.; the real-time operating system includes interrupt management, task scheduling, MEM (memory management); the framework layer includes: system basic capabilities, underlying software services, hardware service capabilities, etc.; the application layer includes: system applications (such as calendar, camera, contacts, browser, etc.), third-party applications (such as third-party social applications, video applications, etc.). After the camera application in the application layer shoots a motion video, the kernel layer calls the interpolation algorithm to execute the video interpolation method in the embodiments of the present application.

[0114] After shooting a motion video, there is a dataset of human motion scenes, and the specific types of the above motion videos include but are not limited to playing basketball, playing table tennis, playing badminton, waving hands, running, jumping. For the dataset of human motion scenes, further processing is needed to filter out video frames containing human bodies from the dataset, and to adjust the image size of the video frames to obtain data suitable for the interpolation algorithm. For data suitable for the interpolation algorithm, a key point detection algorithm is called to detect key points of the data, a mask segmentation algorithm is called to segment the data, and an interpolation algorithm is called to interpolate the data.

[0115] Video interpolation refers to adding one or more frames between every two frames of the original video, shortening the display time between each frame, correcting the illusion formed by the human eye persistence of vision, and thus improving the smoothness of the video. In the process of video interpolation by optical flow method, when the human body moves rapidly and greatly, causing a large displacement, the video interpolation effect is often poor, and the interpolation effect of the human body may have artifacts or some limbs disappear, thereby affecting the overall video smoothness.

[0116] Based on this, the present application provides a video interpolation method, which is shown in Figure 3 ​The specific process of the video frame interpolation method includes the following steps:

[0117] S201, obtaining a first video frame and a third video frame.

[0118] The first video frame and the third video frame include the same preset type object.

[0119] In the embodiments of the present application, a plurality of shooting devices are used to shoot slow-motion videos of human body movements (including playing basketball, playing table tennis, playing badminton, waving hands, running, jumping, etc.). The shooting devices can be cameras, mobile phones, etc. The advantage of this is that the frame interpolation network learns images of different qualities, enhances the generalization of the model, and processes the video into data suitable for the video frame interpolation method. The specific processing steps include: selecting video frames containing human body images from the video, and processing the image size of the video frames to 48X48 for the video frames containing human body images. The first video frame and the third video frame can be two adjacent frames or two non-adjacent frames, as long as the two frames include the same preset type object. In one example, person 1 can be set as the preset type object.

[0120] S202, performing key point detection of the preset type object on the first video frame and the third video frame to obtain a first key point image and a third key point image.

[0121] The first video frame and the third video frame can be subjected to key point detection of the preset type object by using a skeletal key point detection network. The key points include 17 skeletal key points of left and right ankles, left and right knees, left and right hips, left and right wrists, left and right elbows, left and right shoulders, left and right ears, left and right eyes, and back of the head. The skeletal key point detection network can specifically use the human pose as composition tokens network.

[0122] S203, performing mask segmentation processing of the preset type object on the first video frame and the third video frame to obtain a first mask image and a third mask image.

[0123] The first video frame and the third video frame can be subjected to mask segmentation processing of the preset type object by using a portrait segmentation network. In the case of taking person 1 as the preset type object, the first video frame and the third video frame are subjected to mask segmentation processing of the preset type object by using the portrait segmentation network, and two frames of person 1 segmentation masks are obtained. The portrait segmentation network can specifically be a unet network structure. One result of the mask segmentation processing is shown in Figure 4 , which is the mask segmentation processing of the human body jumping and shooting posture.

[0124] S204, performing frame interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame, to obtain an intermediate video frame.

[0125] According to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame, performing frame interpolation prediction to obtain an intermediate video frame, wherein the intermediate video frame refers to a video frame that can be inserted between the first video frame and the third video frame.

[0126] In the embodiments of the present application, key point detection of a preset type object is performed on the first video frame and the third video frame to obtain a first key point image and a third key point image; mask segmentation processing of the preset type object is performed on the first video frame and the third video frame to obtain a first mask image and a third mask image; and then, frame interpolation prediction is performed according to the first key point image, the third key point image, the first mask image and the third mask image to obtain an intermediate video frame. Since the first key point image and the third key point image contain key point information, the intermediate video frame predicted based on the key point image can improve the situation of limb disappearance. Since the first mask image and the third mask image contain the region of the preset type object, the intermediate video frame predicted based on the mask image has a smoother edge and reduces artifacts. In the present application, frame interpolation prediction is performed according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame, which can improve the situation of artifacts or partial limb disappearance in the frame interpolation process for the human body.

[0127] In one example, referring to Figure 5 The above step S204 specifically includes:

[0128] S301, performing feature extraction on the first key point image, the third key point image, the first video frame and the third video frame to obtain a first key point image feature, a third key point image feature, a first video frame feature and a third video frame feature.

[0129] In the traditional frame interpolation network, only the first video frame and the third video frame are subjected to feature extraction to obtain the first video frame feature and the third video frame feature; but in the video frame interpolation method in the embodiments of the present application, in addition to the feature extraction on the video frames, feature extraction is also performed on the key point images obtained by key point detection.

[0130] S302, performing optical flow estimation based on the first key point image feature, the third key point image feature, the first video frame feature and the third video frame feature to obtain a first stage prediction optical flow and a second stage prediction optical flow.

[0131] The first-stage predicted optical flow is used to describe the movement of a pixel point from the first video frame to the second video frame, and the second-stage predicted optical flow is used to describe the movement of a pixel point from the third video frame to the second video frame; the second video frame is a video frame between the first video frame and the third video frame.

[0132] S303, the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image are fused to obtain an intermediate video frame.

[0133] In the embodiment of the application, in addition to extracting features from the video frames, features are also extracted from the key point images obtained through key point detection, and then optical flow estimation is performed based on the first key point image features, the third key point image features, the first video frame features, and the third video frame features, so that the solved optical flow is more accurate, and then the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image are fused to obtain an intermediate video frame, effectively improving the situation that the limbs disappear in the frame insertion result when the human body moves rapidly.

[0134] In one example, referring to Figure 6 The step S301 specifically includes:

[0135] S401, image channel dimension reduction processing is performed on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image.

[0136] A single-channel image refers to an image with only one channel, and each pixel point can only have one value to represent color. The pixel value is between 0 and 255, 0 is black, 255 is white, and the intermediate value is some different levels of gray. The single-channel image has single color, small data volume, less memory occupation, and fast image processing speed.

[0137] S402, feature extraction is performed on the first key point single-channel image and the third key point single-channel image to obtain first key point image features and third key point image features.

[0138] S403, feature extraction is performed on the first video frame and the third video frame to obtain first video frame features and third video frame features.

[0139] In the embodiment of the application, after obtaining the first key point image and the third key point image, the two key point images are processed into single-channel images, and then feature extraction is performed on the single-channel images. The single-channel image has single color, small data volume, less memory occupation, and fast image processing speed.

[0140] In one example, the step S401 specifically includes:

[0141] For each key point in the first key point image, pixels within a preset range centered on the key point are subjected to the single-channel Gaussian distribution processing to obtain a first intermediate image; pixel values of each pixel in the first intermediate image that is not subjected to the single-channel Gaussian distribution processing are single-channeled into a first preset value to obtain a first key point single-channel image.

[0142] For each key point in the third key point image, pixels within a preset range centered on the key point are subjected to the single-channel Gaussian distribution processing to obtain a third intermediate image; pixel values of each pixel in the third intermediate image that is not subjected to the single-channel Gaussian distribution processing are single-channeled into a first preset value to obtain a third key point single-channel image.

[0143] The single-channel Gaussian distribution processing refers to single-channel processing of the pixel points and the processed pixel points conform to the Gaussian distribution. In one example, the effect of the single-channel Gaussian distribution processing of the pixels within a 5X5 range centered on the key point is as shown in Figure 7 . In the figure, the pixel values of each pixel that is not subjected to the single-channel Gaussian distribution processing are single-channeled into 1, thus completing the image channel dimension reduction processing of the key point image.

[0144] In the embodiment of the application, by subjecting the pixels within a preset range centered on the key point to the single-channel Gaussian distribution processing and single-channeling the pixel values of the remaining pixels into a first preset value, the key point image is reduced in dimension to a single-channel image.

[0145] In one example, referring to Figure 8 , the step S303 specifically includes:

[0146] S501, the first stage prediction optical flow, the second stage prediction optical flow, the first video frame, the third video frame are fused to obtain a fusion result.

[0147] Specifically, the first stage prediction optical flow and the first video frame are subjected to affine transformation to obtain a first transformation result; the second stage prediction optical flow and the second video frame are subjected to affine transformation to obtain a second transformation result; and the fusion result is obtained based on the first transformation result and the second transformation result.

[0148] Affine transformation can be realized through a series of composite atomic transformations, including translation, scaling, flipping, rotation and shearing. Affine transformation refers to the process of transforming to another vector space through once linear transformation (multiplication by a matrix) and once translation (addition of a vector) in a vector space.

[0149] The first transformation result is obtained by performing affine transformation on the first video frame based on the first-stage predicted optical flow, and the second transformation result is obtained by performing affine transformation on the third video frame based on the second-stage predicted optical flow. Then, the fusion result is obtained by adding the first transformation result and the second transformation result.

[0150] S502, based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image, an edge refinement map is obtained.

[0151] The edge refinement map is an image in which edge details are enhanced, and can improve the phenomenon that the edges of the preset type object appear to be spiky and uneven.

[0152] S503, the fusion result and the edge refinement map are fused to obtain an intermediate video frame.

[0153] In the embodiments of the present application, first, a fusion result is obtained based on optical flow and video frames, and then an edge refinement map is obtained based on optical flow, video frames, and mask images. The fusion result and the edge refinement map are fused to obtain an intermediate video frame. Since the edge refinement map can improve the phenomenon that the edges of the preset type object appear to be spiky and uneven, the intermediate video frame obtained by fusing the fusion result and the edge refinement map can make the edges of the portrait smoother and reduce artifacts.

[0154] In one example, the step S502 specifically includes:

[0155] Step one, setting a first weight value for the preset type object region in the first mask image and the third mask image;

[0156] Step two, setting a second weight value for the non-pre-set type object region in the first mask image and the third mask image;

[0157] Step three, obtaining an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with the region weight value, and the third mask image with the region weight value.

[0158] The weight value of the preset type object region in the first mask image and the third mask image can be set to 1.5, and the weight value of the non-pre-set type object region can be set to 0.5. The weight of the preset type object region is strengthened, and the weight of the non-pre-set type object region is reduced.

[0159] In the embodiments of the present application, by setting weight values for the preset type object region and the non-pre-set type object region of the mask image respectively, the mask image is added as a weight map to the calculation of the edge refinement map, and an edge refinement map with smooth edges is obtained.

[0160] For an example, see Figure 9 , which is a flowchart of a video interpolation method provided in an embodiment of the present application. First, a human motion dataset is constructed, including slow-motion videos of people playing basketball, table tennis, badminton, waving, running, jumping, etc. The specific construction process can be found in the detailed description of the above step S201; then, two video frames are obtained from the dataset, and the segmentation mask and skeletal key points of the human body are obtained for these two video frames; in the process of predicting the intermediate video frames through the traditional interpolation network, the skeletal key points of the human body are added to the feature extraction part to make the obtained optical flow more accurate, effectively improving the situation where limbs disappear in the interpolation result when the human body moves quickly; the segmentation mask of the human body is added to the fusion network part, and the human body segmentation mask is used as a weight map. During the fusion process, the weight of the portrait area is strengthened and the weight of the non-portrait area is reduced, so that the portrait edge of the fusion result is smoother and artifacts are reduced, thereby obtaining the final result.

[0161] In the second aspect, the present application provides a video frame insertion device, see Figure 10 , the device comprises:

[0162] An acquisition module 601 is configured to acquire a first video frame and a third video frame, wherein the first video frame and the third video frame include objects of the same preset type;

[0163] A key point detection module 602 is configured to perform key point detection on the first video frame and the third video frame for objects of a preset type to obtain a first key point image and a third key point image;

[0164] A mask segmentation module 603 is configured to perform mask segmentation processing of a preset type of object on the first video frame and the third video frame to obtain a first mask image and a third mask image;

[0165] The interpolation prediction module 604 is configured to perform interpolation prediction based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame to obtain an intermediate video frame.

[0166] In a possible embodiment, the interpolation prediction module includes:

[0167] a feature extraction module, configured to extract features from the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features;

[0168] an optical flow solving module configured to perform optical flow estimation based on the first key point image feature, the third key point image feature, the first video frame feature, and the third video frame feature to obtain a first stage predicted optical flow and a second stage predicted optical flow, the first stage predicted optical flow being used to describe a motion of a pixel in the first video frame to the second video frame, and the second stage predicted optical flow being used to describe a motion of a pixel in the third video frame to the second video frame, the second video frame being a video frame between the first video frame and the third video frame;

[0169] an intermediate video frame determining module configured to fuse the first stage predicted optical flow, the second stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame.

[0170] In a possible implementation, the feature extraction module includes:

[0171] a channel dimension reduction module configured to perform image channel dimension reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image;

[0172] a single-channel image feature extraction module configured to perform feature extraction on the first key point single-channel image and the third key point single-channel image to obtain a first key point image feature and a third key point image feature;

[0173] a video frame feature extraction module configured to perform feature extraction on the first video frame and the third video frame to obtain a first video frame feature and a third video frame feature.

[0174] In a possible implementation, the channel dimension reduction module is specifically configured to:

[0175] for each key point in the first key point image, perform single-channel Gaussian distribution processing on pixels within a preset range centered on the key point to obtain a first intermediate image, and single-channelize pixel values of each pixel in the first intermediate image that has not been subjected to the single-channel Gaussian distribution processing into a first preset value to obtain a first key point single-channel image;

[0176] for each key point in the third key point image, perform single-channel Gaussian distribution processing on pixels within a preset range centered on the key point to obtain a third intermediate image, and single-channelize pixel values of each pixel in the third intermediate image that has not been subjected to the single-channel Gaussian distribution processing into a first preset value to obtain a third key point single-channel image.

[0177] In a possible implementation, the intermediate video frame determining module includes:

[0178] a fusion module, configured to fuse the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result;

[0179] an edge refinement map determining module, configured to obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image;

[0180] The intermediate video frame determining unit is configured to fuse the fusion result with the edge refinement map to obtain an intermediate video frame.

[0181] In a possible embodiment, the fusion module includes:

[0182] A first affine transformation module is used to perform an affine transformation on the first-stage predicted optical flow and the first video frame to obtain a first transformation result;

[0183] A second affine transformation module is used to perform an affine transformation on the second-stage predicted optical flow and the second video frame to obtain a second transformation result;

[0184] A fusion unit is used to obtain a fusion result based on the first transformation result and the second transformation result.

[0185] In a possible embodiment, the edge refinement map determining module includes:

[0186] A first weight value setting module, configured to set a first weight value for a preset type of object area in the first mask image and the third mask image;

[0187] A second weight value setting module, configured to set a second weight value for non-preset type object areas in the first mask image and the third mask image;

[0188] An edge refinement map determination unit is used to obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with regional weight values, and the third mask image with regional weight values.

[0189] In a third aspect of the present application, an electronic device is provided, comprising:

[0190] one or more processors and memory;

[0191] The memory is coupled to the one or more processors, and is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform any of the steps of the above-mentioned video frame insertion method. The structure of the electronic device can be seen in Figure 1 , I will not go into details here.

[0192] In a fourth aspect of the present application, a computer-readable storage medium is provided, comprising a computer program. When the computer program is run on an electronic device, the electronic device executes the steps of any of the above-mentioned video frame insertion methods.

[0193] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute any of the steps of the video frame insertion method described in the above embodiments.

[0194] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0195] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device and electronic device embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, refer to the descriptions of the method embodiments.

[0196] The above merely provides the preferred embodiment of the present application, and not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of video interpolation, characterized by, The method comprises: acquiring a first video frame and a third video frame, wherein the first video frame and the third video frame comprise a same preset type object, and the preset type object is a person; performing key point detection of the preset type object on the first video frame and the third video frame to obtain a first key point image and a third key point image; performing mask segmentation processing of the preset type object on the first video frame and the third video frame to obtain a first mask image and a third mask image; performing interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

2. The method of claim 1, wherein, The interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame comprises: performing feature extraction on the first key point image, the third key point image, the first video frame and the third video frame to obtain a first key point image feature, a third key point image feature, a first video frame feature and a third video frame feature; performing optical flow estimation based on the first key point image feature, the third key point image feature, the first video frame feature and the third video frame feature to obtain a first stage prediction optical flow and a second stage prediction optical flow, wherein the first stage prediction optical flow is used to describe the motion of the pixel points of the first video frame to the second video frame, and the second stage prediction optical flow is used to describe the motion of the pixel points of the third video frame to the second video frame; the second video frame is a video frame between the first video frame and the third video frame; performing fusion on the first stage prediction optical flow, the second stage prediction optical flow, the first video frame, the third video frame, the first mask image and the third mask image to obtain an intermediate video frame.

3. The method of claim 2, wherein, The feature extraction on the first key point image, the third key point image, the first video frame and the third video frame to obtain a first key point image feature, a third key point image feature, a first video frame feature and a third video frame feature comprises: performing image channel dimension reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image; performing feature extraction on the first key point single-channel image and the third key point single-channel image to obtain a first key point image feature and a third key point image feature; performing feature extraction on the first video frame and the third video frame to obtain a first video frame feature and a third video frame feature.

4. The method of claim 3, wherein, The image channel dimension reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image comprises: For each key point in the first key point image, pixels within a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a first intermediate image; pixel values of each pixel in the first intermediate image that is not subjected to single-channel Gaussian distribution processing are single-channeled to a first preset value to obtain a first key point single-channel image; For each key point in the third key point image, pixels within a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a third intermediate image; pixel values of each pixel in the third intermediate image that is not subjected to single-channel Gaussian distribution processing are single-channeled to a first preset value to obtain a third key point single-channel image.

5. The method of claim 2, wherein, The fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame comprises: The fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result comprises: Based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image, an edge refinement map is obtained. The fusing the fusion result and the edge refinement map to obtain an intermediate video frame.

6. The method of claim 5, wherein, The fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result comprises: The first-stage predicted optical flow and the first video frame are subjected to affine transformation to obtain a first transformation result; The second-stage predicted optical flow and the second video frame are subjected to affine transformation to obtain a second transformation result; Based on the first transformation result and the second transformation result, a fusion result is obtained.

7. The method of claim 5, wherein, The obtaining an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image comprises: For a preset type object region in the first mask image and the third mask image, a first weight value is set; For a non-pre-set type object region in the first mask image and the third mask image, a second weight value is set; Based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with a region weight value, and the third mask image with a region weight value, an edge refinement map is obtained.

8. A video interframe apparatus characterized by comprising: The apparatus comprises: An acquisition module is configured to acquire a first video frame and a third video frame, wherein the first video frame and the third video frame comprise a same preset type object, and the preset type object is a person; A key point detection module is configured to perform key point detection of the preset type object on the first video frame and the third video frame to obtain a first key point image and a third key point image; The mask segmentation module is configured to perform mask segmentation processing of the first video frame and the third video frame to obtain a first mask image and a third mask image. The frame interpolation prediction module is configured to perform frame interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

9. An electronic device, comprising: Comprising: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is configured to store computer program codes, the computer program codes comprising computer instructions, and the one or more processors are configured to invoke the computer instructions to enable the electronic device to perform the video frame interpolation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program, when executed on an electronic device, causes the electronic device to perform the video frame interpolation method according to any one of claims 1 to 7.

Images