Video frame insertion method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video frame insertion method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the key point detection of a preset type of object and the mask segmentation processing of the preset type of object for a first video frame and a third video frame, and obtaining a first key point image, a third key point image, a first mask image and a third mask image; and performing frame insertion prediction according to the first key point image, the third key point image, the first mask image and the third mask image to obtain an intermediate video frame. Since the key point image contains the key point information, the intermediate video frame predicted based on the key point image can improve the limb disappearance condition, and since the mask image contains the preset type of object region, the edge of the intermediate video frame predicted based on the mask image is smoother, and artifacts are reduced. According to the scheme, the situation that artifacts occur or part of limbs disappear in the frame insertion process of the human body can be improved.

Description

Technical Field

[0001] The present application relates to the field of frame insertion technology, and in particular to a video frame insertion method, device, electronic device and storage medium. Background Art

[0002] Video interpolation refers to the process of adding one or more frames to every two frames of the original video, shortening the display time between each frame, and correcting the illusion caused by the persistence of human vision, thereby improving the smoothness of the video. In the process of video interpolation using the optical flow method, when the human body moves rapidly and significantly, resulting in a large displacement, the video interpolation effect is often poor, and the interpolation effect of the human body will have artifacts or some limbs will disappear, thus affecting the overall smoothness of the video. Summary of the invention

[0003] In view of this, the present application provides a video interpolation method, device, electronic device and storage medium, which are used to improve the situation where artifacts appear or part of the limbs disappear during the interpolation process of the human body.

[0004] In a first aspect of the present application, a video frame insertion method is provided, the method comprising:

[0005] Acquire a first video frame and a third video frame, wherein the first video frame and the third video frame include objects of the same preset type;

[0006] Performing key point detection of preset type objects on the first video frame and the third video frame to obtain a first key point image and a third key point image;

[0007] Performing mask segmentation processing of preset type objects on the first video frame and the third video frame to obtain a first mask image and a third mask image;

[0008] An interpolation prediction is performed according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

[0009] In an embodiment of the present application, the key point detection of the preset type of object is performed on the first video frame and the third video frame to obtain the first key point image and the third key point image; the mask segmentation processing of the preset type of object is performed on the first video frame and the third video frame to obtain the first mask image and the third mask image; then, interpolation prediction is performed based on the first key point image, the third key point image, the first mask image, and the third mask image to obtain an intermediate video frame. Since the first key point image and the third key point image contain key point information, the intermediate video frame predicted based on the key point image can improve the situation of limb disappearance. Since the first mask image and the third mask image contain the preset type of object area, the edge of the intermediate video frame predicted based on the mask image is smoother and artifacts are reduced. In the present application, interpolation prediction is performed based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame to obtain an intermediate video frame, which can improve the situation of artifacts or partial limb disappearance in the interpolation process for the human body.

[0010] In a possible embodiment, performing interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame to obtain an intermediate video frame includes:

[0011] Performing feature extraction on the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features;

[0012] Based on the first key point image feature, the third key point image feature, the first video frame feature and the third video frame feature, an optical flow is estimated to obtain a first-stage predicted optical flow and a second-stage predicted optical flow, wherein the first-stage predicted optical flow is used to describe the motion of pixels from the first video frame to the second video frame, and the second-stage predicted optical flow is used to describe the motion of pixels from the third video frame to the second video frame; the second video frame is a video frame between the first video frame and the third video frame;

[0013] The first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image are fused to obtain an intermediate video frame.

[0014] In an embodiment of the present application, in addition to feature extraction of video frames, feature extraction is also performed on key point images obtained by key point detection, and then optical flow estimation is performed based on the first key point image features, the third key point image features, the first video frame features and the third video frame features, so that the solved optical flow is more accurate, and then the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image and the third mask image are fused to obtain an intermediate video frame, which effectively improves the situation where limbs disappear in the interpolation results when the human body moves rapidly.

[0015] In a possible embodiment, the extracting features of the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features includes:

[0016] Performing image channel dimensionality reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image;

[0017] Performing feature extraction on the first key point single-channel image and the third key point single-channel image to obtain first key point image features and third key point image features;

[0018] Feature extraction is performed on the first video frame and the third video frame to obtain first video frame features and third video frame features.

[0019] In an embodiment of the present application, after obtaining the first key point image and the third key point image, the two key point images are processed into a single-channel image, and then feature extraction is performed on the single-channel image. The single-channel image has a single color, a small amount of data, a small memory occupancy, and a fast image processing speed.

[0020] In a possible embodiment, performing image channel dimensionality reduction processing on the first key point image and the third key point image to obtain the first key point single channel image and the third key point single channel image includes:

[0021] For each key point in the first key point image, pixels within a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a first intermediate image; pixel values ​​of each pixel in the first intermediate image that has not been subjected to single-channel Gaussian distribution processing are subjected to single-channelization to a first preset value to obtain a first key point single-channel image;

[0022] For each key point in the third key point image, pixels within a preset range centered on the key point are processed with single-channel Gaussian distribution to obtain a third intermediate image; the pixel values ​​of each pixel in the third intermediate image that has not been processed with single-channel Gaussian distribution are single-channelized to a first preset value to obtain a third key point single-channel image.

[0023] In an embodiment of the present application, by performing single-channel Gaussian distribution processing on pixels within a preset range centered on the key point, and single-channelizing the pixel values ​​of the remaining pixels to a first preset value, the key point image is reduced in dimension to a single-channel image.

[0024] In a possible embodiment, fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame includes:

[0025] Fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result;

[0026] Obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image;

[0027] The fusion result and the edge refinement map are fused to obtain an intermediate video frame.

[0028] In an embodiment of the present application, a fusion result is first obtained based on the optical flow and the video frame, and then an edge refinement map is obtained based on the optical flow, the video frame, and the mask image. The fusion result and the edge refinement map are fused to obtain an intermediate video frame. Since the edge refinement map can improve the burrs and unevenness on the edges of objects of a preset type, the intermediate video frame obtained by fusing the fusion result and the edge refinement map can make the edge of the portrait smoother and reduce artifacts.

[0029] In a possible embodiment, fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result includes:

[0030] Performing an affine transformation on the first-stage predicted optical flow and the first video frame to obtain a first transformation result;

[0031] Performing an affine transformation on the second-stage predicted optical flow and the second video frame to obtain a second transformation result;

[0032] A fusion result is obtained based on the first transformation result and the second transformation result.

[0033] In the embodiment of the present application, by performing affine transformation on the optical flow and the video frame, a fusion result of the optical flow and the video frame is obtained based on the transformation results of two sets of affine transformations, thereby maintaining the flatness and parallelism of the two-dimensional image.

[0034] In a possible embodiment, obtaining the edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image includes:

[0035] For the preset type object areas in the first mask image and the third mask image, setting a first weight value;

[0036] For the non-preset type object regions in the first mask image and the third mask image, setting a second weight value;

[0037] An edge refinement map is obtained based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with regional weight values, and the third mask image with regional weight values.

[0038] In the embodiment of the present application, by setting weight values ​​for the preset type object area and the non-preset type object area of ​​the mask image respectively, the mask image is added as a weight map to the calculation of the edge refinement map, and an edge refinement map with smooth edges is obtained.

[0039] In a second aspect, the present application provides a video frame insertion device, the device comprising:

[0040] An acquisition module, used to acquire a first video frame and a third video frame, wherein the first video frame and the third video frame include objects of the same preset type;

[0041] A key point detection module, used to perform key point detection of a preset type of object on the first video frame and the third video frame to obtain a first key point image and a third key point image;

[0042] A mask segmentation module, configured to perform mask segmentation processing of preset type objects on the first video frame and the third video frame to obtain a first mask image and a third mask image;

[0043] The interpolation prediction module is used to perform interpolation prediction based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

[0044] In a possible embodiment, the interpolation prediction module includes:

[0045] a feature extraction module, configured to extract features from the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features;

[0046] An optical flow solution module is used to estimate the optical flow based on the first key point image feature, the third key point image feature, the first video frame feature and the third video frame feature to obtain the first stage predicted optical flow and the second stage predicted optical flow, wherein the first stage predicted optical flow is used to describe the motion of the pixel points from the first video frame to the second video frame, and the second stage predicted optical flow is used to describe the motion of the pixel points from the third video frame to the second video frame; the second video frame is a video frame between the first video frame and the third video frame;

[0047] The intermediate video frame determination module is used to fuse the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame.

[0048] In a possible embodiment, the feature extraction module includes:

[0049] A channel dimension reduction module is used to perform image channel dimension reduction processing on the first key point image and the third key point image to obtain a first key point single channel image and a third key point single channel image;

[0050] A single-channel image feature extraction module, used to extract features from the first key point single-channel image and the third key point single-channel image to obtain first key point image features and third key point image features;

[0051] The video frame feature extraction module is used to extract features from the first video frame and the third video frame to obtain first video frame features and third video frame features.

[0052] In a possible embodiment, the channel dimension reduction module is specifically used to:

[0053] For each key point in the first key point image, pixels within a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a first intermediate image; pixel values ​​of each pixel in the first intermediate image that has not been subjected to single-channel Gaussian distribution processing are subjected to single-channelization to a first preset value to obtain a first key point single-channel image;

[0054] For each key point in the third key point image, pixels within a preset range centered on the key point are processed with single-channel Gaussian distribution to obtain a third intermediate image; the pixel values ​​of each pixel in the third intermediate image that has not been processed with single-channel Gaussian distribution are single-channelized to a first preset value to obtain a third key point single-channel image.

[0055] In a possible embodiment, the intermediate video frame determination module includes:

[0056] A fusion module, used for fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result;

[0057] An edge refinement map determining module, configured to obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image;

[0058] The intermediate video frame determination unit is used to fuse the fusion result and the edge refinement map to obtain an intermediate video frame.

[0059] In a possible embodiment, the fusion module includes:

[0060] A first affine transformation module, used for performing affine transformation on the first-stage predicted optical flow and the first video frame to obtain a first transformation result;

[0061] A second affine transformation module, used for performing affine transformation on the second-stage predicted optical flow and the second video frame to obtain a second transformation result;

[0062] A fusion unit is used to obtain a fusion result based on the first transformation result and the second transformation result.

[0063] In a possible embodiment, the edge refinement map determining module includes:

[0064] A first weight value setting module, used for setting a first weight value for a preset type of object area in the first mask image and the third mask image;

[0065] A second weight value setting module, used for setting a second weight value for the non-preset type object area in the first mask image and the third mask image;

[0066] An edge refinement map determination unit is used to obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with regional weight values, and the third mask image with regional weight values.

[0067] In a third aspect of the present application, an electronic device is provided, comprising:

[0068] one or more processors and memory;

[0069] The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method steps described in the first aspect above.

[0070] In a fourth aspect of the present application, a computer-readable storage medium is provided, comprising a computer program. When the computer program is executed on an electronic device, the electronic device executes the method steps described in the first aspect above.

[0071] In an embodiment of the present application, the key point detection of the preset type of object is performed on the first video frame and the third video frame to obtain the first key point image and the third key point image; the mask segmentation processing of the preset type of object is performed on the first video frame and the third video frame to obtain the first mask image and the third mask image; then, interpolation prediction is performed based on the first key point image, the third key point image, the first mask image, and the third mask image to obtain an intermediate video frame. Since the first key point image and the third key point image contain key point information, the intermediate video frame predicted based on the key point image can improve the situation of limb disappearance. Since the first mask image and the third mask image contain the preset type of object area, the edge of the intermediate video frame predicted based on the mask image is smoother and artifacts are reduced. In the present application, interpolation prediction is performed based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame to obtain an intermediate video frame, which can improve the situation of artifacts or partial limb disappearance in the interpolation process for the human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0073] Figure 1 A schematic diagram of a structure of an electronic device;

[0074] Figure 2 A schematic diagram of a system architecture of an electronic device;

[0075] Figure 3A schematic diagram of a first flow chart of a video frame insertion method provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of a result of mask segmentation processing;

[0077] Figure 5 A second flow chart of the video frame insertion method provided in an embodiment of the present application;

[0078] Figure 6 A third flow chart of the video frame insertion method provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of the pixel values ​​of a single key point and its surrounding pixels;

[0080] Figure 8 A fourth flow chart of the video frame insertion method provided in an embodiment of the present application;

[0081] Fig. 9 A fifth flow chart of the video frame insertion method provided in the embodiment of the present application;

[0082] Fig.10 A schematic diagram of the structure of a video frame insertion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0084] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0085] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0086] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0087] See also Figure 1 , Figure 1The figure shows a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may be a mobile phone, a tablet computer, or other user terminals.

[0088] The electronic device includes a processor 110, a transceiver 120, and a display unit 170. The display unit 170 may include a display screen.

[0089] Optionally, the electronic device may further include a memory 130. The processor 110, the transceiver 120 and the memory 130 may communicate with each other through an internal connection path to transmit control and / or data signals. The memory 130 is used to store a computer program, and the processor 110 is used to call and run the computer program from the memory 130.

[0090] Optionally, the electronic device may further include an antenna 140 for transmitting the wireless signal output by the transceiver 120 .

[0091] The processor 110 and the memory 130 may be combined into a processing device, or more commonly, they are independent components, and the processor 110 is used to execute the program code stored in the memory 130 to implement the above functions. In specific implementation, the memory 130 may also be integrated into the processor 110, or may be independent of the processor 110.

[0092] In addition, in order to make the functions of the electronic device more complete, the electronic device may also include one or more of an input unit 160, an audio circuit 180, a camera 190 and a sensor 101, and the audio circuit may also include a speaker 182, a microphone 184, and the like.

[0093] Optionally, the electronic device may further include a power supply 150 for providing power to various devices or circuits in the electronic device.

[0094] Understandably, Figure 1 The operations and / or functions of each module in the electronic device shown are respectively for implementing the corresponding processes in the following method embodiments. For details, please refer to the description in the following method embodiments, and the detailed description is appropriately omitted here to avoid repetition.

[0095] Understandably, Figure 1The processor 110 in the electronic device shown may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (imagesignalprocessor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. A memory can also be set in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

[0096] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0097] It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0098] Understandably, Figure 1 The power supply 150 shown is used to power the processor 110, the memory 130, the display unit 170, the camera 190, the input unit 160 and the transceiver 120. The antenna 140 is used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, the antenna 140 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0099] The transceiver 120 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The transceiver 120 can be one or more devices integrating at least one communication processing module. The transceiver 120 receives electromagnetic waves via the antenna 140, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 110. The transceiver 120 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 140.

[0100] In some embodiments, the antenna 140 and the transceiver 120 of the electronic device are coupled so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0101] The electronic device implements the display function through a GPU, a display unit 170, and an application processor. The GPU is a microprocessor for image processing, which connects the display unit 170 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0102] The display unit 170 is used to display images, videos, etc. The display unit 170 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (AMOLED), a flexible light emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display units 170, where N is a positive integer greater than 1.

[0103] The electronic device can realize the shooting function through the ISP, the camera 190, the video codec, the GPU, the display unit 170 and the application processor.

[0104] The ISP is used to process the data fed back by the camera 190. For example, when recording a video, the camera is turned on, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image.

[0105] ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 190. Camera 190 is used to capture static images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to ISP for conversion into a digital image signal. ISP outputs the digital image signal to DSP for processing. DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 190, where N is a positive integer greater than 1. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0106] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0107] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0108] The memory 130 can be used to store computer executable program codes, which include instructions. The memory 130 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 130 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the memory 130 and / or instructions stored in a memory provided in the processor.

[0109] The electronic device can implement audio functions such as music playing and recording through the audio circuit 180, the speaker 182, the microphone 184, and the application processor.

[0110] The audio circuit 180 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio circuit 180 can also be used to encode and decode audio signals. In some embodiments, the audio circuit 180 can be arranged in the processor 110, or some functional modules of the audio circuit 180 can be arranged in the processor 110.

[0111] The speaker 182, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or listen to a hands-free call through the speaker 182.

[0112] Microphone 184, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can input the sound signal into microphone 184 by speaking close to microphone 184. The electronic device can be provided with at least one microphone 184. In other embodiments, the electronic device can be provided with two microphones 184, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 184 to realize the collection of sound signals, noise reduction, identification of sound sources, and realization of directional recording function, etc.

[0113] The video frame insertion method in the embodiment of the present application can be Figure 2 The system architecture shown is implemented in Figure 2 , the system architecture of the electronic device includes a kernel part, a framework layer part and an application layer part; the kernel part includes a driver layer and a real-time operating system, the driver layer includes a GPU (graphics processor), a display driver (specifically an LCD driver in the figure), a TP driver (touch screen driver), buttons, etc.; the real-time operating system includes interrupt management, task scheduling, MEM (memory management); the framework layer includes: basic system capabilities, underlying software services, hardware service capabilities, etc.; the application layer includes: system applications (such as calendars, cameras, contacts, browsers, etc.), third-party applications (such as social applications and video applications provided by third parties, etc.). After the camera application of the application layer shoots a motion video, the kernel layer will call the interpolation algorithm to execute the video interpolation method in the embodiment of the present application.

[0114] Shooting sports videos also provides a data set of human motion scenes. The specific types of the above sports videos include but are not limited to playing basketball, playing table tennis, playing badminton, waving, running, and jumping. The data set of human motion scenes needs to be further processed to filter out video frames containing human bodies and adjust the image size of the video frames to obtain data suitable for the interpolation algorithm. For data suitable for the interpolation algorithm, call the key point detection algorithm to perform key point detection on the data, call the mask segmentation algorithm to perform mask segmentation on the data, and call the interpolation algorithm to implement interpolation of the data.

[0115] Video interpolation refers to the process of adding one or more frames to every two frames of the original video, shortening the display time between each frame, and correcting the illusion caused by the persistence of human vision, thereby improving the smoothness of the video. In the process of video interpolation using the optical flow method, when the human body moves rapidly and significantly, resulting in a large displacement, the video interpolation effect is often poor, and the interpolation effect of the human body will have artifacts or some limbs will disappear, thus affecting the overall smoothness of the video.

[0116] Based on this, the present application provides a video frame insertion method, see Figure 3,The specific process of the video frame insertion method includes:

[0117] S201. Acquire a first video frame and a third video frame.

[0118] The first video frame and the third video frame include objects of the same preset type.

[0119] In the embodiments of the present application, a variety of shooting devices will be used to shoot slow-motion videos of human movements (including playing basketball, table tennis, badminton, waving, running, jumping, etc.). The shooting devices can be, for example, cameras, mobile phones, etc. The advantage of this is that the interpolation network can learn images of different qualities, enhance the generalization of the model, and process the video into data suitable for the video interpolation method. The specific processing steps include: screening video frames containing human images from the video, and for video frames containing human images, processing the image size of the video frames to 48X48. Obtain the first video frame and the third video frame from the data suitable for the video interpolation method. The first video frame and the third video frame can be two adjacent frames or two non-adjacent frames, as long as the two frames include the same preset type object. In an example, person 1 can be set as a preset type object.

[0120] S202: Perform key point detection on the first video frame and the third video frame for objects of a preset type to obtain a first key point image and a third key point image.

[0121] The skeleton key point detection network can be used to detect the key points of the preset type of objects in the first video frame and the third video frame, and the key points include 17 skeleton key points, namely, left and right ankles, left and right knees, left and right hips, left and right wrists, left and right elbows, left and right shoulders, left and right ears, left and right eyes, and the back of the head. The skeleton key point detection network can specifically adopt a human pose as composition tokens network.

[0122] S203: Perform mask segmentation processing of preset type objects on the first video frame and the third video frame to obtain a first mask image and a third mask image.

[0123] The portrait segmentation network can be used to perform mask segmentation processing of preset type objects on the first video frame and the third video frame. In the case where person 1 is the preset type object, the portrait segmentation network is used to perform mask segmentation processing of preset type objects on the first video frame and the third video frame, and two frames of person 1 segmentation masks are obtained. The portrait segmentation network can specifically be a unet network structure. One result of the mask segmentation processing is as follows Figure 4 As shown, the mask segmentation processing of the human body's jump shooting posture is performed.

[0124] S204. Perform interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame to obtain an intermediate video frame.

[0125] According to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame, interpolation prediction is performed to obtain an intermediate video frame, where the intermediate video frame refers to a video frame that can be inserted between the first video frame and the third video frame.

[0126] In an embodiment of the present application, the key point detection of the preset type of object is performed on the first video frame and the third video frame to obtain the first key point image and the third key point image; the mask segmentation processing of the preset type of object is performed on the first video frame and the third video frame to obtain the first mask image and the third mask image; then, interpolation prediction is performed based on the first key point image, the third key point image, the first mask image, and the third mask image to obtain an intermediate video frame. Since the first key point image and the third key point image contain key point information, the intermediate video frame predicted based on the key point image can improve the situation of limb disappearance. Since the first mask image and the third mask image contain the preset type of object area, the edge of the intermediate video frame predicted based on the mask image is smoother and artifacts are reduced. In the present application, interpolation prediction is performed based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame to obtain an intermediate video frame, which can improve the situation of artifacts or partial limb disappearance in the interpolation process for the human body.

[0127] For an example, see Figure 5 , the above step S204 specifically includes:

[0128] S301. Perform feature extraction on the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features.

[0129] In a traditional interpolation network, only features of the first video frame and the third video frame are extracted to obtain features of the first video frame and features of the third video frame; however, the video interpolation method in the embodiment of the present application, in addition to extracting features from the video frames, also extracts features from the key point images obtained by key point detection.

[0130] S302 , performing optical flow estimation based on the first key point image feature, the third key point image feature, the first video frame feature, and the third video frame feature to obtain a first-stage predicted optical flow and a second-stage predicted optical flow.

[0131] The first stage predicted optical flow is used to describe the movement of pixels from the first video frame to the second video frame, and the second stage predicted optical flow is used to describe the movement of pixels from the third video frame to the second video frame; the second video frame is a video frame between the first video frame and the third video frame.

[0132] S303 , fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame.

[0133] In an embodiment of the present application, in addition to feature extraction of video frames, feature extraction is also performed on key point images obtained by key point detection, and then optical flow estimation is performed based on the first key point image features, the third key point image features, the first video frame features and the third video frame features, so that the solved optical flow is more accurate, and then the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image and the third mask image are fused to obtain an intermediate video frame, which effectively improves the situation where limbs disappear in the interpolation results when the human body moves rapidly.

[0134] For an example, see Figure 6 , the above step S301 specifically includes:

[0135] S401. Perform image channel dimensionality reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image.

[0136] A single-channel image is an image with only one channel. Each pixel can only have one value to represent the color. Its pixel value is between 0 and 255, where 0 is black, 255 is white, and the intermediate values ​​are some different levels of gray. Single-channel images have a single color, small data volume, low memory usage, and fast image processing speed.

[0137] S402: Perform feature extraction on the first key point single-channel image and the third key point single-channel image to obtain first key point image features and third key point image features.

[0138] S403: Perform feature extraction on the first video frame and the third video frame to obtain first video frame features and third video frame features.

[0139] In an embodiment of the present application, after obtaining the first key point image and the third key point image, the two key point images are processed into a single-channel image, and then feature extraction is performed on the single-channel image. The single-channel image has a single color, a small amount of data, a small memory occupancy, and a fast image processing speed.

[0140] In one example, the above step S401 specifically includes:

[0141] For each key point in the first key point image, pixels within a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a first intermediate image; pixel values ​​of each pixel in the first intermediate image that has not been subjected to single-channel Gaussian distribution processing are subjected to single-channelization to a first preset value to obtain a first key point single-channel image;

[0142] For each key point in the third key point image, pixels within a preset range centered on the key point are processed with single-channel Gaussian distribution to obtain a third intermediate image; the pixel values ​​of each pixel in the third intermediate image that has not been processed with single-channel Gaussian distribution are single-channelized to a first preset value to obtain a third key point single-channel image.

[0143] The above-mentioned single-channel Gaussian distribution processing refers to single-channel processing of pixels and the processed pixels conform to Gaussian distribution. In an example, the effect of single-channel Gaussian distribution processing of pixels in a 5X5 range centered on the key point is as follows: Figure 7 As shown in the figure, the pixel value of each pixel that has not been processed by single-channel Gaussian distribution is single-channelized to 1, thus completing the image channel dimensionality reduction processing of the key point image.

[0144] In an embodiment of the present application, by performing single-channel Gaussian distribution processing on pixels within a preset range centered on the key point, and single-channelizing the pixel values ​​of the remaining pixels to a first preset value, the key point image is reduced in dimension to a single-channel image.

[0145] For an example, see Figure 8 , the above step S303 specifically includes:

[0146] S501, fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result.

[0147] Specifically, the first-stage predicted optical flow and the first video frame are subjected to affine transformation to obtain a first transformation result; the second-stage predicted optical flow and the second video frame are subjected to affine transformation to obtain a second transformation result; and a fusion result is obtained based on the first transformation result and the second transformation result.

[0148] Affine transformation can be achieved by combining a series of atomic transformations, including translation, scaling, flipping, rotation and shearing. Affine transformation refers to the process of performing a linear transformation (multiplying a matrix) and a translation (adding a vector) in a vector space to transform it to another vector space.

[0149] The first stage predicts that the optical flow plus the first video frame will be transformed into a first transformation result by affine transformation, and the second stage predicts that the optical flow plus the third video frame will be transformed into a second transformation result by affine transformation; then the first transformation result and the second transformation result are added to obtain a fusion result.

[0150] S502: Obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image.

[0151] The edge refinement map is an image with enhanced edge details, which can improve the burr and unevenness of the edges of preset type objects.

[0152] S503: Fuse the fusion result with the edge refinement map to obtain an intermediate video frame.

[0153] In an embodiment of the present application, a fusion result is first obtained based on the optical flow and the video frame, and then an edge refinement map is obtained based on the optical flow, the video frame, and the mask image. The fusion result and the edge refinement map are fused to obtain an intermediate video frame. Since the edge refinement map can improve the burrs and unevenness on the edges of objects of a preset type, the intermediate video frame obtained by fusing the fusion result and the edge refinement map can make the edge of the portrait smoother and reduce artifacts.

[0154] In one example, the above step S502 specifically includes:

[0155] Step 1: setting a first weight value for a preset type of object area in the first mask image and the third mask image;

[0156] Step 2: setting a second weight value for the non-preset type object area in the first mask image and the third mask image;

[0157] Step three, based on the first stage predicted optical flow, the second stage predicted optical flow, the first video frame, the third video frame, the first mask image with regional weight values, and the third mask image with regional weight values, an edge refinement map is obtained.

[0158] The weight value of the preset type object area in the first mask image and the third mask image can be set to 1.5, and the weight value of the non-preset type object area can be set to 0.5. The weight of the preset type object area is strengthened, and the weight of the non-preset type object area is reduced.

[0159] In the embodiment of the present application, by setting weight values ​​for the preset type object area and the non-preset type object area of ​​the mask image respectively, the mask image is added as a weight map to the calculation of the edge refinement map, and an edge refinement map with smooth edges is obtained.

[0160] For an example, see Fig. 9 , which is a flow chart of a video interpolation method provided in an embodiment of the present application. First, a human motion data set is constructed, including slow-motion videos of a human body playing basketball, table tennis, badminton, waving, running, jumping, etc. The specific construction process can be referred to the detailed description of the above step S201; then two video frames are obtained from the data set, and the segmentation mask and the skeleton key points of the human body are obtained for the two video frames; in the process of predicting the intermediate video frame through the traditional interpolation network, the skeleton key points of the human body are added to the feature extraction part to make the obtained optical flow more accurate, and effectively improve the situation where the limbs disappear in the interpolation result when the human body moves quickly; the segmentation mask of the human body is added to the fusion network part, and the human body segmentation mask is used as a weight map. In the fusion process, the weight of the portrait area is strengthened and the weight of the non-portrait area is reduced, so that the portrait edge of the fusion result is smoother and artifacts are reduced to obtain the final result.

[0161] In a second aspect, the present application provides a video frame insertion device, see Fig.10 , the device comprises:

[0162] An acquisition module 601 is used to acquire a first video frame and a third video frame, wherein the first video frame and the third video frame include objects of the same preset type;

[0163] A key point detection module 602 is used to perform key point detection of a preset type of object on the first video frame and the third video frame to obtain a first key point image and a third key point image;

[0164] A mask segmentation module 603 is used to perform mask segmentation processing of preset type objects on the first video frame and the third video frame to obtain a first mask image and a third mask image;

[0165] The interpolation prediction module 604 is used to perform interpolation prediction based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

[0166] In a possible embodiment, the interpolation prediction module includes:

[0167] a feature extraction module, configured to extract features from the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features;

[0168] An optical flow solution module is used to estimate the optical flow based on the first key point image feature, the third key point image feature, the first video frame feature and the third video frame feature to obtain the first stage predicted optical flow and the second stage predicted optical flow, wherein the first stage predicted optical flow is used to describe the motion of the pixel points from the first video frame to the second video frame, and the second stage predicted optical flow is used to describe the motion of the pixel points from the third video frame to the second video frame; the second video frame is a video frame between the first video frame and the third video frame;

[0169] The intermediate video frame determination module is used to fuse the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame.

[0170] In a possible embodiment, the feature extraction module includes:

[0171] A channel dimension reduction module is used to perform image channel dimension reduction processing on the first key point image and the third key point image to obtain a first key point single channel image and a third key point single channel image;

[0172] A single-channel image feature extraction module, used to extract features from the first key point single-channel image and the third key point single-channel image to obtain first key point image features and third key point image features;

[0173] The video frame feature extraction module is used to extract features from the first video frame and the third video frame to obtain first video frame features and third video frame features.

[0174] In a possible embodiment, the channel dimension reduction module is specifically used to:

[0175] For each key point in the first key point image, pixels within a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a first intermediate image; pixel values ​​of each pixel in the first intermediate image that has not been subjected to single-channel Gaussian distribution processing are subjected to single-channelization to a first preset value to obtain a first key point single-channel image;

[0176] For each key point in the third key point image, pixels within a preset range centered on the key point are processed with single-channel Gaussian distribution to obtain a third intermediate image; the pixel values ​​of each pixel in the third intermediate image that has not been processed with single-channel Gaussian distribution are single-channelized to a first preset value to obtain a third key point single-channel image.

[0177] In a possible embodiment, the intermediate video frame determination module includes:

[0178] A fusion module, used for fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result;

[0179] An edge refinement map determining module, configured to obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image;

[0180] The intermediate video frame determination unit is used to fuse the fusion result and the edge refinement map to obtain an intermediate video frame.

[0181] In a possible embodiment, the fusion module includes:

[0182] A first affine transformation module, used for performing affine transformation on the first-stage predicted optical flow and the first video frame to obtain a first transformation result;

[0183] A second affine transformation module, used for performing affine transformation on the second-stage predicted optical flow and the second video frame to obtain a second transformation result;

[0184] A fusion unit is used to obtain a fusion result based on the first transformation result and the second transformation result.

[0185] In a possible embodiment, the edge refinement map determining module includes:

[0186] A first weight value setting module, used for setting a first weight value for a preset type of object area in the first mask image and the third mask image;

[0187] A second weight value setting module, used for setting a second weight value for the non-preset type object area in the first mask image and the third mask image;

[0188] An edge refinement map determination unit is used to obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with regional weight values, and the third mask image with regional weight values.

[0189] In a third aspect of the present application, an electronic device is provided, comprising:

[0190] one or more processors and memory;

[0191] The memory is coupled to the one or more processors, and the memory is used to store computer program codes, the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform any of the steps of the above-mentioned video frame insertion method. The structure of the electronic device can be seen in Figure 1 , I will not go into details here.

[0192] In a fourth aspect of the present application, a computer-readable storage medium is provided, comprising a computer program, which, when executed on an electronic device, enables the electronic device to execute any of the steps of the above-mentioned video frame insertion method.

[0193] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute any of the steps of the video frame insertion method described in the above embodiments.

[0194] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk Solid State Disk (SSD), etc.

[0195] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0196] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A video frame insertion method, characterized in that: The method comprises: Acquire a first video frame and a third video frame, wherein the first video frame and the third video frame include objects of the same preset type; Performing key point detection of preset type objects on the first video frame and the third video frame to obtain a first key point image and a third key point image; Performing mask segmentation processing of preset type objects on the first video frame and the third video frame to obtain a first mask image and a third mask image; An interpolation prediction is performed according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

2. The method according to claim 1, characterized in that The step of performing interpolation prediction according to the first key point image, the third key point image, the first mask image, the third mask image, the first video frame, and the third video frame to obtain an intermediate video frame includes: Performing feature extraction on the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features; Based on the first key point image feature, the third key point image feature, the first video frame feature and the third video frame feature, an optical flow is estimated to obtain a first-stage predicted optical flow and a second-stage predicted optical flow, wherein the first-stage predicted optical flow is used to describe the motion of pixels from the first video frame to the second video frame, and the second-stage predicted optical flow is used to describe the motion of pixels from the third video frame to the second video frame; the second video frame is a video frame between the first video frame and the third video frame; The first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image are fused to obtain an intermediate video frame.

3. The method according to claim 2, characterized in that The extracting features of the first key point image, the third key point image, the first video frame, and the third video frame to obtain first key point image features, third key point image features, first video frame features, and third video frame features includes: Performing image channel dimensionality reduction processing on the first key point image and the third key point image to obtain a first key point single-channel image and a third key point single-channel image; Performing feature extraction on the first key point single-channel image and the third key point single-channel image to obtain first key point image features and third key point image features; Feature extraction is performed on the first video frame and the third video frame to obtain first video frame features and third video frame features.

4. The method according to claim 3, characterized in that The performing image channel dimensionality reduction processing on the first key point image and the third key point image to obtain a first key point single channel image and a third key point single channel image includes: For each key point in the first key point image, pixels within a preset range centered on the key point are subjected to single-channel Gaussian distribution processing to obtain a first intermediate image; pixel values ​​of each pixel in the first intermediate image that has not been subjected to single-channel Gaussian distribution processing are subjected to single-channelization to a first preset value to obtain a first key point single-channel image; For each key point in the third key point image, pixels within a preset range centered on the key point are processed with single-channel Gaussian distribution to obtain a third intermediate image; the pixel values ​​of each pixel in the third intermediate image that has not been processed with single-channel Gaussian distribution are single-channelized to a first preset value to obtain a third key point single-channel image.

5. The method according to claim 2, characterized in that: The step of fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image to obtain an intermediate video frame includes: Fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result; Obtain an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image; The fusion result and the edge refinement map are fused to obtain an intermediate video frame.

6. The method according to claim 5, characterized in that The fusing the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, and the third video frame to obtain a fusion result includes: Performing an affine transformation on the first-stage predicted optical flow and the first video frame to obtain a first transformation result; Performing an affine transformation on the second-stage predicted optical flow and the second video frame to obtain a second transformation result; A fusion result is obtained based on the first transformation result and the second transformation result.

7. The method according to claim 5, characterized in that The step of obtaining an edge refinement map based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image, and the third mask image comprises: For the preset type object areas in the first mask image and the third mask image, setting a first weight value; For the non-preset type object regions in the first mask image and the third mask image, setting a second weight value; An edge refinement map is obtained based on the first-stage predicted optical flow, the second-stage predicted optical flow, the first video frame, the third video frame, the first mask image with regional weight values, and the third mask image with regional weight values.

8. A video frame insertion device, characterized in that: The device comprises: An acquisition module, used to acquire a first video frame and a third video frame, wherein the first video frame and the third video frame include objects of the same preset type; A key point detection module, used to perform key point detection of preset type objects on the first video frame and the third video frame to obtain a first key point image and a third key point image; A mask segmentation module, configured to perform mask segmentation processing of preset type objects on the first video frame and the third video frame to obtain a first mask image and a third mask image; The interpolation prediction module is used to perform interpolation prediction based on the first key point image, the third key point image, the first mask image, the third mask image, the first video frame and the third video frame to obtain an intermediate video frame.

9. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the video interpolation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when being executed on an electronic device, enables the electronic device to execute the video frame insertion method according to any one of claims 1 to 7.

Images