Video processing method and device and electronic equipment

Abstract

The invention provides a video processing method and apparatus, and an electronic device. The method comprises the steps of determining a first video frame and a second video frame adjacent to each other in a video; determining a frame insertion moment based on the playing moment of the first video frame and the playing moment of the second video frame; the first video frame and the second video frame are processed based on a video frame insertion model and the frame insertion moment, a frame insertion image between the first video frame and the second video frame is obtained, and the video frame insertion model comprises an optical flow module, a conversion module and a correction module. The optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after operator fusion processing; and inserting the frame insertion image between the first video frame and the second video frame.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of video processing technology, and more particularly to a video processing method, device, and electronic device. Background Art

[0002] Video interpolation technology can increase the frame rate of a video. For example, if the frame rate of the original video is 30 frames, the electronic device can insert a new video frame between every two adjacent video frames of the original video, thereby obtaining a video with a frame rate of 60 frames.

[0003] Currently, electronic devices can process the semantic information and optical flow information of two adjacent video frames based on the interpolation model to obtain an interpolated image between two adjacent video frames. However, the semantic information processing is very complex, and it takes a long time to obtain optical flow information based on convolution processing, which leads to low efficiency of interpolation. Summary of the invention

[0004] The present disclosure provides a video processing method, device and electronic device, which are used to solve one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides a video processing method, the video processing method comprising:

[0006] Determining adjacent first and second video frames in the video;

[0007] Determining a frame insertion time based on a playback time of the first video frame and a playback time of the second video frame;

[0008] Based on the video interpolation model and the interpolation time, the first video frame and the second video frame are processed to obtain an interpolation image between the first video frame and the second video frame, wherein the video interpolation model includes an optical flow module, a conversion module and a correction module, the optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after operator fusion processing;

[0009] The interpolation frame image is inserted between the first video frame and the second video frame.

[0010] In a second aspect, the present disclosure provides a video processing device, the video processing device comprising a first determination module, a second determination module, a processing module and a frame insertion module, wherein:

[0011] The first determination module is used to determine adjacent first video frames and second video frames in the video;

[0012] The second determination module is used to determine the frame insertion time based on the playback time of the first video frame and the playback time of the second video frame;

[0013] The processing module is used to process the first video frame and the second video frame based on the video interpolation model and the interpolation time to obtain an interpolation image between the first video frame and the second video frame, wherein the video interpolation model includes an optical flow module, a conversion module and a correction module, the optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after operator fusion processing;

[0014] The interpolation module is used to insert the interpolation frame image between the first video frame and the second video frame.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video processing method as described in the first aspect and various possible aspects of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the video processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0019] The present disclosure provides a video processing method, device and electronic device, wherein the electronic device can determine adjacent first and second video frames in a video to be processed, determine the interpolation time based on the playback time of the first video frame and the playback time of the second video frame, process the first and second video frames based on the video interpolation model and the interpolation time, and obtain an interpolation image between the first and second video frames, wherein the video interpolation model includes an optical flow module, a conversion module and a correction module, the optical flow module and the correction module are partial convolutional neural networks, the conversion module is a module after operator fusion processing, and the electronic device can insert an interpolation image between the first and second video frames. In the above method, since the optical flow module and the correction module are partial convolutional neural networks, the electronic device can obtain optical flow information and correction information at a faster speed, and since the conversion module is a module after multiple operators are fused, the data reading and writing speed is faster, and the video interpolation model does not need to process the semantic information of the first and second video frames, thereby improving the efficiency of determining the interpolation image, thereby improving the efficiency of video interpolation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of obtaining a video to be subjected to frame insertion processing provided by an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of another method of obtaining a video to be processed by frame insertion provided in an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of the structure of a video frame insertion model provided by an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the structure of an optical flow module provided in an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of a method for determining an interpolated frame image provided by an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of an interpolated image provided by an embodiment of the present disclosure;

[0029] Fig. 9 A schematic diagram of a scene of interpolated image occlusion provided by an embodiment of the present disclosure;

[0030] Fig.10 A schematic diagram of a video processing method provided by an embodiment of the present disclosure;

[0031] Fig.11 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure;

[0032] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0035] Electronic device: is a device with wireless transceiver function. Electronic devices can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted. The electronic device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a vehicle-mounted electronic device, a wireless terminal in self-driving, a wireless electronic device in remote medical care, a wireless electronic device in smart grid, a wireless electronic device in transportation safety, a wireless electronic device in smart city, a wireless electronic device in smart home, a wearable electronic device, etc. The electronic device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access electronic device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote electronic device, a mobile device, a UE electronic device, a wireless communication device, a UE agent or a UE device, etc. The electronic device can also be fixed or mobile.

[0036] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0037] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including a video. The video may include a video frame 1, a video frame 2, a video frame 3, and a video frame 4. An electronic device ( Figure 1The electronic device may perform frame interpolation on the video (not shown). The electronic device may determine video frame A based on video frame 1 and video frame 2, determine video frame B based on video frame 2 and video frame 3, and determine video frame C based on video frame 3 and video frame 4. The electronic device may insert video frame A between video frame 1 and video frame 2, insert video frame B between video frame 2 and video frame 3, and insert video frame C between video frame 3 and video frame 4. In this way, the electronic device may obtain a video after interpolation, and the video after interpolation has 3 more video frames than the video before interpolation, thereby improving the frame rate of the video.

[0038] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0039] In the related art, video interpolation technology can improve the frame rate of the video. For example, if the frame rate of the video is 30 frames, the electronic device can insert a new video frame between every two adjacent video frames in the original video, so that the electronic device can adjust the frame rate of the video to 60 frames. At present, the electronic device can process two adjacent video frames based on the interpolation model to obtain an interpolated image between two adjacent video frames. For example, the interpolation model can determine the semantic information of two adjacent video frames, and determine the optical flow information of two adjacent video frames relative to the interpolation moment. The interpolation model can process the optical flow information and the semantic information to obtain an interpolated image between two adjacent video frames. However, in the above method, the complexity of semantic information processing is high, and the interpolation model is based on the traditional convolution processing method to obtain optical flow information, which takes a long time, resulting in low interpolation efficiency.

[0040] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides a video processing method, in which an electronic device can determine adjacent first and second video frames in a video to be processed, determine the interpolation time based on the playback time of the first video frame and the playback time of the second video frame, and process the first and second video frames based on the video interpolation model and the interpolation time to obtain an interpolation image between the first and second video frames, wherein the video interpolation model includes an optical flow module, a conversion module, and a correction module, the optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after operator fusion processing, and the electronic device can insert an interpolation image between the first and second video frames. In the above method, since the optical flow module and the correction module are partial convolutional neural networks, the electronic device can obtain optical flow information and correction information at a faster speed, and since the conversion module is a module after multiple operators are fused, the data reading and writing speed is faster, and the video interpolation model does not need to process the semantic information of the first and second video frames, thereby improving the efficiency of determining the interpolation image, thereby improving the efficiency of video interpolation.

[0041] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0042] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0043] Figure 2 A flowchart of a video processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0044] S201: Determine a first video frame and a second video frame that are adjacent to each other in a video.

[0045] The execution subject of the embodiment of the present disclosure may be an electronic device, or a video processing device arranged in the electronic device. The video processing device may be implemented based on software, or may be implemented based on a combination of software and hardware, which is not limited in the embodiment of the present disclosure.

[0046] The video may be a video to be processed by frame insertion. For example, if the electronic device performs frame insertion on video 1, then the video 1 is the video to be processed; if the electronic device performs frame insertion on video 2, then the video 2 is the video to be processed.

[0047] Optionally, the electronic device can determine the video to be processed by interpolation based on the following feasible implementation method: in response to a touch operation on a video processing control, display a video page, and in response to a touch operation on any video in the video page, determine the video corresponding to the touch operation as the video to be processed by interpolation.

[0048] Among them, the video processing control can be a video interpolation control. When the user clicks on the video interpolation application, the electronic device can display the page of the video interpolation application, and the page of the interpolation application can include the video processing control (the control is a functional control).

[0049] Optionally, the video page may include multiple pre-stored videos. For example, the electronic device may shoot multiple videos and store them in a database. When the electronic device displays the video page, the video page may include the multiple videos pre-stored in the database.

[0050] Optionally, when a user clicks on any video in a video page, the electronic device may determine the video clicked by the user as a video to be processed by frame insertion. Figure 3 , the process of obtaining the video to be interpolated is explained.

[0051] Figure 3 A schematic diagram of obtaining a video to be interpolated provided by an embodiment of the present disclosure. Figure 3 , including: an electronic device. The display page of the electronic device is a video page, wherein the video page may include video A, video B, video C, and video D. When the user clicks on video A, the electronic device may determine video A as a video to be inserted.

[0052] It should be noted that the electronic device in the embodiments of the present disclosure may be any device with end-to-end computing capabilities, and the embodiments of the present disclosure are not limited to this.

[0053] Optionally, the electronic device may determine the live video as the video to be subjected to frame insertion processing. For example, the live broadcast device may collect the live video of the anchor during the live broadcast and send the live video to the server. Before the server sends the live video to other devices, it may perform frame insertion processing on the live video, thereby improving the frame rate of the live video.

[0054] Next, combine Figure 4 , the process of obtaining the video to be inserted into the frame is explained.

[0055] Figure 4 This is another schematic diagram of obtaining a video to be processed by frame insertion provided by an embodiment of the present disclosure. Figure 4, including a live broadcast device and a server. The live broadcast device can collect live video during the live broadcast process and continuously send the live video to the server. After the server receives the live video, it can insert new video frames between adjacent video frames, thereby increasing the frame rate of the live video, thereby improving the user's experience when watching the live broadcast.

[0056] It should be noted that the electronic device can obtain the video to be processed by frame interpolation based on any feasible implementation method (such as receiving the video to be processed by frame interpolation sent by other devices, etc.), and the embodiments of the present disclosure are not limited to this.

[0057] Optionally, the first video frame and the second video frame are any pair of adjacent video frames in the video. For example, if the video includes video frame 1, video frame 2, and video frame 3, the adjacent video frames may include video frame 1 and video frame 2, video frame 2 and video frame 3, wherein in video frame 1 and video frame 2, video frame 1 may be the first video frame, and video frame 2 may be the second video frame, and in video frame 2 and video frame 3, video frame 2 may be the first video frame, and video frame 3 may be the second video frame. For example, in Figure 1 In the illustrated embodiment, the video before interpolation processing may include three groups of adjacent video frames, the first group of adjacent video frames being: video frame 1 and video frame 2, the second group of adjacent video frames being: video frame 2 and video frame 3, and the third group of adjacent video frames being: video frame 3 and video frame 4.

[0058] It should be noted that the above examples are merely illustrative of adjacent first and second video frames, wherein "first" and "second" do not include a sequential relationship. For example, in video frame 1 and video frame 2, video frame 1 may be the second video frame, and video frame 2 may be the first video frame. The embodiments of the present disclosure do not limit this.

[0059] It should be noted that the electronic device can obtain the adjacent first video frame and second video frame in the video based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0060] S202: Determine a frame insertion time based on the playback time of the first video frame and the playback time of the second video frame.

[0061] The interpolation moment may be a moment between the playback moment of the first video frame and the playback moment of the second video frame. For example, the playback moment of the first video frame may be the playback moment of the first video frame in the video, and the playback moment of the second video frame may be the playback moment of the second video frame in the video. If the playback moment of the first video frame is earlier than the playback moment of the second video frame, the interpolation moment is later than the playback moment of the first video frame and earlier than the playback moment of the second video frame. For example, if the playback moment of the first video frame is moment A, and the playback moment of the second video frame is moment B, the interpolation moment may be any moment between moment A and moment B. For example, if the playback moment of the first video frame is moment A, and the playback moment of the second video frame is video B, the interpolation moment may be a moment in the middle of moment A and video B.

[0062] It should be noted that the electronic device can determine the playback time of the first video frame based on the timestamp of the first video frame, and determine the playback time of the second video frame based on the timestamp of the second video frame. The electronic device can also determine the playback time of the first video frame and the playback time of the second video frame based on any other feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0063] It should be noted that in the embodiments of the present disclosure, the electronic device may determine the frame insertion time based on any feasible implementation method, and the electronic device may also determine the middle time between two adjacent video frames as the frame insertion time, and the embodiments of the present disclosure are not limited to this.

[0064] S203: Based on the video interpolation model and the interpolation time, the first video frame and the second video frame are processed to obtain an interpolation image between the first video frame and the second video frame.

[0065] Optionally, the interpolated image may be an image inserted between the first video frame and the second video frame. For example, two adjacent video frames in a video are video frame 1 and video frame 2, and video frame 1 and video frame 2 are continuous video frames. If the electronic device determines that the interpolated image between video frame 1 and video frame 2 is image A, the electronic device may insert image A between video frame 1 and video frame 2.

[0066] Among them, the video interpolation model is a pre-trained interpolation model, and the electronic device can input any two video frames into the video interpolation model. The video interpolation model can determine the interpolation images corresponding to the two video frames based on the two video frames.

[0067] The video interpolation model may include an optical flow module, a conversion module, and a correction module. The optical flow module is used to obtain optical flow information. For example, the optical flow module may determine the optical flow information of the first video frame at the interpolation time based on the interpolation time, and the optical flow module may also determine the optical flow information of the second video frame at the interpolation time based on the interpolation time.

[0068] Among them, the optical flow module can be used to obtain the optical flow information of the video frame, and the optical flow module can be a partial convolutional neural network. For example, the neural network of the optical flow module of the embodiment of the present disclosure is not a traditional convolutional neural network. The optical flow module of the embodiment of the present disclosure is a partial convolutional neural network, wherein, in the partial convolution, if the convolution can calculate the output based on at least one valid input value, then in the next step, the mask of the pixel point is marked as 1, that is, the mask of the pixel point is 1, and in the volume set of the next layer, the value of the mask at this point is updated to 1. Therefore, when the image is processed by partial convolution, the convolution speed is faster, thereby improving the efficiency of obtaining optical flow information, and thereby improving the efficiency of interpolation.

[0069] Among them, the conversion module can process the optical flow information and the video frame, and then predict the video frame after the optical flow processing. For example, the conversion module can process the optical flow information of the first video frame at the time of interpolation and the first video frame, and then predict the image of the first video frame at the time of interpolation. For example, the conversion module can add the motion vector based on the optical flow to the corresponding grid coordinates to generate the sampling position coordinates, and then perform grid sampling on the input image based on the sampled position coordinates to obtain the image after the optical flow.

[0070] Optionally, the conversion module (warp module) may be a module after the operator fusion processing. For example, in the model processing process, each access to data requires a certain amount of time, and the multi-node processing process will result in a long processing time and low processing efficiency. Therefore, the conversion module after the operator fusion processing in the embodiment of the present disclosure may include one processing node, thereby saving the time for data storage, improving the processing efficiency of the video interpolation model, and improving the efficiency of interpolation.

[0071] The correction module is used to obtain correction information (such as occlusion information and residual information) of the interpolated image, thereby improving the accuracy of the interpolated image. The correction module can be a part of the convolutional neural network, so the correction module is more efficient in obtaining the correction information, thereby improving the processing efficiency of the video interpolation model and improving the efficiency of interpolation.

[0072] Next, combine Figure 5 , the structure of the video frame insertion model is explained.

[0073] Figure 5 This is a structural diagram of a video frame insertion model provided by an embodiment of the present disclosure. Figure 5, including: a video interpolation model. The video interpolation model may include an optical flow module, a conversion module, a correction module and an integration module (the integration module may integrate the information of the first three modules to obtain an interpolated image). The optical flow module is connected to the conversion module, the conversion module is also connected to the correction module, the correction module is also connected to the integration module, the optical flow module is the first module of the video interpolation model, and the integration module is the last module of the video interpolation model.

[0074] Next, combine Figure 6 , the structure of the optical flow module is explained.

[0075] Figure 6 This is a schematic diagram of the structure of an optical flow module provided by an embodiment of the present disclosure. Figure 6 , including: an optical flow module. The optical flow module may include a partial convolutional network 1, a partial convolutional network 2, ..., a partial convolutional network n. When performing convolution processing on an image, the partial convolutional network in the optical flow module can retain valid pixel information in the mask, thereby improving the efficiency of the convolution.

[0076] It should be noted that the structure of the convolution layer of the correction module is similar to that of the optical flow module, both of which are partial convolutional networks, and will not be described in detail in the embodiments of the present disclosure.

[0077] Optionally, the electronic device can input the first video frame and the second video frame into the video interpolation model, and the video interpolation model can output the interpolated image between the first video frame and the second video frame. Since the optical flow module and the correction module of the video interpolation model are partial convolutional neural networks, the convolution efficiency of the optical flow module and the correction module is relatively high, and the conversion module in the video interpolation model is a module after operator fusion processing, which can save the time for reading and writing data. Therefore, the video interpolation model can quickly determine the interpolated image and improve the efficiency of determining the interpolated image. In low-latency scenarios (such as live broadcast scenarios), electronic devices can quickly generate high-frame-rate live videos based on the video interpolation model without reducing the live broadcast delay, thereby improving the user experience.

[0078] S204: insert an interpolation frame image between the first video frame and the second video frame.

[0079] Optionally, after the electronic device determines the interpolated image between the first video frame and the second video frame, it can insert the interpolated image between the first video frame and the second video frame, so that the first video frame, the interpolated image and the third video frame can constitute a continuous video frame, thereby improving the frame rate of the video.

[0080] The disclosed embodiment provides a video processing method, in which an electronic device can determine adjacent first and second video frames in a video to be processed, determine an interpolation time based on the playback time of the first video frame and the playback time of the second video frame, process the first and second video frames based on a video interpolation model and the interpolation time, and obtain an interpolation image between the first and second video frames, and the electronic device can insert the interpolation image between the first and second video frames. In the above method, since the optical flow module and the correction module in the video interpolation model and the conversion module are modules after operator fusion processing, the electronic device can obtain optical flow information and correction information at a faster speed, and the conversion module can read and write data at a faster speed, and the video interpolation model does not need to process the semantic information of the first and second video frames, thereby improving the efficiency of determining the interpolation image, thereby improving the efficiency of video interpolation.

[0081] exist Figure 2 Based on the embodiment shown below, combined with Figure 7 , in the above-mentioned video processing method, based on the video interpolation model and the interpolation time, the first video frame and the second video frame are processed to obtain the interpolation image between the first video frame and the second video frame.

[0082] Figure 7 A schematic diagram of a method for determining an interpolated image provided by an embodiment of the present disclosure. Figure 7 The method flow includes:

[0083] S701. Perform partial convolution processing on a first video frame based on an optical flow module and a frame insertion time to obtain a first optical flow feature of the first video frame at the frame insertion time.

[0084] The first optical flow feature may include optical flow information of the first video frame at the interpolation time. For example, the first optical flow feature may include optical flow information between the playback time and the interpolation time of the first video frame. For example, the optical flow module may perform partial convolution processing on the first video frame based on the interpolation time, and then obtain the first optical flow feature, which may include optical flow information of pixels in the first video frame at the interpolation time.

[0085] Optionally, before the optical flow module performs partial convolution processing on the first video frame, the electronic device may downsample the first video frame to obtain a downsampled image corresponding to the first video frame. In this way, the electronic device may perform convolution processing on the downsampled image corresponding to the first video frame based on the optical flow module. Since the number of pixels in the downsampled image corresponding to the first video frame is smaller than the number of pixels in the first video frame, the efficiency of the convolution can be improved, and thus the efficiency of obtaining the first optical flow feature can be improved.

[0086] Optionally, after the optical flow module obtains the first optical flow feature, the conversion module may perform upsampling processing on the first optical flow feature (feature image), and scale transform the first optical flow feature based on the size of the first video frame.

[0087] S702: Perform partial convolution processing on the second video frame based on the optical flow module and the frame insertion time to obtain a second optical feature of the second video frame at the frame insertion time.

[0088] The second optical flow feature may include optical flow information of the second video frame at the interpolation time. For example, the optical flow module may perform partial convolution processing on the second video frame based on the interpolation time, and then obtain the second optical flow feature, which may include optical flow information of pixels in the second video frame at the interpolation time.

[0089] Optionally, before the optical flow module performs partial convolution processing on the second video frame, the electronic device may downsample the second video frame to obtain a downsampled image corresponding to the second video frame. In this way, the electronic device may perform convolution processing on the downsampled image corresponding to the second video frame based on the optical flow module. Since the number of pixels in the downsampled image corresponding to the second video frame is smaller than the number of pixels of the second video frame, the efficiency of the convolution can be improved, and thus the efficiency of obtaining the second optical flow feature can be improved.

[0090] Optionally, after the optical flow module obtains the second optical flow feature, the conversion module may perform upsampling processing on the second optical flow feature (feature image), and scale transform the second optical flow feature based on the size of the second video frame.

[0091] It should be noted that the conversion module performs upsampling and scaling processing on the first optical flow feature and the second optical flow feature, which are all node processing after operator fusion, which can improve processing efficiency.

[0092] S703: Determine an interpolated frame image based on the first video frame, the first optical flow feature, the second video frame, and the second optical flow feature.

[0093] Optionally, the electronic device may determine the interpolated image based on the following feasible implementation methods: based on the conversion module, processing the first video frame and the first optical flow feature to obtain a first predicted interpolated image; based on the conversion module, processing the second video frame and the second optical flow feature to obtain a second predicted interpolated image; based on the first video frame, the first predicted interpolated image, the second video frame and the second predicted interpolated image, determining the interpolated image.

[0094] The first predicted interpolated frame image may be an image obtained by warping the first video frame to the interpolation time based on the first optical flow feature. For example, if the playback time of the first video frame is earlier than the interpolation time, the first optical flow feature may be forward-traced optical flow information. Therefore, the conversion module may forward-trace the pixels in the first video frame based on the first optical flow feature, and then obtain the first predicted interpolated frame image. For example, the conversion module may determine the sampling coordinates of the pixels of the first video frame based on the first optical flow feature, and then sample the pixels of the first video frame based on the sampling coordinates to obtain the first predicted interpolated frame image.

[0095] The second predicted interpolated frame image may be an image obtained by warping the second video frame to the interpolated frame time based on the second optical flow feature. For example, if the playback time of the second video frame is later than the interpolated frame time, the second optical flow feature may be the optical flow information traced back. Therefore, the conversion module may trace back the pixels in the second video frame based on the second optical flow feature, and then obtain the second predicted interpolated frame image. For example, the conversion module may determine the sampling coordinates of the pixels of the second video frame based on the second optical flow feature, and then sample the pixels of the second video frame based on the sampling coordinates to obtain the second predicted interpolated frame image.

[0096] It should be noted that, although the first predicted interpolated frame image and the second predicted interpolated frame image are both images at the interpolation moment predicted by the conversion module, since the first predicted interpolated frame image is obtained by forward tracing based on the first video frame, and the second predicted interpolated frame image is obtained by backward tracing based on the second video frame, there is a difference between the first predicted interpolated frame image and the second predicted interpolated frame image.

[0097] Next, combine Figure 8 , the interpolated image is explained.

[0098] Figure 8 A schematic diagram of an interpolated image provided by an embodiment of the present disclosure. Figure 8 , including a first video frame and a second video frame. The first video frame and the second video frame both include a circle. The circle in the first video frame is on the left, and the circle in the second video frame is on the right. Electronic device ( Figure 8 (not shown) can determine that in the first video frame-second video frame, the circle moves to the right. The electronic device can collect pixels in the first video frame based on the first optical flow feature to obtain a first predicted interpolated image, and the electronic device can collect pixels in the second video frame based on the second optical flow feature to obtain a second predicted interpolated image. Due to the difference between the first optical flow feature and the second optical flow feature, there are also differences between the first predicted interpolated image and the second predicted interpolated image (different positions of objects, etc.).

[0099] Among them, the electronic device determines the interpolated frame image based on the first video frame, the first predicted interpolated frame image, the second video frame and the second predicted interpolated frame image. Specifically, it can be: based on the correction module, the first video frame, the first predicted interpolated frame image, the second video frame and the second predicted interpolated frame image are partially convolved to obtain predicted occlusion information and predicted residual information of the interpolated frame image, and the interpolated frame image is determined based on the first predicted interpolated frame image, the second predicted interpolated frame image, the predicted occlusion information and the predicted residual information.

[0100] The predicted occlusion information is the occlusion information of the interpolated frame image. For example, both the first video frame and the second video frame may include object 1 and object 2, and object 1 may occlude object 2 in the interpolated frame image between the first video frame and the second video frame. The correction module can predict the occlusion information.

[0101] Next, combine Fig. 9 , the scene of interpolated image occlusion is explained.

[0102] Fig. 9 A schematic diagram of a scene of interpolated image occlusion provided by an embodiment of the present disclosure. Fig. 9 , including: a first video frame and a second video frame. The first video frame and the second video frame both include a circle and a rectangle, the circle in the first video frame is located on the left side of the rectangle, and the circle in the second video frame is located on the right side of the rectangle.

[0103] See also Fig. 9 Although the rectangle in the first video frame and the second video frame does not block the circle, based on the optical flow between the first video frame and the second video frame, the electronic device ( Fig. 9 (not shown) can determine that the circle is moving to the right, so in the interpolated image, the circle will be blocked by the rectangle, and the correction module can predict the blocking information.

[0104] It should be noted that during the training process of the video interpolation model, the correction module can learn the ability to predict the occlusion information.

[0105] Among them, the predicted residual information can be the residual information between the first predicted interpolation image, the second predicted interpolation image and the real interpolation image predicted by the correction module. For example, the conversion module warps the first video frame to the interpolation moment based on the first optical flow feature to obtain the first predicted interpolation image. There is a residual between the first predicted interpolation image and the real interpolation image. The correction module of the video interpolation model can learn the residual during the training process. Therefore, during the use of the video interpolation model, the correction module can predict the residual. For example, after the warp module outputs the first predicted interpolation image and the second predicted interpolation image, the correction module can predict the real occlusion information of the interpolation image and the residual information between the current predicted interpolation image and the real interpolation image based on the first video frame, the first predicted interpolation image, the second video frame and the second predicted interpolation image.

[0106] Optionally, the electronic device determines the interpolated image based on the first predicted interpolated image, the second predicted interpolated image, the predicted occlusion information and the predicted residual information. Specifically, it may be: determining a first weight of the first interpolated image in the predicted occlusion information, determining a second weight of the second interpolated image in the predicted occlusion information, and determining the interpolated image based on the first weight, the first interpolated image, the second weight, the second interpolated image and the predicted residual information.

[0107] Optionally, since the interpolation moment can be close to the playback moment of the first video frame, or close to the playback moment of the second video frame, the first predicted interpolation image and the second predicted interpolation image have different effects on the interpolation image. For example, if the duration between the interpolation moment and the playback moment of the first video frame is less than the duration between the interpolation moment and the playback moment of the second video frame, it means that the interpolation moment is close to the playback moment of the first video frame, and the interpolation image is greatly affected by the first video frame; if the duration between the interpolation moment and the playback moment of the second video frame is less than the duration between the interpolation moment and the playback moment of the first video frame, it means that the interpolation moment is close to the playback moment of the second video frame, and the interpolation image is greatly affected by the second video frame.

[0108] Optionally, the electronic device may determine the first weight based on the following formula:

[0109] A=Ot*(1-t)

[0110] Among them, A is the first weight, Ot is the predicted occlusion information, and t is the interpolation time.

[0111] It should be noted that, in the above formula, Ot and the duration between the playback time of the first video frame and the playback time of the second video frame are all normalized.

[0112] Similarly, based on the above formula, the electronic device can determine the second weight as:

[0113] B=(1-Ot)*t

[0114] Among them, B is the second weight, Ot is the predicted occlusion information, and t is the interpolation time.

[0115] Optionally, after the electronic device determines the first weight and the second weight, it can determine the interpolated image based on the following feasible implementation method: multiply the first interpolated image by the first weight to obtain the first image, multiply the second interpolated image by the second weight to obtain the second image, and add the first image, the second image and the predicted residual information to obtain the interpolated image.

[0116] Optionally, the electronic device may determine the interpolated image based on the following formula:

[0117]

[0118] Among them, It can be an interpolation image, Ot is predicted occlusion information, t is the interpolation time, It0 is the first predicted interpolation image, It1 is the second predicted interpolation image, and Rt is the prediction residual information.

[0119] Based on the above formula, the electronic device can determine the interpolated image between the first video frame and the second video frame.

[0120] Optionally, based on any one of the above embodiments, the video processing method further includes: determining the running status of the main GPU, and when the running status is an idle state, executing the video processing method based on the main GPU; when the running status is an occupied state, determining an idle slave GPU from at least one slave GPU, and executing the video processing method based on the idle slave GPU.

[0121] The main GPU is a GPU for decoding videos. For example, when an electronic device obtains a video to be interpolated, the electronic device can decode the video in the main GPU to obtain all video frames of the video.

[0122] The running state may be an idle state and an occupied state. For example, the idle state may be a state where the GPU does not perform frame insertion processing on the video frames in the video, and the occupied state may be a state where the GPU performs frame insertion processing on the video frames in the video.

[0123] Optionally, when the main GPU is in an idle state, the electronic device may preferentially perform video frame interpolation processing based on the main GPU. Since the main GPU includes all video frames, the main GPU may quickly perform video frame interpolation processing, thereby improving the efficiency of video frame interpolation.

[0124] Optionally, when the operation state of the master GPU is occupied, the electronic device may determine an idle slave GPU from at least one slave GPU, and perform video frame insertion processing based on the idle slave GPU. For example, if the master GPU is performing video frame insertion processing, the master GPU may copy multiple video frames to be inserted to the idle slave GPU. Although copying multiple video frames to the slave GPU takes time, the slave GPU can perform frame insertion processing on the video in parallel, thereby improving the efficiency of video frame insertion.

[0125] Optionally, when the master GPU copies video frames to the slave GPU, the master GPU may copy multiple video frames to the slave GPU at one time, thereby reducing the number of times the video frames are copied and improving the efficiency of video interpolation. For example, the master GPU performs interpolation processing on the 1st, 2nd, 3rd and 4th frames, and the master GPU may copy the 4th, 5th, 6th and 7th frames to the slave GPU for interpolation processing, so that only the 4th frame is duplicated, and the 1st, 2nd, 3rd, 5th, 6th and 7th frames are only copied once (if two frames are copied, the master GPU processes the 1st and 2nd frames, the slave GPU1 processes the 2nd and 3rd frames, and the slave GPU2 processes the 3rd and 4th frames, so that the 2nd and 3rd frames are duplicated), thereby improving the efficiency of interpolation.

[0126] Optionally, before the electronic device performs video interpolation based on multiple GPUs, it can also determine the number of GPUs to be used, thereby saving GPU resources. For example, the electronic device can determine that the upper limit of the duration of video interpolation (the live broadcast end needs to limit the interpolation duration to avoid high latency) is t1, the duration of the main GPU decoding the video is t2, and the time it takes to copy the video frame between GPUs is t3. Then the duration of other GPUs processing video interpolation is: t2+t3. Therefore, the number of GPUs scheduled by the electronic device is (t2+t3) / t1, and this value can be rounded up. In this way, the electronic device can reasonably schedule GPU resources based on the upper limit of the duration of video interpolation to improve the utilization of the GPU.

[0127] The disclosed embodiment provides a method for determining an interpolated image, which performs partial convolution processing on a first video frame based on an optical flow module and an interpolation time to obtain a first optical flow feature of the first video frame at the interpolation time, performs partial convolution processing on a second video frame based on the optical flow module and the interpolation time to obtain a second optical feature of the second video frame at the interpolation time, and determines the interpolated image based on the first video frame, the first optical flow feature, the second video frame, and the second optical flow feature. In this way, the electronic device can accurately determine the residual between the predicted interpolated image and the real interpolated image based on the first predicted interpolated image traced forward and the second predicted interpolated image traced backward, and then accurately and quickly generate the interpolated image between the first video frame and the second video frame in combination with the predicted occlusion information. Since the calculation rate of the optical flow module, the warp module, and the correction module in the video interpolation model is relatively fast, the electronic device can quickly determine the interpolated image, thereby improving the efficiency of video interpolation.

[0128] Based on any of the above embodiments, Fig.10 , the process of the video processing method is explained.

[0129] Fig.10 A schematic diagram of a video processing method provided by an embodiment of the present disclosure. Fig.10 , including: a first video frame, a second video frame, an optical flow module, a conversion module, a correction module and an integration module. Electronic device ( Fig.10 (not shown) can input the first video frame and the second video frame to the optical flow module, and the optical flow module defaults the frame insertion time to be the middle time between the playback time of the first video frame and the playback time of the second video frame. The optical flow module can output the first optical flow feature and the second optical flow feature.

[0130] See also Fig.10 The conversion module can process the first optical flow feature, the first video frame, the second optical flow feature, and the second video frame, and then transfer the optical flow of the first video frame to the interpolation time to obtain the first predicted interpolation image, and transfer the optical flow of the second video frame to the interpolation time to obtain the second predicted interpolation image. The correction module can process the first predicted interpolation image, the second predicted interpolation image, the first video frame, and the second video frame to obtain the predicted residual information and the predicted occlusion information.

[0131] See also Fig.10 The integration module can obtain an interpolated frame image between the first video frame and the second video frame based on the prediction residual information, the prediction occlusion information, the first prediction interpolated frame image and the second prediction interpolated frame image. Since the optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after the operator fusion processing, the optical flow module, the correction module and the conversion module have a high efficiency in processing data, thereby improving the efficiency of determining the interpolated frame image and improving the efficiency of video interpolation.

[0132] Fig.11 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure. Fig.11 , comprising a video processing device 110, the video processing device 110 comprising a first determining module 111, a second determining module 112, a processing module 113 and a frame inserting module 114, wherein:

[0133] The first determining module 111 is used to determine adjacent first video frames and second video frames in a video;

[0134] The second determination module 112 is used to determine the frame insertion time based on the playback time of the first video frame and the playback time of the second video frame;

[0135] The processing module 113 is used to process the first video frame and the second video frame based on the video interpolation model and the interpolation time to obtain an interpolation image between the first video frame and the second video frame, wherein the video interpolation model includes an optical flow module, a conversion module and a correction module, the optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after operator fusion processing;

[0136] The interpolation module 114 is configured to insert the interpolation frame image between the first video frame and the second video frame.

[0137] According to one or more embodiments of the present disclosure, the processing module 113 is specifically used for:

[0138] Based on the optical flow module and the frame insertion time, performing partial convolution processing on the first video frame to obtain a first optical flow feature of the first video frame at the frame insertion time;

[0139] Based on the optical flow module and the frame insertion time, performing partial convolution processing on the second video frame to obtain a second optical flow feature of the second video frame at the frame insertion time;

[0140] The interpolated frame image is determined based on the first video frame, the first optical flow feature, the second video frame, and the second optical flow feature.

[0141] According to one or more embodiments of the present disclosure, the processing module 113 is specifically used for:

[0142] Based on the conversion module, the first video frame and the first optical flow feature are processed to obtain a first predicted interpolated frame image;

[0143] Based on the conversion module, the second video frame and the second optical flow feature are processed to obtain a second predicted interpolated frame image;

[0144] The interpolation frame image is determined based on the first video frame, the first predicted interpolation frame image, the second video frame and the second predicted interpolation frame image.

[0145] According to one or more embodiments of the present disclosure, the processing module 113 is specifically used for:

[0146] Based on the correction module, performing partial convolution processing on the first video frame, the first predicted interpolated frame image, the second video frame and the second predicted interpolated frame image to obtain predicted occlusion information and predicted residual information of the interpolated frame image;

[0147] The interpolation frame image is determined based on the first predicted interpolation frame image, the second predicted interpolation frame image, the predicted occlusion information, and the predicted residual information.

[0148] According to one or more embodiments of the present disclosure, the processing module 113 is specifically used for:

[0149] Determining a first weight of the first predicted interpolated frame image in the predicted occlusion information;

[0150] Determining a second weight of the second predicted interpolated frame image in the predicted occlusion information;

[0151] The interpolation frame image is determined based on the first weight, the first predicted interpolation frame image, the second weight, the second predicted interpolation frame image, and the prediction residual information.

[0152] According to one or more embodiments of the present disclosure, the processing module 113 is specifically used for:

[0153] multiplying the first interpolated frame image by the first weight to obtain a first image;

[0154] multiplying the second interpolated frame image by the second weight to obtain a second image;

[0155] The first image, the second image and the prediction residual information are added together to obtain the interpolated frame image.

[0156] According to one or more embodiments of the present disclosure, the processing module 113 is further configured to:

[0157] Determine the operating state of the master GPU, where the master GPU is a GPU that decodes the video;

[0158] When the running state is an idle state, executing the video processing method according to any one of claims 1 to 6 based on the main GPU;

[0159] When the running state is the occupied state, a slave GPU in an idle state is determined from at least one slave GPU, and the video processing method according to any one of claims 1 to 6 is executed based on the slave GPU in the idle state.

[0160] The video processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0161] Fig.12 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.12 , which shows a schematic diagram of the structure of an electronic device 1200 suitable for implementing the embodiment of the present disclosure. The electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0162] like Fig.12 As shown, the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 to a random access memory (RAM) 1203. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0163] Typically, the following devices may be connected to the I / O interface 1205: input devices 1206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1209. The communication device 1209 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 1200 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0164] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1209, or installed from a storage device 1208, or installed from a ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0165] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0166] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0167] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0168] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, various methods that may be involved in the above embodiments are implemented.

[0169] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements various methods that may be involved in the above embodiments when executed by a processor.

[0170] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0171] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0173] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0174] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0176] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0177] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0178] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from a user, the method of sending a prompt message to the user can be, for example, a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0179] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0180] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0181] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0182] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0183] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: include: Determining adjacent first and second video frames in the video; Determining a frame insertion time based on a playback time of the first video frame and a playback time of the second video frame; Based on the video interpolation model and the interpolation time, the first video frame and the second video frame are processed to obtain an interpolation image between the first video frame and the second video frame, wherein the video interpolation model includes an optical flow module, a conversion module and a correction module, the optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after operator fusion processing; The interpolation frame image is inserted between the first video frame and the second video frame.

2. The method according to claim 1, characterized in that The processing of the first video frame and the second video frame based on the video interpolation model and the interpolation time to obtain an interpolation image between the first video frame and the second video frame includes: Based on the optical flow module and the frame insertion time, performing partial convolution processing on the first video frame to obtain a first optical flow feature of the first video frame at the frame insertion time; Based on the optical flow module and the frame insertion time, performing partial convolution processing on the second video frame to obtain a second optical flow feature of the second video frame at the frame insertion time; The interpolated frame image is determined based on the first video frame, the first optical flow feature, the second video frame, and the second optical flow feature.

3. The method according to claim 2, characterized in that The determining the interpolated frame image based on the first video frame, the first optical flow feature, the second video frame, and the second optical flow feature includes: Based on the conversion module, the first video frame and the first optical flow feature are processed to obtain a first predicted interpolated frame image; Based on the conversion module, the second video frame and the second optical flow feature are processed to obtain a second predicted interpolated frame image; The interpolation frame image is determined based on the first video frame, the first predicted interpolation frame image, the second video frame and the second predicted interpolation frame image.

4. The method according to claim 3, characterized in that The determining the interpolated frame image based on the first video frame, the first predicted interpolated frame image, the second video frame, and the second predicted interpolated frame image includes: Based on the correction module, performing partial convolution processing on the first video frame, the first predicted interpolated frame image, the second video frame and the second predicted interpolated frame image to obtain predicted occlusion information and predicted residual information of the interpolated frame image; The interpolation frame image is determined based on the first predicted interpolation frame image, the second predicted interpolation frame image, the predicted occlusion information, and the predicted residual information.

5. The method according to claim 4, characterized in that The determining the interpolated frame image based on the first predicted interpolated frame image, the second predicted interpolated frame image, the predicted occlusion information and the predicted residual information includes: Determining a first weight of the first predicted interpolated frame image in the predicted occlusion information; Determining a second weight of the second predicted interpolated frame image in the predicted occlusion information; The interpolation frame image is determined based on the first weight, the first predicted interpolation frame image, the second weight, the second predicted interpolation frame image, and the prediction residual information.

6. The method according to claim 5, characterized in that The determining the interpolated frame image based on the first weight, the first interpolated frame image, the second weight, the second interpolated frame image and the residual information includes: multiplying the first interpolated frame image by the first weight to obtain a first image; multiplying the second interpolated frame image by the second weight to obtain a second image; The first image, the second image and the prediction residual information are added together to obtain the interpolated frame image.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to an electronic device, wherein a master GPU and at least one slave GPU are provided in the electronic device, and the method further comprises: Determine the operating state of the master GPU, where the master GPU is a GPU that decodes the video; When the running state is an idle state, executing the video processing method according to any one of claims 1 to 6 based on the main GPU; When the running state is the occupied state, a slave GPU in an idle state is determined from at least one slave GPU, and the video processing method according to any one of claims 1 to 6 is executed based on the slave GPU in the idle state.

8. A video processing device, characterized in that: It includes a first determination module, a second determination module, a processing module and a frame insertion module, wherein: The first determination module is used to determine adjacent first video frames and second video frames in the video; The second determination module is used to determine the frame insertion time based on the playback time of the first video frame and the playback time of the second video frame; The processing module is used to process the first video frame and the second video frame based on the video interpolation model and the interpolation time to obtain an interpolation image between the first video frame and the second video frame, wherein the video interpolation model includes an optical flow module, a conversion module and a correction module, the optical flow module and the correction module are partial convolutional neural networks, and the conversion module is a module after operator fusion processing; The interpolation module is used to insert the interpolation frame image between the first video frame and the second video frame.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the video processing method according to any one of claims 1 to 7 is implemented.

Images