A video frame interpolation method and apparatus

Abstract

The application discloses a video frame interpolation method and device. The method comprises the following steps: acquiring a first video frame at a first time and a second video frame at a second time; generating a first mask corresponding to a third time according to a first motion vector diagram or a second motion vector diagram, performing motion compensation on the first video frame according to a third motion vector diagram, performing motion compensation on the second video frame according to a fourth motion vector diagram, obtaining more accurate first and second reference frames, and performing image fusion on the first and second reference frames according to a non-protection area in the first mask to obtain a third video frame at the third time, so that the first and second video frames are jointly referenced, and the image fusion is performed according to the non-protection area in the first mask, thereby ensuring the quality of video frame interpolation.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a video frame interpolation method and apparatus. Background Technology

[0002] To achieve a smoother user experience, smartphones and smart TVs are increasingly adopting higher screen refresh rates. For example, 60Hz is now mainstream for smartphones, with some leading manufacturers offering 90Hz or even 120Hz screens. However, video frame rates haven't kept pace with screen refresh rates. Common video frame rates are 24FPS, 30FPS, and 60FPS, which differ from screen refresh rates. Therefore, video frame interpolation (VFI) technology is commonly used. VFI involves generating one or more intermediate video frames between two consecutive original video frames. VFI can be used to increase the video frame rate, making it match the screen refresh rate.

[0003] Therefore, ensuring the quality of video frames during interpolation is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a video frame interpolation method and apparatus to ensure the quality of video frames during interpolation.

[0005] Firstly, this application provides a video frame interpolation method. This method can be executed by the video frame interpolation apparatus provided in this application, which can be the electronic device provided in this application. The method includes: The process involves: acquiring a first video frame at a first time point and a second video frame at a second time point; generating a first mask corresponding to a third time point based on a first motion vector map or a second motion vector map, the first mask including a non-protected area; the first motion vector map being the motion vector map from the first time point to the second time point, the second motion vector map being the motion vector map from the second time point to the first time point, and the third time point being located between the first time point and the second time point; performing motion compensation on the first video frame based on the third motion vector map to obtain a first reference frame for the third time point, and performing motion compensation on the second video frame based on a fourth motion vector map to obtain a second reference frame for the third time point, the third motion vector map being the motion vector map from the third time point to the first time point, and the fourth motion vector map being the motion vector map from the third time point to the second time point; and performing image fusion on the first reference frame and the second reference frame based on the non-protected area in the first mask to obtain the third video frame for the third time point.

[0006] In the above method, after obtaining the first video frame and the second video frame, a first mask corresponding to the third moment is generated based on the first motion vector map or the second motion vector map. Motion compensation is then performed on the first video frame based on the third motion vector map to obtain the first reference frame at the third moment. Motion compensation is then performed on the second video frame based on the fourth motion vector map to obtain the second reference frame at the third moment. This results in more accurate first and second reference frames, which are used to reference both the first and second video frames. Furthermore, image fusion is performed based on the unprotected area in the first mask to obtain the third video frame at the third moment, thus ensuring the quality of video frame interpolation.

[0007] In one possible design, each pixel in the unprotected region of the first mask corresponds to a first weight value and a second weight value; the step of image fusion of the first reference frame and the second reference frame based on the unprotected region of the first mask includes: Based on the position of the first pixel in the unprotected area, obtain the second pixel at the corresponding position in the first reference frame and the third pixel at the corresponding position in the second reference frame; perform a weighted operation on the pixel value of the second pixel based on the first weight value corresponding to the first pixel to obtain a first operation result; perform a weighted operation on the pixel value of the third pixel based on the second weight value corresponding to the first pixel to obtain a second operation result; and determine the sum of the first operation result and the second operation result as the pixel value of the pixel at the corresponding position in the third video frame.

[0008] In the above method, the pixel value of the corresponding pixel in the third video frame is obtained by combining the first weight value and the second weight value corresponding to the first pixel with the pixel value of the second pixel and the pixel value of the third pixel. Thus, any pixel value is referenced to the pixel value of the corresponding pixel in the first video frame and the second video frame, and image fusion is performed based on the non-protected area, thereby ensuring the quality of video frame interpolation.

[0009] In one possible design, generating the first mask corresponding to the third moment based on the first motion vector image includes: Based on the coordinates and motion vector values ​​of each pixel in the first motion vector image, the coordinates of each pixel at the third time point are determined, and a second mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the second mask; the iteration operation includes mean blurring and binarization of the second mask; erosion and mean blurring are performed on the second mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0010] In the above method, the second mask can be denoised through at least one iteration operation, and the pixel motion degree from the first video frame to the third video frame can be more clearly defined, thereby obtaining a more accurate first mask.

[0011] In one possible design, generating the first mask corresponding to the third moment based on the second motion vector image includes: Based on the coordinates and motion vector values ​​of each pixel in the second motion vector image, the coordinates of each pixel at the third time point are determined respectively, and a third mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the third mask; the iteration operation includes mean blurring and binarization of the third mask; erosion and mean blurring are performed on the third mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0012] In the above method, the third mask can be denoised through at least one iteration operation, and the pixel motion degree from the second video frame to the third video frame can be more clearly defined, thus obtaining a more accurate first mask.

[0013] In one possible design, the method further includes: Based on the first video frame and the second video frame, a first motion vector diagram from the first time point to the second time point is generated; based on the coordinates and motion vector value of the fourth pixel in the first motion vector diagram, the coordinates of the fourth pixel at the third time point are determined, wherein the fourth pixel is any pixel in the first motion vector diagram; at least one pixel adjacent to the coordinates of the fourth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the fourth pixel from the first time point to the third time point; if the motion vector value of the fifth pixel in the third motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the fifth pixel.

[0014] In the above manner, the coordinates of the fourth pixel at the third moment can be determined, and at least one adjacent pixel can be assigned a value according to the motion displacement. Thus, the motion vector values ​​of adjacent pixels can be mutually verified to obtain a more accurate motion vector value.

[0015] In one possible design, before performing motion compensation on the first video frame based on the third motion vector to obtain the first reference frame at the third moment, the method further includes: Based on the third motion vector image, a fourth mask corresponding to the third moment is generated; based on the fourth mask, pixels in the third motion vector image that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0016] In the above method, pixels that meet the repair conditions can be selected through the fourth mask, and a more accurate fourth mask can be obtained by updating these pixels that meet the repair conditions.

[0017] In one possible design, selecting pixels in the third motion vector image that satisfy the repair conditions based on the fourth mask includes: Select the sixth pixel in the fourth mask that meets the first repair condition, the first repair condition including that the pixel does not have a valid motion vector value; The step of updating the motion vector value of the pixel that meets the repair condition based on the motion vector value of at least one pixel associated with the pixel that meets the repair condition includes: Starting from the sixth pixel, a first set of pixels corresponding to the sixth pixel is searched in the fourth mask. The pixels in the first set of pixels are the pixels closest to the sixth pixel in at least one direction and have motion vector values. The seventh pixel corresponding to the sixth pixel in the third motion vector map and the second set of pixels corresponding to the first set of pixels are obtained. The motion vector value of the seventh pixel is determined based on the distance between the pixels in the second set of pixels and the seventh pixel, and the motion vector value of the pixels in the second set of pixels.

[0018] In the above method, by using the second pixel set corresponding to the first pixel set, the motion vector value of the nearest pixel of the seventh pixel in at least one direction can be obtained. Thus, by using the distance of the seventh pixel to its neighboring pixels and the motion vector value of the seventh pixel to its neighboring pixels, the motion vector value of the pixel can be repaired more accurately.

[0019] In one possible design, selecting pixels in the third motion vector image that satisfy the repair conditions based on the fourth mask includes: The eighth pixel in the fourth mask that meets the second repair condition is selected. The second repair condition includes the value of the eighth pixel indicating that the ninth pixel corresponding to the eighth pixel in the third motion vector map does not have a motion vector value, and at least one pixel adjacent to the ninth pixel in the first direction has a motion vector value. The step of updating the motion vector value of the pixel that meets the repair condition based on the motion vector value of at least one pixel associated with the pixel that meets the repair condition includes: Obtain the ninth pixel in the third motion vector diagram that corresponds to the eighth pixel; determine the motion vector value of the ninth pixel based on the motion vector value of at least one pixel adjacent to the ninth pixel in the first direction.

[0020] In the above method, the motion vector value of the ninth pixel is directly determined by the motion vector value of at least one pixel adjacent to the ninth pixel in the first direction, thereby improving the efficiency of the motion vector value of the repaired pixel.

[0021] In one possible design, the method further includes: Based on the first video frame and the second video frame, a second motion vector diagram from the second time point to the first time point is generated; based on the coordinates and motion vector value of the tenth pixel in the second motion vector diagram, the coordinates of the tenth pixel at the third time point are determined, where the tenth pixel is any pixel in the second motion vector diagram; at least one pixel adjacent to the coordinates of the tenth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the tenth pixel from the first time point to the third time point; if the motion vector value of the eleventh pixel in the fourth motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the eleventh pixel.

[0022] In the above method, the coordinates of the tenth pixel at the third moment can be determined, and at least one adjacent pixel can be assigned a value according to the motion displacement. Thus, the motion vector values ​​of adjacent pixels can be mutually verified to obtain a more accurate motion vector value.

[0023] In one possible design, before performing motion compensation on the second video frame based on the fourth motion vector to obtain the second reference frame at the third moment, the method further includes: Based on the fourth motion vector map, a fifth mask corresponding to the third moment is generated; based on the fifth mask, pixels in the fourth motion vector map that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0024] In the above method, pixels that meet the repair conditions can be selected through the fifth mask, and a more accurate fifth mask can be obtained by updating these pixels that meet the repair conditions.

[0025] In one possible design, selecting pixels in the fourth motion vector image that satisfy the repair conditions based on the fifth mask includes: The twelfth pixel in the fifth mask that meets the third repair condition is selected. The third repair condition includes that the pixel does not have a valid motion vector value. The step of updating the motion vector value of the pixel that meets the repair conditions based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions includes: Starting from the twelfth pixel, a third pixel set corresponding to the twelfth pixel is searched in the fifth mask. The pixels in the third pixel set are the pixels closest to the twelfth pixel in at least one direction and have motion vector values. The thirteenth pixel corresponding to the twelfth pixel in the fourth motion vector map and the fourth pixel set corresponding to the third pixel set are obtained. The motion vector value of the thirteenth pixel is determined based on the distance between the pixels in the fourth pixel set and the thirteenth pixel, and the motion vector values ​​of the pixels in the fourth pixel set.

[0026] In the above method, by using the fourth pixel set corresponding to the third pixel set, the motion vector value of the twelfth pixel in at least one direction can be obtained. Thus, by using the distance between the twelfth pixel and its neighboring pixels and the motion vector value of the neighboring pixels, the motion vector value of the pixel can be repaired more accurately.

[0027] In one possible design, selecting pixels in the fourth motion vector image that satisfy the repair conditions based on the fifth mask includes: Select the fourteenth pixel in the fifth mask that meets the fourth repair condition. The fourth repair condition includes that the value of the fourteenth pixel indicates that the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image does not have a motion vector value, and that at least one pixel adjacent to the fifteenth pixel in the second direction has a motion vector value. The step of updating the motion vector value of the pixel that meets the repair condition based on the motion vector value of at least one pixel associated with the pixel that meets the repair condition includes: Obtain the fifteenth pixel in the fourth motion vector diagram that corresponds to the fourteenth pixel; determine the motion vector value of the fourteenth pixel based on the motion vector value of at least one pixel adjacent to the fourteenth pixel in the second direction.

[0028] In the above method, the motion vector value of the fourteenth pixel is directly determined by the motion vector value of at least one pixel adjacent to the fourteenth pixel in the second direction, thereby improving the efficiency of the motion vector value of the repaired pixel.

[0029] Secondly, this application provides a video frame interpolation apparatus, comprising: The acquisition module is used to acquire the first video frame at the first moment and the second video frame at the second moment. A generation module is used to generate a first mask corresponding to a third time step based on a first motion vector map or a second motion vector map. The first mask includes a non-protected area. The first motion vector map is a motion vector map from the first time step to the second time step, and the second motion vector map is a motion vector map from the second time step to the first time step. The third time step is located between the first time step and the second time step. The processing module is configured to perform motion compensation on the first video frame based on a third motion vector map to obtain a first reference frame at the third time point, and to perform motion compensation on the second video frame based on a fourth motion vector map to obtain a second reference frame at the third time point, wherein the third motion vector map is the motion vector map from the third time point to the first time point, and the fourth motion vector map is the motion vector map from the third time point to the second time point; and The third video frame at the third moment is obtained by performing image fusion on the first reference frame and the second reference frame based on the unprotected area in the first mask.

[0030] In one possible design, in the unprotected region of the first mask, each pixel corresponds to a first weight value and a second weight value; the processing module is specifically used for: Based on the position of the first pixel in the unprotected area, obtain the second pixel at the corresponding position in the first reference frame and the third pixel at the corresponding position in the second reference frame; the first pixel is any pixel in the unprotected area; perform a weighted operation on the pixel value of the second pixel according to the first weight value corresponding to the first pixel to obtain a first operation result; perform a weighted operation on the pixel value of the third pixel according to the second weight value corresponding to the first pixel to obtain a second operation result; and determine the pixel value of the pixel at the corresponding position in the third video frame by summing the first operation result and the second operation result.

[0031] In one possible design, if the first mask is generated based on the first motion vector image, the pixel value of the first pixel in the first mask is used to indicate the first weight value, and the second weight value is equal to 1 minus the first weight value; or if the first mask is generated based on the second motion vector image, the pixel value of the first pixel in the first mask is used to indicate the second weight value, and the first weight value is equal to 1 minus the second weight value.

[0032] In one possible design, the generation module is specifically used for: Based on the coordinates and motion vector values ​​of each pixel in the first motion vector image, the coordinates of each pixel at the third time point are determined, and a second mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the second mask; the iteration operation includes mean blurring and binarization of the second mask; erosion and mean blurring are performed on the second mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0033] In one possible design, the generation module is specifically used for: Based on the coordinates and motion vector values ​​of each pixel in the second motion vector image, the coordinates of each pixel at the third time point are determined respectively, and a third mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the third mask; the iteration operation includes mean blurring and binarization of the third mask; erosion and mean blurring are performed on the third mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0034] In one possible design, the generation module is further configured to: Based on the first video frame and the second video frame, a first motion vector diagram from the first time point to the second time point is generated; based on the coordinates and motion vector value of the fourth pixel in the first motion vector diagram, the coordinates of the fourth pixel at the third time point are determined, wherein the fourth pixel is any pixel in the first motion vector diagram; at least one pixel adjacent to the coordinates of the fourth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the fourth pixel from the first time point to the third time point; if the motion vector value of the fifth pixel in the third motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the fifth pixel.

[0035] In one possible design, the generation module is further configured to: Based on the third motion vector image, a fourth mask corresponding to the third moment is generated; based on the fourth mask, pixels in the third motion vector image that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0036] In one possible design, the generation module is specifically used for: A sixth pixel in the fourth mask that meets the first repair condition is selected, the first repair condition including that the pixel does not have a valid motion vector value; starting from the sixth pixel, a first pixel set corresponding to the sixth pixel is searched in the fourth mask, the pixels in the first pixel set are the pixels closest to the sixth pixel in at least one direction and have motion vector values; a seventh pixel corresponding to the sixth pixel in the third motion vector map and a second pixel set corresponding to the first pixel set are obtained; the motion vector value of the seventh pixel is determined according to the distance between the pixels in the second pixel set and the seventh pixel, and the motion vector value of the pixels in the second pixel set.

[0037] In one possible design, the generation module is specifically used for: Select an eighth pixel in the fourth mask that meets the second repair condition, the second repair condition including that the value of the eighth pixel indicates that the ninth pixel corresponding to the eighth pixel in the third motion vector image does not have a motion vector value, and that at least one pixel adjacent to the ninth pixel in the first direction has a motion vector value; obtain the ninth pixel corresponding to the eighth pixel in the third motion vector image; determine the motion vector value of the ninth pixel based on the motion vector value of at least one pixel adjacent to the ninth pixel in the first direction.

[0038] In one possible design, the generation module is further configured to: Based on the first video frame and the second video frame, a second motion vector diagram from the second time point to the first time point is generated; based on the coordinates and motion vector value of the tenth pixel in the second motion vector diagram, the coordinates of the tenth pixel at the third time point are determined, where the tenth pixel is any pixel in the second motion vector diagram; at least one pixel adjacent to the coordinates of the tenth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the tenth pixel from the first time point to the third time point; if the motion vector value of the eleventh pixel in the fourth motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the eleventh pixel.

[0039] In one possible design, the generation module is further configured to: Based on the fourth motion vector map, a fifth mask corresponding to the third moment is generated; based on the fifth mask, pixels in the fourth motion vector map that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0040] In one possible design, the generation module is specifically used for: Select the twelfth pixel in the fifth mask that meets the third repair condition, the third repair condition including that the pixel does not have a valid motion vector value; starting from the twelfth pixel, search in the fifth mask to obtain a third pixel set corresponding to the twelfth pixel, the pixels in the third pixel set being the pixels closest to the twelfth pixel in at least one direction and having a motion vector value; obtain the thirteenth pixel corresponding to the twelfth pixel in the fourth motion vector map, and the fourth pixel set corresponding to the third pixel set; determine the motion vector value of the thirteenth pixel based on the distance between the pixels in the fourth pixel set and the thirteenth pixel, and the motion vector value of the pixels in the fourth pixel set.

[0041] In one possible design, the generation module is specifically used for: Select the fourteenth pixel in the fifth mask that meets the fourth repair condition. The fourth repair condition includes that the value of the fourteenth pixel indicates that the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image does not have a motion vector value, and that at least one pixel adjacent to the fifteenth pixel in the second direction has a motion vector value; obtain the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image; determine the motion vector value of the fourteenth pixel based on the motion vector value of at least one pixel adjacent to the fourteenth pixel in the second direction.

[0042] Thirdly, an electronic device is provided, the electronic device comprising: one or more processors; one or more memories; wherein the one or more memories store one or more computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of the first aspects above.

[0043] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium including computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first aspects above.

[0044] For the beneficial effects in aspects two through four above, please refer to the beneficial effects in aspect one, which will not be repeated here. Attached Figure Description

[0045] Figure 1 A schematic diagram of a system architecture to which a video frame interpolation method can be applied, provided in an embodiment of this application; Figure 2A schematic diagram of the video frame interpolation module in a system architecture diagram of a video frame interpolation method applicable to an embodiment of this application; Figure 3 A flowchart illustrating the steps of a video frame display method that can be implemented using a video frame interpolation method provided in this application embodiment; Figure 4 A flowchart illustrating the steps of a video frame interpolation method provided in this application embodiment; Figure 5 A flowchart illustrating the steps of generating a first mask in a video frame interpolation method provided in this application embodiment; Figure 6 A flowchart illustrating the steps of a video frame display method that can be implemented using a video frame interpolation method provided in this application embodiment; Figure 7 A schematic diagram of a video frame interpolation device provided in an embodiment of this application; Figure 8 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0047] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” means one or more (including two); “and / or” describes the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0048] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0049] In this application's embodiments, the term "multiple" refers to two or more. Therefore, in this application's embodiments, "multiple" can also be understood as "at least two." "At least one" can be understood as one or more, such as one, two, or more. For example, "including at least one" means including one, two, or more, and is not limited to which ones are included. For example, "including at least one of A, B, and C" could mean A, B, C; A and B; A and C; B and C; or A and B and C. Similarly, the understanding of descriptions such as "at least one" is similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone; A and B existing simultaneously; or B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0050] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects, and are not used to limit the order, sequence, priority or importance of multiple objects.

[0051] For ease of understanding, the terms involved in the embodiments of this application are explained, and the explanation of these terms is also part of the inventive content of the embodiments of this application.

[0052] Application scenarios

[0053] This application provides a video frame interpolation method, which can be applied to... Figure 1 The system architecture is shown. It should be understood that... Figure 1 The system architecture shown is only one example; in some other examples, it may have more complex architectures. Figure 1 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits. Figure 1The system architecture shown can be implemented by any device capable of displaying video, such as a terminal device, and is not limited here.

[0054] Figure 1 The system architecture shown may include a graphics processing unit (GPU) 101, a central processing unit (CPU) 102, a video processing chip 103, a graphics rendering engine 104, a video frame interpolation module 105, and a display screen 106. Specifically, it may be as follows: GPU101 is used to run graphics rendering engine 104 or video frame interpolation module 105.

[0055] CPU 102 is used to run graphics rendering engine 104 or video frame interpolation module 105.

[0056] It should be noted that only one component of GPU101 and CPU102 may be present. Figure 1 The system architecture shown can also include both GPU 101 and CPU 102. For example, when both GPU 101 and CPU 102 are present, the graphics rendering engine 104 or the video frame interpolation module 105 can be run in a hybrid manner. Since the advantage of GPU 101 is its strong parallel processing performance, which can process more pixels in the video frame at the same time, the graphics rendering engine 104 can be run by GPU 101. The advantage of CPU 102 is its faster running speed, so the video frame interpolation module 105 can be run by CPU 102.

[0057] The video processing chip 103 is used to output video frame streams to the video frame interpolation module 105.

[0058] The graphics rendering engine 104 is used to output video frame streams to the video frame interpolation module 105.

[0059] It should be noted that although both the video processing chip 103 and the graphics rendering engine 104 output video frames, their methods may differ. The input to the video processing chip 103 can be video frames, which are then preprocessed (e.g., noise reduction) before being output as preprocessed video frames. The input to the graphics rendering engine 104 can be non-video data (e.g., rendering models), which are then rendered to generate video frames.

[0060] In some cases, the video processing chip 103 and the graphics rendering engine 104 may not both be included. For example, in a video call scenario, graphics rendering is not required. The video processing chip 103 can directly acquire video frames captured by the camera and output them to the video frame interpolation module 105. In this case, the system architecture may not include the graphics rendering engine 104.

[0061] The video frame interpolation module 105 is used to generate interpolated video frames based on the video frame stream (including at least two video frames) output by the video processing chip 103 or the graphics rendering engine 104, according to a video frame interpolation algorithm. For example, it can generate a third video frame at a third moment based on the first video frame at a first moment and the second video frame at a second moment, with the third moment located between the first and second moments. The video frame interpolation module 105 can be specifically system software or application software.

[0062] Display screen 106 is used to display video frames. When the video frame interpolation module 105 generates interpolated video frames, the display screen 106 receives the original video frame stream input from the video processing chip 103 or the graphics rendering engine 104, as well as the newly generated interpolated video frames, and displays them.

[0063] in, Figure 1 The intermediate video frame interpolation module 105 may specifically include various sub-modules, such as... Figure 2 As shown. Obviously, Figure 2 The submodules shown are only examples; in some other examples, they may have more than one. Figure 2 The diagram shows more or fewer sub-modules. Each sub-module can be implemented by corresponding program code in the software running the video frame interpolation module 105. Each sub-module can implement its function by calling the corresponding function in the software. The specific functions of each sub-module are as follows: The input frame detection module 201 is used to detect whether the first and second input video frames have the same content and whether their frame sizes are consistent.

[0064] The consistency of content between two video frames refers to whether the content of the first and second video frames is the same. This can be determined by comparing whether the pixel values ​​of the two video frames are greater than a preset ratio (such as 99%). If the content of the two video frames is the same, it is obviously not necessary to interpolate the video frames to obtain a consistent video frame; one of the video frames can be directly selected. When the sizes of the two video frames are different, such as the first video frame being a landscape video frame and the second video frame being a portrait video frame, video frame interpolation cannot be performed.

[0065] The scene switching detection module 202 is used to detect whether there is a scene switching between the first video frame and the second video frame.

[0066] Scene switching can be detected using existing technologies such as optical flow. When there is a scene switch between two video frames, if the content of the video frames is too different, video frame interpolation cannot be performed.

[0067] The motion vector diagram module 203 is used to obtain a motion vector diagram based on the input first and second video frames, and to repair the motion vector diagram. It should be noted that in this embodiment, the motion vector (MV) represents the direction and distance of pixel movement. The motion vector diagram is used to describe the motion vector corresponding to each pixel in the video frame to be processed. For example, the motion vector diagram can be a diagram of the same size as the video frame to be processed, or it can be a matrix corresponding to each pixel in a video frame to be processed. In some cases, the motion vector diagram may show a phenomenon where some pixels lack motion vector values ​​(which can be called gaps or holes). Therefore, motion vector diagrams containing gaps or holes can be repaired so that pixels in these areas also have motion vectors.

[0068] The motion compensation module 204 is used to perform motion compensation (MC) on the input first video frame and second video frame based on the motion vector diagram to obtain the first reference frame and second reference frame to be interpolated.

[0069] Mask generation module 205: Used to generate masks, which are used to occlude (fully or partially) the video frames to be processed. The occluded area can be called the protected area. The mask can control the area of ​​the video frame to be processed or the processing procedure. Different parts of the image can be processed separately through the mask. It should be noted that this application provides a fusion mask, which can not only indicate whether a pixel is occluded, but also characterize the similarity between corresponding pixels in the two video frames when fused. For example, the value of the pixel in the mask can be used to characterize the weight value of the fusion of the two video frames.

[0070] The two-frame fusion module 206 performs two-frame fusion on the first reference frame and the second reference frame according to the fusion mask to obtain the interpolated third video frame.

[0071] Figure 1 The video frame interpolation method implemented by the system architecture shown can achieve, for example: Figure 3 The video frame display method is shown below. The specific steps are as follows: Step 301: The video processing chip 103 or the graphics rendering engine 104 outputs the video frame to the video frame interpolation module 105.

[0072] Step 302: The video frame interpolation module 105 receives and buffers the video frame as the target video frame, and determines whether the target video frame is the first video frame of the video frame stream. If so, proceed to step 303; otherwise, proceed to step 304.

[0073] Step 303: The video frame interpolation module 105 outputs the target video frame to the display screen 106 for display.

[0074] Step 304: The video frame interpolation module 105 obtains the interpolated video frame based on the target video frame and the preceding video frame.

[0075] The preceding video frame refers to the video frame adjacent to the target video frame before the video processing chip 103 or the graphics rendering engine 104 receives it.

[0076] Step 305: The video frame interpolation module 105 outputs the interpolated video frames to the display screen 106 for display.

[0077] If the number of interpolated video frames is greater than the target number of frames, return to step 304; otherwise, proceed to step 306. For example, if the target video frame is not the first video frame in the video frame stream, the preceding video frame corresponds to time T, and the corresponding time is T+1. In "T+1", "1" refers to one selected unit of time, which is the interval between the input video frames, such as 5 milliseconds (ms), 10 ms, etc. In "T+X", "+" refers to X selected units of time after time T. When X is a decimal, it represents the time proportional to one selected unit of time.

[0078] The interpolated video frame can be one or more frames. For example, if there is only one interpolated video frame, it corresponds to time T+0.5. If there are three interpolated video frames, they correspond to times T+0.25, T+0.5, and T+0.75 respectively. If the unit duration is selected as 10ms, then T+0.25 is 2.5ms after time T, T+0.5 is 5.0ms after time T, and T+0.75 is 7.5ms after time T. In the case of three interpolated video frames, steps 304 and 305 are repeated three times: interpolating and displaying the video frame at time T+0.25, interpolating and displaying the video frame at time T+0.5, and interpolating and displaying the video frame at time T+0.75.

[0079] Step 306: The video frame interpolation module 105 outputs the target video frame to the display screen 106 for display.

[0080] Step 307: The video frame interpolation module 105 updates the target video frame to the preceding video frame.

[0081] After step 307, you can return to step 301 and wait to acquire a new target video frame.

[0082] It should be noted that, Figure 3In the video frame display method shown, video frame interpolation can be used to make the video frame rate adapt to the screen refresh rate. The key step to achieve this is step 304. The following section will combine... Figure 4 This application provides a detailed description of a video frame interpolation method.

[0083] like Figure 4 As shown, the steps of the video frame interpolation method provided in this application can be as follows: Step 401: Obtain the first video frame at the first moment and the second video frame at the second moment.

[0084] In this context, the first moment and the second moment are in a sequential relationship. Whether the first moment or the second moment comes first does not affect the result of video frame interpolation. We can assume that the first moment is before the second moment, and the first video frame can have the same size as the second video frame.

[0085] Step 402: Generate the first mask corresponding to the third moment based on the first motion vector map or the second motion vector map.

[0086] Wherein, the first mask includes a non-protected area, the first motion vector diagram is the motion vector diagram from the first time point to the second time point, the second motion vector diagram is the motion vector diagram from the second time point to the first time point, and the third time point is located between the first time point and the second time point.

[0087] Step 403: Perform motion compensation on the first video frame according to the third motion vector to obtain the first reference frame at the third moment, and perform motion compensation on the second video frame according to the fourth motion vector to obtain the second reference frame at the third moment.

[0088] Wherein, the third motion vector diagram is the motion vector diagram from the third time point to the first time point, and the fourth motion vector diagram is the motion vector diagram from the third time point to the second time point.

[0089] Step 404: Based on the unprotected area in the first mask, perform image fusion on the first reference frame and the second reference frame to obtain the third video frame at the third moment.

[0090] It should be noted that the first mask can be entirely unprotected or may include both protected and unprotected areas. When the first mask is entirely unprotected, the pixel values ​​of each pixel can indicate how to perform video frame image fusion. When the first mask includes both protected and unprotected areas, the protected area can be used to indicate pixel areas in the first and second video frames that do not participate in video frame image fusion; these can be pixel areas in the first and second video frames where pixels do not move. Obviously, from the first moment to the second moment, since the pixels in the protected area do not move, there is no need to perform image fusion on the protected area; the corresponding pixel values ​​in the protected area of ​​either the first or second video frame can be directly copied. The unprotected area, however, indicates pixel areas that change from the first moment to the second moment; these can be pixel areas in the first and second video frames where pixels move. Therefore, it is necessary to combine the first and second video frames to perform video frame image fusion and estimate the corresponding pixel values ​​in the protected area of ​​the third video frame.

[0091] We can define the following: the first time point is time T, and the first video frame of time T is F0; the second time point is time T+1, and the second video frame of time T is F1; the third time point between the first and second time points is t, where t∈(0,1), where "1" in the interval (0,1) represents one selected unit of time, and "∈" represents "belongs to" in set theory; the third time point is time T+t, and the third video frame of time T is F0. t For example, t=0.5 can be chosen; the first motion vector diagram is Flow. 01 The second motion vector diagram is Flow. 10 The third motion vector diagram is Flow. t0 The fourth motion vector diagram is Flow. t1 The first mask (which can be called the fusion mask) is called MergeMask. Based on the above definition, the specific implementation of the video frame interpolation method provided in this application is described in detail: In one possible implementation of step 402, a first mask corresponding to the third moment can be generated based on the first motion vector image, specifically as follows: Based on the coordinates and motion vector values ​​of each pixel in the first motion vector image, the coordinates of each pixel at the third time point are determined, and a second mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the second mask; the iteration operation includes mean blurring and binarization of the second mask; erosion and mean blurring are performed on the second mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0092] For example, the default pixel value of a pixel in the second mask is a first preset value. For any pixel among the pixels, after obtaining the coordinates of the pixel at the third time, the pixel value of the corresponding pixel in the second mask is set to the second preset value, indicating that the corresponding pixel in the second mask has a valid motion vector value.

[0093] The iterative operation can involve first applying mean blurring to the second mask, followed by binarization. Mean blurring makes the binarization process more explicit.

[0094] After repeated iterative operations, as well as erosion and mean blurring processes, the proximity of each pixel at the third moment to the corresponding pixel in the first or second video frame can be represented by the first mask. The weight value during fusion in step 404 can be set according to the pixel value of each pixel in the first mask.

[0095] The steps of the above implementation method can be as follows: Figure 5 As shown, Figure 5 The steps shown are merely examples of the above implementation method. The above implementation method can also be implemented through other steps, which are not limited here.

[0096] Step 501: Generate the second mask.

[0097] The size of the second mask can be the same as that of the first video frame F0 and the second video frame F1, and also the same as that of the first motion vector diagram Flow. 01 The dimensions are consistent. The pixel value of each pixel in the second mask can represent whether a pixel has moved to the coordinates of that pixel at the third time t. A first preset value (e.g., 0) indicates that no pixel has moved to the coordinates of that pixel, and a second preset value (e.g., 1) indicates that a pixel has moved to the coordinates of that pixel. The pixel value of each pixel in the second mask can be initialized to 0, indicating that no pixel has moved to the coordinates of that pixel. Then, for Flow... 01 For each pixel on the mask, if the pixel's coordinates are (x, y) and its motion vector value is (u, v), then the coordinates (xt, yt) of the position reached by the pixel at the third time t are calculated using the following formula. Then, the pixel value of the second mask at the position (xt, yt) is set to 1.

[0098]

[0099] Here, "round" means rounding to the nearest integer.

[0100] Step 502: Perform mean blurring on the second mask.

[0101] The parameters in the mean blur algorithm include window size, and the values ​​of these parameters are not limited.

[0102] Step 503: Perform binarization on the mean after blurring.

[0103] The parameters of the binarization (threshold) algorithm include the binarization threshold, the value of which is not limited.

[0104] Clearly, steps 502 and 503 are steps in the iterative operation.

[0105] Step 504: Determine whether the number of iterations is greater than or equal to the preset number.

[0106] If yes, proceed to step 505; otherwise, proceed to step 502.

[0107] Step 505: Etch the second mask after the iterative operation.

[0108] The erosion algorithm includes parameters such as window size, and the values ​​of these parameters are not limited.

[0109] Step 506: Perform mean blurring on the second mask after etching to obtain the first mask.

[0110] The window size in the parameters of the mean fuzzy algorithm can be set to be the same as the window size in the erosion algorithm in step 505.

[0111] The video frame interpolation method provided in this application can be directly run on GPU and CPU to achieve video frame interpolation. In the video frame interpolation method provided in this application, a first mask can be generated based on a first motion vector map and a second motion vector map, thereby providing a basis for generating the interpolated video frame.

[0112] In another possible implementation of step 402, the first mask corresponding to the third moment can be generated based on the second motion vector diagram, specifically as follows: Based on the coordinates and motion vector values ​​of each pixel in the second motion vector image, the coordinates of each pixel at the third time point are determined respectively, and a third mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the third mask; the iteration operation includes mean blurring and binarization of the third mask; erosion and mean blurring are performed on the third mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0113] The above implementation method for generating the first mask based on the second motion vector image can refer to the description of generating the first mask based on the first motion vector image. Furthermore, the above implementation method also has similar implementation steps to steps 501 to 506, which can also refer to the description of steps 501 to 506, and will not be repeated here.

[0114] In the video frame interpolation method provided in this application, the pixel values ​​in each of the first, second, third, and fourth motion vector maps can contain two components, representing the motion amplitude of the pixel in the horizontal and vertical directions, respectively. The first and second motion vector maps can be directly obtained from the first and second video frames using existing techniques such as optical flow and block matching.

[0115] One way to generate a third motion vector graphic is as follows: Based on the first video frame and the second video frame, a first motion vector diagram from the first time point to the second time point is generated; based on the coordinates and motion vector value of the fourth pixel in the first motion vector diagram, the coordinates of the fourth pixel at the third time point are determined, wherein the fourth pixel is any pixel in the first motion vector diagram; at least one pixel adjacent to the coordinates of the fourth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the fourth pixel from the first time point to the third time point; if the motion vector value of the fifth pixel in the third motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the fifth pixel.

[0116] For example, the first motion vector diagram (Flow) 01 In the image, the coordinates of the fourth pixel are (x, y), and the motion vector value of the fourth pixel is (u, v). Then, what is the coordinate of the fourth pixel at time t (x, y)? t , y t It can be calculated using the following formula: x t =x+u t; y t =y+v t; Where, x t The y-coordinate represents the horizontal coordinate of the fourth pixel at time t in the third moment. t Let represent the vertical coordinate of the fourth pixel at time t, and u represent the horizontal displacement of the fourth pixel from time 1 to time 3. This represents the vertical displacement of the fourth pixel from the first moment to the third moment. " indicates multiplication.

[0117] It should be noted that since the value of the motion vector may not be an integer, x t , y t It may not be an integer either; at least one pixel adjacent to the coordinates of the fourth pixel at the third time point can be (x... t , y t The pixels at integer positions around (x) t , y t If there are four adjacent pixels (top, bottom, left, and right), then the motion vector values ​​of these four pixels are all assigned (i.e., set) to (-u). t, -v t). It should be noted that this assignment is based on coordinates (x, y). t , y t )to (x t , y t The motion vector values ​​of pixels at integer positions around the first motion vector map are assigned once. When all pixels in the first motion vector map have been traversed, the third motion vector map (Flow) is then used. t0 In the third motion vector map, some pixels may have their motion vector values ​​set multiple times. If the motion vector value of the fifth pixel is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the fifth pixel. If the motion vector of the fifth pixel is set only once, the motion vector value set only once is determined as the motion vector value of the fifth pixel. The motion vector of the fifth pixel may not be set at all, in which case the fifth pixel is the pixel whose motion vector needs to be repaired later.

[0118] The above-described method for generating the third motion vector image is only one example. There are many other ways to implement it. For example, an approximate pixel with the coordinates of the fourth pixel at the third time can be obtained. The coordinates of the approximate pixel can be the rounded value of the coordinates of the fourth pixel at the third time. The motion vector value of the approximate pixel is set according to the motion displacement of the adjacent coordinates of the fourth pixel in the first motion vector image. For example, the negative number of the motion displacement of the adjacent coordinate to the left of the fourth pixel is used as the motion vector value of the approximate pixel.

[0119] It should be noted that, before step 403, a fourth mask corresponding to the third time step can be generated, and the third motion vector image can be repaired based on the fourth mask. One possible implementation is as follows: Based on the third motion vector image, a fourth mask corresponding to the third moment is generated; based on the fourth mask, pixels in the third motion vector image that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0120] For example, the pixel values ​​of those pixels in the fourth mask that have valid motion vector values ​​(which can be understood as non-zero motion vector values) can be set to the second preset value. Alternatively, the pixel values ​​of the coordinates of the pixels in the fourth mask that have motion vector values ​​can be set to the second or third preset value. The fourth mask can be set to a first preset value as a default value. For example, a first preset value of 0 indicates that the pixel has no valid motion vector value, while a second preset value of 1 indicates that the pixel has a valid motion vector value. Pixels in the fourth mask with the first preset value can be selected for repair, or pixels with the first preset value and their adjacent pixels can be selected for repair. This can be flexibly implemented according to the specific scenario and is not limited here.

[0121] One way to repair the third motion vector image based on the fourth mask is as follows: Select the sixth pixel in the fourth mask that meets the first repair condition. Starting from the sixth pixel, search in the fourth mask to obtain the first pixel set corresponding to the sixth pixel. Obtain the seventh pixel in the third motion vector image that corresponds to the sixth pixel, and the second pixel set that corresponds to the first pixel set. Determine the motion vector value of the seventh pixel based on the distance between the pixels in the second pixel set and the seventh pixel, and the motion vector value of the pixels in the second pixel set.

[0122] The first repair condition includes that the pixel does not have a valid motion vector value (which can be understood as a non-zero motion vector value), and the pixel in the first pixel set is the pixel closest to the sixth pixel in at least one direction and has a motion vector value.

[0123] For example, the fourth mask is FlowMask0. Traversing any pixel of FlowMask0, if the pixel value is 1, it indicates that the pixel is in the third motion vector map (FlowMask0). t0 The corresponding pixel in the third motion vector map already has a motion vector value, so this pixel does not need to be repaired; if the pixel value is 0, it means that this pixel is in the third motion vector map (Flow). t0 If there is no valid motion vector value for the corresponding pixel in the () field, then the pixel is the sixth pixel.

[0124] The first repair condition may also include whether the pixel needs to participate in the image fusion of the first reference frame and the second reference frame. Based on the first mask, it can be determined whether the pixel needs to participate in the image fusion of the first reference frame and the second reference frame. For example, if the weight value of the pixel participating in the fusion is 0 in the first mask, then it will not participate in the image fusion of the first reference frame and the second reference frame, and there is no need to repair the pixel.

[0125] Next, starting from the sixth pixel, search in the four directions of up, down, left, and right until a pixel with a corresponding pixel value of 1 in each of the four directions (which can be called a valid pixel) is found or the pixel reaches the boundary of the fourth mask. The resulting set of pixels is the first set of pixels.

[0126] The sixth pixel corresponds to the seventh pixel in the third motion vector diagram, and the first pixel set corresponds to the second pixel set in the third motion vector diagram. Therefore, the motion vector value of the seventh pixel can be calculated as follows: The distance between the seventh pixel and the effective pixels to the left of the second pixel set is nLeft pixels. The variable left=1 indicates that a valid pixel was found to the left. This valid pixel is in Flow. t0 The obtained motion vector value is (uLeft, vLeft). If no valid pixel is found when searching to the left, then we can set nLeft=1 and left=0. The distance between the seventh pixel and the valid pixel to the right in the second pixel set is nRight pixels. The variable right=1 indicates that a valid pixel was found when searching to the right. This valid pixel is in Flow. t0 The obtained motion vector value is (uRight, vRight). If no valid pixel is found when searching to the right, then nRight=1 and right=0 can be set. The distance between the seventh pixel and the valid pixel below in the second pixel set is nBottom pixels. The variable bottom=1 indicates that a valid pixel was found when searching downwards. This valid pixel is in Flow. t0 The obtained motion vector value is (uBottom, vBottom). If no valid pixel is found when searching downwards, then nBottom=1 and bottom=0 can be set. The distance between the seventh pixel and the valid pixel above the second pixel set is nTop pixels. The variable top=1 indicates that a valid pixel was found when searching upwards. This valid pixel is in Flow. t0 The obtained motion vector value is (uTop, vTop). If no valid pixels are found when searching upwards, then nTop=1, top=0 can be set.

[0127] Therefore, the sum of the motion vectors of the seventh pixel (uSum, vSum) is calculated using the following formula: ; ; The sum of coefficients for the seventh pixel, cSum, can be calculated using the following formula. If the calculated cSum value is 0, then cSum is set to a non-zero value (e.g., 1, 2, etc.): ; The motion vector value (u,v) of the seventh pixel is calculated using the following formula: ; ; Furthermore, the pixel value of FlowMask0 in the sixth pixel is assigned to 1, indicating that there is a valid motion vector value in the seventh pixel corresponding to the sixth pixel.

[0128] There are several ways to implement the repair of the third motion vector image based on the fourth mask. Another method is as follows: Select the eighth pixel in the fourth mask that meets the second repair condition, and obtain the ninth pixel in the third motion vector map that corresponds to the eighth pixel; determine the motion vector value of the ninth pixel based on the motion vector value of at least one pixel adjacent to the ninth pixel in the first direction.

[0129] The second repair condition includes the value of the eighth pixel indicating that the ninth pixel corresponding to the eighth pixel in the third motion vector diagram does not have a motion vector value, and at least one pixel adjacent to the ninth pixel in the first direction has a motion vector value.

[0130] For example, traversing any pixel of the fourth mask FlowMask0, if that pixel is in the third motion vector (FlowMask0)... t0 If a pixel in the first direction (e.g., adjacent pixels to the left and right or adjacent pixels to the top and bottom) does not have a motion vector value, then that pixel is the eighth pixel. Taking adjacent pixels to the left and right as an example, the average of the motion vector values ​​of the ninth pixel's adjacent pixels can be used as the ninth pixel's motion vector value, and the pixel value of the eighth pixel can be assigned a value of 1, indicating that the ninth pixel corresponding to the eighth pixel has a valid motion vector value. Using a fourth mask can reduce the workload of motion vector image repair, only repairing the areas that need repair.

[0131] The two methods described above for repairing the third motion vector image based on the fourth mask can be implemented individually or in combination, and no limitation is made here.

[0132] Accordingly, one way to generate the fourth motion vector image is as follows: Based on the first video frame and the second video frame, a second motion vector diagram from the second time point to the first time point is generated; based on the coordinates and motion vector value of the tenth pixel in the second motion vector diagram, the coordinates of the tenth pixel at the third time point are determined, where the tenth pixel is any pixel in the second motion vector diagram; at least one pixel adjacent to the coordinates of the tenth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the tenth pixel from the first time point to the third time point; if the motion vector value of the eleventh pixel in the fourth motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the eleventh pixel.

[0133] The specific implementation method for generating the fourth motion vector image can be referred to the method for generating the third motion vector image, and will not be described again here.

[0134] It should be noted that, before step 403, a fifth mask corresponding to the third time step can also be generated, and the fourth motion vector diagram can be repaired based on the fifth mask. One possible implementation is as follows: Based on the fourth motion vector map, a fifth mask corresponding to the third moment is generated; based on the fifth mask, pixels in the fourth motion vector map that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0135] The above implementation method is similar to the corresponding steps of the fourth mask. You can refer to the generation of the fourth mask and the repair of the third motion vector, which will not be repeated here.

[0136] One way to repair the third motion vector image based on the fourth mask is as follows: Select the twelfth pixel in the fifth mask that meets the third repair condition; starting from the twelfth pixel, search in the fifth mask to obtain the third pixel set corresponding to the twelfth pixel; obtain the thirteenth pixel in the fourth motion vector image that corresponds to the twelfth pixel, and the fourth pixel set that corresponds to the third pixel set; determine the motion vector value of the thirteenth pixel based on the distance between the pixels in the fourth pixel set and the thirteenth pixel, and the motion vector value of the pixels in the fourth pixel set.

[0137] The third repair condition includes that the pixel does not have a valid motion vector value, and the pixels in the third pixel set are the pixels closest to the twelfth pixel in at least one direction and have a motion vector value.

[0138] There are several ways to implement the repair of the third motion vector image based on the fourth mask. Another method is as follows: Select the fourteenth pixel in the fifth mask that meets the fourth repair condition, and obtain the fifteenth pixel in the fourth motion vector diagram that corresponds to the fourteenth pixel; determine the motion vector value of the fourteenth pixel based on the motion vector value of at least one pixel adjacent to the fourteenth pixel in the second direction.

[0139] The fourth repair condition includes the value of the fourteenth pixel indicating that the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image does not have a motion vector value, and at least one pixel adjacent to the fifteenth pixel in the second direction has a motion vector value.

[0140] The two methods described above for repairing the fourth motion vector image based on the fifth mask can be implemented individually or in combination. No specific method is required here. The specific implementation method can refer to the steps for repairing the third motion vector image using the fourth mask, which will not be repeated here.

[0141] In the video frame interpolation method provided in this application, the repair of the third motion vector map and the fourth motion vector map provides a basis for motion compensation of the video frame, and the motion compensation of the video frame can be the process described in step 403.

[0142] The execution process of step 403 can be as follows: For example, according to the third motion vector diagram Flow t0 Motion compensation is performed on the first video frame F0 to obtain the first reference frame Warp0 corresponding to the third time t; based on the fourth motion vector diagram Flow... t0 Motion compensation is performed on the second video frame F1 to obtain the second reference frame Warp1 corresponding to the third time t.

[0143] For any first pixel in the unprotected region of the first mask (MergeMask), let the coordinates of the first pixel be (x, y). Then, the first pixel corresponds to the second pixel P in the first reference frame. t0 The coordinates are (x, y), and the first pixel corresponds to the third pixel P in the second reference frame. t1 The coordinates are (x, y). If the second pixel is in Flow t0 If the corresponding motion vector value is (u0, v0), then P t0The pixel value is equal to F0(x+u0,y+v0), where F0(x+u0,y+v0) represents the pixel value of F0 at (x+u0,y+v0). If the third pixel is in Flow... t1 If the corresponding motion vector value is (u1, v1), then P t1 The pixel value is equal to F1(x+u1,y+v1), where F1(x+u1,y+v1) represents the pixel value of F1 at (x+u1,y+v1).

[0144] It should be noted that since motion vector values ​​are not necessarily integers, (x+u0,y+v0) or (x+u1,y+v1) may also not be integers. In this case, we can use the values ​​of the nearest integer coordinates of the neighboring pixels of (x+u0,y+v0) or (x+u1,y+v1) (nearest neighbor method), or we can use bilinear interpolation or bicubic interpolation methods to calculate using the pixels of the surrounding integer coordinates (such as the four coordinates of up, down, left, and right) of (x+u0,y+v0) or (x+u1,y+v1).

[0145] After motion compensation is applied to the video frames, image fusion can be performed on the first reference frame and the second reference frame. The specific process is as follows: In one possible implementation, in the unprotected region of the first mask, each pixel corresponds to a first weight value and a second weight value; the specific steps of step 404 can be as follows: Based on the position of the first pixel in the unprotected area, obtain the second pixel at the corresponding position in the first reference frame and the third pixel at the corresponding position in the second reference frame; perform a weighted operation on the pixel value of the second pixel based on the first weight value corresponding to the first pixel to obtain a first operation result; perform a weighted operation on the pixel value of the third pixel based on the second weight value corresponding to the first pixel to obtain a second operation result; and determine the sum of the first operation result and the second operation result as the pixel value of the pixel at the corresponding position in the third video frame.

[0146] Therefore, it can be seen that the first reference frame and the second reference frame can be fused according to the first mask, thereby compensating for the accuracy of the individual reference frame.

[0147] It should be noted that there are several ways to set the first and second weight values. For example, the first and second weight values ​​can be set in advance, or they can be set in the following ways: If the first mask is generated based on the first motion vector image, then the pixel value of the first pixel in the first mask is used to indicate the first weight value, and the second weight value is equal to 1 minus the first weight value; or, if the first mask is generated based on the second motion vector image, then the pixel value of the first pixel in the first mask is used to indicate the second weight value, and the first weight value is equal to 1 minus the second weight value.

[0148] Taking the first mask as an example generated based on the first motion vector image, it is clear that when the pixel value of the first pixel in the first mask is 1, it means that the second pixel corresponding to the first pixel in the first reference frame is valid, and the pixel value of the second pixel can be directly used as the pixel value of the corresponding pixel in the third video frame. If the pixel value of the first pixel is 0, it means that the second pixel corresponding to the first pixel in the first video frame is invalid, and the pixel value of the second pixel will inevitably have a larger error. The pixel value of the third pixel corresponding to the second video frame can be used as the pixel value of the corresponding pixel in the third video frame. When the pixel value of the first pixel is greater than 0 and less than 1, it is fused based on the pixel value of the second pixel, the weight value of the second pixel, the pixel value of the third pixel, and the weight value of the third pixel.

[0149] For example, if MergeMask is based on Flow 01 If the MergeMask is generated based on Flow, then the pixel value of each pixel represents the first weight value; conversely, if the MergeMask is based on Flow... 10 The generated value, MergeMask, represents the second weight value for each pixel, and the third video frame F is set. t The pixel value at a certain coordinate is P t The pixel value of Warp0 at the same coordinates is P0, the pixel value of Warp1 at the same coordinates is P1, and the pixel value of MergeMask at the same coordinates is a, a∈[0.0, 1.0].

[0150] If MergeMask is based on Flow 01 If generated, then P t It is obtained by calculation using the following formula: ; If MergeMask is generated based on Flow10, then P t It is obtained by calculation using the following formula: .

[0151] Thus obtain For pixel values ​​with multiple color components, the calculation of the color components can refer to the formula above.

[0152] In summary, when the video frame interpolation method in steps 401-404 is applied to video frame display, the flow of the video frame display method can be as follows: Figure 6 As shown.

[0153] Step 601: Obtain the first video frame at the first moment, the second video frame at the second moment, and the third moment.

[0154] Step 602: Determine whether the first video frame and the second video frame meet the interpolation conditions.

[0155] For example, the interpolation conditions can be that the size and scene of the first video frame and the second video frame are the same, but the content of the first video frame and the second video frame are different.

[0156] If yes, proceed to step 603; otherwise, proceed to step 609.

[0157] Step 603: Obtain the first motion vector image and the second motion vector image.

[0158] Step 604: Generate the first mask at the third time step.

[0159] Step 605: Generate a third motion vector map and a fourth mask, as well as a fourth motion vector map and a fifth mask, based on the first motion vector map and the second motion vector map.

[0160] Step 606: Repair the third motion vector image based on the first mask and the fourth mask, and update the third motion vector image; repair the fourth motion vector image based on the first mask and the fifth mask, and update the fourth motion vector image.

[0161] Step 607: Perform motion compensation on the first video frame based on the third motion vector diagram to obtain the first reference frame; perform motion compensation on the second video frame based on the fourth motion vector diagram to obtain the second reference frame.

[0162] Step 608: Based on the first mask, perform image fusion on the first reference frame and the second reference frame to obtain the third video frame and output it.

[0163] Step 609: Output the first video frame or the second video frame.

[0164] like Figure 7 As shown, this application provides a video frame interpolation apparatus, comprising: The acquisition module 701 is used to acquire the first video frame at the first moment and the second video frame at the second moment. The generation module 702 is used to generate a first mask corresponding to a third time step based on a first motion vector map or a second motion vector map. The first mask includes a non-protected area. The first motion vector map is a motion vector map from the first time step to the second time step, and the second motion vector map is a motion vector map from the second time step to the first time step. The third time step is located between the first time step and the second time step. Processing module 703 is configured to perform motion compensation on the first video frame based on the third motion vector map to obtain a first reference frame at the third time point, and to perform motion compensation on the second video frame based on the fourth motion vector map to obtain a second reference frame at the third time point, wherein the third motion vector map is the motion vector map from the third time point to the first time point, and the fourth motion vector map is the motion vector map from the third time point to the second time point; and The third video frame at the third moment is obtained by performing image fusion on the first reference frame and the second reference frame based on the unprotected area in the first mask.

[0165] In one possible design, in the unprotected region of the first mask, each pixel corresponds to a first weight value and a second weight value; the processing module 703 is specifically used for: Based on the position of the first pixel in the unprotected area, obtain the second pixel at the corresponding position in the first reference frame and the third pixel at the corresponding position in the second reference frame; the first pixel is any pixel in the unprotected area; perform a weighted operation on the pixel value of the second pixel according to the first weight value corresponding to the first pixel to obtain a first operation result; perform a weighted operation on the pixel value of the third pixel according to the second weight value corresponding to the first pixel to obtain a second operation result; and determine the pixel value of the pixel at the corresponding position in the third video frame by summing the first operation result and the second operation result.

[0166] In one possible design, if the first mask is generated based on the first motion vector image, the pixel value of the first pixel in the first mask is used to indicate the first weight value, and the second weight value is equal to 1 minus the first weight value; or if the first mask is generated based on the second motion vector image, the pixel value of the first pixel in the first mask is used to indicate the second weight value, and the first weight value is equal to 1 minus the second weight value.

[0167] In one possible design, the generation module 702 is specifically used for: Based on the coordinates and motion vector values ​​of each pixel in the first motion vector image, the coordinates of each pixel at the third time point are determined, and a second mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the second mask; the iteration operation includes mean blurring and binarization of the second mask; erosion and mean blurring are performed on the second mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0168] In one possible design, the generation module 702 is specifically used for: Based on the coordinates and motion vector values ​​of each pixel in the second motion vector image, the coordinates of each pixel at the third time point are determined respectively, and a third mask corresponding to the third time point is generated based on the coordinates of each pixel at the third time point; at least one iteration operation is performed on the third mask; the iteration operation includes mean blurring and binarization of the third mask; erosion and mean blurring are performed on the third mask after the iteration operation to obtain the first mask corresponding to the third time point.

[0169] In one possible design, the generation module 702 is further configured to: Based on the first video frame and the second video frame, a first motion vector diagram from the first time point to the second time point is generated; based on the coordinates and motion vector value of the fourth pixel in the first motion vector diagram, the coordinates of the fourth pixel at the third time point are determined, wherein the fourth pixel is any pixel in the first motion vector diagram; at least one pixel adjacent to the coordinates of the fourth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the fourth pixel from the first time point to the third time point; if the motion vector value of the fifth pixel in the third motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the fifth pixel.

[0170] In one possible design, the generation module 702 is further configured to: Based on the third motion vector image, a fourth mask corresponding to the third moment is generated; based on the fourth mask, pixels in the third motion vector image that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0171] In one possible design, the generation module 702 is specifically used for: A sixth pixel in the fourth mask that meets the first repair condition is selected, the first repair condition including that the pixel does not have a valid motion vector value; starting from the sixth pixel, a first pixel set corresponding to the sixth pixel is searched in the fourth mask, the pixels in the first pixel set are the pixels closest to the sixth pixel in at least one direction and have motion vector values; a seventh pixel corresponding to the sixth pixel in the third motion vector map and a second pixel set corresponding to the first pixel set are obtained; the motion vector value of the seventh pixel is determined according to the distance between the pixels in the second pixel set and the seventh pixel, and the motion vector value of the pixels in the second pixel set.

[0172] In one possible design, the generation module 702 is specifically used for: Select an eighth pixel in the fourth mask that meets the second repair condition, the second repair condition including that the value of the eighth pixel indicates that the ninth pixel corresponding to the eighth pixel in the third motion vector image does not have a motion vector value, and that at least one pixel adjacent to the ninth pixel in the first direction has a motion vector value; obtain the ninth pixel corresponding to the eighth pixel in the third motion vector image; determine the motion vector value of the ninth pixel based on the motion vector value of at least one pixel adjacent to the ninth pixel in the first direction.

[0173] In one possible design, the generation module 702 is further configured to: Based on the first video frame and the second video frame, a second motion vector diagram from the second time point to the first time point is generated; based on the coordinates and motion vector value of the tenth pixel in the second motion vector diagram, the coordinates of the tenth pixel at the third time point are determined, where the tenth pixel is any pixel in the second motion vector diagram; at least one pixel adjacent to the coordinates of the tenth pixel at the third time point is obtained, and the motion vector value of the at least one pixel is set according to the motion displacement of the tenth pixel from the first time point to the third time point; if the motion vector value of the eleventh pixel in the fourth motion vector diagram is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the eleventh pixel.

[0174] In one possible design, the generation module 702 is further configured to: Based on the fourth motion vector map, a fifth mask corresponding to the third moment is generated; based on the fifth mask, pixels in the fourth motion vector map that meet the repair conditions are selected; based on the motion vector value of at least one pixel associated with the pixel that meets the repair conditions, the motion vector value of the pixel that meets the repair conditions is updated.

[0175] In one possible design, the generation module 702 is specifically used for: Select the twelfth pixel in the fifth mask that meets the third repair condition, the third repair condition including that the pixel does not have a valid motion vector value; starting from the twelfth pixel, search in the fifth mask to obtain a third pixel set corresponding to the twelfth pixel, the pixels in the third pixel set being the pixels closest to the twelfth pixel in at least one direction and having a motion vector value; obtain the thirteenth pixel corresponding to the twelfth pixel in the fourth motion vector map, and the fourth pixel set corresponding to the third pixel set; determine the motion vector value of the thirteenth pixel based on the distance between the pixels in the fourth pixel set and the thirteenth pixel, and the motion vector value of the pixels in the fourth pixel set.

[0176] In one possible design, the generation module 702 is specifically used for: Select the fourteenth pixel in the fifth mask that meets the fourth repair condition. The fourth repair condition includes that the value of the fourteenth pixel indicates that the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image does not have a motion vector value, and that at least one pixel adjacent to the fifteenth pixel in the second direction has a motion vector value; obtain the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image; determine the motion vector value of the fourteenth pixel based on the motion vector value of at least one pixel adjacent to the fourteenth pixel in the second direction.

[0177] This application also provides an electronic device that can have the following features: Figure 8 The structure shown indicates that the electronic device can be a computer device, or a chip or chip system that enables the computer device to implement the above methods.

[0178] like Figure 8The illustrated electronic device may include at least one processor 801, which is coupled to a memory to read and execute instructions in the memory to implement the video frame interpolation step provided in the embodiments of this application. Optionally, the electronic device may also include a communication interface 802 for supporting the electronic device in receiving or sending signaling or data. The communication interface 802 in the electronic device can be used to interact with other electronic devices. The processor 801 can be used to implement the video frame interpolation step performed by the electronic device. Optionally, the electronic device may also include a memory 803 storing computer instructions. The memory 803 may be coupled to the processor 801 and / or the communication interface 802 to support the processor 801 in calling the computer instructions in the memory 803 to implement the video frame interpolation step; in addition, the memory 803 may also be used to store data involved in the method embodiments of this application, such as data and instructions necessary to support the interaction of the communication interface 802, and / or configuration information necessary for the electronic device to execute the method described in the embodiments of this application.

[0179] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are invoked and executed by a computer, the computer can perform the methods involved in any possible design of the above-described method embodiments. In this application, the computer-readable storage medium is not limited; for example, it can be RAM (random-access memory), ROM (read-only memory), etc.

[0180] This application also provides a chip that may include a processor and an interface circuit for performing the methods involved in any of the above-described method embodiments and any possible implementations of the method embodiments, wherein "coupling" refers to two components being directly or indirectly combined with each other, and such combination may be fixed or movable.

[0181] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, implementation can be entirely or partially in the form of computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0182] The steps of the methods or algorithms described in the embodiments of this application can be directly embedded in hardware, software units executed by a processor, or a combination of both. The software units can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a terminal device. Optionally, the processor and storage medium can also be housed in different components within the terminal device.

[0183] These computer instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0184] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the scope of the invention. Accordingly, this specification and drawings are merely exemplary descriptions of the invention as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A video frame interpolation method, characterized in that, include: Acquire the first video frame at the first moment and the second video frame at the second moment; Based on the first motion vector map or the second motion vector map, a first mask corresponding to the third moment is generated, the first mask including a non-protected area; the first motion vector map is the motion vector map from the first moment to the second moment, the second motion vector map is the motion vector map from the second moment to the first moment, and the third moment is located between the first moment and the second moment; Motion compensation is performed on the first video frame according to the third motion vector to obtain the first reference frame at the third moment, and motion compensation is performed on the second video frame according to the fourth motion vector to obtain the second reference frame at the third moment. The third motion vector is the motion vector from the third moment to the first moment, and the fourth motion vector is the motion vector from the third moment to the second moment. Based on the unprotected area in the first mask, the first reference frame and the second reference frame are image fused to obtain the third video frame at the third moment; The method further includes: Based on the first video frame and the second video frame, generate the first motion vector diagram from the first moment to the second moment; Based on the coordinates and motion vector value of the fourth pixel in the first motion vector image, the coordinates of the fourth pixel at the third moment are determined, wherein the fourth pixel is any pixel in the first motion vector image. Obtain at least one pixel adjacent to the coordinates of the fourth pixel at the third time, and set the motion vector value of the at least one pixel according to the motion displacement of the fourth pixel from the first time to the third time. If the motion vector value of the fifth pixel in the third motion vector image is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the fifth pixel, where the fifth pixel is any pixel in the third motion vector image.

2. The method as described in claim 1, characterized in that, In the unprotected area of ​​the first mask, each pixel corresponds to a first weight value and a second weight value; The step of image fusion of the first reference frame and the second reference frame based on the unprotected area in the first mask includes: Based on the position of the first pixel in the unprotected area, obtain the second pixel at the corresponding position in the first reference frame and the third pixel at the corresponding position in the second reference frame; the first pixel can be any pixel in the unprotected area; The pixel value of the second pixel is weighted according to the first weight value corresponding to the first pixel to obtain the first calculation result; The pixel value of the third pixel is weighted according to the second weight value corresponding to the first pixel to obtain the second calculation result; The sum of the first calculation result and the second calculation result is determined as the pixel value of the pixel at the corresponding position in the third video frame.

3. The method as described in claim 2, characterized in that: If the first mask is generated based on the first motion vector image, then the pixel value of the first pixel in the first mask is used to indicate the first weight value, and the second weight value is equal to 1 minus the first weight value; or If the first mask is generated based on the second motion vector image, then the pixel value of the first pixel in the first mask is used to indicate the second weight value, which is equal to 1 minus the second weight value.

4. The method according to any one of claims 1-3, characterized in that, The step of generating the first mask corresponding to the third moment based on the first motion vector image includes: Based on the coordinates and motion vector values ​​of each pixel in the first motion vector image, the coordinates of each pixel at the third time moment are determined respectively, and a second mask corresponding to the third time moment is generated based on the coordinates of each pixel at the third time moment. Perform at least one iteration operation on the second mask; the iteration operation includes mean blurring and binarization of the second mask; The second mask after the iterative operation is subjected to erosion and mean blurring to obtain the first mask corresponding to the third time step.

5. The method according to any one of claims 1-3, characterized in that, The step of generating the first mask corresponding to the third moment based on the second motion vector image includes: Based on the coordinates and motion vector values ​​of each pixel in the second motion vector image, the coordinates of each pixel at the third time moment are determined respectively, and a third mask corresponding to the third time moment is generated based on the coordinates of each pixel at the third time moment. Perform at least one iteration operation on the third mask; the iteration operation includes mean blurring and binarization of the third mask; The third mask after the iterative operation is subjected to erosion and mean blurring to obtain the first mask corresponding to the third time step.

6. The method according to any one of claims 1-3, characterized in that, Before performing motion compensation on the first video frame based on the third motion vector to obtain the first reference frame at the third moment, the method further includes: Based on the third motion vector diagram, generate the fourth mask corresponding to the third moment; Based on the fourth mask, select the pixels in the third motion vector image that meet the repair conditions; The motion vector value of the pixel that satisfies the repair condition is updated based on the motion vector value of at least one pixel associated with the pixel that satisfies the repair condition.

7. The method as described in claim 6, characterized in that: The step of selecting pixels in the third motion vector image that meet the repair conditions based on the fourth mask includes: Select the sixth pixel in the fourth mask that meets the first repair condition, the first repair condition including that the pixel does not have a valid motion vector value; The step of updating the motion vector value of the pixel that meets the repair condition based on the motion vector value of at least one pixel associated with the pixel that meets the repair condition includes: Starting from the sixth pixel, a first set of pixels corresponding to the sixth pixel is searched in the fourth mask. The pixels in the first set of pixels are the pixels closest to the sixth pixel in at least one direction and have motion vector values. Obtain the seventh pixel corresponding to the sixth pixel in the third motion vector image, and the second pixel set corresponding to the first pixel set; The motion vector value of the seventh pixel is determined based on the distance between the pixels in the second pixel set and the seventh pixel, as well as the motion vector value of the pixels in the second pixel set.

8. The method as described in claim 6, characterized in that: The step of selecting pixels in the third motion vector image that meet the repair conditions based on the fourth mask includes: The eighth pixel in the fourth mask that meets the second repair condition is selected. The second repair condition includes the value of the eighth pixel indicating that the ninth pixel corresponding to the eighth pixel in the third motion vector map does not have a motion vector value, and at least one pixel adjacent to the ninth pixel in the first direction has a motion vector value. The step of updating the motion vector value of the pixel that meets the repair condition based on the motion vector value of at least one pixel associated with the pixel that meets the repair condition includes: Obtain the ninth pixel in the third motion vector image that corresponds to the eighth pixel; The motion vector value of the ninth pixel is determined based on the motion vector value of at least one pixel adjacent to the ninth pixel in the first direction.

9. The method according to any one of claims 1-3, characterized in that, The method further includes: Based on the first video frame and the second video frame, generate the second motion vector diagram from the second time point to the first time point; Based on the coordinates and motion vector value of the tenth pixel in the second motion vector image, the coordinates of the tenth pixel at the third moment are determined, and the tenth pixel is any pixel in the second motion vector image. Obtain at least one pixel adjacent to the coordinates of the tenth pixel at the third time, and set the motion vector value of the at least one pixel according to the motion displacement of the tenth pixel from the first time to the third time; If the motion vector value of the eleventh pixel in the fourth motion vector image is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the eleventh pixel.

10. The method according to any one of claims 1-3, characterized in that, Before performing motion compensation on the second video frame based on the fourth motion vector diagram to obtain the second reference frame at the third moment, the method further includes: Based on the fourth motion vector diagram, generate the fifth mask corresponding to the third moment; Based on the fifth mask, select the pixels in the fourth motion vector image that meet the repair conditions; The motion vector value of the pixel that satisfies the repair condition is updated based on the motion vector value of at least one pixel associated with the pixel that satisfies the repair condition.

11. The method as described in claim 10, characterized in that: The step of selecting pixels in the fourth motion vector image that meet the repair conditions based on the fifth mask includes: The twelfth pixel in the fifth mask that meets the third repair condition is selected. The third repair condition includes that the pixel does not have a valid motion vector value. The step of updating the motion vector value of the pixel that meets the repair condition based on the motion vector value of at least one pixel associated with the pixel that meets the repair condition includes: Starting from the twelfth pixel, a third pixel set corresponding to the twelfth pixel is obtained in the fifth mask. The pixels in the third pixel set are the pixels closest to the twelfth pixel in at least one direction and have motion vector values. Obtain the thirteenth pixel in the fourth motion vector image that corresponds to the twelfth pixel, and the fourth pixel set that corresponds to the third pixel set; The motion vector value of the thirteenth pixel is determined based on the distance between the pixels in the fourth pixel set and the thirteenth pixel, as well as the motion vector values ​​of the pixels in the fourth pixel set.

12. The method as described in claim 10, characterized in that: The step of selecting pixels in the fourth motion vector image that meet the repair conditions based on the fifth mask includes: Select the fourteenth pixel in the fifth mask that meets the fourth repair condition. The fourth repair condition includes that the value of the fourteenth pixel indicates that the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image does not have a motion vector value, and that at least one pixel adjacent to the fifteenth pixel in the second direction has a motion vector value. The step of updating the motion vector value of the pixel that meets the repair condition based on the motion vector value of at least one pixel associated with the pixel that meets the repair condition includes: Obtain the fifteenth pixel in the fourth motion vector image that corresponds to the fourteenth pixel; The motion vector value of the fourteenth pixel is determined based on the motion vector value of at least one pixel adjacent to the fourteenth pixel in the second direction.

13. A video frame interpolation device, characterized in that, include: The acquisition module is used to acquire the first video frame at the first moment and the second video frame at the second moment. A generation module is used to generate a first mask corresponding to a third time step based on a first motion vector map or a second motion vector map. The first mask includes a non-protected area. The first motion vector map is a motion vector map from the first time step to the second time step, and the second motion vector map is a motion vector map from the second time step to the first time step. The third time step is located between the first time step and the second time step. The processing module is configured to perform motion compensation on the first video frame based on the third motion vector to obtain the first reference frame at the third moment, and to perform motion compensation on the second video frame based on the fourth motion vector to obtain the second reference frame at the third moment. The third motion vector is the motion vector from the third moment to the first moment, and the fourth motion vector is the motion vector from the third moment to the second moment. as well as Used to perform image fusion on the first reference frame and the second reference frame based on the unprotected area in the first mask, to obtain the third video frame at the third moment; The generation module is also used for: Based on the first video frame and the second video frame, generate the first motion vector diagram from the first moment to the second moment; Based on the coordinates and motion vector value of the fourth pixel in the first motion vector image, the coordinates of the fourth pixel at the third moment are determined, wherein the fourth pixel is any pixel in the first motion vector image. Obtain at least one pixel adjacent to the coordinates of the fourth pixel at the third time, and set the motion vector value of the at least one pixel according to the motion displacement of the fourth pixel from the first time to the third time. If the motion vector value of the fifth pixel in the third motion vector image is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the fifth pixel, where the fifth pixel is any pixel in the third motion vector image.

14. The apparatus as claimed in claim 13, characterized in that, In the unprotected region of the first mask, each pixel corresponds to a first weight value and a second weight value; the processing module is specifically used for: Based on the position of the first pixel in the unprotected area, obtain the second pixel at the corresponding position in the first reference frame and the third pixel at the corresponding position in the second reference frame; the first pixel can be any pixel in the unprotected area; The pixel value of the second pixel is weighted according to the first weight value corresponding to the first pixel to obtain the first calculation result; The pixel value of the third pixel is weighted according to the second weight value corresponding to the first pixel to obtain the second calculation result; The sum of the first calculation result and the second calculation result is determined as the pixel value of the pixel at the corresponding position in the third video frame.

15. The apparatus as claimed in claim 14, characterized in that: If the first mask is generated based on the first motion vector image, then the pixel value of the first pixel in the first mask is used to indicate the first weight value, and the second weight value is equal to 1 minus the first weight value; or If the first mask is generated based on the second motion vector image, then the pixel value of the first pixel in the first mask is used to indicate the second weight value, which is equal to 1 minus the second weight value.

16. The apparatus according to any one of claims 13-15, characterized in that, The generation module is specifically used for: Based on the coordinates and motion vector values ​​of each pixel in the first motion vector image, the coordinates of each pixel at the third time moment are determined respectively, and a second mask corresponding to the third time moment is generated based on the coordinates of each pixel at the third time moment. Perform at least one iteration operation on the second mask; the iteration operation includes mean blurring and binarization of the second mask; The second mask after the iterative operation is subjected to erosion and mean blurring to obtain the first mask corresponding to the third time step.

17. The apparatus according to any one of claims 13-15, characterized in that, The generation module is specifically used for: Based on the coordinates and motion vector values ​​of each pixel in the second motion vector image, the coordinates of each pixel at the third time moment are determined respectively, and a third mask corresponding to the third time moment is generated based on the coordinates of each pixel at the third time moment. Perform at least one iteration operation on the third mask; the iteration operation includes mean blurring and binarization of the third mask; The third mask after the iterative operation is subjected to erosion and mean blurring to obtain the first mask corresponding to the third time step.

18. The apparatus according to any one of claims 13-15, characterized in that, The generation module is also used for: Based on the third motion vector diagram, generate the fourth mask corresponding to the third moment; Based on the fourth mask, select the pixels in the third motion vector image that meet the repair conditions; The motion vector value of the pixel that satisfies the repair condition is updated based on the motion vector value of at least one pixel associated with the pixel that satisfies the repair condition.

19. The apparatus as claimed in claim 18, characterized in that, The generation module is specifically used for: Select the sixth pixel in the fourth mask that meets the first repair condition, the first repair condition including that the pixel does not have a valid motion vector value; Starting from the sixth pixel, a first set of pixels corresponding to the sixth pixel is searched in the fourth mask. The pixels in the first set of pixels are the pixels closest to the sixth pixel in at least one direction and have motion vector values. Obtain the seventh pixel corresponding to the sixth pixel in the third motion vector image, and the second pixel set corresponding to the first pixel set; The motion vector value of the seventh pixel is determined based on the distance between the pixels in the second pixel set and the seventh pixel, as well as the motion vector value of the pixels in the second pixel set.

20. The apparatus as claimed in claim 18, characterized in that, The generation module is specifically used for: The eighth pixel in the fourth mask that meets the second repair condition is selected. The second repair condition includes the value of the eighth pixel indicating that the ninth pixel corresponding to the eighth pixel in the third motion vector map does not have a motion vector value, and at least one pixel adjacent to the ninth pixel in the first direction has a motion vector value. Obtain the ninth pixel in the third motion vector image that corresponds to the eighth pixel; The motion vector value of the ninth pixel is determined based on the motion vector value of at least one pixel adjacent to the ninth pixel in the first direction.

21. The apparatus according to any one of claims 13-15, characterized in that, The generation module is also used for: Based on the first video frame and the second video frame, generate the second motion vector diagram from the second time point to the first time point; Based on the coordinates and motion vector value of the tenth pixel in the second motion vector image, the coordinates of the tenth pixel at the third moment are determined, and the tenth pixel is any pixel in the second motion vector image. Obtain at least one pixel adjacent to the coordinates of the tenth pixel at the third time, and set the motion vector value of the at least one pixel according to the motion displacement of the tenth pixel from the first time to the third time; If the motion vector value of the eleventh pixel in the fourth motion vector image is set multiple times, the average value of the multiple set motion vector values ​​is determined as the motion vector value of the eleventh pixel.

22. The apparatus according to any one of claims 13-15, characterized in that, The generation module is also used for: Based on the fourth motion vector diagram, generate the fifth mask corresponding to the third moment; Based on the fifth mask, select the pixels in the fourth motion vector image that meet the repair conditions; The motion vector value of the pixel that satisfies the repair condition is updated based on the motion vector value of at least one pixel associated with the pixel that satisfies the repair condition.

23. The apparatus as claimed in claim 22, characterized in that, The generation module is specifically used for: The twelfth pixel in the fifth mask that meets the third repair condition is selected. The third repair condition includes that the pixel does not have a valid motion vector value. Starting from the twelfth pixel, a third pixel set corresponding to the twelfth pixel is obtained in the fifth mask. The pixels in the third pixel set are the pixels closest to the twelfth pixel in at least one direction and have motion vector values. Obtain the thirteenth pixel in the fourth motion vector image that corresponds to the twelfth pixel, and the fourth pixel set that corresponds to the third pixel set; The motion vector value of the thirteenth pixel is determined based on the distance between the pixels in the fourth pixel set and the thirteenth pixel, as well as the motion vector values ​​of the pixels in the fourth pixel set.

24. The apparatus as claimed in claim 22, characterized in that, The generation module is specifically used for: Select the fourteenth pixel in the fifth mask that meets the fourth repair condition. The fourth repair condition includes that the value of the fourteenth pixel indicates that the fifteenth pixel corresponding to the fourteenth pixel in the fourth motion vector image does not have a motion vector value, and that at least one pixel adjacent to the fifteenth pixel in the second direction has a motion vector value. Obtain the fifteenth pixel in the fourth motion vector image that corresponds to the fourteenth pixel; The motion vector value of the fourteenth pixel is determined based on the motion vector value of at least one pixel adjacent to the fourteenth pixel in the second direction.

25. An electronic device, characterized in that, The electronic device includes: one or more processors; one or more memories; wherein the one or more memories store one or more computer instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 12.

Images