Image frame generation method and device and electronic equipment

Abstract

The invention discloses an image frame generation method and device and electronic equipment, and belongs to the technical field of image processing. The method comprises the following steps: generating a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame and a motion vector between the first image frame and the second image frame, the second image frame is the next image frame of the first image frame, and the first image frame and the second image frame are color images; generating a second intermediate image according to the optical flow from the first image frame to the second image frame; and fusing the first intermediate image and the second intermediate image according to the error graph to obtain a target frame insertion image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a method, device and electronic device for generating an image frame. Background Art

[0002] Frame interpolation refers to generating additional image frames between two adjacent image frames based on the original image frames without re-rendering the scene, thereby increasing the frame rate with less computing power and improving the user's visual experience.

[0003] However, existing interpolated images cannot take into account both the processing effect of large-scale motion and the rendering effect of semi-transparent areas. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device and electronic device for generating an image frame, which can solve the technical problem that the existing interpolated images cannot take into account both the processing effect of large-scale motion and the rendering effect of semi-transparent areas.

[0005] In a first aspect, an embodiment of the present application provides a method for generating an image frame, the method comprising:

[0006] Generate a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame, wherein the second image frame is an image frame next to the first image frame, and the first image frame and the second image frame are color images;

[0007] generating a second intermediate image according to an optical flow from the first image frame to the second image frame;

[0008] The first intermediate image and the second intermediate image are fused according to the error map to obtain a target interpolated image.

[0009] In a second aspect, an embodiment of the present application provides a device for generating an image frame, the device comprising:

[0010] A first generating module, configured to generate a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame, wherein the second image frame is an image frame next to the first image frame, and the first image frame and the second image frame are color images;

[0011] A second generating module, configured to generate a second intermediate image according to the optical flow from the first image frame to the second image frame;

[0012] The first fusion module is used to fuse the first intermediate image and the second intermediate image according to the error map to obtain a target interpolated image.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method provided in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method provided in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method provided in the first aspect.

[0017] In the image frame generation method, device and electronic device of the present application, the first intermediate image generated by combining the motion vector can effectively process large-scale motion; and the second intermediate image generated based on the optical flow can effectively process the semi-transparent area, and the error map is used to characterize the error of each pixel in the first intermediate image. Therefore, the first intermediate image and the second intermediate image are fused based on the error map to obtain a target interpolation image with a smaller error, which can take into account the processing effect of large-scale motion and the rendering effect of semi-transparent areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of a method for generating an image frame provided by an embodiment of the present application;

[0019] Figure 2 is a structural schematic diagram of an image frame generation device provided by another embodiment of the present application;

[0020] Figure 3 is a structural schematic diagram of an electronic device provided by another embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] In order to solve the above technical problems, the present application provides a method for generating an image frame. The following is a detailed description of the method for generating an image frame provided by the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0025] like Figure 1 As shown, Figure 1 : is a flow chart of a method for generating an image frame provided by an embodiment of the present application. The embodiment of the present application provides a method for generating an image frame, which may include:

[0026] S101, generating a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame, wherein the second image frame is an image frame next to the first image frame, and the first image frame and the second image frame are color images;

[0027] In this embodiment, the first image frame can be any image frame in the original video, and the second image frame is the image frame after the first image frame in the original video. The motion vector and depth image of each pixel can be calculated based on the first image frame and the second image frame, and the first intermediate image is generated based on the first image frame, the second image frame, the motion vector and the depth image, wherein the motion vector can be used to describe the displacement of each pixel in the image frame, and the depth image is used to describe the spatial depth of each pixel in the image frame. The first intermediate image is a new intermediate image frame generated between the first image frame and the second image frame. Inserting the intermediate image frame into the original video can improve the frame rate of the original video. The first image frame, the second image frame and the first intermediate image can all be RGB images.

[0028] Specifically, the system can perform spatial transformation and interpolation on each pixel by combining the first image frame, the second image frame and the motion vector information, and calculate the position and color of each pixel at the first time point, that is, obtain the first intermediate image.

[0029] In some embodiments, generating a first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame includes:

[0030] Generate a third motion vector and a fourth motion vector according to the first motion vector, the second motion vector and the first time point, wherein the first motion vector is a motion vector from the second time point to the third time point, the second motion vector is a motion vector from the third time point to the second time point, the first time point is a time point corresponding to the image frame to be inserted between the first image frame and the second image frame, the third motion vector is a motion vector from the first time point to the second time point, the fourth motion vector is a motion vector from the first time point to the third time point, the second time point is a time point corresponding to the first image frame, and the third time point is a time point corresponding to the second image frame;

[0031] generating an occlusion map according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image;

[0032] The first intermediate image is generated according to the first image frame, the second image frame, the occlusion map, the third motion vector, and the fourth motion vector.

[0033] In this embodiment, the first time point, the second time point and the third time point are used to describe the relative positions of the image frames in time, wherein the second time point is the time position of the first image frame, the third time point is the time position of the second image frame, the first time point is the time point between the second time point and the third time point, and the first time point is the time position of the image frame to be inserted between the first image frame and the second image frame. Then, the first motion vector is used to characterize the object motion from the second time point to the third time point; the second motion vector is used to characterize the object motion from the third time point to the second time point.

[0034] Then, when the first motion vector and the second motion vector are known, the third motion vector can be generated according to the first motion vector, and the fourth motion vector can be generated according to the second motion vector.

[0035] Specifically, in some embodiments, generating the third motion vector and the fourth motion vector according to the first motion vector, the second motion vector, and the first time point corresponding to the intermediate image frame includes:

[0036] forward projecting the first motion vector to the first time point to obtain the third motion vector;

[0037] The second motion vector is forward projected to the first time point to obtain the fourth motion vector.

[0038] In this embodiment, forward projection refers to forward projection of a known motion vector according to time to derive the motion of the object at other time points. Through forward projection, the motion vector of the corresponding time point can be calculated according to the existing motion vector, thereby obtaining the motion information of the object at different time points. Therefore, the first motion vector can be forward projected to the first time point to obtain the third motion vector. The second motion vector is forward projected to the second time point to obtain the fourth motion vector.

[0039] For example, the first time point can be set to the t position moment, t is greater than 0 and less than 1, the second time point is set to the 0 position moment, and the third time point is set to the 1 position moment, then the first motion vector is expressed as The second motion vector is expressed as At this time, the third motion vector and the fourth motion vector can be calculated based on the following forward projection formula:

[0040]

[0041] Among them, x is used to represent the x-th pixel point, is the third motion vector, Represents the fourth motion vector.

[0042] In this way, by forward projecting the first motion vector and the second motion vector to the first time point, the motion information corresponding to the first time point can be accurately calculated, thereby obtaining the third motion vector and the fourth motion vector. In this way, the motion state of the intermediate frame can be estimated and adjusted more accurately, effectively improving the image quality of the generated first intermediate image.

[0043] After determining the third motion vector and the fourth motion vector, in the process of generating the interpolated first intermediate image, the first intermediate image may be deformed due to the application of the motion vector and depth information. This image deformation may cause pixels in some areas of the first intermediate image to be occluded or expose blank areas, resulting in artifacts.

[0044] In order to solve the above problem, occlusion judgment can be performed on each pixel in the inserted first intermediate image. The occlusion judgment can identify these occluded or exposed areas, calculate and generate an occlusion map. The pixels of the occlusion map correspond to the pixels of the intermediate image one by one. Each pixel value of the occlusion map indicates whether the pixel at the corresponding position is occluded or leaked during the interpolation process. Therefore, the occlusion map can indicate which pixels are occluded or leaked during the deformation process, thereby helping to accurately process these areas in the subsequent pixel fusion process and avoid the generation of artifacts.

[0045] An occlusion map may be generated according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image.

[0046] Specifically, in some embodiments, generating an occlusion map according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image includes:

[0047] Reversely deforming the first depth image to the first time point according to the third motion vector to obtain a first deformed depth image;

[0048] Reversely deforming the second depth image to the first time point according to the fourth motion vector to obtain a second deformed depth image;

[0049] Forward projecting the first depth image and the second depth image to the first time point to obtain a third depth image;

[0050] The occlusion map is generated according to a comparison result of the third depth image, the first deformed depth image, and the second deformed depth image.

[0051] In this embodiment, the depth image is used to characterize the scene depth information of the corresponding image frame. Specifically, the pixels of the depth image correspond one-to-one to the pixels of the image frame, and each pixel value of the depth image is used to represent the depth information of the pixel at the corresponding position in the three-dimensional space. Depth information refers to the distance from the camera or viewing angle to the surface of the object corresponding to the pixel. Smaller depth values ​​usually indicate objects that are closer to the camera, and larger depth values ​​indicate objects that are farther away. Among them, the first depth image corresponds to the first image frame, and the second depth image corresponds to the second image frame.

[0052] In this embodiment, when the first depth image and the second depth image, and the third motion vector and the fourth motion vector are known, the coordinates of each pixel in the first depth image and the second depth image can be projected to the first time point according to the corresponding motion vector to obtain the third depth image.

[0053] For example, the first depth image may be set to d0, and the second depth image may be set to d1, and then the first depth image and the second depth image may be projected to the first time point according to the following projection formula:

[0054]

[0055] Among them, d t Used to represent the third depth image. The first motion vector is represented as The second motion vector is expressed as x is used to represent the x-th pixel point, and position t is the first time point.

[0056] In addition to forward projection, the first depth image may be reversely deformed to the first time point according to the third motion vector to obtain a first deformed depth image; and the second depth image may be reversely deformed to the first time point according to the fourth motion vector to obtain a second deformed depth image.

[0057] Among them, the reverse deformation is to calculate the spatial position relationship between each pixel in the first deformed depth image and the second deformed depth image of the target, and the original first depth image and the second depth image, and use the relationship to adjust the pixel position of the depth image, so that the pixels originally belonging to the original first depth image and the second depth image can be mapped to the position at the first time point, so as to obtain the first deformed depth image and the second deformed depth image.

[0058] Therefore, the first deformed depth image is the result of the reverse deformation calculation of the first depth image of the previous frame. Then the depth information of each pixel in the first deformed depth image in the first depth image is mapped to the spatial position at the first time point. Similarly, the second deformed depth image is the result of the reverse deformation calculation of the second depth image of the next frame. Then the depth information of each pixel in the second deformed depth image in the second depth image is mapped to the spatial position at the first time point.

[0059] For example, the first time point may be set to the t position moment, where t is less than 1 and greater than 0, the second time point may be set to the 0 position moment, and the third time point may be set to the 1 position moment. is the third motion vector, Represents the fourth motion vector, then the reverse deformation process can be calculated by the following reverse deformation formula:

[0060]

[0061] Wherein, d0 represents the first depth image, d1 represents the second depth image, represents the first deformed depth image, represents the second deformed depth image, is the third motion vector, Represents the fourth motion vector.

[0062] Since the first deformed depth image, the second deformed depth image, and the third depth image have different motions, their depth values ​​at the first time point are usually different. By comparing these depth images, it is possible to determine which areas are occluded areas and which areas need to be further fused, thereby generating an occlusion map.

[0063] Through the above method, the depth image calculation method of reverse deformation and forward projection can be combined, and the depth information of the second time point and the third time point can be mapped to the first time point in different ways to obtain multiple depth images of the first time point. Then, the multiple depth images are compared to more accurately calculate the occlusion map, thereby avoiding artifacts and unrealistic image overlap and improving the quality of the inserted image frame.

[0064] In some embodiments, the occlusion map includes a plurality of occlusion pixels, and generating the occlusion map according to a comparison result of the third depth image, the first deformed depth image, and the second deformed depth image includes:

[0065] For any first pixel in the third depth image, obtain a second pixel in the first deformed depth image whose position corresponds to the first pixel, and a third pixel in the second deformed depth image whose position corresponds to the first pixel;

[0066] When the absolute value of the first difference is greater than a first threshold value and the absolute value of the second difference is less than or equal to the first threshold value, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first value;

[0067] When the absolute value of the second difference is greater than the first threshold and the absolute value of the first difference is less than or equal to the first threshold, the pixel value of the occluded pixel point corresponding to the position of the first pixel point is determined as a second value;

[0068] When the absolute value of the first difference is less than or equal to the first threshold, and the absolute value of the second difference is less than or equal to the first threshold, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first pixel point as a third value, wherein the first value is less than the third value, and the third value is less than the second value;

[0069] The first difference is a difference between a first depth value of the first pixel and a second depth value of the second pixel, and the second difference is a difference between a first depth value of the first pixel and a third depth value of the third pixel.

[0070] In this embodiment, an occlusion map can be generated by comparing pixels in different depth images, which can be used to determine which pixels are occluded and provide accurate information for subsequent pixel fusion and processing of inserted target interpolation images.

[0071] Specifically, the third depth image is a depth image at the first time point obtained by forward projection, and the first deformed depth image and the second deformed depth image are both depth images at the first time point obtained by reverse deformation. Therefore, the pixel points corresponding to the positions in the third depth image, the first deformed depth image, and the second deformed depth image represent the depth values ​​of the position of the same object. Therefore, the depth information in the third depth image, the first deformed depth image, and the second deformed depth image can be compared to determine whether the position of each pixel point is blocked.

[0072] Specifically, the first deformed depth image and the second deformed depth image may be compared with the third depth image pixel by pixel to determine whether each pixel is blocked. Take any first pixel in the third depth image as an example.

[0073] For example, the second pixel corresponding to the first pixel in the first deformed depth image and the third pixel corresponding to the first pixel in the second deformed depth image can be obtained; then the absolute value of the first difference between the first depth value of the first pixel and the second depth value of the second pixel can be calculated, and the absolute value of the second difference between the first depth value of the first pixel and the third depth value of the third pixel can also be calculated. If the absolute value of the first difference is greater than the set first threshold, it can be considered that the pixel corresponding to the first pixel in the first intermediate image is blocked by the first image frame, and if the absolute value of the second difference is greater than the set first threshold, it can be considered that the pixel corresponding to the first pixel in the first intermediate image is blocked by the second image frame.

[0074] Then, if the absolute value of the first difference is greater than the first threshold, and the absolute value of the second difference is less than or equal to the first threshold, it can be considered that the pixel corresponding to the first pixel is only blocked by the first image frame but not blocked by the second image frame, and the pixel value of the blocked pixel corresponding to the first pixel is determined as the first value, which may be 0. If the absolute value of the second difference is greater than the first threshold, and the absolute value of the first difference is less than or equal to the first threshold, it can be considered that the pixel corresponding to the first pixel is only blocked by the second image frame but not blocked by the first image frame, and the pixel value of the blocked pixel corresponding to the first pixel is determined as the second value, which may be 1. If the absolute value of the first difference and the absolute value of the second difference are both less than or equal to the set first threshold, it can be considered that the pixel corresponding to the first pixel in the first intermediate image is not blocked, and the pixel value of the blocked pixel corresponding to the first pixel can be determined as the third value, which may be 0.5.

[0075] For example, for the x-th occluded pixel point O(x) in the occlusion map O, the calculation formula of O(x) can be:

[0076]

[0077] Among them, x represents the xth pixel, d t It is used to represent the third depth image, the first value is 0, the second value is 1, the third value is 0.5, TH is the first threshold, represents the first deformed depth image, represents the second deformed depth image.

[0078] In the above manner, the differences between images of different depths can be compared pixel by pixel to accurately identify which pixels are occluded, thereby generating an occlusion map.

[0079] In some embodiments, generating the first intermediate image according to the first image frame, the second image frame, the occlusion map, the third motion vector, and the fourth motion vector includes:

[0080] Reversely deforming the first image frame to the first time point according to the third motion vector to obtain a first deformed image frame;

[0081] Reversely deforming the second image frame to the first time point according to the fourth motion vector to obtain a second deformed image frame;

[0082] The first deformed image frame and the second deformed image frame are fused according to the occlusion map to obtain the first intermediate image.

[0083] In this embodiment, the first image frame can be reversely deformed to the first time point according to the third motion vector to obtain a first deformed image frame; the second image frame can be reversely deformed to the first time point according to the fourth motion vector to obtain a second deformed image frame.

[0084] Among them, the reverse deformation is achieved by calculating the spatial position relationship between each pixel in the first deformed image frame and the second deformed image frame of the target, and the original first image frame and the second image frame, and adjusting the pixel position of the image frame using the relationship, so that the pixels originally belonging to the original first image frame and the second image frame can be mapped to the position of the first time point, thereby obtaining the first deformed image frame and the second deformed image frame.

[0085] Therefore, the first deformed image frame is the result of the reverse deformation calculation of the first image frame of the previous frame. Then the color information of each pixel in the first deformed image frame in the first image frame is mapped to the spatial position at the first time point. Similarly, the second deformed image frame is the result of the reverse deformation calculation of the second image frame of the next frame. Then the color information of each pixel in the second deformed image frame in the second image frame is mapped to the spatial position at the first time point.

[0086] For example, the formula for the reverse deformation can be as follows:

[0087]

[0088] Wherein, I0 is used to represent the first image frame, and I1 is used to represent the second image frame. is the third motion vector, represents the fourth motion vector, is the first deformed image frame, is the second deformed image frame.

[0089] After the first deformed image frame and the second deformed image frame are obtained by reverse deformation, the first deformed image frame and the second deformed image frame may be fused pixel by pixel. During the fusion process, pixel values ​​at corresponding positions in the occlusion map may be used as fusion weights for fusion.

[0090] Specifically, the fusion formula can be:

[0091]

[0092] in, is the first intermediate image, is the first deformed image frame, is the second deformed image frame, ⊙ represents pixel-by-pixel multiplication, and O represents the occlusion map.

[0093] In this way, the color information of the previous frame and the next frame is accurately mapped to the first time point through reverse deformation, ensuring that the pixel value of the inserted first intermediate image reflects the motion change between the two original image frames. In addition, the first deformed image frame and the second deformed image frame can be fused pixel by pixel using the occlusion map as a weight, effectively avoiding unnatural transitions in the occlusion area, reducing artifacts and retaining correct visual information, thereby obtaining a more accurate and smooth interpolated image frame.

[0094] In this embodiment, the error map may be calculated according to the occlusion information of each pixel in the occlusion map in combination with the first deformed image frame and the second deformed image frame obtained by deformation.

[0095] Specifically, in some embodiments, the error map includes a plurality of error pixels, and generating the error map of the first intermediate image according to the first image frame, the second image frame, the first depth image corresponding to the first image frame, the second depth image corresponding to the second image frame, and the motion vector between the first image frame and the second image frame includes:

[0096] For any first occluded pixel point in the occlusion map, when the pixel value of the first occluded pixel point is a third value, determining a first error pixel value according to a first deformed image frame and a second deformed image frame, wherein the first error pixel value is an error pixel value of an error pixel point whose position corresponds to the first occluded pixel point, the first deformed image frame is an image frame obtained by reversely deforming the first image frame to the first time point according to the third motion vector, and the second deformed image frame is an image frame obtained by reversely deforming the second image frame to the first time point according to the fourth motion vector;

[0097] When the first occluded pixel point is the first value or the second value, the first error pixel value is determined as a fourth value.

[0098] In this embodiment, the error map is to evaluate the error degree of each pixel by comparing the pixel differences between the deformed image frames and combining the occlusion map, so that the subsequent pixel fusion can generate the interpolated frame more accurately. The error map includes multiple error pixels, so for each error pixel, the occlusion pixel in the corresponding occlusion map can be checked, and the corresponding error value can be determined based on different occlusion situations, thereby forming a complete error map.

[0099] Taking any first occluded pixel point in the occlusion image as an example, when the pixel value of the first occluded pixel point is the third value, it can be considered that the position of the first occluded pixel point is not occluded, so the first error pixel value can be determined according to the first deformed image frame and the second deformed image frame, wherein the first error pixel value is the error pixel value of the error pixel point corresponding to the position of the first occluded pixel point; when the first occluded pixel point is the first value or the second value, it can be considered that the position of the first occluded pixel point is occluded, then the error here is large, and the first error pixel value is determined as the fourth value. The fourth value can be 0.

[0100] When the pixel value of the first occluded pixel point is the third value, the calculation formula of the error pixel value can be:

[0101]

[0102] in, represents the i-th channel of the first image frame, represents the i-th channel of the second image frame. Each image frame includes three channels: R, G, and B. M Then represents the error map, and O represents the occlusion map.

[0103] In this way, the degree of occlusion of each pixel can be determined based on the occlusion map, and the error map can be calculated when the occlusion is small, so that the difference between the previous and next frames can be measured more accurately. This method helps reduce artifacts caused by occlusion or deformation, making the final interpolated frame more realistic and smooth.

[0104] S102: Generate a second intermediate image according to the optical flow from the first image frame to the second image frame.

[0105] In the embodiment of the present application, the optical flow represents the motion vector of each pixel in the image between two consecutive image frames, so the pixel position and color of the second intermediate image can be inferred by the optical flow. Specifically, the optical flow can help to infer the changes between the first image frame and the second image frame, and then generate the pixel information of the second intermediate image during the interpolation process. Therefore, using the optical flow information between the first image frame and the second image frame, the pixel motion between them can be calculated, and then the second intermediate image can be generated.

[0106] Specifically, in some embodiments, generating a second intermediate image according to the optical flows of the first image frame and the second image frame includes:

[0107] Determining a second optical flow and a third optical flow according to the first optical flow from the second time point to the third time point, wherein the second optical flow is the optical flow from the first time point to the second time point, the third optical flow is the optical flow from the first time point to the third time point, the second time point is the time point corresponding to the first image frame, the third time point is the time point corresponding to the second image frame, and the first time point is the time point corresponding to the second intermediate image;

[0108] Performing a reverse spatial transformation on the first image frame to the first time point according to the second optical flow to obtain a third deformed image frame;

[0109] Performing a reverse spatial transformation on the second image frame to the first time point according to the third optical flow to obtain a fourth deformed image frame;

[0110] The third deformed image frame and the fourth deformed image frame are fused to obtain the second intermediate image.

[0111] In this embodiment, first, the first optical flow from the second time point to the third time point, that is, the first optical flow between the first image frame and the second image frame, can be calculated. Specifically, a block matching search algorithm can be adopted, by dividing the image into small blocks and calculating the matching degree of each block between the first image frame and the second image frame, and using the sum of absolute difference (SAD) as a cost function to measure the similarity between the small blocks. Specifically, for each small block, the sum of the absolute values ​​of the pixel value differences between the corresponding areas in the second image frame and the first image frame can be calculated. The matching result with the minimum cost function is the optical flow estimation, which represents the displacement or movement of each pixel, thereby obtaining the first optical flow.

[0112] After obtaining the first optical flow, the motion trajectory of each pixel can be estimated by the optical flow between the first image frame and the second image frame, and then the optical flow information can be used to infer the optical flow of the interpolated frame at the first time point relative to the first image frame and the second image frame, that is, the second optical flow and the third optical flow. Specifically, the optical flow of the interpolated frame at the first time point is obtained by interpolating the motion information between the first image frame and the second image frame, reflecting the motion of each pixel in the interpolated frame. This ensures that the displacement information of the pixels is coherent when generating the interpolated frame, avoiding motion distortion or jumps.

[0113] Specifically, the second optical flow and the third optical flow can be calculated by the following formula:

[0114]

[0115] in, is the second optical flow, is the third optical flow, t is the first time point, is the first optical flow.

[0116] After obtaining the second optical flow and the third optical flow, the displacement information of each pixel in the first image frame and the second image frame can be reversely spatially transformed according to the second optical flow and the third optical flow from the interpolated frame at the first time point to the first image frame and the second image frame, that is, the pixel positions of the previous and next frames are mapped to the positions at the first time point, so as to obtain the third deformed image frame and the fourth deformed image frame respectively.

[0117] Then, the deformed third deformed image frame and the fourth deformed image frame are fused to obtain a final optical flow interpolation frame at the first time point, that is, a second intermediate image.

[0118] For example, the fusion may be performed according to the following fusion formula:

[0119]

[0120] in, is the second optical flow, is the third optical flow, t is the first time point, is the third deformed image frame, is the fourth deformed image frame, is the second intermediate image.

[0121] In this way, smooth and coherent in-between frames can be generated, avoiding motion artifacts or visual inconsistencies.

[0122] S103: Fusing the first intermediate image and the second intermediate image according to the error map to obtain a target interpolated image.

[0123] In an embodiment of the present application, after obtaining the first intermediate image and its error map obtained by interpolating between the first image frame and the second image frame based on depth information and motion vectors, and the second intermediate image interpolated based on optical flow information, the first intermediate image and the second intermediate image at the first time point can be fused, and a pixel-by-pixel fusion method can be used to generate the final interpolated frame at the first time point, that is, the target interpolated frame image.

[0124] Specifically, the error map provides weight information for each pixel in the first intermediate image, and the fusion ratio of the first intermediate image and the second intermediate image can be determined according to the value of the error map. When the error is small or the pixels are relatively consistent, the first intermediate image can be relied on more; while in areas with large errors or complex motion, the second intermediate image can be relied on more. Through this weighted fusion, while ensuring smooth visual effects, the appearance of artifacts and ghosting can be reduced, resulting in a more accurate and natural interpolated frame.

[0125] For example, the fusion of the first intermediate image and the second intermediate image may be completed by the following fusion formula, which is:

[0126]

[0127] Among them, t is the first time point, M is the error map, is the first intermediate image, is the second intermediate image, and ⊙ represents the pixel-by-pixel dot product.

[0128] In the present application, the first intermediate image generated by combining the motion vector can effectively process large-scale motion, and the second intermediate image generated based on the optical flow can effectively process the semi-transparent area, and the error map is used to characterize the error of each pixel in the first intermediate image. Therefore, the first intermediate image and the second intermediate image are fused based on the error map to obtain a target interpolation image with a smaller error, which can take into account both the processing effect of large-scale motion and the rendering effect of semi-transparent areas.

[0129] Figure 2 is a schematic diagram of the structure of an image frame generation device provided by another embodiment of the present application, such as Figure 2 As shown, the image frame generating device may include:

[0130] A first generating module 201 is used to generate a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame, wherein the second image frame is an image frame next to the first image frame, and the first image frame and the second image frame are color images;

[0131] A second generating module 202, configured to generate a second intermediate image according to the optical flow from the first image frame to the second image frame;

[0132] The first fusion module 203 is used to fuse the first intermediate image and the second intermediate image according to the error map to obtain a target interpolated frame image.

[0133] In the present application, the first intermediate image generated by combining the motion vector can effectively process large-scale motion, and the second intermediate image generated based on the optical flow can effectively process the semi-transparent area, and the error map is used to characterize the error of each pixel in the first intermediate image. Therefore, the first intermediate image and the second intermediate image are fused based on the error map to obtain a target interpolation image with a smaller error, which can take into account both the processing effect of large-scale motion and the rendering effect of semi-transparent areas.

[0134] In another optional example, the first generating module 201 includes:

[0135] A first generating unit is configured to generate a third motion vector and a fourth motion vector according to a first motion vector, a second motion vector and a first time point, wherein the first motion vector is a motion vector from the second time point to a third time point, the second motion vector is a motion vector from the third time point to the second time point, the first time point is a time point corresponding to an image frame to be inserted between the first image frame and the second image frame, the third motion vector is a motion vector from the first time point to the second time point, the fourth motion vector is a motion vector from the first time point to the third time point, the second time point is a time point corresponding to the first image frame, and the third time point is a time point corresponding to the second image frame;

[0136] A second generating unit, configured to generate an occlusion map according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image;

[0137] The third generating unit is used to generate the first intermediate image according to the first image frame, the second image frame, the occlusion map, the third motion vector and the fourth motion vector.

[0138] In another optional example, the second generating unit includes:

[0139] A first deformation subunit, configured to reversely deform the first depth image to the first time point according to the third motion vector to obtain a first deformed depth image;

[0140] a second deformation subunit, configured to reversely deform the second depth image to the first time point according to the fourth motion vector to obtain a second deformed depth image;

[0141] A first projection subunit, configured to forward project the first depth image and the second depth image to the first time point to obtain a third depth image;

[0142] The first generating subunit is configured to generate the occlusion map according to a comparison result of the third depth image, the first deformed depth image, and the second deformed depth image.

[0143] In another optional example, the first generating subunit is specifically used for:

[0144] For any first pixel in the third depth image, obtain a second pixel in the first deformed depth image whose position corresponds to the first pixel, and a third pixel in the second deformed depth image whose position corresponds to the first pixel;

[0145] When the absolute value of the first difference is greater than a first threshold value and the absolute value of the second difference is less than or equal to the first threshold value, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first value;

[0146] When the absolute value of the second difference is greater than the first threshold and the absolute value of the first difference is less than or equal to the first threshold, the pixel value of the occluded pixel point corresponding to the position of the first pixel point is determined as a second value;

[0147] When the absolute value of the first difference is less than or equal to the first threshold, and the absolute value of the second difference is less than or equal to the first threshold, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first pixel point as a third value, wherein the first value is less than the third value, and the third value is less than the second value;

[0148] The first difference is a difference between a first depth value of the first pixel and a second depth value of the second pixel, and the second difference is a difference between a first depth value of the first pixel and a third depth value of the third pixel.

[0149] In another optional example, the first generating module 201 further includes:

[0150] a first determining unit, for determining, for any first occluded pixel point in the occlusion map, a first error pixel value according to a first deformed image frame and a second deformed image frame, when a pixel value of the first occluded pixel point is a third value, wherein the first error pixel value is an error pixel value of an error pixel point corresponding to a position of the first occluded pixel point, the first deformed image frame is an image frame obtained by reversely deforming the first image frame to the first time point according to the third motion vector, and the second deformed image frame is an image frame obtained by reversely deforming the second image frame to the first time point according to the fourth motion vector;

[0151] The second determining unit is used to determine the first error pixel value as a fourth value when the first occluded pixel point is the first value or the second value.

[0152] The image frame generation device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0153] The image frame generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0154] The image frame generation device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0155] Alternatively, if Figure 3 As shown, an embodiment of the present application also provides an electronic device 100, including a processor 110, a memory 119, and a program or instruction stored in the memory 119 and executable on the processor 110. When the program or instruction is executed by the processor 110, each process of the above-mentioned image frame generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0156] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0157] Please refer to Figure 4 , Figure 4 The hardware structure diagram of an electronic device for implementing an embodiment of the present application is shown in FIG. The electronic device 100 includes, but is not limited to, a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.

[0158] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 120 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0159] The processor 120 is configured to generate a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame, wherein the second image frame is an image frame next to the first image frame, and the first image frame and the second image frame are color images;

[0160] A processor 120, configured to generate a second intermediate image according to an optical flow from the first image frame to the second image frame;

[0161] The processor 120 is configured to fuse the first intermediate image and the second intermediate image according to the error map to obtain a target interpolated image.

[0162] In the present application, the first intermediate image generated by combining the motion vector can effectively process large-scale motion, and the second intermediate image generated based on the optical flow can effectively process the semi-transparent area, and the error map is used to characterize the error of each pixel in the first intermediate image. Therefore, the first intermediate image and the second intermediate image are fused based on the error map to obtain a target interpolation image with a smaller error, which can take into account both the processing effect of large-scale motion and the rendering effect of semi-transparent areas.

[0163] In another optional example, the processor 120 is further configured to:

[0164] Generate a third motion vector and a fourth motion vector according to the first motion vector, the second motion vector and the first time point, wherein the first motion vector is a motion vector from the second time point to the third time point, the second motion vector is a motion vector from the third time point to the second time point, the first time point is a time point corresponding to the image frame to be inserted between the first image frame and the second image frame, the third motion vector is a motion vector from the first time point to the second time point, the fourth motion vector is a motion vector from the first time point to the third time point, the second time point is a time point corresponding to the first image frame, and the third time point is a time point corresponding to the second image frame;

[0165] generating an occlusion map according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image;

[0166] The first intermediate image is generated according to the first image frame, the second image frame, the occlusion map, the third motion vector, and the fourth motion vector.

[0167] In another optional example, the processor 120 is further configured to:

[0168] Reversely deforming the first depth image to the first time point according to the third motion vector to obtain a first deformed depth image;

[0169] Reversely deforming the second depth image to the first time point according to the fourth motion vector to obtain a second deformed depth image;

[0170] Forward projecting the first depth image and the second depth image to the first time point to obtain a third depth image;

[0171] The occlusion map is generated according to a comparison result of the third depth image, the first deformed depth image, and the second deformed depth image.

[0172] In another optional example, the processor 120 is further configured to:

[0173] For any first pixel in the third depth image, obtain a second pixel in the first deformed depth image whose position corresponds to the first pixel, and a third pixel in the second deformed depth image whose position corresponds to the first pixel;

[0174] When the absolute value of the first difference is greater than a first threshold value and the absolute value of the second difference is less than or equal to the first threshold value, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first value;

[0175] When the absolute value of the second difference is greater than the first threshold and the absolute value of the first difference is less than or equal to the first threshold, the pixel value of the occluded pixel point corresponding to the position of the first pixel point is determined as a second value;

[0176] When the absolute value of the first difference is less than or equal to the first threshold, and the absolute value of the second difference is less than or equal to the first threshold, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first pixel point as a third value, wherein the first value is less than the third value, and the third value is less than the second value;

[0177] The first difference is a difference between a first depth value of the first pixel and a second depth value of the second pixel, and the second difference is a difference between a first depth value of the first pixel and a third depth value of the third pixel.

[0178] In another optional example, the processor 120 is further configured to:

[0179] For any first occluded pixel point in the occlusion map, when the pixel value of the first occluded pixel point is a third value, determining a first error pixel value according to a first deformed image frame and a second deformed image frame, wherein the first error pixel value is an error pixel value of an error pixel point whose position corresponds to the first occluded pixel point, the first deformed image frame is an image frame obtained by reversely deforming the first image frame to the first time point according to the third motion vector, and the second deformed image frame is an image frame obtained by reversely deforming the second image frame to the first time point according to the fourth motion vector;

[0180] When the first occluded pixel point is the first value or the second value, the first error pixel value is determined as a fourth value.

[0181] It should be understood that in the embodiment of the present application, the input unit 124 may include a graphics processor (Graphics Processing Unit, GPU) 1241 and a microphone 1242, and the graphics processor 1241 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 126 may include a display panel 1261, and the display panel 1261 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 127 includes a touch panel 1271 and at least one of other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include two parts: a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0182] The memory 129 can be used to store software programs and various data. The memory 129 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 129 may include a volatile memory or a non-volatile memory, or the memory 129 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 129 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0183] The processor 120 may include one or more processing units; optionally, the processor 120 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 120.

[0184] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image frame generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0185] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0186] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image frame generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0187] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0188] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image frame generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0189] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0191] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for generating an image frame, characterized in that: include: Generate a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame, wherein the second image frame is an image frame next to the first image frame, and the first image frame and the second image frame are color images; generating a second intermediate image according to an optical flow from the first image frame to the second image frame; The first intermediate image and the second intermediate image are fused according to the error map to obtain a target interpolated image.

2. The method according to claim 1, characterized in that The generating a first intermediate image according to the first image frame, the second image frame, the first depth image corresponding to the first image frame, the second depth image corresponding to the second image frame, and the motion vector between the first image frame and the second image frame includes: Generate a third motion vector and a fourth motion vector according to the first motion vector, the second motion vector and the first time point, wherein the first motion vector is a motion vector from the second time point to the third time point, the second motion vector is a motion vector from the third time point to the second time point, the first time point is a time point corresponding to the image frame to be inserted between the first image frame and the second image frame, the third motion vector is a motion vector from the first time point to the second time point, the fourth motion vector is a motion vector from the first time point to the third time point, the second time point is a time point corresponding to the first image frame, and the third time point is a time point corresponding to the second image frame; generating an occlusion map according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image; The first intermediate image is generated according to the first image frame, the second image frame, the occlusion map, the third motion vector, and the fourth motion vector.

3. The method according to claim 2, characterized in that The generating an occlusion map according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image includes: Reversely deforming the first depth image to the first time point according to the third motion vector to obtain a first deformed depth image; Reversely deforming the second depth image to the first time point according to the fourth motion vector to obtain a second deformed depth image; Forward projecting the first depth image and the second depth image to the first time point to obtain a third depth image; The occlusion map is generated according to a comparison result of the third depth image, the first deformed depth image, and the second deformed depth image.

4. The method according to claim 3, characterized in that The occlusion map includes a plurality of occlusion pixels, and generating the occlusion map according to a comparison result of the third depth image, the first deformed depth image, and the second deformed depth image includes: For any first pixel in the third depth image, obtain a second pixel in the first deformed depth image whose position corresponds to the first pixel, and a third pixel in the second deformed depth image whose position corresponds to the first pixel; When the absolute value of the first difference is greater than a first threshold value and the absolute value of the second difference is less than or equal to the first threshold value, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first value; When the absolute value of the second difference is greater than the first threshold and the absolute value of the first difference is less than or equal to the first threshold, the pixel value of the occluded pixel point corresponding to the position of the first pixel point is determined as a second value; When the absolute value of the first difference is less than or equal to the first threshold, and the absolute value of the second difference is less than or equal to the first threshold, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first pixel point as a third value, wherein the first value is less than the third value, and the third value is less than the second value; The first difference is a difference between a first depth value of the first pixel and a second depth value of the second pixel, and the second difference is a difference between a first depth value of the first pixel and a third depth value of the third pixel.

5. The method according to claim 2, characterized in that: The error map includes a plurality of error pixels, and generating the error map of the first intermediate image according to the first image frame, the second image frame, the first depth image corresponding to the first image frame, the second depth image corresponding to the second image frame, and the motion vector between the first image frame and the second image frame includes: For any first occluded pixel point in the occlusion map, when the pixel value of the first occluded pixel point is a third value, determining a first error pixel value according to a first deformed image frame and a second deformed image frame, wherein the first error pixel value is an error pixel value of an error pixel point whose position corresponds to the first occluded pixel point, the first deformed image frame is an image frame obtained by reversely deforming the first image frame to the first time point according to the third motion vector, and the second deformed image frame is an image frame obtained by reversely deforming the second image frame to the first time point according to the fourth motion vector; When the first occluded pixel point is the first value or the second value, the first error pixel value is determined as a fourth value.

6. A device for generating an image frame, characterized in that: include: A first generating module, configured to generate a first intermediate image and an error map of the first intermediate image according to a first image frame, a second image frame, a first depth image corresponding to the first image frame, a second depth image corresponding to the second image frame, and a motion vector between the first image frame and the second image frame, wherein the second image frame is an image frame next to the first image frame, and the first image frame and the second image frame are color images; A second generating module, configured to generate a second intermediate image according to the optical flow from the first image frame to the second image frame; The first fusion module is used to fuse the first intermediate image and the second intermediate image according to the error map to obtain a target interpolated image.

7. The device according to claim 6, characterized in that The first generation module comprises: A first generating unit is configured to generate a third motion vector and a fourth motion vector according to a first motion vector, a second motion vector and a first time point, wherein the first motion vector is a motion vector from the second time point to a third time point, the second motion vector is a motion vector from the third time point to the second time point, the first time point is a time point corresponding to an image frame to be inserted between the first image frame and the second image frame, the third motion vector is a motion vector from the first time point to the second time point, the fourth motion vector is a motion vector from the first time point to the third time point, the second time point is a time point corresponding to the first image frame, and the third time point is a time point corresponding to the second image frame; A second generating unit, configured to generate an occlusion map according to the third motion vector, the fourth motion vector, the first depth image, and the second depth image; The third generating unit is used to generate the first intermediate image according to the first image frame, the second image frame, the occlusion map, the third motion vector and the fourth motion vector.

8. The device according to claim 7, characterized in that The second generating unit comprises: A first deformation subunit, configured to reversely deform the first depth image to the first time point according to the third motion vector to obtain a first deformed depth image; a second deformation subunit, configured to reversely deform the second depth image to the first time point according to the fourth motion vector to obtain a second deformed depth image; A first projection subunit, configured to forward project the first depth image and the second depth image to the first time point to obtain a third depth image; The first generating subunit is configured to generate the occlusion map according to a comparison result of the third depth image, the first deformed depth image, and the second deformed depth image.

9. The device according to claim 8, characterized in that The first generating subunit is specifically used for: For any first pixel in the third depth image, obtain a second pixel in the first deformed depth image whose position corresponds to the first pixel, and a third pixel in the second deformed depth image whose position corresponds to the first pixel; When the absolute value of the first difference is greater than a first threshold value and the absolute value of the second difference is less than or equal to the first threshold value, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first value; When the absolute value of the second difference is greater than the first threshold and the absolute value of the first difference is less than or equal to the first threshold, the pixel value of the occluded pixel point corresponding to the position of the first pixel point is determined as a second value; When the absolute value of the first difference is less than or equal to the first threshold, and the absolute value of the second difference is less than or equal to the first threshold, determining the pixel value of the occluded pixel point corresponding to the first pixel point at the same position as the first pixel point as a third value, wherein the first value is less than the third value, and the third value is less than the second value; The first difference is a difference between a first depth value of the first pixel and a second depth value of the second pixel, and the second difference is a difference between a first depth value of the first pixel and a third depth value of the third pixel.

10. The device according to claim 7, characterized in that The first generation module also includes: a first determining unit, for determining, for any first occluded pixel point in the occlusion map, a first error pixel value according to a first deformed image frame and a second deformed image frame, when a pixel value of the first occluded pixel point is a third value, wherein the first error pixel value is an error pixel value of an error pixel point corresponding to a position of the first occluded pixel point, the first deformed image frame is an image frame obtained by reversely deforming the first image frame to the first time point according to the third motion vector, and the second deformed image frame is an image frame obtained by reversely deforming the second image frame to the first time point according to the fourth motion vector; The second determining unit is used to determine the first error pixel value as a fourth value when the first occluded pixel point is the first value or the second value.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for generating an image frame as described in any one of claims 1 to 5 are implemented.

Images