Frame insertion method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the field of image processing, and provides a frame insertion method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: dividing a screen into a plurality of partitions, and judging the type of each partition; for the static partition, taking pixel data belonging to the static partition in the first image or the second image as target pixel data belonging to the static partition; for the dynamic partition, generating pixel motion information of the dynamic partition according to pixel data belonging to the dynamic partition in the first image and the second image, moving the pixel data belonging to the dynamic partition in the first image and the second image according to the pixel motion information, and fusing the moved pixel data to generate target pixel data belonging to the dynamic partition; and obtaining a frame insertion image between the first image and the second image according to the target pixel data belonging to each partition. According to the frame insertion method disclosed by the embodiment of the invention, the frame insertion cost can be reduced, the frame insertion efficiency is improved, and a relatively high frame insertion effect can be ensured.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to a frame interpolation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Frame interpolation plays a crucial role in improving image smoothness and has evolved from a professional post-production tool into a key technology for enhancing visual experience in many consumer scenarios. Currently, the traditional frame interpolation techniques used in the industry are mainly based on optical flow algorithms. Optical flow algorithms do not rely on large amounts of data for training; instead, they describe the movement of each pixel between two consecutive frames using optical flow information. An optical flow field is obtained based on the fact that "an object in the previous frame has moved to a certain position in the next frame." The optical flow information includes forward optical flow information (from frame t to frame t+1) and backward optical flow information (from frame t+1 to frame t). Pixels in frame t are moved according to the forward optical flow information, and pixels in frame t+1 are moved according to the backward optical flow information. Finally, the two moved images are merged to obtain the intermediate frame.

[0003] However, traditional frame interpolation techniques require calculating the optical flow information of each pixel, which makes frame interpolation too costly and inefficient. Summary of the Invention

[0004] In view of this, this disclosure proposes a frame interpolation method, apparatus, electronic device, storage medium, and program product. The frame interpolation method of the embodiments of this disclosure can reduce frame interpolation costs, improve frame interpolation efficiency, and ensure high frame interpolation effect.

[0005] According to one aspect of this disclosure, a frame interpolation method is provided, the method comprising: dividing a screen into multiple partitions; determining the type of each partition based on attribute information of a first image and a second image; when any partition is of type static partition, using pixel data belonging to the static partition in the first image or the second image as target pixel data belonging to the static partition; when any partition is of type dynamic partition, generating pixel motion information of the dynamic partition based on the pixel data belonging to the dynamic partition in the first image and the second image, moving the pixel data belonging to the dynamic partition in the first image and the second image based on the pixel motion information, and fusing the moved pixel data to generate target pixel data belonging to the dynamic partition; and obtaining an interpolated frame image between the first image and the second image based on the target pixel data belonging to each partition.

[0006] In one possible implementation, the attribute information includes the identifier of the object corresponding to each pixel in the image, and the step of determining the type of each partition based on the attribute information of the first image and the second image includes: for the first partition among multiple partitions, determining whether the objects included in the first partition in the first image and the second image are consistent based on the attribute information; if the objects included in the first partition in the first image and the second image are consistent, determining the first partition as a static partition; if the objects included in the first partition in the first image and the second image are inconsistent, determining the first partition as a dynamic partition.

[0007] In one possible implementation, the method further includes: determining the identifier of each object in the three-dimensional scene through depth testing and template testing; when rendering the three-dimensional scene to obtain an arbitrary image, determining the identifier of the object corresponding to each pixel in the arbitrary image based on the identifier of each object and the position of each object in the arbitrary image.

[0008] In one possible implementation, the attribute information includes depth information corresponding to each pixel in the image. The step of determining the type of each partition based on the attribute information of the first image and the second image includes: for the first partition among multiple partitions, determining whether the motion amplitude of the first partition in the first image or the second image is greater than a first threshold based on the attribute information; if the motion amplitude of the first partition in the first image or the second image is greater than the first threshold, determining the first partition as a dynamic partition; if the motion amplitude of the first partition in the first image and the second image is less than or equal to the first threshold, determining the first partition as a static partition.

[0009] In one possible implementation, generating pixel motion information of the dynamic partition based on pixel data belonging to the dynamic partition in the first image and the second image, and moving the pixel data belonging to the dynamic partition in the first image and the second image based on the pixel motion information, includes: for a first object among a plurality of objects included in the dynamic partition in the first image and the second image, generating pixel motion information corresponding to the first object based on pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition, wherein the pixel motion information corresponding to the first object includes at least one of the motion vector of the first object and optical flow information corresponding to the first object; and moving the pixel data corresponding to the first object and belonging to the dynamic partition in the first image and the second image based on the pixel motion information corresponding to the first object.

[0010] In one possible implementation, if the first object is an opaque rigid body and is partially occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object and the optical flow information corresponding to the first object; if the first object is an opaque rigid body and is not occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object; if the first object is a flexible object or a transparent object, the pixel motion information corresponding to the first object includes the optical flow information corresponding to the first object.

[0011] In one possible implementation, if the first object is an opaque rigid body and is partially occluded, the step of moving the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to the pixel motion information corresponding to the first object includes: calculating a weighted average result of the motion vector of the first object and the optical flow information corresponding to the first object, and moving the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to the weighted average result.

[0012] According to another aspect of this disclosure, a frame interpolation apparatus is provided, the apparatus comprising: a partitioning module, configured to divide a screen into multiple partitions and determine the type of each partition based on attribute information of a first image and a second image; a first generation module, configured to, when any partition is of type static partition, use pixel data belonging to the static partition in the first image or the second image as target pixel data belonging to the static partition; a second generation module, configured to, when any partition is of type dynamic partition, generate pixel motion information of the dynamic partition based on the pixel data belonging to the dynamic partition in the first image and the second image, move the pixel data belonging to the dynamic partition in the first image and the second image based on the pixel motion information, and fuse the moved pixel data to generate target pixel data belonging to the dynamic partition; and a third generation module, configured to obtain an interpolated frame image between the first image and the second image based on the target pixel data belonging to each partition.

[0013] In one possible implementation, the attribute information includes the identifier of the object corresponding to each pixel in the image, and the step of determining the type of each partition based on the attribute information of the first image and the second image includes: for the first partition among multiple partitions, determining whether the objects included in the first partition in the first image and the second image are consistent based on the attribute information; if the objects included in the first partition in the first image and the second image are consistent, determining the first partition as a static partition; if the objects included in the first partition in the first image and the second image are inconsistent, determining the first partition as a dynamic partition.

[0014] In one possible implementation, the apparatus further includes: a first determining module, configured to determine the identifier of each object in the three-dimensional scene through depth testing and template testing; and a second determining module, configured to, when rendering the three-dimensional scene to obtain an arbitrary image, determine the identifier of the object corresponding to each pixel in the arbitrary image based on the identifier of each object and the position of each object in the arbitrary image.

[0015] In one possible implementation, the attribute information includes depth information corresponding to each pixel in the image. The step of determining the type of each partition based on the attribute information of the first image and the second image includes: for the first partition among multiple partitions, determining whether the motion amplitude of the first partition in the first image or the second image is greater than a first threshold based on the attribute information; if the motion amplitude of the first partition in the first image or the second image is greater than the first threshold, determining the first partition as a dynamic partition; if the motion amplitude of the first partition in the first image and the second image is less than or equal to the first threshold, determining the first partition as a static partition.

[0016] In one possible implementation, generating pixel motion information of the dynamic partition based on pixel data belonging to the dynamic partition in the first image and the second image, and moving the pixel data belonging to the dynamic partition in the first image and the second image based on the pixel motion information, includes: for a first object among a plurality of objects included in the dynamic partition in the first image and the second image, generating pixel motion information corresponding to the first object based on pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition, wherein the pixel motion information corresponding to the first object includes at least one of the motion vector of the first object and optical flow information corresponding to the first object; and moving the pixel data corresponding to the first object and belonging to the dynamic partition in the first image and the second image based on the pixel motion information corresponding to the first object.

[0017] In one possible implementation, if the first object is an opaque rigid body and is partially occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object and the optical flow information corresponding to the first object; if the first object is an opaque rigid body and is not occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object; if the first object is a flexible object or a transparent object, the pixel motion information corresponding to the first object includes the optical flow information corresponding to the first object.

[0018] In one possible implementation, if the first object is an opaque rigid body and is partially occluded, the step of moving the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to the pixel motion information corresponding to the first object includes: calculating a weighted average result of the motion vector of the first object and the optical flow information corresponding to the first object, and moving the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to the weighted average result.

[0019] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0020] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0022] According to the frame interpolation method of this disclosure, the screen is divided into multiple partitions, and the type of each partition is determined based on the attribute information of the first image and the second image. When any partition is a static partition, the pixel data belonging to the static partition in the first image or the second image is used as the target pixel data belonging to the static partition. When any partition is a dynamic partition, pixel motion information of the dynamic partition is generated based on the pixel data belonging to the dynamic partition in the first image and the second image. The pixel data belonging to the dynamic partition in the first image and the second image is moved according to the pixel motion information, and the moved pixel data is fused to generate the target pixel data belonging to the dynamic partition. Based on the target pixel data belonging to each partition, the interpolated frame image between the first image and the second image is obtained. This method ensures the frame interpolation effect of static and dynamic partitions, and eliminates the need to calculate the pixel motion information of the static partition or obtain the target pixel data belonging to the static partition based on the pixel motion information of the static partition, thereby reducing the frame interpolation cost and improving the frame interpolation efficiency.

[0023] This embodiment of the disclosure, by accurately identifying the type of partition, confines complex calculations to dynamic partitions that are subject to change, thus eliminating the calculation process for static partitions. This allows dynamic partitions to produce higher or slightly lower frame interpolation effects depending on the actual situation, while static partitions always guarantee high-quality frame interpolation.

[0024] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0026] Figure 1 An exemplary application scenario of the frame interpolation method according to embodiments of this disclosure is shown.

[0027] Figure 2 A schematic diagram illustrating the flow of a frame interpolation method according to an embodiment of the present disclosure is shown.

[0028] Figure 3 This diagram illustrates the division of a screen according to an embodiment of the present disclosure.

[0029] Figure 4 This diagram illustrates one method for determining the partition type according to an embodiment of the present disclosure.

[0030] Figure 5 This diagram illustrates a process for determining the identifier of an object corresponding to each pixel in an image according to an embodiment of the present disclosure.

[0031] Figure 6 A schematic diagram illustrating another method for determining the partition type according to an embodiment of the present disclosure is shown.

[0032] Figure 7 This diagram illustrates the process of obtaining target pixel data belonging to a dynamic partition according to an embodiment of the present disclosure.

[0033] Figure 8 A schematic diagram showing the structure of a frame interpolation apparatus according to an embodiment of the present disclosure is provided.

[0034] Figure 9 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0035] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0036] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0037] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0038] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0039] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0040] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0041] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0042] Traditional frame interpolation techniques have the following drawbacks:

[0043] 1. The core of traditional frame interpolation technology is the assumption that pixel intensity remains constant over a short period of time and that motion is linear and continuous. Once real-world scenarios violate these assumptions, frame interpolation will fail.

[0044] 2. Highly dependent on the accuracy of optical flow calculation. The optical flow information of a pixel is typically solved by minimizing an energy function. This function includes a data term and a smoothing term. The data term is used to determine whether the pixel colors match, and the smoothing term is used to determine whether the movements of adjacent pixels are similar. The process of minimizing this energy function is inherently complex and prone to getting trapped in local optima, resulting in incorrect optical flow information. If the optical flow information calculated in the first step is inaccurate, then subsequent pixel pushing and fusion based on incorrect optical flow information will inevitably fail. In other words, traditional frame interpolation techniques are highly dependent on the accuracy of optical flow information; any tiny error in optical flow calculation will be amplified into a noticeable flaw in the generated intermediate frame.

[0045] 3. Occlusion issues degrade frame interpolation quality. Suppose a pixel visible in frame t is obscured by a foreground object in frame t+1. Traditional frame interpolation techniques attempt to find a corresponding point for this pixel in frame t+1, but this will inevitably fail since the corresponding pixel doesn't exist. In this case, traditional frame interpolation techniques will either "fill" the pixel with incorrect background information or attempt to pull a non-existent pixel from frame t, resulting in unsightly ghosting or tearing.

[0046] 4. Difficulty in balancing computational complexity and frame interpolation quality. To achieve better frame interpolation results, very complex and time-consuming optimization algorithms are needed to calculate the optical flow information of each pixel. This makes frame interpolation too costly and inefficient, making it almost impossible for processors to use traditional frame interpolation techniques for real-time frame interpolation.

[0047] In view of this, this disclosure provides a frame interpolation method, apparatus, electronic device, storage medium, and program product. The frame interpolation method of this disclosure can reduce frame interpolation costs, improve frame interpolation efficiency, and ensure high frame interpolation results.

[0048] Furthermore, the frame interpolation method of this disclosure calculates global motion vectors and optical flow information for the object, and combines the motion vectors and optical flow information to generate image data, thereby avoiding tearing problems inside the object.

[0049] Furthermore, the frame interpolation method of this disclosure selects at least one of motion vectors and optical flow information to generate image data based on the type of object, transparency level, and occlusion situation, which can reduce ghosting generated during motion.

[0050] Figure 1 An exemplary application scenario of the frame interpolation method according to embodiments of this disclosure is shown.

[0051] The frame interpolation method is executed by a processor, which may be a graphics processor or a central processing unit, etc. The specific type of processor is not limited in the embodiments disclosed herein.

[0052] like Figure 1 As shown, it is assumed that the processor is a graphics processing unit (GPU). The GPU's rendering pipeline renders images (including a first image and a second image) and their attribute information based on the 3D scene. The image attribute information may include the identifier of the object corresponding to each pixel in the image, the depth information of each pixel, the type (rigid or flexible) and transparency of the objects in the image, which objects in the image are occluded, and whether they are completely or partially occluded. This embodiment of the present disclosure does not limit the specific content of the image attribute information.

[0053] The graphics processor executes a frame interpolation method to generate an interpolated image between the first image and the second image based on the first image, the second image, and the attribute information of the first image and the second image.

[0054] Figure 2 A schematic diagram illustrating the flow of a frame interpolation method according to an embodiment of the present disclosure is shown.

[0055] like Figure 2 As shown, in one possible implementation, the frame interpolation method includes the following steps S21 to S24.

[0056] Step S21: Divide the screen into multiple partitions and determine the type of each partition based on the attribute information of the first and second images.

[0057] Step S22: When any partition is of type static partition, the pixel data belonging to the static partition in the first image or the second image is taken as the target pixel data belonging to the static partition.

[0058] As can be seen from step S22, in the frame interpolation method provided in this embodiment, the pixel motion information of the static partition is not calculated, and the target pixel data belonging to the static partition is not generated based on the pixel motion information of the static partition. This reduces the frame interpolation cost required to calculate pixel motion information and generate target pixel data, and improves the efficiency of frame interpolation for the first image and the second image.

[0059] Step S23: When any partition is a dynamic partition, pixel motion information of the dynamic partition is generated based on the pixel data of the dynamic partition in the first image and the second image. The pixel data of the dynamic partition in the first image and the second image is moved according to the pixel motion information. The moved pixel data is then fused to generate target pixel data of the dynamic partition.

[0060] As can be seen from step S23, in the frame interpolation method provided in this embodiment, the pixel motion information of the dynamic partition is still calculated, and the target pixel data belonging to the dynamic partition is generated based on the pixel motion information of the dynamic partition, thus ensuring that the dynamic partition has a good frame interpolation effect.

[0061] Since static partitioning and dynamic partitioning are processed differently, steps S22 and S23 can be executed in parallel to process different types of partitions, further improving frame interpolation efficiency.

[0062] Step S24: Based on the target pixel data belonging to each partition, obtain the interpolated frame image between the first image and the second image.

[0063] For example, the frame interpolation method in this embodiment of the present disclosure divides the screen into multiple partitions, generates target pixel data belonging to each partition, and fuses the target pixel data belonging to each partition to obtain an interpolated image between the first image and the second image.

[0064] In step S21, the screen is first divided into multiple partitions. The screen division in this embodiment is a logical division, not a physical splitting of the screen.

[0065] Figure 3 This diagram illustrates the division of a screen according to an embodiment of the present disclosure.

[0066] like Figure 3 As shown, the screen size is 256×128. Assuming the screen is divided into 8 partitions (partition 1-partition 8), each partition could have a size of 64×64. In practical scenarios, the screen can be divided into more or fewer partitions depending on user needs. This disclosure does not limit the specific method of screen division.

[0067] It can determine the type of each partition and, based on the type of each partition, determine the generation method of the target pixel data belonging to each partition.

[0068] The type of each partition can be determined based on the attribute information of the first and second images. The dimensions of the first and second images are consistent with the screen dimensions; therefore, the pixel data belonging to the same partition in each image corresponds. The degree of difference between the pixel data belonging to the same partition in the first and second images can be determined based on their attribute information. If the difference is large, the partition type can be identified as a dynamic partition. If the difference is small, the partition type can be identified as a static partition.

[0069] It should be understood that the partition type determined based on the attribute information of the first and second images is only valid during the generation of the interpolated image between the first and second images, and is invalid when generating interpolated images between other two images.

[0070] It should be understood that only when the pixel data belonging to the same partition in the first and second images differs significantly (i.e., when the partition type is dynamic partitioning) will the pixel data belonging to that partition in the interpolated image obtained through pixel motion information differ significantly from the first and second images. When the pixel data belonging to the same partition in the first and second images differs little (i.e., when the partition type is static partitioning), the pixel data belonging to that partition in the interpolated image obtained through pixel motion information differs little from both the first and second images. Therefore, when any partition is of static partitioning type, step S22 can be executed, using the pixel data belonging to that static partition in the first or second image as the target pixel data belonging to that static partition. For example, if partition 1 is of static partitioning type, step S22 can be executed, using the pixel data belonging to partition 1 in the first or second image as the target pixel data belonging to partition 1. This method ensures the interpolation effect of static partitioning without requiring the calculation of pixel motion information in the static partition or the determination of target pixel data belonging to the static partition based on pixel motion information, thus reducing interpolation costs and improving interpolation efficiency.

[0071] When any partition is a dynamic partition, step S23 can be executed to generate pixel motion information for that dynamic partition based on the pixel data belonging to that dynamic partition in the first and second images. The pixel motion information indicates the movement pattern of pixels in the first and second images. In one example, the pixel motion information can be the optical flow information described above. In another example, for some pixels in the dynamic partition, motion vectors can be generated instead of optical flow information as pixel motion information, or the pixel motion information can be a weighted average of motion vectors and optical flow information. Introducing motion vectors can further improve the frame interpolation effect.

[0072] The pixel data belonging to the dynamic partition in the first and second images are moved according to the pixel motion information of the dynamic partition. The two moved pixel data are then merged to generate the target pixel data belonging to the dynamic partition.

[0073] For example, if partition 2 is a dynamic partition, pixel motion information for partition 2 can be generated based on the pixel data belonging to partition 2 in the first and second images. Then, the pixel data belonging to partition 2 in the first and second images can be moved based on this pixel motion information. Finally, the moved pixel data is fused to generate the target pixel data belonging to partition 2. This method ensures the frame interpolation effect of the dynamic partition.

[0074] Pixel data fusion can be achieved using existing technologies, which will not be elaborated upon here. Examples of generating pixel motion information and moving pixel data can be found in [link to documentation]. Figure 7 And related descriptions.

[0075] Finally, step S24 is executed to obtain an interpolated frame image between the first image and the second image based on the target pixel data belonging to each partition. This step can be implemented based on existing technology. For example, the target pixel data of each partition can be stitched together, and the stitched image can be used as the interpolated frame image. Alternatively, the edge pixel data of adjacent partitions can be smoothed based on the stitched image, and the smoothed image can be used as the interpolated frame image, ultimately obtaining a high-quality intermediate frame (interpolated frame image) that has a smooth transition in dynamic partitions and a clear and sharp image in static partitions. The embodiments of this disclosure do not limit the specific method of obtaining the interpolated frame image based on the target pixel data.

[0076] According to the frame interpolation method of this disclosure, the screen is divided into multiple partitions, and the type of each partition is determined based on the attribute information of the first image and the second image. When any partition is a static partition, the pixel data belonging to the static partition in the first image or the second image is used as the target pixel data belonging to the static partition. When any partition is a dynamic partition, pixel motion information of the dynamic partition is generated based on the pixel data belonging to the dynamic partition in the first image and the second image. The pixel data belonging to the dynamic partition in the first image and the second image is moved according to the pixel motion information, and the moved pixel data is fused to generate the target pixel data belonging to the dynamic partition. Based on the target pixel data belonging to each partition, the interpolated frame image between the first image and the second image is obtained. This method ensures the frame interpolation effect of static and dynamic partitions, and eliminates the need to calculate the pixel motion information of the static partition or obtain the target pixel data belonging to the static partition based on the pixel motion information of the static partition, thereby reducing the frame interpolation cost and improving the frame interpolation efficiency.

[0077] This embodiment of the disclosure, by accurately identifying the type of partition, confines complex calculations to dynamic partitions that are subject to change, thus eliminating the calculation process for static partitions. This allows dynamic partitions to produce higher or slightly lower frame interpolation effects depending on the actual situation, while static partitions always guarantee high-quality frame interpolation. Figure 4 This diagram illustrates one method for determining the partition type according to an embodiment of the present disclosure.

[0078] like Figure 4 As shown, in one possible implementation, the attribute information includes the identifier of the object corresponding to each pixel in the image. Determining the type of each partition based on the attribute information of the first and second images includes the following steps:

[0079] For the first partition among multiple partitions, determine whether the objects included in the first partition are the same in the first image and the second image based on the attribute information.

[0080] In Example 1, if the objects included in the first partition are the same in the first image and the second image, the first partition is determined to be a static partition.

[0081] In Example 2, if the objects included in the first partition are inconsistent in the first image and the second image, the first partition is determined to be a dynamic partition.

[0082] For example, attribute information can include the identifier of the object corresponding to each pixel in the image. Taking the first partition of multiple partitions as an example (the first partition can be any one of the multiple partitions), the attribute information can be used to determine whether the objects included in the first partition are the same in the first image and the second image. For example, the first union of the identifiers of the objects corresponding to each pixel of the first partition in the first image and the second union of the identifiers of the objects corresponding to each pixel of the first partition in the second image can be calculated. If the first union and the second union are different, it means that the objects included in the first partition are inconsistent in the first image and the second image; if the first union and the second union are the same, it means that the objects included in the first partition are consistent in the first image and the second image.

[0083] If the objects included in the first partition are the same in the first image and the second image, the first partition can be determined as a static partition.

[0084] When the objects included in the first partition are inconsistent in the first image and the second image, the first partition can be determined to be a dynamic partition.

[0085] The following describes an exemplary method for determining the identifier of the object corresponding to each pixel in an image. Figure 5 This diagram illustrates a process for determining the identifier of an object corresponding to each pixel in an image according to an embodiment of the present disclosure.

[0086] like Figure 5 As shown, in one possible implementation, the method further includes the following steps S51-S52:

[0087] Step S51: Determine the identifier of each object in the 3D scene through depth testing and template testing.

[0088] Step S52: When rendering an arbitrary image from a 3D scene, determine the object identifier corresponding to each pixel in the arbitrary image based on the identifier of each object and the position of each object in the arbitrary image.

[0089] For example, an add-on module can be added to the rendering pipeline, and a buffer can be allocated to this add-on module. This add-on module is used to perform depth testing and stencil testing to determine the identifier of each object in the 3D scene. Different objects have different identifiers. Both depth testing and stencil testing can be implemented using existing technologies, and their details will not be elaborated upon here.

[0090] When rendering an arbitrary image from a 3D scene, the additional module can determine the object identifier corresponding to each pixel in the rendered image based on the identifier of each object in the 3D scene and the position of each object in the rendered image, and store the object identifier corresponding to each pixel in a buffer.

[0091] The rendered image can be either the first image or the second image mentioned above.

[0092] Figure 6 A schematic diagram illustrating another method for determining the partition type according to an embodiment of the present disclosure is shown.

[0093] like Figure 6 As shown, in one possible implementation, the attribute information includes depth information corresponding to each pixel in the image. Determining the type of each partition based on the attribute information of the first and second images includes the following steps:

[0094] For the first partition among multiple partitions, determine whether the motion amplitude of the first partition in the first image or the second image is greater than a first threshold based on the attribute information.

[0095] In Example 1, if the motion amplitude of the first partition in the first image or the second image is greater than the first threshold, the first partition is determined to be a dynamic partition.

[0096] In Example 2, if the motion amplitude of the first partition in the first image and the second image is less than or equal to the first threshold, the first partition is determined to be a static partition.

[0097] For example, when rendering an image, depth information corresponding to each pixel in the image can also be obtained. This depth information can be three-dimensional motion data. The image's attribute information can include the depth information corresponding to each pixel in the image.

[0098] Taking the first partition among multiple partitions as an example (the first partition can be any one of the multiple partitions), the motion amplitude (motionVec) of the partition in the first image or the second image can be determined based on the depth information corresponding to the pixels in the first partition. For example, in the world coordinate system, the maximum value of the single-dimensional motion data of each pixel belonging to the first partition in the first image can be used as the motion amplitude of the first partition in the first image. This disclosure does not limit the specific method for determining the motion amplitude of a partition in an image based on depth information.

[0099] A first threshold can be preset according to the application scenario requirements. If the motion amplitude of the first partition in the first image or the second image is greater than the first threshold, the first partition can be determined as a dynamic partition. If the motion amplitude of the first partition in both the first image and the second image is less than or equal to the first threshold, the first partition can be determined as a static partition.

[0100] Those skilled in the art will understand that there are many other ways to calculate the motion amplitude. For example, the average value of the three-dimensional motion data of each pixel belonging to the first partition in the first image can be calculated first, and the average values ​​of each pixel can be compared. The largest average value can be used as the motion amplitude of the first partition in the first image. This disclosure does not limit the method of calculating the motion amplitude.

[0101] Those skilled in the art should understand that the two methods for determining the partition type described above can also be used simultaneously. If both methods determine that the partition type is a static partition, then step S22 is executed; if both methods determine that the partition type is a dynamic partition, then step S23 is executed. This disclosure does not limit the specific implementation method for determining the partition type.

[0102] The following example, using partition 2 as a dynamic partition, illustrates an exemplary method for obtaining target pixel data belonging to a dynamic partition. Figure 7 This diagram illustrates the process of obtaining target pixel data belonging to a dynamic partition according to an embodiment of the present disclosure.

[0103] like Figure 7 As shown, in one possible implementation, pixel motion information of the dynamic partition is generated based on pixel data belonging to the dynamic partition in the first image and the second image, and pixel data belonging to the dynamic partition in the first image and the second image is moved according to the pixel motion information (step S23), including the following steps:

[0104] For a first object among multiple objects included in a dynamic partition in a first image and a second image, pixel motion information corresponding to the first object is generated based on pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition. The pixel motion information corresponding to the first object includes at least one of the motion vector of the first object and the optical flow information corresponding to the first object.

[0105] Based on the pixel motion information corresponding to the first object, move the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition.

[0106] For example, a dynamic partition (such as partition 2) may include multiple objects in the first and second images. Taking the first object among multiple objects as an example, pixel motion information corresponding to the first object and belonging to the dynamic partition (such as partition 2) in the first and second images can be generated. This pixel motion information includes at least one of the motion vector of the first object and the optical flow information corresponding to the first object. Specifically, generating the motion vector of the first object, or generating the optical flow information corresponding to the first object, or generating both the motion vector and the optical flow information corresponding to the first object, depends on the type, transparency, whether the first object is occluded, and the degree of occlusion of the first object. The type, transparency, whether the first object is occluded, and the degree of occlusion of the first object can be determined during the depth testing and template testing processes described above and can be used as part of the attribute information of the first object.

[0107] An exemplary correlation between the type of the first object, its transparency, whether the first object is occluded, the degree of occlusion of the first object, and the content of the pixel motion information corresponding to the first object is as follows:

[0108] In Example 1, if the first object is an opaque rigid body and is partially occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object and the optical flow information corresponding to the first object.

[0109] In Example 2, if the first object is an opaque rigid body and is not occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object.

[0110] In Example 3, if the first object is a flexible object or a transparent object, the pixel motion information corresponding to the first object includes the optical flow information corresponding to the first object.

[0111] For example, if the first object is an opaque rigid body and is partially occluded, pixel motion information corresponding to the first object can be generated based on the pixel data in the first and second images that correspond to the first object and belong to a dynamic partition (such as partition 2). This includes optical flow information corresponding to the first object and the motion vector of the object.

[0112] The generation of optical flow information can be achieved based on existing optical flow algorithms, which will not be elaborated here. As described in the prior art, optical flow information may include forward optical flow information and backward optical flow information.

[0113] Motion vectors can be generated based on existing bounding box algorithms. The motion vectors can include forward and backward motion vectors. Assuming the first image is frame t and the second image is frame t+1, the first bounding box of the first object in the pixel data belonging to a dynamic partition (e.g., partition 2) in the first image can be calculated, and the first position information of the first object can be calculated based on the first bounding box. The second bounding box of the first object in the pixel data belonging to the dynamic partition (e.g., partition 2) in the second image can be calculated, and the second position information of the first object can be calculated based on the second bounding box. The forward motion vector can be the vector obtained by subtracting the first and second position information from the second position information. The backward motion vector can also be the vector obtained by subtracting the first and second position information from the second position information.

[0114] For example, the first position information can be coordinates (x1, y1, z1), the second position information can be coordinates (x2, y2, z2), the forward motion vector can be (x2-x1, y2-y1, z2-z1), and the backward motion vector can be (x1-x2, y1-y2, z1-z2).

[0115] It should be understood that the method of generating motion vectors is not limited to the examples described above, as long as the motion vectors can reflect the degree of motion of the first object in the dynamic partitions of the first and second images. This disclosure does not limit the specific calculation method of the motion vectors.

[0116] In one possible implementation, if the first object is an opaque rigid body and is partially occluded, the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition is moved according to the pixel motion information corresponding to the first object. This includes: calculating the weighted average result of the motion vector of the first object and the optical flow information corresponding to the first object, and moving the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to the weighted average result.

[0117] For example, if the first object is an opaque rigid body and is partially occluded, the pixel motion information corresponding to the first object may include the motion vector of the first object and the optical flow information corresponding to the first object. A weighted average of the motion vector of the first object and the optical flow information corresponding to the first object can be calculated. Based on the weighted average result, the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition (such as partition 2) is moved.

[0118] The weights of motion vectors and optical flow information can be preset, and this disclosure does not limit this.

[0119] The weighted average of motion vectors and optical flow information can be obtained by weighting the forward motion vector and the forward optical flow information to obtain the forward weighted average result, or by weighting the backward motion vector and the backward optical flow information to obtain the backward weighted average result.

[0120] For example, the forward motion vector is the global motion vector of the first object, and the forward optical flow information includes the optical flow of each pixel of the first object. Therefore, the forward motion vector can be transformed in reverse first to obtain the forward local motion vector of each pixel of the first object. Then, the forward local motion vector of each pixel is used to perform a weighted average with the optical flow of the same pixel in the forward optical flow information.

[0121] Similarly, the backward motion vector is the global motion vector of the first object, and the backward optical flow information includes the optical flow of each pixel of the first object. Therefore, the backward motion vector can be transformed in reverse first to obtain the backward local motion vector of each pixel of the first object. Then, the backward local motion vector of each pixel is used to perform a weighted average with the optical flow of the same pixel in the backward optical flow information.

[0122] Moving pixel data in the first and second images that correspond to the first object and belong to a dynamic partition (such as partition 2) based on the weighted average result can be done by moving the pixel data of the corresponding pixel in the first image based on the forward weighted average result of each pixel, and moving the pixel data of the corresponding pixel in the second image based on the backward weighted average result of each pixel.

[0123] Since the forward local motion vector and the backward local motion vector are obtained by inverse transformation based on the unified global motion vector, the entire vector field is smooth. After moving the pixel data by applying the weighted average result of motion vector and optical flow information, the motion amplitude of a certain pixel inside the first object will not differ significantly from the motion amplitude of other pixels, that is, there will be no tearing problem.

[0124] If the first object is an opaque rigid body and is not occluded, pixel motion information corresponding to the first object, including the motion vector of the first object, can be generated based on the pixel data in the first and second images that correspond to the first object and belong to a dynamic partition (such as partition 2). It is not necessary to generate optical flow information for the first object.

[0125] To move pixel data in the first and second images that correspond to the first object and belong to a dynamic partition (e.g., partition 2), based on motion vectors, one could perform a reverse transformation on the forward motion vector to obtain the forward local motion vector for each pixel of the first object, and then use the forward local motion vector of each pixel to move the pixel data of the corresponding pixel in the first image. Similarly, after performing a reverse transformation on the backward motion vector to obtain the backward local motion vector for each pixel of the first object, one could use the backward local motion vector of each pixel to move the pixel data of the corresponding pixel in the second image.

[0126] If the first object is a flexible or transparent object, pixel motion information corresponding to the first object, including optical flow information, can be generated based on the pixel data in the first and second images that correspond to the first object and belong to a dynamic partition (such as partition 2). It is not necessary to generate the motion vector of the first object.

[0127] The pixel data corresponding to the first object and belonging to a dynamic partition (such as partition 2) in the first image and the second image can be moved according to the optical flow information corresponding to the first object. This can be done by using the forward optical flow information of each pixel to move the pixel data of the corresponding pixel in the first image, and using the backward optical flow information of each pixel to move the pixel data of the corresponding pixel in the second image.

[0128] It should be understood that in practical applications, the generation method of target pixel data corresponding to the first object and belonging to the dynamic partition should not be limited to the above example. For example, if the first object is an opaque rigid body and is not occluded, the weighted average result can also be used as pixel motion information for the boundary of the first object in the dynamic partition. Based on the pixel motion information, the pixel data of the boundary of the first object can be moved to ensure the edge consistency and integrity of the object.

[0129] It should be understood that the pixel motion information corresponding to the first object generated for the dynamic partition (such as partition 2) is only used to generate pixel data that corresponds to the first object and belongs to the dynamic partition (such as partition 2), and will not be used when generating target pixel data that corresponds to the first object but belongs to other partitions (such as partition 1).

[0130] By combining global motion vectors and optical flow information to obtain target pixel data, and integrating the advantages of high-level semantic information and low-level visual cues, it can robustly handle the challenges in optical flow calculation such as occlusion, exposure, and rapid motion, and ultimately provide reliable motion estimation for high-quality frame interpolation.

[0131] This disclosure also proposes a frame interpolation device. Figure 8 A schematic diagram showing the structure of a frame interpolation apparatus according to an embodiment of the present disclosure is provided.

[0132] In one possible implementation, such as Figure 8As shown, the frame interpolation device includes: a partitioning module 61, a first generation module 62, a second generation module 63, and a third generation module 64.

[0133] The partitioning module 61 is used to divide the screen into multiple partitions and determine the type of each partition based on the attribute information of the first image and the second image.

[0134] The first generation module 62 is used to take the pixel data belonging to the static partition in the first image or the second image as the target pixel data belonging to the static partition when any partition is of the static partition type.

[0135] The second generation module 63 is used to generate pixel motion information of the dynamic partition based on the pixel data of the dynamic partition in the first image and the second image when any partition is a dynamic partition, move the pixel data of the dynamic partition in the first image and the second image based on the pixel motion information, and fuse the moved pixel data to generate target pixel data of the dynamic partition.

[0136] The third generation module 64 is used to obtain an interpolated frame image between the first image and the second image based on the target pixel data belonging to each partition.

[0137] In one possible implementation, the attribute information includes the identifier of the object corresponding to each pixel in the image. The type of each partition is determined based on the attribute information of the first image and the second image, including: for the first partition among multiple partitions, determining whether the objects included in the first partition in the first image and the second image are consistent based on the attribute information; if the objects included in the first partition in the first image and the second image are consistent, the first partition is determined to be a static partition; if the objects included in the first partition in the first image and the second image are inconsistent, the first partition is determined to be a dynamic partition.

[0138] In one possible implementation, the apparatus further includes: a first determining module, configured to determine the identifier of each object in the three-dimensional scene through depth testing and template testing; and a second determining module, configured to determine the identifier of each pixel in an arbitrary image based on the identifier of each object and the position of each object in the arbitrary image when rendering the three-dimensional scene to obtain an arbitrary image.

[0139] In one possible implementation, the attribute information includes depth information corresponding to each pixel in the image. The type of each partition is determined based on the attribute information of the first image and the second image, including: for the first partition among multiple partitions, determining whether the motion amplitude of the first partition in the first image or the second image is greater than a first threshold based on the attribute information; if the motion amplitude of the first partition in the first image or the second image is greater than the first threshold, the first partition is determined to be a dynamic partition; if the motion amplitude of the first partition in the first image and the second image is less than or equal to the first threshold, the first partition is determined to be a static partition.

[0140] In one possible implementation, generating pixel motion information for a dynamic partition based on pixel data belonging to a dynamic partition in the first and second images, and moving the pixel data belonging to the dynamic partition in the first and second images based on the pixel motion information, includes: for a first object among multiple objects included in the dynamic partition in the first and second images, generating pixel motion information corresponding to the first object based on pixel data in the first and second images that corresponds to the first object and belongs to the dynamic partition, wherein the pixel motion information corresponding to the first object includes at least one of the motion vector of the first object and optical flow information corresponding to the first object; and moving the pixel data corresponding to the first object and belonging to the dynamic partition in the first and second images based on the pixel motion information corresponding to the first object.

[0141] In one possible implementation, if the first object is an opaque rigid body and is partially occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object and the optical flow information corresponding to the first object; if the first object is an opaque rigid body and is not occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object; if the first object is a flexible object or a transparent object, the pixel motion information corresponding to the first object includes the optical flow information corresponding to the first object.

[0142] In one possible implementation, if the first object is an opaque rigid body and is partially occluded, the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition is moved according to the pixel motion information corresponding to the first object. This includes: calculating the weighted average result of the motion vector of the first object and the optical flow information corresponding to the first object, and moving the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to the weighted average result.

[0143] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0144] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0145] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0146] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0147] Figure 9 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 9 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0148] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0149] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0150] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0151] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0152] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0153] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0154] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0155] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0157] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A frame interpolation method, characterized in that, The method includes: The screen is divided into multiple partitions, and the type of each partition is determined based on the attribute information of the first and second images. When any partition is of type static partition, the pixel data belonging to the static partition in the first image or the second image shall be used as the target pixel data belonging to the static partition; When any partition is a dynamic partition, pixel motion information of the dynamic partition is generated based on the pixel data belonging to the dynamic partition in the first image and the second image. The pixel data belonging to the dynamic partition in the first image and the second image is moved based on the pixel motion information. The moved pixel data is then fused to generate target pixel data belonging to the dynamic partition. Based on the target pixel data belonging to each partition, an interpolated image is obtained between the first image and the second image.

2. The method according to claim 1, characterized in that, The attribute information includes the identifier of the object corresponding to each pixel in the image. The step of determining the type of each partition based on the attribute information of the first and second images includes: For the first partition among multiple partitions, determine whether the objects included in the first partition in the first image and the second image are the same based on the attribute information; If the objects included in the first partition are the same in the first image and the second image, the first partition is determined to be a static partition; If the objects included in the first partition are inconsistent in the first image and the second image, the first partition is determined to be a dynamic partition.

3. The method according to claim 2, characterized in that, The method further includes: The identifier of each object in the 3D scene is determined through depth testing and template testing; When rendering the three-dimensional scene to obtain an arbitrary image, the identifier of the object corresponding to each pixel in the arbitrary image is determined based on the identifier of each object and the position of each object in the arbitrary image.

4. The method according to claim 1, characterized in that, The attribute information includes the depth information corresponding to each pixel in the image. The step of determining the type of each partition based on the attribute information of the first and second images includes: For the first partition among multiple partitions, determine whether the motion amplitude of the first partition in the first image or the second image is greater than a first threshold based on the attribute information; If the motion amplitude of the first partition in the first image or the second image is greater than the first threshold, the first partition is determined to be a dynamic partition. If the motion amplitude of the first partition in the first image and the second image is less than or equal to the first threshold, the first partition is determined to be a static partition.

5. The method according to claim 1, characterized in that, The step of generating pixel motion information for the dynamic partition based on pixel data belonging to the dynamic partition in the first image and the second image, and moving pixel data belonging to the dynamic partition in the first image and the second image based on the pixel motion information, includes: For the first object among the multiple objects included in the first image and the second image in the dynamic partition, pixel motion information corresponding to the first object is generated based on the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition. The pixel motion information corresponding to the first object includes at least one of the motion vector of the first object and the optical flow information corresponding to the first object. Shift the pixel data in the first image and the second image that correspond to the first object and belong to the dynamic partition according to the pixel motion information corresponding to the first object.

6. The method according to claim 5, characterized in that, If the first object is an opaque rigid body and is partially occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object and the optical flow information corresponding to the first object. If the first object is an opaque rigid body and is not occluded, the pixel motion information corresponding to the first object includes the motion vector of the first object; If the first object is a flexible or transparent object, the pixel motion information corresponding to the first object includes the optical flow information corresponding to the first object.

7. The method according to claim 6, characterized in that, If the first object is an opaque rigid body and is partially obscured. The step of moving pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to pixel motion information corresponding to the first object includes: Calculate the weighted average of the motion vector of the first object and the optical flow information corresponding to the first object, and move the pixel data in the first image and the second image that corresponds to the first object and belongs to the dynamic partition according to the weighted average result.

8. A frame interpolation device, characterized in that, The device includes: The partitioning module is used to divide the screen into multiple partitions and determine the type of each partition based on the attribute information of the first and second images. The first generation module is used to take the pixel data belonging to the static partition in the first image or the second image as the target pixel data belonging to the static partition when any partition is of the static partition type. The second generation module is used to generate pixel motion information of the dynamic partition based on the pixel data of the first image and the second image belonging to the dynamic partition when any partition is of type dynamic partition, move the pixel data of the first image and the second image belonging to the dynamic partition based on the pixel motion information, and fuse the moved pixel data to generate target pixel data belonging to the dynamic partition. The third generation module is used to obtain the interpolated frame image between the first image and the second image based on the target pixel data belonging to each partition.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images