Frame interpolation method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a frame interpolation method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining a first color frame and a first transparent texture corresponding to a first moment and a second color frame and a second transparent texture corresponding to a second moment; performing feature extraction on the first transparent texture and the second transparent texture to obtain a first group of transparent features corresponding to the first transparent texture and a second group of transparent features corresponding to the second transparent texture; determining a target selection factor according to the first group of transparent features and the second group of transparent features; and mixing the first color frame and the second color frame according to a target selection factor to generate an interpolation frame between the first moment and the second moment. In this way, the quality of the interpolation frame can be improved under the condition that the game fluency is guaranteed.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to an earlier U.S. provisional patent application filed on May 15, 2024, application number 63 / 648,126 entitled “METHODFOR MOBILE GAME FRAME INTERPOLATION WITH TRANSPARENT TEXTURE”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and more particularly to a frame interpolation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0004] Frame interpolation is a technique that takes two frames as input and outputs an interpolated frame. This technique is often used for temporal upsampling of video to increase the video's frame rate.

[0005] Taking video games as an example, video games sometimes cannot maintain a sustainable or consistent frame rate (FPS) output due to performance bottlenecks. Unstable FPS output can degrade the user's gaming experience; for instance, after playing for a period of time, frame drops can cause the interaction between the user and the game to become choppy. Although related technologies have proposed methods for interpolating video game frames, existing technologies are not comprehensive, resulting in poor interpolation quality, such as artifacts and flickering. Summary of the Invention

[0006] This application discloses a frame interpolation method, apparatus, electronic device, storage medium, and program product.

[0007] The technical solution of this application is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a frame interpolation method, the method comprising:

[0009] Obtain the first color frame and the first transparent texture corresponding to the first time step, and the second color frame and the second transparent texture corresponding to the second time step;

[0010] Feature extraction is performed on the first transparent texture and the second transparent texture to obtain the first set of transparent features corresponding to the first transparent texture and the second set of transparent features corresponding to the second transparent texture.

[0011] The target selection factor is determined based on the first set of transparency features and the second set of transparency features;

[0012] The first color frame and the second color frame are mixed according to the target selection factor to generate an interpolation frame between the first time and the second time.

[0013] In a second aspect, the embodiments of the present application provide a frame interpolation device, which comprises an acquisition unit, a feature extraction unit and an interpolation unit, wherein:

[0014] The acquisition unit is configured to acquire a first color frame and a first transparent texture corresponding to a first time, and a second color frame and a second transparent texture corresponding to a second time.

[0015] The feature extraction unit is configured to perform feature extraction on the first transparent texture and the second transparent texture to obtain a first group of transparent features corresponding to the first transparent texture and a second group of transparent features corresponding to the second transparent texture.

[0016] The interpolation unit is configured to determine a target selection factor according to the first group of transparent features and the second group of transparent features, and mix the first color frame and the second color frame according to the target selection factor to generate an interpolation frame between the first time and the second time.

[0017] In a third aspect, the embodiments of the present application provide an electronic device, which comprises a processor and a memory for storing a computer program capable of running on the processor; wherein the processor is configured to execute the method of the first aspect when running the computer program.

[0018] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0019] In a fifth aspect, the embodiments of the present application provide a computer program product, which comprises a computer program or instructions, and the computer program or instructions are executed by a processor to implement the method of the first aspect.

[0020] This application provides a frame interpolation method, apparatus, electronic device, storage medium, and program product. First, it acquires a first color frame and a first transparent texture corresponding to a first time step, and a second color frame and a second transparent texture corresponding to a second time step. Then, it extracts features from the first and second transparent textures to obtain a first set of transparent features corresponding to the first transparent texture and a second set of transparent features corresponding to the second transparent texture. Next, it determines a target selection factor based on the first and second transparent features, and mixes the first and second color frames according to the target selection factor to generate an interpolated frame between the first and second time steps. In this way, by using a separate transparent texture as additional input to calculate the target selection factor, the interpolation algorithm can determine whether the target pixel in the interpolated frame is a pixel from the first color frame, a pixel from the second color frame, or a mixture of pixels from both color frames. Compared with related technologies, this method has minimal overhead and low power consumption, extending battery life in electronic devices. Furthermore, it can be executed in real-time on most mobile devices, offering strong flexibility. Additionally, it reduces artifacts and flickering on transparent objects in the generated interpolated frame, thus improving the quality of the interpolated frame without sacrificing the smoothness of gameplay. Attached Figure Description

[0021] Figure 1 A flowchart illustrating the frame interpolation method provided in this application embodiment. Figure 1 ;

[0022] Figure 2 A flowchart illustrating the frame interpolation method provided in this application embodiment. Figure 2 ;

[0023] Figure 3 This is a schematic diagram of the interpolation algorithm architecture provided in the embodiments of this application;

[0024] Figure 4 A schematic diagram illustrating the workflow of the transparent test sub-component provided in an embodiment of this application;

[0025] Figure 5A This is a schematic diagram of the first color frame at position t0 provided in an embodiment of this application;

[0026] Figure 5B This is a schematic diagram of the second color frame at position t1 provided in an embodiment of this application;

[0027] Figure 6A This is a schematic diagram of an interpolated frame without transparency testing provided in an embodiment of this application;

[0028] Figure 6B This is a schematic diagram of the test area provided in an embodiment of this application;

[0029] Figure 6CA schematic diagram of an interpolated frame after transparent test repair is provided for an embodiment of the present application.

[0030] Figure 7 A schematic diagram of the component structure of a frame interpolation device is provided for an embodiment of the present application.

[0031] Figure 8 A schematic diagram of the component structure of an electronic device is provided for an embodiment of the present application.

[0032] Figure 9 A schematic diagram of the component structure of a chip is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings, which are only used for reference and are not intended to limit the embodiments of the present application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0036] It should also be noted that the terms "first", "second", "third" used in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first", "second", "third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In order to facilitate the understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below, and the following related technologies can be combined with the technical solutions of the embodiments of the present application in any way, which all belong to the protection scope of the embodiments of the present application.

[0038] Interpolation refers to a method of generating a frame in the time-space between two frames. Specifically, the frame interpolation technique takes two frames as input and outputs an interpolated frame. This technique is often used for temporal upsampling of video to improve the frame rate of the video.

[0039] Taking video games as an example, it is considered that video games sometimes cannot maintain a sustainable or consistent frame-per-second output due to performance bottlenecks. Unstable frame-per-second output can reduce the user's game experience, for example, after playing the game for a period of time, the interaction between the user and the game can become unsmooth due to frame loss.

[0040] In the related art, although methods of interpolating video game frames have been proposed, there are still artifacts, flickering, and the like in the generated interpolated frame results on transparent objects. Although the related art also proposes several methods of processing transparent objects. Exemplarily, some technologies use a similarity measure to detect areas on rendered color textures that have mixed transparent objects, calculate additional optical flow, and mix the optical flow and game motion vector warping results to mitigate artifacts introduced by using only pure game motion vectors. Some technologies also utilize additional optical flow, but utilize Deep Learning Super Sampling (DLSS) frames to generate an Artificial Intelligence (AI) network to determine how to use the optical flow and game motion vectors. In addition, in addition to using optical flow, some technologies can separate transparent objects and opaque objects in the game rendering pipeline, perform interpolation on the source frame without rendering transparent objects, and mix the transparent objects back into the interpolation result. Some technologies also separate transparent objects and opaque objects in the game rendering pipeline, but it further calculates the brightness of the transparent objects and uses the brightness difference between the two source frames to determine the mixing factor in the interpolated frame.

[0041] That is, the related art either relies on optical flow to predict the correct motion on transparent objects or separates transparent objects from real rendering frames in the game rendering pipeline and mixes the transparent objects back after performing interpolation on frames without transparent objects. However, due to the limitations of computing power and power consumption on mobile devices, it is particularly challenging to obtain satisfactory optical flow on mobile devices. In addition, in many games, it is difficult to render game frames without mixing transparent objects onto the game frames, which makes it infeasible or impractical to apply the related art.

[0042] Although optical flow calculation can use different parameter settings to simplify the calculation on mobile devices with acceptable time and power consumption, the optical flow calculation has poor quality under such settings that still produce flickering transparent effects; or it is difficult to perform optical flow calculation in 60 FPS→120 FPS interpolation due to increased time and power consumption at a higher frame rate, i.e., Frames Per Second (FPS).

[0043] To solve the above problems, an embodiment of the present application provides a frame interpolation method, specifically a method for interpolating color frames using a transparent texture. Compared with related technologies, the present method uses a separate transparent texture as an additional input, so that the artifacts and flickering on the transparent object of the generated interpolation frame are reduced, and the method has high robustness. In addition, the present method has little overhead and low power consumption; and can be executed in real time on most mobile devices, so that better quality interpolation frames can be generated without sacrificing the smoothness of game playing.

[0044] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0045] In an embodiment of the present application, Figure 1 The flow of a frame interpolation method provided by an embodiment of the present application is shown in Figure 1 As shown in Figure 1 , the method can include:

[0046] S101, obtaining a first color frame and a first transparent texture corresponding to a first time, and a second color frame and a second transparent texture corresponding to a second time.

[0047] In an embodiment of the present application, the transparent texture refers to a texture containing only transparent objects rendered on the actual frame. Specifically, the transparent texture can be a texture with a transparent effect in vision, which is commonly used in graphic design, game development and visual effects to increase the visual level and realism. The implementation of the transparent texture depends on the RGBA color model. In the RGBA color model, the Alpha (A) channel represents the transparency, and its value ranges from 0 (completely transparent) to 1 (completely opaque). By adjusting the value of the Alpha channel, different degrees of transparency can be achieved. In the rendering process, the transparent object needs to be mixed with the background, and the Alpha blending technique is usually used to handle the transparency.

[0048] In an embodiment of the present application, the color frame (Color frame) can also be referred to as a "color frame", which refers to a game rendering frame with transparent objects and opaque objects, and is used as an input of an interpolation algorithm. The opaque object refers to an object that completely blocks the object behind it, which usually uses an opaque material and a shader (Shader); the transparent object refers to an object that allows light to pass through, which usually allows the object on the other side to be clearly visible. Here, the first color frame and the second color frame can be used as inputs of the interpolation algorithm, and finally an interpolation frame is output.

[0049] In an embodiment of the present application, the first time can be a previous time point, denoted by t0; and the second time can be a current time point, denoted by t1.

[0050] In the embodiments of the present application, the interpolation algorithm can first obtain data corresponding to the first time and the second time, such as color frames and transparent textures, and can further include motion vectors and depth information. For example, the data corresponding to the previous time point t0, such as the first color frame, the first motion vector, the first depth information and the first transparent texture, are denoted as c0, mv0, d0 and tt0 respectively. Similarly, the data corresponding to the current time point t1, such as the second color frame, the second motion vector, the second depth information and the second transparent texture, are denoted as c1, mv1, d1 and tt1 respectively.

[0051] It should be further noted that for the interpolation algorithm, the data corresponding to the previous time point t0 has been reserved in the internal cache. In this way, after the data corresponding to the current time point t1 is ready, the interpolation algorithm can use the data from t0 and t1 to generate the interpolated frame. In addition, after each interpolated frame is generated, the data corresponding to the current time point t1 can be written into the internal cache as the data corresponding to t0 for the next round of interpolation algorithm.

[0052] S102, feature extraction is performed on the first transparent texture and the second transparent texture to obtain a first group of transparent features corresponding to the first transparent texture and a second group of transparent features corresponding to the second transparent texture.

[0053] In the embodiments of the present application, the first transparent texture and the second transparent texture can be input into the transparent feature extraction component to obtain the first group of transparent features and the second group of transparent features. For example, feature extraction is performed on the first transparent texture to obtain the first group of transparent features corresponding to the first transparent texture, and feature extraction is performed on the second transparent texture to obtain the second group of transparent features corresponding to the second transparent texture.

[0054] In the embodiments of the present application, the transparent feature extraction component can process the input transparent texture into features including luminance, alpha value, etc. For example, if the input transparent texture depends on the RGBA color model, the fourth channel data can be directly extracted as the alpha value; if the input transparent texture is in RGB format or without alpha channel, a threshold segmentation or edge detection algorithm can be used to generate the alpha value. In addition, the luminance of the transparent object is affected by the ambient light, and needs to be converted from RGB to YUV color space to extract the Y component as the luminance feature; or even the luminance value can be corrected in combination with an ambient light influence model, which can suppress the luminance distortion of the transparent object caused by refraction.

[0055] In the embodiments of the present application, the first group of transparent features can include at least one of the following: a first luminance feature and a first alpha value; and the second group of transparent features can include at least one of the following: a second luminance feature and a second alpha value.

[0056] In a specific implementation, the first set of transparent features can include a first luminance feature, a first alpha value and other optional features; and the second set of transparent features can include a second luminance feature, a second alpha value and other optional features. Exemplarily, the other optional features can be edge intensity, texture complexity, etc., which are not limited herein.

[0057] S103, determining a target selection factor according to the first set of transparent features and the second set of transparent features.

[0058] In the embodiments of the present application, the determination of the target selection factor according to the first set of transparent features and the second set of transparent features can include: performing a transparency test on the first set of transparent features and the second set of transparent features to determine the target selection factor.

[0059] In some embodiments, the transparency test subcomponent is configured to perform the transparency test on the first set of transparent features and the second set of transparent features to obtain the target selection factor. As shown in the figure, the method can include: Figure 2

[0060] S201, determining an initial selection factor.

[0061] S202, performing selection factor calculation according to the first set of transparent features and the second set of transparent features to determine a calculated selection factor.

[0062] S203, performing correction processing on the initial selection factor according to the calculated selection factor to obtain a target selection factor.

[0063] In the embodiments of the present application, the initial selection factor can be the selection factor obtained in the previous step. Accordingly, the determination of the initial selection factor can include: obtaining a first motion vector and first depth information corresponding to a first time, and a second motion vector and second depth information corresponding to a second time; performing re-projection on a first color frame and a second color frame according to the first motion vector and the second motion vector to determine a mapping relationship between an interpolated frame and the first color frame and the second color frame; and performing occlusion test according to the mapping relationship, the first depth information and the second depth information to determine the initial selection factor.

[0064] It should be noted that the first motion vector mv0 and the second motion vector mv1 can be used to map the pixel positions at the first time t0 and the second time t1 to a third time (i.e., an intermediate time point, denoted as t 0.5 ), to generate re-projected candidate pixel positions, which are used to construct the mapping relationship between t0-t 0.5 -t1. Wherein, the pixel at the first time t0 is projected to t 0.5 ​The pixel at the second time t1 is inversely projected to t 0.5 , through the second motion vector mv1, so as to establish the association with t0 and t1 for each pixel in the interpolation frame t 0.5 , to obtain the mapping relationship between t0-t 0.5 -t1, or the corresponding relationship between t0-t 0.5 -t1.

[0065] In this way, after the re-projection and the establishment of the mapping relationship between t0-t 0.5 -t1, each pixel in the interpolation frame t 0.5 will have a reference pixel at its position at the first time t0 and another reference pixel at its position at the second time t1. Then it is further determined whether to use the reference pixel from the first color frame, or the reference pixel from the second color frame, or the mixed reference pixel from both, as the pixel at the corresponding position in the interpolation frame t 0.5 ; that is, to determine the initial selection factor.

[0066] In the embodiments of the present application, the initial selection factor can include a first initial selection factor s0, a second initial selection factor s1 and a third initial selection factor s 0.5 , which are respectively used to indicate the contribution weight of each pixel to the interpolation result. The first initial selection factor s0 is used to indicate the initial contribution weight of the pixel in the first color frame to the interpolation frame, the second initial selection factor s1 is used to indicate the initial contribution weight of the pixel in the second color frame to the interpolation frame, and the third initial selection factor s 0.5 is used to indicate the initial contribution weight of directly using part of the pixels in the second color frame.

[0067] It should be further noted that, in the embodiments of the present application, the higher s0 is, the more pixels in the first color frame are used, and the higher s1 is, the more pixels in the second color frame are used. In addition, in order to process some special elements (for example, almost opaque objects that should be kept at the same position as t1, etc.), an additional factor (that is, the third initial selection factor s 0.5 ) can be used here to indicate the weight of the pixel at the same position from the second color frame at the time t1, without using the motion vector.

[0068] In a possible implementation, the initial selection factor can be determined by comparing the depth information in the occlusion test. Illustratively, the initial selection factor of a pixel with greater depth information is set to 1, and the initial selection factor of a pixel with smaller depth information is set to 0. That is, if the first depth information is greater than the second depth information, s0 corresponding to the current pixel is set to 1 and s1 is set to 0; if the first depth information is smaller than the second depth information, s0 corresponding to the current pixel is set to 0 and s1 is set to 1.

[0069] It can be understood that, in the embodiment of the present application, the calculation of the selection factor can include first calculation of the selection factor a0, second calculation of the selection factor a1 and third calculation of the selection factor a 0.5 In some embodiments, the calculation of the selection factor according to the first set of transparent features and the second set of transparent features can include: determining the third calculation of the selection factor; inputting the third calculation of the factor, the first set of transparent features and the second set of transparent features into the first calculation model to obtain the first calculation of the selection factor; and determining the second calculation of the selection factor according to the first calculation of the selection factor and the third calculation of the selection factor.

[0070] In the embodiment of the present application, the third set of transparent features corresponding to the interpolation frame is determined according to the mapping relationship between the interpolation frame and the first color frame and the second color frame; and the third set of transparent features is input into the second calculation model to obtain the third calculation of the selection factor.

[0071] Here, the transparent feature function is represented as follows:

[0072]

[0073] wherein θ is a smooth range for controlling the transition of the interpolation weight, which defines the smooth range of the difference (x-y). When the absolute value of (x-y) is ≤ θ, the output value is smoothly transitioned between 0 and 1; when (x-y) > θ, the output value is clamped to 1; and when (x-y) < -θ, the output value is clamped to 0.

[0074] That is, the range of the interpolation weight is smoothly transitioned within the range [-θ, θ] by the parameter θ, and is truncated to 0 or 1 when out of the range. Illustratively, the smaller the value of θ, the more suitable for fast switching scenarios; and the greater the value of θ, the more suitable for smooth transition scenarios. The value of θ can be set according to actual conditions, which is not limited here.

[0075] For the second calculation model, the third calculation of the selection factor a 0.5 is calculated as follows:

[0076] a 0.5 =f s (tt 0.5, y, θ) (2)

[0077] For the first calculation model, the first calculation selection factor a0is calculated as follows:

[0078] a0= (1 - a 0.5 ) · f s (tt0, tt1, θ) (3)

[0079] After a 0.5 and a0are obtained, the second calculation selection factor a1is calculated as follows:

[0080] a1= 1 - a0- a 0.5 (4)

[0081] Here, tt0represents the first set of transparency features, tt1represents the second set of transparency features, and tt 0.5 represents the third set of transparency features.

[0082] It is also appreciated that the transparent test is performed in a similar manner as the occlusion test. In the occlusion test, the comparison is based on depth information, while in the transparent test, the comparison is based on transparency features.

[0083] In another possible implementation, determining the calculation selection factor based on the first set of transparency features and the second set of transparency features can include:

[0084] comparing the first set of transparency features and the second set of transparency features;

[0085] determining that the first calculation selection factor is higher than the second calculation selection factor when the first luminance feature is lower than the second luminance feature and the first alpha value is higher than the second alpha value;

[0086] determining that the first calculation selection factor is lower than the second calculation selection factor when the first luminance feature is higher than the second luminance feature and the first alpha value is lower than the second alpha value.

[0087] In the embodiments of the present application, if the luminance feature of a certain pixel is low and the alpha value is large (close to opaque), then the corresponding calculation selection factor is increased, and vice versa. Exemplarily, it can be determined from the transparency features which source can have less transparent objects; then the selection factor of the source with less transparent objects is increased, while the selection factor of the source with more transparent objects is decreased.

[0088] In addition, it is noted that the larger the alpha value of a certain pixel, the more opaque (i.e., the lower the transparency and the higher the opacity) the pixel is; the smaller the alpha value of a certain pixel, the more transparent (i.e., the higher the transparency and the lower the opacity) the pixel is.

[0089] In some embodiments, the first calculation selection factor is set to 1 and the second calculation selection factor is set to 0 when the first luminance feature is lower than the second luminance feature and the first alpha value is higher than the second alpha value; otherwise, the first calculation selection factor is set to 0 and the second calculation selection factor is set to 1 when the first luminance feature is higher than the second luminance feature and the first alpha value is lower than the second alpha value.

[0090] That is, the embodiments of the present application can give priority to the pixel with high opacity. Exemplarily, the selection factor corresponding to the pixel with lower luminance and larger alpha value can be set to 1 and the selection factor of the other pixel can be set to 0.

[0091] In some embodiments, the calculation of the selection factor according to the first set of transparent features and the second set of transparent features further comprises: setting the first calculation selection factor and the second calculation selection factor to 0.5 when the difference between the first luminance feature and the second luminance feature satisfies a first difference range and the difference between the first alpha value and the second alpha value satisfies a second difference range.

[0092] In the embodiments of the present application, the first difference range is used to measure that the first luminance feature is relatively close to the second luminance feature, and the second difference range is used to measure that the first alpha value is relatively close to the second alpha value.

[0093] That is, the embodiments of the present application can also consider the equal mixing of similar feature pixels. Exemplarily, if the luminance and the alpha value of two pixels are relatively close, the first calculation selection factor and the second calculation selection factor can also be set to 0.5 to mix the two pixels.

[0094] It can be understood that after the first calculation selection factor, the second calculation selection factor and the third calculation selection factor are determined, the initial selection factor can be updated by the first calculation selection factor, the second calculation selection factor and the third calculation selection factor. In some embodiments, the initial selection factor is updated by the calculation selection factor to obtain a target selection factor, which can comprise:

[0095] The first target selection factor is obtained by modifying and normalizing the first initial selection factor and the first calculation selection factor;

[0096] The second target selection factor is obtained by modifying and normalizing the second initial selection factor and the second calculation selection factor;

[0097] The third target selection factor is obtained by modifying and normalizing the third initial selection factor and the third calculation selection factor.

[0098] In the embodiments of the present application, the first target selection factor is denoted as s'0, the second target selection factor is denoted as s'1, and the third target selection factor is denoted as s'2. 0.5 Exemplarily, the correction formula of the selection factor is as follows:

[0099] s'0←s0·a0 (5)

[0100] s'0←s0·a0 (5) 0.5 0.5 ·a 0.5 (6)

[0101] s'1←s1·a1(7)

[0102] In the embodiments of the present application, normalization processing can also be performed to prevent weight overflow and ensure that the color value after mixing is legal. In a possible implementation manner, the normalization formula is as follows:

[0103]

[0104]

[0105]

[0106] In another possible implementation manner, the normalization formula is as follows:

[0107]

[0108]

[0109]

[0110] In some embodiments, for special pixels, the first target selection factor and the second target selection factor are both set to 0, and the third target selection factor is set to 1, in a case where the alpha value of the current pixel is lower than a second threshold value. Wherein, for these special elements (for example, almost opaque objects that should be kept at the same position as t1), the value of s'2 can be equal to 1, and the pixel at the corresponding position in the second color frame corresponding to the current time point t1 is directly used. 0.5

[0111] That is, for the detection area with low opacity, the offset calculation of the motion vector can be skipped, and the pixel at the corresponding position in the second color frame corresponding to the current time point t1 is directly copied.

[0112] ​​It should be noted that the transparency test described above can be implemented by a transparency test sub-component. The transparency test sub-component can be easily added to the current frame interpolation algorithm as a test sub-component in addition to the typical occlusion test sub-component. In addition, it can be used interchangeably with other optional test sub-components to improve the quality of the interpolation result while maintaining the high flexibility of the interpolation algorithm.

[0113] That is, the target selection factor after the transparency test will help to eliminate the movement of the transparent object along with the motion vector of the background object, and try to pass the transparent object at t1 to the same position at t 0.5 This can prevent the transparent object from leaving artifacts or flickering on the interpolated frame, such as left-right flickering when the camera or character moves or rotates.

[0114] S104, mixing the first color frame and the second color frame according to the target selection factor to generate an interpolated frame between the first time and the second time.

[0115] In the embodiments of the present application, a third time between the first time and the second time is taken as an example, which can refer to an intermediate time point, denoted by t 0.5 , then the interpolated frame corresponding to the third time can be denoted by c 0.5 In some embodiments, mixing the first color frame and the second color frame according to the target selection factor to generate an interpolated frame between the first time and the second time can include:

[0116] According to the first target selection factor, the second target selection factor and the third target selection factor, the first color frame, the second color frame and part of the pixels in the second color frame are weighted and calculated to obtain the interpolated frame.

[0117] In the embodiments of the present application, the first target selection factor can be used to indicate the contribution weight of the pixels in the first color frame to the interpolated frame, the second target selection factor can be used to indicate the contribution weight of the pixels in the second color frame to the interpolated frame, and the third target selection factor can be used to indicate the contribution weight of directly using part of the pixels in the second color frame.

[0118] It should be noted that each pixel in the interpolated frame c 0.5 may be a pixel from the first color frame corresponding to the first time t0, or may be a pixel from the second color frame corresponding to the second time t1, or may also be a pixel mixture from both, which can be selected and mixed according to the target selection factor, and output the final interpolation result (i.e. "interpolated frame").

[0119] Exemplarily, the calculation formula of the interpolation mixture is as follows:

[0120] c0.5 =s′0·c0+s′ 0.5 ·c 1，static +s′1·c1 (14)

[0121] In the embodiments of this application, c 1，static This refers to directly copying a portion of pixels from the second color frame in the interpolated frame to avoid motion offset.

[0122] In some embodiments, mixing the first color frame and the second color frame according to a target selection factor to generate an interpolated frame between the first time moment and the second time moment may include:

[0123] When the opacity of the target pixel is detected to meet the first condition, some pixels in the second color frame are directly determined as the target pixels at the corresponding positions in the interpolation frame;

[0124] When the opacity of the target pixel does not meet the first condition, the pixels at the corresponding positions in the first color frame and the second color frame are weighted according to the first target selection factor and the second target selection factor to obtain the target pixel at the corresponding position in the interpolation frame.

[0125] In this embodiment, the opacity of the target pixel satisfying the first condition may include: the alpha value of the target pixel is lower than a first threshold, or in other words, the target pixel is in a low opacity region. In this case, the interpolation frame directly uses a portion of the pixels in the second color frame corresponding to the second time t1; at this time, the values ​​of s′0 and s′1 are equal to 0, and s′... 0.5 The value of is 1.

[0126] In this embodiment, the opacity of the target pixel does not satisfy the first condition, including: the alpha value of the target pixel is higher than the second threshold, or the target pixel is in a high opacity region; or the target pixel is in a similar region to be blended. In this case, the interpolation frame can be obtained by weighting the pixels at corresponding positions in the first color frame c0 and the second color frame c1 based on s′0 and s′1, where s′ 0. The value of is 0.

[0127] In one possible implementation, if the alpha value of a pixel in the second color frame c1 is higher than a third threshold and the luminance feature is lower than a preset luminance threshold, then s′0 and s′ 0.5 The value of is set to 0, and the value of s′1 is set to 1. At this time, the pixel at the corresponding position in the second color frame c1 can be directly used.

[0128] In another possible implementation, for the regions that are invalidated due to occlusion or transparency test markers, the neighborhood interpolation or the previous time point of the history frame data can also be used to fill in. Exemplarily, if the transparent object occludes the background, the pixels can be copied from the corresponding position in the first color frame c0.

[0129] The embodiment provides a frame interpolation method, in particular, a new method for interpolating frames of mobile phone games using transparent textures. Compared with related technologies, the method uses a separate transparent texture as an additional input and generates an interpolated frame with reduced artifacts and flicker on transparent objects. In order to alleviate the problem of transparent objects in frame interpolation, the method uses a method similar to occlusion test using depth information, which can be processing the input transparent texture into transparent features including luminance, alpha value and the like, and using any number of transparent features to detect regions with transparent particles and effects. The interpolation algorithm tests the features at the corresponding positions in the two input frames (for example, the previous rendering frame and the current rendering frame) with each other to determine whether to use the pixels of the first color frame, the pixels of the second color frame or the mixed pixels of the two color frames. In this way, compared with related technologies, the method is much more convenient when dealing with related problems (for example, optical flow) of transparent objects, and at the same time, better or similar quality can be achieved on the interpolated frame; and the overhead of the method is extremely small, and can be executed in real time on most of the latest mobile devices, so that the interpolated frame with better quality can be generated without sacrificing the smoothness of playing games.

[0130] In another embodiment of the present application, based on the frame interpolation method of the foregoing embodiment, the technical solution of the embodiment of the present application is part of the frame interpolation algorithm, which aims to reduce the related problems of transparent objects in the interpolation result, and the frame interpolation algorithm can provide improved game experience and increased battery life for users.

[0131] In the embodiment of the present application, the method proposed herein is a supplement to the conventional frame interpolation algorithm, and uses an additional input texture (abbreviated as "transparent texture" herein) including only transparent objects. The method can be extracting transparent features (including one or more of luminance, alpha value and other optional features) from the input transparent texture, and using the extracted transparent features for transparency test. The test will detect the regions on the current color frame (abbreviated as "color frame" hereinafter) with transparent objects and opaque objects, where the opaque objects are mixed with the transparent objects to a large extent. Then, by obtaining the pixels at the corresponding positions in the previous color frame, the mixed regions on the current color frame are repaired by using a method similar to the de-occlusion region repair.

[0132] In the embodiments of the present application, since the method does not involve complex calculations such as optical flow, the artifacts introduced by transparent objects can be reduced, the impact on power consumption is small, and it is easy to execute on a mobile device. Moreover, the method of the embodiments of the present application can be applied to any game that obtains a transparent texture in a game rendering pipeline or estimates a transparent texture from a color frame, while not destroying the integrity of the original rendering pipeline, wherein the transparent object is removed in the original rendering pipeline after capture.

[0133] The overall architecture of the frame interpolation algorithm is described below.

[0134] (1) Interpolation algorithm overview

[0135] The frame interpolation algorithm architecture, as shown in Figure 3 , can include three main components: a re-projection component 301, a transparent feature extraction component 302, and a warping component 303. Among them, the transparent feature extraction component 302 and the warping component 303 are involved here. In the warping component 303, it can include: constructing a mapping relationship between t0-t 0.5 -t1 303-1, an occlusion test sub-component 303-2, a transparent test sub-component 303-3, an optional other test sub-component 303-4, and a target pixel selection and repair / mixing sub-component 303-5. Among them, the other test sub-component 303-4 is an optional sub-component.

[0136] Exemplarily, the transparent feature extraction component 302 can extract brightness, alpha value and other optional features from the transparent texture and use them for the transparent test sub-component 303-3, and the transparent test sub-component 303-3 can be used with and interchangeably with the other test sub-component 303-4.

[0137] In order to perform frame interpolation in a real game, the interpolation algorithm will be executed on a separate thread outside the rendering thread of the game. The interpolation algorithm will reserve an internal cache for the previous time point t0, which can include a first color frame, a first motion vector, a first depth information and a first transparent texture, which are denoted as c0, mv0, d0, tt0 respectively. Once the data of the current time point t1 (similarly, denoted as c1, mv1, d1, tt1 respectively) is ready in the rendering pipeline of the game, the interpolation algorithm will start to use the data from t0 and t1 to generate an interpolated frame. In the following description, an example of interpolating one frame between each real game frame is taken here, wherein the interpolation algorithm will output an interpolated frame corresponding to time point t 0.5 . 0.5 .

[0138] As shown in Figure 3 , after re-projection and construction of the mapping relationship between t0-t 0.5 -t1, c 0.5each pixel in c will have a reference to its position at time point t0 and another reference to its position at time point t1. The next task will be to determine whether to use the pixel from the first color frame (time point t0), or the pixel from the second color frame (time point t1), or a pixel blend from both, as the c 0.5 pixel color in c. This is achieved by various test subcomponents including occlusion test, transparency test, etc. In the embodiments of the present application, the interpolation algorithm uses selection factors s0, s 0.5 , s1 to indicate the degree of contribution of each pixel to the interpolation result, where s0 is higher indicating more use of the color at time point t0, and s1 is higher indicating more use of the color at time point t1. In addition, in order to handle some special elements (e.g. almost opaque objects that should be kept at the same position as t1, etc.), an additional factor s 0.5 may also be used here to indicate the proportion of pixels from the same position in the second color frame (time point t1) without applying the motion vector. After all tests, the color can be selected and blended according to the resulting target selection factors, and the final interpolation result, i.e. the final interpolated frame c 0.5 is output.

[0139] (2) Transparency object detection workflow.

[0140] The embodiments of the present application propose a lightweight but effective method to solve the related problems of transparent objects, such as artifacts and object flickering, by introducing an additional transparency test subcomponent in the warping step in the frame interpolation method. Wherein, the transparency test subcomponent takes the processed transparent features from a separate transparency texture as input and updates the target color selection factors, as the occlusion test. Exemplarily, the workflow of the transparency test subcomponent is shown in Figure 4 , which can include:

[0141] S401, obtaining a transparency feature vector.

[0142] S402, color selection factor calculation.

[0143] S403, input the initial selection factors of the previous step.

[0144] S404, modify the selection factors.

[0145] S404, output the target selection factors to the next step.

[0146] In the embodiments of the present application, after obtaining the transparency feature vector, the selection factor calculation is performed by the aforementioned formulas (1)~(4), and the calculation selection factors (a0, a1 and a 0.5 ) can be obtained; the initial selection factors (s0, s1 and s0.5 After that, you can select factors (a0, a1, and a) based on the calculation. 0.5 ) and initial selection factors (s0, s1 and s 0.5 The selection factors can be modified, for example, by the correction process of formulas (5) to (7) above, and / or by the normalization process of formulas (8) to (10) above (or by the normalization process of formulas (11) to (13) above), so that the target selection factors (s′0, s′1 and s′) can be calculated. 0.5 ), and output it to the next step.

[0147] In this embodiment, the transparency test is performed similarly to the occlusion test. In the occlusion test, depth information is compared (rather than transparency features), and the selection factor for pixels with greater depth information is set to 1, while the selection factor for pixels with less depth information is set to 0. In the transparency test, the selection factor for a pixel with lower brightness and a higher alpha value can be set to 1, while the selection factor for another pixel can be set to 0. If the brightness and alpha values ​​of two pixels are similar, both selection factors can be set to 0.5 to blend the two pixels.

[0148] In one specific embodiment, for c 0.5 For each pixel in the image, the transparency test will be performed on pixels at t0, t... 0.5 The transparent features extracted at the corresponding position of t1 are compared. Specifically, it determines which source has fewer transparent objects from the transparent features, increasing the selection factor for sources with fewer transparent objects and decreasing the selection factor for sources with more transparent objects. If special transparent objects (such as low-opacity objects) are detected, the pixel at the corresponding position in the current color frame can be directly used without using motion vector offset; and this pixel is superimposed on the interpolated frame to maintain the temporal continuity of such objects.

[0149] It should be noted that the selection factor after performing the transparency test will help eliminate the movement of transparent objects with the motion vector of the background object, and attempt to transfer the transparent object at t1 to t2. 0.5 The same position at the same location. This prevents transparent objects from leaving artifacts or flickering on the interpolated frame, such as flickering left and right when the camera or character moves or rotates.

[0150] (3) Example of transparent object detection.

[0151] In an exemplary scenario of a certain game, Figure 5A and Figure 5B as well as Figure 6A , Figure 6B and Figure 6CIt is demonstrated how the transparency test helps to reduce the flickering problem of the transparent texture. Among others, Figure 5A denotes the first color frame at t0, Figure 5B denotes the second color frame at t1. Figure 5A and Figure 5B The key object in

[0152] wherein, Figure 6A denotes the interpolated frame at t 0.5 without transparency test, Figure 6B denotes the test region, Figure 6C denotes the interpolated frame after transparency test repair. As can be seen, Figure 6A shows the result without applying the transparency test, the key object has a clear skew to the right, which skewed frame will be shown between the two normal frames as shown in Figure 5A and Figure 5B which will negatively affect the player experience.

[0153] Figure 6B The detection region detected by the transparency test is highlighted using gray color. Some selection factors can be assigned to the pixels in the detection region, so that the interpolation result will select the frame without the key object as the color source. In addition, the detection region corresponds to the key object in t1, and the pixels in the detection region will be directly copied from the second color frame at t1, so as to ensure that the key objects in c 0.5 and c1 are located at the same position. After the transparency test, the output is Figure 6C As can be seen from Figure 6C , the key object is no longer skewed.

[0154] In some embodiments, the present method can be easily added to the current frame interpolation algorithm as a test subcomponent in addition to the typical occlusion test subcomponent. It can be used interchangeably with other test subcomponents to improve the quality of the interpolation result while maintaining the high flexibility of the interpolation algorithm.

[0155] In some embodiments, the present method can be a lightweight optical flow or a lightweight AI module to detect transparent objects. Although the lightweight optical flow or the lightweight AI module can achieve comparable interpolation quality compared to the technical solutions of the embodiments of the present application, the lightweight optical flow or the lightweight AI module still uses a large number of modifications to achieve similar power consumption as the embodiments of the present application without sacrificing the quality of the interpolation result.

[0156] The foregoing is a detailed description of the specific implementation of the foregoing embodiments. As can be seen from the foregoing embodiments, the technical solutions of the foregoing embodiments provide a lightweight solution to solve the common problem introduced by transparent objects in the frame interpolation method. Experimentally, in a game with very high quality (1620x736 resolution) and 60FPS interpolated to 120FPS, the interpolation algorithm of the embodiments of the present application introduces less than 10mW of additional power consumption, and the average power consumption of the interpolation algorithm is 692.4mW. Compared with related technologies (such as optical flow or AI-based methods), the present method significantly reduces the computing power and power consumption used during runtime, while achieving comparable results in reducing the transparent object problem in the frame interpolation algorithm.

[0157] In yet another embodiment of the present application, based on the same inventive concept as the foregoing embodiments, Figure 7 A schematic diagram of the composition structure of a frame interpolation device provided by the embodiments of the present application is shown in FIG. 7. As shown in FIG. 7, the frame interpolation device 700 can include an acquisition unit 701, a feature extraction unit 702, and an interpolation unit 703, wherein: Figure 7

[0158] The acquisition unit 701 is configured to acquire a first color frame and a first transparent texture corresponding to a first time, and a second color frame and a second transparent texture corresponding to a second time;

[0159] The feature extraction unit 702 is configured to perform feature extraction on the first transparent texture and the second transparent texture to obtain a first group of transparent features corresponding to the first transparent texture and a second group of transparent features corresponding to the second transparent texture;

[0160] The interpolation unit 703 is configured to determine a target selection factor according to the first group of transparent features and the second group of transparent features, and to mix the first color frame and the second color frame according to the target selection factor to generate an interpolated frame between the first time and the second time.

[0161] In some embodiments, the interpolation unit 703 is further configured to perform transparent testing according to the first group of transparent features and the second group of transparent features to determine the target selection factor.

[0162] In some embodiments, the interpolation unit 703 is further configured to determine an initial selection factor, to perform selection factor calculation according to the first group of transparent features and the second group of transparent features to determine a calculated selection factor, and to perform correction processing on the initial selection factor according to the calculated selection factor to obtain the target selection factor.

[0163] ​In some embodiments, the interpolation unit 703 is further configured to obtain a first motion vector and first depth information corresponding to the first time, and a second motion vector and second depth information corresponding to the second time; perform re-projection on the first color frame and the second color frame according to the first motion vector and the second motion vector, to determine a mapping relationship between the interpolated frame and the first color frame and the second color frame; and perform occlusion testing according to the mapping relationship, the first depth information and the second depth information, to determine the initial selection factor.

[0164] In some embodiments, the target selection factor includes a first target selection factor, a second target selection factor and a third target selection factor; correspondingly, the interpolation unit 703 is further configured to perform weighted calculation on the first color frame, the second color frame and part of the pixels in the second color frame according to the first target selection factor, the second target selection factor and the third target selection factor, to obtain the interpolated frame; wherein the first target selection factor is used to indicate the contribution weight of the pixels in the first color frame to the interpolated frame, the second target selection factor is used to indicate the contribution weight of the pixels in the second color frame to the interpolated frame, and the third target selection factor is used to indicate the contribution weight of directly using part of the pixels in the second color frame.

[0165] In some embodiments, the interpolation unit 703 is further configured to, when detecting that the opacity of the target pixel satisfies a first condition, directly determine part of the pixels in the second color frame as the target pixel at the corresponding position in the interpolated frame; and / or, when detecting that the opacity of the target pixel does not satisfy the first condition, perform weighted calculation on the pixels at the corresponding position in the first color frame and the second color frame according to the first target selection factor and the second target selection factor, to obtain the target pixel at the corresponding position in the interpolated frame.

[0166] In some embodiments, the interpolation unit 703 is further configured to, when the opacity of the current pixel satisfies the first condition, set the first target selection factor and the second target selection factor to 0, and set the third target selection factor to 1.

[0167] In some embodiments, the initial selection factor includes a first initial selection factor, a second initial selection factor and a third initial selection factor, and the calculation selection factor includes a first calculation selection factor, a second calculation selection factor and a third calculation selection factor; correspondingly, the interpolation unit 703 is further configured to perform correction and normalization processing according to the first initial selection factor and the first calculation selection factor, to obtain the first target selection factor; perform correction and normalization processing according to the second initial selection factor and the second calculation selection factor, to obtain the second target selection factor; and perform correction and normalization processing according to the third initial selection factor and the third calculation selection factor, to obtain the third target selection factor.

[0168] In some embodiments, the first set of transparency features comprises at least one of: a first luminance feature and a first alpha value; and the second set of transparency features comprises at least one of: a second luminance feature and a second alpha value.

[0169] In some embodiments, the interpolation unit 703 is further configured to compare the first set of transparency features and the second set of transparency features; determine that the first calculation selection factor is higher than the second calculation selection factor when the first luminance feature is lower than the second luminance feature and the first alpha value is higher than the second alpha value; and determine that the first calculation selection factor is lower than the second calculation selection factor when the first luminance feature is higher than the second luminance feature and the first alpha value is lower than the second alpha value.

[0170] In some embodiments, the interpolation unit 703 is further configured to set the first calculation selection factor and the second calculation selection factor to 0.5 when a difference between the first luminance feature and the second luminance feature satisfies a first difference range and a difference between the first alpha value and the second alpha value satisfies a second difference range.

[0171] In the embodiments of the present application, the "unit" can be a part of circuit, a part of processor, a part of program or software, etc., and of course can be a module, and also can be non-modular. Moreover, the components in the embodiments of the present application can be integrated in one processing unit, or can be physically present individually, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of software function module.

[0172] In addition, those skilled in the art should understand that the above description of the frame interpolation device in the embodiments of the present application can be understood with reference to the description of the frame interpolation method in the embodiments of the present application.

[0173] The embodiments of the present application provide a frame interpolation device, which uses a separate transparent texture as an additional input, so that the artifacts and flicker on the transparent object of the generated interpolation frame are reduced. In addition, the method has little overhead and low power consumption; and can be executed in real time on most mobile devices, so that better quality interpolation frames can be generated without sacrificing the smoothness of the game.

[0174] In still another embodiment of the present application, Figure 8 A schematic diagram of the structure of an electronic device is provided in the embodiments of the present application. As shown in Figure 8 The electronic device 800 includes a processor 810 and a memory 820, the memory 820 can store a computer program, and the processor 810 can call and run the computer program from the memory 820 to implement the method in the embodiments of the present application.

[0175] The memory 820 can be a separate device independent of the processor 810, or can be integrated in the processor 810.

[0176] In some embodiments, as shown in Figure 8 The electronic device 800 can further include a transceiver 830, which can be controlled by the processor 810 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0177] The transceiver 830 can include a transmitter and a receiver. The transceiver 830 can further include an antenna, and the number of antennas can be one or more.

[0178] In some embodiments, the present application further provides another composition of an electronic device, wherein the electronic device can include the frame interpolation apparatus 700 according to any one of the preceding embodiments.

[0179] Figure 9 A schematic diagram of a chip according to an embodiment of the present application is shown in Figure 9 The chip 900 includes a processor 910, which can call and run a computer program from a memory to implement the method according to an embodiment of the present application.

[0180] In some embodiments, as shown in Figure 9 The chip 900 can further include a memory 920. The processor 910 can call and run a computer program from the memory 920 to implement the method according to an embodiment of the present application.

[0181] The memory 920 can be a separate device independent of the processor 910, or can be integrated in the processor 910.

[0182] In some embodiments, as shown in Figure 9 The chip 900 can further include an input interface 930. The processor 910 can control the input interface 930 to communicate with other devices or chips, specifically, to obtain information or data sent by other devices or chips.

[0183] In some embodiments, as shown in Figure 9 The chip 900 can further include an output interface 940. The processor 910 can control the output interface 940 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0184] In some embodiments, the chip can be applied to the electronic device according to an embodiment of the present application, and details are not described herein for brevity.

[0185] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0186] It can be understood that the processor of the embodiments of the present application can be an integrated circuit chip with information processing capability. In the implementation process, each step of the above method embodiments can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The processor mentioned above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware code processor execution, or executed by hardware and software module combination in the code processor. The software module can be located in the random access memory, the flash memory, the read only memory, the programmable read only memory or the electrically erasable programmable memory, the register, etc. The storage medium is mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method.

[0187] It is also understood that the memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable Programmable ROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0188] It is also understood that the above-mentioned memory is exemplary but not limiting, for example, the memory in the embodiments of the present application can also be Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synch Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM), etc. That is, the memory in the embodiments of the present application is intended to include, but not be limited to, these and any other suitable types of memory.

[0189] The embodiments of the present application also provide a computer readable storage medium for storing a computer program.

[0190] In some embodiments, the computer readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program is executed by at least one processor to realize the corresponding process realized by the electronic device in each method of the embodiments of the present application. For brevity, details are not repeated here.

[0191] The embodiments of the present application further provide a computer program product comprising computer program instructions.

[0192] In some embodiments, the computer program product can be applied to the electronic device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding process realized by the electronic device in each method of the embodiments of the present application. For brevity, details are not repeated here.

[0193] The embodiments of the present application further provide a computer program.

[0194] In some embodiments, the computer program can be applied to the electronic device in the embodiments of the present application, and when the computer program runs on the computer, the computer executes the corresponding process realized by the electronic device in each method of the embodiments of the present application. For brevity, details are not repeated here.

[0195] Those skilled in the art can understand that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described device and unit can refer to the corresponding process in the foregoing method embodiments, and details are not repeated here.

[0197] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0198] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment of the present application according to actual needs.

[0199] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0200] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and various program code storage media.

[0201] It should be noted that in the present application, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0202] The serial numbers of the above embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0203] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0204] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0205] The features disclosed in several method or device embodiments of the present application can be combined, without conflict, to form new method or device embodiments.

[0206] The above description is merely a specific implementation of the present application. The protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A frame interpolation method, characterized in that, The method includes: Obtain the first color frame and the first transparent texture corresponding to the first time step, and the second color frame and the second transparent texture corresponding to the second time step; Feature extraction is performed on the first transparent texture and the second transparent texture to obtain a first set of transparent features corresponding to the first transparent texture and a second set of transparent features corresponding to the second transparent texture; The target selection factor is determined based on the first set of transparency features and the second set of transparency features; The first color frame and the second color frame are mixed according to the target selection factor to generate an interpolated frame between the first time point and the second time point.

2. The method according to claim 1, characterized in that, The step of determining the target selection factor based on the first set of transparency features and the second set of transparency features includes: Based on the first set of transparency features and the second set of transparency features, a transparency test is performed to determine the target selection factor.

3. The method according to claim 2, characterized in that, The step of performing a transparency test based on the first set of transparency features and the second set of transparency features to determine the target selection factor includes: Determine the initial selection factor; The selection factor is calculated based on the first set of transparent features and the second set of transparent features to determine the selection factor. The initial selection factor is corrected based on the calculated selection factor to obtain the target selection factor.

4. The method according to claim 3, characterized in that, The determination of the initial selection factor includes: Obtain the first motion vector and first depth information corresponding to the first time moment, and the second motion vector and second depth information corresponding to the second time moment; The first color frame and the second color frame are reprojected based on the first motion vector and the second motion vector to determine the mapping relationship between the interpolated frame and the first color frame and the second color frame; Based on the mapping relationship, the first depth information, and the second depth information, an occlusion test is performed to determine the initial selection factor.

5. The method according to any one of claims 1 to 4, characterized in that, The target selection factors include a first target selection factor, a second target selection factor, and a third target selection factor; The step of mixing the first color frame and the second color frame according to the target selection factor to generate an interpolated frame between the first time moment and the second time moment includes: The first color frame, the second color frame, and a portion of pixels in the second color frame are weighted and calculated based on the first target selection factor, the second target selection factor, and the third target selection factor to obtain the interpolated frame. Wherein, the first target selection factor is used to indicate the contribution weight of the pixels in the first color frame to the interpolation frame, the second target selection factor is used to indicate the contribution weight of the pixels in the second color frame to the interpolation frame, and the third target selection factor is used to indicate the contribution weight of directly using some pixels in the second color frame.

6. The method according to claim 5, characterized in that, The step of mixing the first color frame and the second color frame according to the target selection factor to generate an interpolated frame between the first time moment and the second time moment includes: When the opacity of the target pixel is detected to meet the first condition, some pixels in the second color frame are directly determined as the target pixel at the corresponding position in the interpolation frame; When the opacity of the target pixel does not meet the first condition, the pixels at corresponding positions in the first color frame and the second color frame are weighted according to the first target selection factor and the second target selection factor to obtain the target pixel at the corresponding position in the interpolation frame.

7. The method according to claim 6, characterized in that, The method further includes: When the opacity of the current pixel meets the first condition, the first target selection factor and the second target selection factor are both set to 0, and the third target selection factor is set to 1.

8. The method according to claim 3, characterized in that, The initial selection factors include a first initial selection factor, a second initial selection factor, and a third initial selection factor, and the calculated selection factors include a first calculated selection factor, a second calculated selection factor, and a third calculated selection factor; The step of revising the initial selection factor based on the calculated selection factor to obtain the target selection factor includes: The first target selection factor is obtained by modifying and normalizing the first initial selection factor and the first calculated selection factor. The second target selection factor is obtained by modifying and normalizing the second initial selection factor and the second calculated selection factor. The third target selection factor is obtained by modifying and normalizing the third initial selection factor and the third calculated selection factor.

9. The method according to claim 8, characterized in that, The first set of transparency features includes at least one of the following: a first brightness feature and a first alpha value; The second set of transparency features includes at least one of the following: a second brightness feature and a second alpha value.

10. The method according to claim 9, characterized in that, The step of calculating the selection factor based on the first set of transparency features and the second set of transparency features, and determining the selection factor, includes: Compare the first set of transparent features and the second set of transparent features; When the first brightness feature is lower than the second brightness feature and the first alpha value is higher than the second alpha value, it is determined that the first calculation selection factor is higher than the second calculation selection factor. When the first luminance feature is higher than the second luminance feature and the first alpha value is lower than the second alpha value, it is determined that the first calculation selection factor is lower than the second calculation selection factor.

11. The method according to claim 9, characterized in that, The step of calculating the selection factor based on the first set of transparency features and the second set of transparency features further includes: When the difference between the first brightness feature and the second brightness feature meets the first difference range, and the difference between the first alpha value and the second alpha value meets the second difference range, both the first calculation selection factor and the second calculation selection factor are set to 0.

5.

12. A frame interpolation device, characterized in that, The frame interpolation device includes an acquisition unit, a feature extraction unit, and an interpolation unit, wherein: The acquisition unit is configured to acquire the first color frame and the first transparent texture corresponding to the first time moment, and the second color frame and the second transparent texture corresponding to the second time moment. The feature extraction unit is configured to extract features from the first transparent texture and the second transparent texture to obtain a first set of transparent features corresponding to the first transparent texture and a second set of transparent features corresponding to the second transparent texture. The interpolation unit is configured to determine a target selection factor based on the first set of transparency features and the second set of transparency features; and to mix the first color frame and the second color frame based on the target selection factor to generate an interpolated frame between the first time point and the second time point.

13. An electronic device, characterized in that, The electronic device includes a processor and a memory for storing a computer program capable of running on the processor; wherein the processor is configured to perform the method as described in any one of claims 1 to 11 when running the computer program.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1 to 11.

Images