Image frame insertion method and device, terminal equipment, storage medium and program product

Abstract

The invention discloses an image frame insertion method and device, terminal equipment, a storage medium and a program product, and the method comprises the steps: determining a motion vector and dynamic object information between a first image and a second image, and determining depth information and a semitransparent map corresponding to the second image; wherein the first image is a frame of image before the second image; constructing an intermediate offset frame according to the motion vector, the dynamic object information and the depth information; determining a first color reference image in the first image and the second image according to the semitransparent map; wherein the first color reference image is used for carrying out color information sampling on the semitransparent object in the middle offset frame; and determining first color information corresponding to the semitransparent object according to the first color reference image, and determining a frame insertion image between the first image and the second image according to the first color information.

Description

Technical Field

[0001] This invention relates to the field of image frame interpolation technology, and in particular to an image frame interpolation method and apparatus, terminal equipment, storage medium and program product. Background Technology

[0002] Frame interpolation can effectively improve the smoothness and realism of videos or animations. The basic principle of frame interpolation is to insert an extra image between two adjacent frames, thus making the video or animation play more smoothly.

[0003] However, in the actual frame interpolation process, if the motion of the semi-transparent object is not processed, the semi-transparent object in the intermediate frame generated by the interpolation will exhibit significant jitter that does not match the actual motion, making the image containing the semi-transparent object completely unacceptable.

[0004] Because common technologies lack image interpolation methods for semi-transparent objects that can be applied to mobile terminals, there is a problem that the obtained interpolated images do not match the actual motion, which greatly reduces the performance and effect of image interpolation processing. Summary of the Invention

[0005] This application provides an image frame interpolation method and apparatus, terminal device, storage medium and program product, which can solve the problem that the interpolated image does not match the actual motion, and greatly improve the performance and effect of image frame interpolation processing.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide an image frame interpolation method, the method comprising:

[0008] Determine the motion vectors and dynamic object information between the first image and the second image, and determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image;

[0009] Based on the motion vector, the dynamic object information, and the depth information, an intermediate offset frame is constructed;

[0010] Based on the semi-transparent texture, a first color reference image is determined in the first image and the second image; wherein, the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame;

[0011] The first color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated image between the first image and the second image is determined based on the first color information.

[0012] Secondly, embodiments of this application provide an image frame interpolation apparatus, which includes: a determining unit and a constructing unit.

[0013] The determining unit is used to determine the motion vector and dynamic object information between the first image and the second image, and to determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image;

[0014] The construction unit is used to construct an intermediate offset frame based on the motion vector, the dynamic object information, and the depth information;

[0015] The determining unit is further configured to determine a first color reference image in the first image and the second image based on the semi-transparent texture; wherein the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame; determine the first color information corresponding to the semi-transparent object based on the first color reference image, and determine the interpolated frame image between the first image and the second image based on the first color information.

[0016] Thirdly, embodiments of this application provide a terminal device, the terminal device comprising: a processor and a memory; wherein,

[0017] The memory is used to store computer programs that can run on the processor;

[0018] The processor is configured to execute the method as described in the first aspect when running the computer program.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0020] Fifthly, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in the first aspect above.

[0021] This application provides an image frame interpolation method, apparatus, terminal device, storage medium, and program product. It determines motion vectors and dynamic object information between a first image and a second image, and determines depth information and a semi-transparent texture corresponding to the second image. The first image is a frame preceding the second image. An intermediate offset frame is constructed based on the motion vectors, dynamic object information, and depth information. A first color reference image is determined between the first and second images based on the semi-transparent texture. This first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame. First color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated frame image between the first and second images is determined based on the first color information. In other words, in this application, based on the construction of the intermediate offset frame between two frames, the color information corresponding to the semi-transparent object can be further determined by acquiring and using the semi-transparent texture of the subsequent frame image. This allows for full utilization of the color information corresponding to the semi-transparent object during the generation of the interpolated frame image, ensuring that the final interpolated frame image reflects the true state of the semi-transparent object. This solves the problem of the interpolated frame image not matching the actual motion, greatly improving the performance and effect of image frame interpolation processing. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the implementation process of the image frame interpolation method proposed in the embodiments of this application. Figure 1 ;

[0023] Figure 2 This is a schematic diagram of the implementation process of the image frame interpolation method proposed in the embodiments of this application. Figure 2 ;

[0024] Figure 3A This is a schematic representation of the dynamic region proposed in the embodiments of this application. Figure 1 ;

[0025] Figure 3B This is a schematic representation of the dynamic region proposed in the embodiments of this application. Figure 2 ;

[0026] Figure 3C This is a schematic diagram illustrating the modification of MV proposed in the embodiments of this application;

[0027] Figure 4 This is a schematic diagram (3) illustrating the implementation process of the image frame interpolation method proposed in this application.

[0028] Figure 5 This is a schematic diagram illustrating the implementation of the motion vector-based frame interpolation method proposed in this application. Figure 1 ;

[0029] Figure 6This is a schematic diagram illustrating the implementation of the motion vector-based frame interpolation method proposed in this application. Figure 2 ;

[0030] Figure 7 This is a schematic diagram of the implementation process of the image frame interpolation method proposed in the embodiments of this application. Figure 4 ;

[0031] Figure 8 This is a schematic diagram of the composition structure of the image frame interpolation device proposed in the embodiments of this application.

[0032] Figure 9 This is a schematic diagram of the composition structure of the terminal device proposed in the embodiments of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0034] Frame interpolation mainly includes motion prediction and motion compensation, which can effectively improve the smoothness and realism of videos or animations. The basic principle of frame interpolation is to insert an extra image between two adjacent frames, thereby making the video or animation smoother during playback.

[0035] Frame interpolation involves inserting additional images between two adjacent keyframes to create smoother video or animation effects. Frame interpolation is widely used in film, television, animation, and games. In these fields, it allows for the creation of more realistic and engaging animation effects. For example, in film production, frame interpolation can be used to create special effects scenes such as explosions and volcanic eruptions. In game development, it can be used to create character animations such as jumping and running.

[0036] There are many ways to implement frame interpolation, the most common being manual and automatic interpolation. Manual interpolation involves manually drawing each image and then layering it onto adjacent frames one by one to create the animation effect. This method produces very detailed animation, but it is time-consuming and labor-intensive. Automatic interpolation, on the other hand, uses computer algorithms to automatically generate intermediate frames, thus shortening the production cycle and reducing costs, while improving the speed and quality of the animation. This method leverages the computing speed and algorithm optimization of computers to quickly generate high-quality animation effects.

[0037] Currently, a common frame interpolation method can use a combination of motion vector (MV) and optical flow to generate intermediate frames, and uses a large number of computational methods to accelerate the operation.

[0038] Since semi-transparent (or transparent) objects lack both depth and dynamic range (MV) information, optical flow can typically be used to calculate their offset between frames for such objects. This offset is then combined with that of an opaque object whose offset is calculated using MV to obtain the final interpolated image. However, this frame interpolation method for semi-transparent (or transparent) objects is computationally complex and consumes a lot of power, making it unsuitable for mobile terminals.

[0039] In actual frame interpolation, if the motion of a semi-transparent object is not processed, the semi-transparent object in the intermediate frame generated by the interpolation will exhibit significant jitter that does not match the actual motion, making the image containing the semi-transparent object completely unacceptable.

[0040] In other words, common technologies lack image interpolation methods for semi-transparent objects that can be applied to mobile terminals, resulting in the problem that the obtained interpolated images do not match the actual motion, which greatly reduces the performance and effect of image interpolation processing.

[0041] To address the aforementioned issues, in the embodiments of this application, based on the construction of the intermediate offset frame between two consecutive images, the color information corresponding to the semi-transparent object can be further determined by acquiring and using the semi-transparent texture of the subsequent frame image. This allows for full utilization of the color information corresponding to the semi-transparent object during the generation of the interpolated image, ensuring that the final interpolated image reflects the true state of the semi-transparent object. This solves the problem of the interpolated image not matching the actual motion, significantly improving the performance and effectiveness of image interpolation processing.

[0042] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0043] One embodiment of this application provides an image frame interpolation method, which can be applied to an image frame interpolation device or a terminal device, and this application does not specifically limit it. Below, using an image frame interpolation device as an example, the image frame interpolation method proposed in this application embodiment will be described by way of example.

[0044] Furthermore, in the embodiments of the application, Figure 1 This is a schematic diagram of the implementation process of the image frame interpolation method proposed in the embodiments of this application. Figure 1 ,like Figure 1 As shown, the image frame interpolation method may include the following steps:

[0045] Step 101: Determine the motion vector and dynamic object information between the first image and the second image, and determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image.

[0046] In the embodiments of this application, the image interpolation device can first determine the motion vector and dynamic object information between the first image and the second image, and at the same time, it can determine the depth information and semi-transparent texture corresponding to the second image.

[0047] It should be noted that, in the embodiments of this application, the first image can be a frame image preceding the second image, wherein the first image and the second image can be two adjacent frames, for example, the first image can be a previous frame image and the second image can be a subsequent frame image; the first image and the second image can also be two non-adjacent frames, for example, the first image can be a third frame image and the second image can be a fifth frame image; this application does not impose any specific limitations.

[0048] It is understood that, in the embodiments of this application, the first image and the second image can be two consecutive frames in a video sequence, two consecutive frames in a game animation, or other forms of two consecutive frames; this application does not impose any specific limitations.

[0049] It should be noted that, in the embodiments of this application, the motion vector between the first image and the second image can represent the positional offset between these two frames, describing the change from the first image to the second image. Here, MV can be at the pixel level, meaning each pixel position can correspond to one MV.

[0050] It is understood that in the embodiments of this application, MV may include motion vectors corresponding to different coordinate components. For example, for two-dimensional coordinates (x, y), MV may include motion vectors of the x component and motion vectors of the y component, namely MVx and MVy, respectively.

[0051] It should be noted that, in the embodiments of this application, the dynamic object information between the first image and the second image can be used to determine the dynamic object portion from the first image to the second image, that is, it can be used to determine the specific location of the dynamic object or dynamic target. The dynamic object information can be a moving mask.

[0052] It is understood that, in the embodiments of this application, the Moving Mask can mark the positions of dynamic objects such as people, vehicles, and props in the image content. The value of the indication information corresponding to each pixel in the Moving Mask can be a first value or a second value, wherein the first value indicates that the pixel does not belong to a dynamic object, and the second value indicates that the pixel belongs to a dynamic object.

[0053] It should be noted that, in the embodiments of this application, the depth information corresponding to the second image can be used to determine the distance between pixels in the second image and the shooting lens. The depth information can be at the pixel level, meaning that each pixel position can correspond to a single depth information.

[0054] It should be noted that, in the embodiments of this application, the semi-transparent texture corresponding to the second image can be used to determine the semi-transparent information of semi-transparent objects in the second image. The semi-transparent information of the semi-transparent objects can be at the pixel level, meaning that each pixel position can correspond to one piece of semi-transparent information.

[0055] It is understood that, in the embodiments of this application, the semi-transparency information of a pixel location may include at least the pixel values ​​of different channels corresponding to that pixel location (e.g., pixels of RGB channels) and transparency (alpha).

[0056] Furthermore, in the embodiments of this application, the image interpolation device can employ various methods to obtain motion vectors and dynamic object information between the first image and the second image, as well as depth information and semi-transparent textures corresponding to the second image. For example, the image interpolation device can identify motion vectors (MV), dynamic object information, depth information, and semi-transparent textures through shader tags and / or render target (RT) tag information. Other methods may also be used, and this application does not impose specific limitations.

[0057] Step 102: Construct intermediate offset frames based on motion vectors, dynamic object information, and depth information.

[0058] In the embodiments of this application, after determining the motion vector and dynamic object information between the first image and the second image, as well as the depth information and semi-transparent texture corresponding to the second image, the intermediate offset frame can be constructed based on the motion vector, dynamic object information and depth information.

[0059] It should be noted that, in the embodiments of this application, the intermediate offset frame can be understood as a frame of image between the first image and the second image that includes the pixel positions of dynamic objects. The intermediate offset frame is determined using the motion vector and dynamic object information between the first image and the second image.

[0060] It is understood that, in the embodiments of this application, the intermediate offset frame can be understood as an offset map between the first image and the second image initially constructed, wherein the intermediate offset frame only includes the position information of the pixels and does not include the color information of the pixels.

[0061] Furthermore, in the embodiments of this application, Figure 2 This is a schematic diagram of the implementation process of the image frame interpolation method proposed in the embodiments of this application. Figure 2 ,like Figure 2 As shown, the method for constructing intermediate offset frames based on motion vectors, dynamic object information, and depth information by an image frame interpolation device may include the following steps:

[0062] Step 201: Determine the corrected motion vector based on the motion vector and dynamic object information.

[0063] In the embodiments of this application, after determining the motion vector and dynamic object information between the first image and the second image, the image interpolation device can first determine the corrected motion vector based on the motion vector and dynamic object information.

[0064] It should be noted that, in the embodiments of this application, dynamic object information can be used to correct the motion vector. Specifically, the dynamic object part (such as a dynamic region) indicated by the motion vector can be determined first based on the motion vector, and then the dynamic object part indicated by the motion vector can be compared with the dynamic object information. If the two are different or inconsistent, the dynamic object information can be used to correct the motion vector, thereby obtaining the corrected motion vector.

[0065] It is understood that in the embodiments of this application, the motion vector on one of the channels can reflect whether the main area of ​​the dynamic object in the image is offset, but it cannot accurately reflect whether the edge of the dynamic object is offset. Therefore, the offset reflected may deviate from the actual offset, and subsequent correction processing is required.

[0066] It is understood that in the embodiments of this application, the accuracy of the obtained dynamic object information is relatively high, and it can more realistically reflect the real position of the dynamic object (dynamic object or dynamic target). Therefore, the motion vector can be corrected according to the dynamic object information, and the finally obtained corrected motion vector can accurately reflect the actual offset of the two frames of images.

[0067] For example, in some embodiments, when correcting the motion vector according to the dynamic object information, if the size of the dynamic region represented by the Moving Mask and the MV is the same, then the MV is directly used as the corrected MV (corrected motion vector); if the size of the dynamic region represented by the Moving Mask and the MV is different, then the deviation between the Moving Mask and the MV is further determined, and the offset information of the deviation part of the MV is adjusted to obtain the corrected MV.

[0068] For example, in some embodiments, the size of the dynamic region represented by the modified MV is greater than or equal to the size of the dynamic region represented by the MV. Here, the dynamic region can be understood as the region where the dynamic object is located.

[0069] For example, in some embodiments, if the size of the dynamic region represented by the Moving Mask is equal to the size of the dynamic region represented by the MV, it indicates that the dynamic region represented by the MV is accurate. If the size of the dynamic region represented by the Moving Mask is greater than the size of the dynamic region represented by the MV, it indicates that the dynamic region represented by the MV is inaccurate. In this case, the value of the MV is adjusted based on the Moving Mask so that the size of the dynamic region represented by the MV is the same as the size of the dynamic region represented by the Moving Mask.

[0070] Exemplary, in some embodiments, Figure 3A This is a schematic representation of the dynamic region proposed in the embodiments of this application. Figure 1 , Figure 3B This is a schematic representation of the dynamic region proposed in the embodiments of this application. Figure 2 The dynamic region represented by MV is as follows Figure 3A or Figure 3B As shown in 300, the dynamic region represented by the Moving Mask is as follows: Figure 3B As shown in 310. Figure 3C This is a schematic diagram of the correction of the MV proposed in an embodiment of this application. If the size of the dynamic region represented by the Moving Mask is larger than the size of the dynamic region represented by the MV, then there will be a deviation portion 320. In this case, the MV can be corrected based on the deviation portion 320 so that the size of the dynamic region represented by the MV is the same as the size of the dynamic region represented by the Moving Mask. The resulting corrected dynamic region represented by the MV is as follows: Figure 3C As shown in Figure 330.

[0071] It should be noted that, in the embodiments of this application, the offset information included for each pixel in the MV includes: horizontal offset, vertical offset, and indication information indicating whether it belongs to a dynamic object. For each pixel in the MV, information from the following three channels is included to indicate the offset information of the corresponding pixel: a first channel indicating the horizontal offset, a second channel indicating the vertical offset, and a third channel indicating dynamic object information. For pixels in the compensation portion, the value of their offset information is set to the offset information of the pixel closest to that pixel that belongs to the dynamic region.

[0072] It should be noted that, in the embodiments of this application, the three channels of each pixel in MV can be understood as r channel, g channel and b channel, wherein the r channel and g channel store horizontal and vertical offset information, and the b channel is used to indicate whether it belongs to a dynamic object.

[0073] It is understood that, in the embodiments of this application, MV may include motion vectors corresponding to different coordinate components, and correspondingly, the modified motion vector may also include motion vectors corresponding to different coordinate components.

[0074] Step 202: Determine the offset parameters based on the corrected motion vector.

[0075] In the embodiments of this application, after determining the corrected motion vector based on the motion vector and dynamic object information, the offset parameter can be further determined based on the corrected motion vector.

[0076] It is understood that, in the embodiments of this application, the corrected motion vector may include motion vectors corresponding to different coordinate components. For example, for two-dimensional coordinates (x, y), the corrected motion vector is denoted as MV_final, which may include the motion vector of the x component and the motion vector of the y component, namely MV_final.x and MV_final.y, respectively.

[0077] It should be noted that, in the embodiments of this application, the offset parameter can also be understood as a motion vector. The offset parameter can represent the positional offset between the intermediate offset frame and the second image, and is used to describe the change from the intermediate offset frame to the second image.

[0078] Furthermore, in the embodiments of this application, when determining the offset parameter based on the modified motion vector, the final offset parameter can be calculated in conjunction with the offset coefficient. For example, the product or quotient between the modified motion vector and the offset coefficient can be used as the corresponding offset parameter. Of course, other calculations can also be performed based on the modified motion vector and the offset coefficient to obtain the offset parameter.

[0079] For example, in some embodiments, assuming the offset parameter is represented as offset, where the offset parameter of the x component can be denoted as offset.x, the offset parameter of the y component can be denoted as offset.y, the corrected motion vectors of the corresponding components can be represented as MV_final.x and MV_final.y, and the offset coefficient can be represented as a, then the offset parameter of the corresponding component can be calculated using the following formula:

[0080]

[0081]

[0082] It should be noted that, in the embodiments of this application, the offset coefficient can be any pre-set value greater than 0. For example, corresponding to the above formula, the value of the offset coefficient 'a' can be pre-set to 2.

[0083] Step 203: Construct intermediate offset frames based on offset parameters, dynamic object information, and depth information.

[0084] In the embodiments of this application, after determining the offset parameters based on the corrected motion vector, an intermediate offset frame can be constructed further based on the offset parameters, dynamic object information, and depth information.

[0085] Furthermore, in the embodiments of this application, during the construction of the intermediate offset frame, the offset parameter can be referenced to determine the offset pixel position corresponding to any pixel position in the second image, that is, the second image is sampled according to the offset parameter.

[0086] For example, in some embodiments, assuming that one of the pixel positions in the second image is (x0, y0), the offset parameters corresponding to the pixel position include the offset parameter offset.x of the x component and the offset parameter offset.y of the y component. Then, based on the offset parameters, the corresponding offset coordinates can be determined as (x0-offset.x, y0-offset.y).

[0087] Furthermore, in the embodiments of this application, during the construction of the intermediate offset frame, the offset parameters, depth information, and dynamic object information corresponding to any pixel position in the second image can be used as the offset parameters, depth information, and dynamic object information corresponding to the corresponding offset pixel position. For example, the offset parameters, depth information, and dynamic object information corresponding to any pixel position in the second image can be packaged into data, and then the packaged data can be stored as data of the offset pixel position determined based on the offset parameters.

[0088] It should be noted that, in the embodiments of this application, if multiple different pixel positions in the second image are the same pixel position after offsetting based on the corresponding offset parameters, then it is necessary to compare the specific data corresponding to these multiple pixel positions, obtain the comparison result, and then determine which pixel position's packaged data should be stored in the offset pixel position's data based on the comparison result.

[0089] For example, in some embodiments, during the comparison of specific data at different pixel locations, one or more of the following data can be selected for comparison: offset parameters, depth information, and dynamic object information. Specifically, for depth information and dynamic object information, dynamic object information can be compared first. If it cannot be determined from the dynamic object information which pixel location's corresponding packaged data should be stored as the data at the offset pixel location, then the depth information is further compared to obtain the comparison result.

[0090] In other words, in the embodiments of this application, during the data comparison process, the priority of depth information is lower than the priority of dynamic object information.

[0091] It should be noted that, in the embodiments of this application, during the construction of the intermediate offset frame, if there is a pixel position that does not have a corresponding original pixel position (pixel position before offset), then there is no corresponding data for that pixel position. In this case, it is possible to directly store the specific data of the corresponding pixel position in the second image as the data of that pixel position.

[0092] Step 103: Determine a first color reference image in the first image and the second image based on the semi-transparent texture; wherein, the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame.

[0093] In the embodiments of this application, after constructing an intermediate offset frame based on motion vectors, dynamic object information, and depth information, a first color reference image corresponding to the semi-transparent object in the intermediate offset frame can be further determined in the first image and the second image based on the semi-transparent texture.

[0094] It should be noted that, in the embodiments of this application, the first color reference image can be used to determine the color of the semi-transparent object in the intermediate offset frame. The semi-transparent object in the intermediate offset frame can be determined using the semi-transparent texture of the second image.

[0095] It should be noted that, in the embodiments of this application, a semi-transparent object can be understood as an object or target with high transparency. The transparency of the semi-transparent object in the intermediate offset frame can be determined by sampling the semi-transparent texture of the second image. The smaller the value of the transparency alpha, the higher the transparency; the minimum value can be considered completely transparent. Conversely, the larger the value of the transparency alpha, the lower the transparency; the maximum value can be considered completely opaque.

[0096] It is understood that, in the embodiments of this application, when determining the first color reference image corresponding to the semi-transparent object in the intermediate offset frame based on the semi-transparent texture in the first image and the second image, for the first pixel position in the intermediate offset frame, the first grayscale indicator parameter corresponding to the second pixel position in the second image and the second grayscale indicator parameter corresponding to the third pixel position in the second image can be determined first based on the semi-transparent texture; then, the first color reference image is determined in the first image and the second image based on the first grayscale indicator parameter and the second grayscale indicator parameter.

[0097] It should be noted that, in the embodiments of this application, the grayscale indication information is an indication value determined based on the grayscale value corresponding to the pixel position, representing whether the grayscale value exceeds the grayscale threshold. The grayscale indication information can be a binary numerical value; for example, the value of the grayscale indication information can be 0 or 1.

[0098] It should be noted that, in the embodiments of this application, the preset grayscale threshold can be used to determine grayscale indication information. The value of the grayscale threshold can be any value greater than 0, and this application does not impose any specific limitation.

[0099] It should be noted that, in the embodiments of this application, the first pixel position can be any pixel position in the intermediate offset frame, and the second pixel position can be a pixel position in the second image that corresponds to the first pixel position. That is, the coordinate position of the second pixel position in the second image is the same as the coordinate position of the first pixel position in the intermediate offset frame, i.e., the coordinate value of the first pixel position is equal to the coordinate value of the second pixel position.

[0100] It should be noted that, in the embodiments of this application, the second pixel position is the pixel position corresponding to the third pixel position after offsetting it according to the offset parameter. That is to say, after offsetting the third pixel position based on the offset parameter, the corresponding pixel position determined can be the second pixel position.

[0101] Accordingly, in the embodiments of this application, the third pixel position and the first pixel position have an inter-frame reference positional relationship. The image information corresponding to the third pixel position can be used to determine the image information corresponding to the first pixel position.

[0102] For example, in some embodiments, the first pixel position can be represented as (x1, y1), that is, the first pixel position can be the pixel position with coordinates (x1, y1) in the intermediate offset frame. The second pixel position can also be represented as (x1, y1), that is, the second pixel position can be the pixel position with coordinates (x1, y1) in the second image. The third pixel position can be represented as (x1+offset.x, y1+offset.y), that is, the third pixel position can be the pixel position with coordinates (x1+offset.x, y1+offset.y) in the second image. The offset parameters corresponding to the pixel position include the offset parameter offset.x of the x component and the offset parameter offset.y of the y component.

[0103] Furthermore, in the embodiments of this application, when determining the first grayscale indicator parameter corresponding to the second pixel position in the second image and the second grayscale indicator parameter corresponding to the third pixel position in the second image based on the semi-transparent texture, the grayscale value corresponding to the second pixel position can be determined based on the semi-transparent texture, and the first grayscale indicator parameter can be determined based on the first grayscale value; at the same time, the grayscale value corresponding to the third pixel position can be determined based on the semi-transparent texture, and the second grayscale indicator parameter can be determined based on the second grayscale value.

[0104] It should be noted that, in the embodiments of this application, when determining the grayscale indicator parameter for any pixel position in the second image, the grayscale value corresponding to the pixel position can be determined first by using a semi-transparent texture, and then the grayscale value can be compared with a preset grayscale threshold, so that the corresponding grayscale indicator parameter can be further determined based on the comparison result.

[0105] It is understood that, in the embodiments of this application, the corresponding grayscale value (grayscale value) can be calculated through the RGB channel of the semi-transparent texture.

[0106] Furthermore, in the embodiments of this application, when determining the corresponding grayscale indicator parameter based on the comparison result, if the grayscale value is greater than a preset grayscale threshold, the value of the grayscale indicator parameter can be set to a first value; if the grayscale value is less than or equal to the preset grayscale threshold, the value of the grayscale indicator parameter can be set to a second value. The first value and the second value are different; for example, the first value is 1 and the second value is 0. This application does not impose specific limitations on this.

[0107] For example, in some embodiments, the first grayscale indicator parameter corresponding to the second pixel position in the second image and the second grayscale indicator parameter corresponding to the third pixel position in the second image can be determined according to the method described above. Here, bLum1 can be used to represent the first grayscale indicator parameter, and bLum2 can be used to represent the second grayscale indicator parameter.

[0108] Furthermore, in the embodiments of this application, when determining the first color reference image in the first image and the second image according to the first grayscale indicator parameter and the second grayscale indicator parameter, if the first grayscale indicator parameter is less than the second grayscale indicator parameter, the first image can be selected as the first color reference image; if the first grayscale indicator parameter is greater than or equal to the second grayscale indicator parameter, the second image can be selected as the first color reference image.

[0109] It is understood that, in the embodiments of this application, after comparing the first grayscale indicator parameter and the second grayscale indicator parameter, if the first grayscale indicator parameter is less than the second grayscale indicator parameter, i.e., bLum1 < bLum2, for example, bLum1 is 0 and bLum2 is 1, then it can be considered that the relevant data corresponding to the semi-transparent object in the second image is not accurate and cannot provide a more accurate reference for the color information of the semi-transparent object in the intermediate offset frame. At this time, the first image can be selected as the first color reference image.

[0110] It is understood that, in the embodiments of this application, after comparing the first grayscale indicator parameter and the second grayscale indicator parameter, if the first grayscale indicator parameter is greater than or equal to the second grayscale indicator parameter, i.e., bLum1≥bLum2, for example, bLum1 is 1 and bLum2 is 0, then it can be considered that the relevant data corresponding to the semi-transparent object in the second image is relatively accurate, and can provide a more accurate reference for the color information of the semi-transparent object in the intermediate offset frame. At this time, the second image can be selected as the first color reference image.

[0111] Step 104: Determine the first color information corresponding to the semi-transparent object based on the first color reference image, and determine the interpolated image between the first image and the second image based on the first color information.

[0112] In the embodiments of this application, after determining the first color reference image corresponding to the semi-transparent object in the intermediate offset frame based on the semi-transparent texture in the first image and the second image, the first color information corresponding to the semi-transparent object can be further determined based on the first color reference image, and the interpolated frame image between the first image and the second image can be determined based on the first color information.

[0113] It should be noted that, in the embodiments of this application, after determining the first color reference image to be referenced, the first color reference image can be further sampled using the offset parameter, so as to obtain the color information corresponding to each pixel position of the semi-transparent object, thereby determining the first color information corresponding to the semi-transparent object.

[0114] Furthermore, in the embodiments of this application, Figure 4 This is a schematic diagram (3) illustrating the implementation process of the image frame interpolation method proposed in this application. Figure 4 As shown, after constructing the intermediate offset frame based on the motion vector, dynamic object information, and depth information, i.e., after step 102, the image frame interpolation method may further include the following steps:

[0115] Step 105: Determine the first depth information corresponding to the first pixel position and the second depth information corresponding to the third pixel position.

[0116] In the embodiments of this application, after constructing an intermediate offset frame based on motion vectors, dynamic object information, and depth information, for the first pixel position in the intermediate offset frame, the first depth information corresponding to the first pixel position can be determined first, and the second depth information corresponding to the third pixel position in the second image can be determined simultaneously.

[0117] It should be noted that, in the embodiments of this application, after the construction of the intermediate offset frame is completed, each pixel position corresponds to data such as offset parameters, depth information and dynamic object information. At this time, for any pixel position in the intermediate offset frame, such as the first pixel position, the first depth information stored therein can be read, denoted as depth1. At the same time, the depth information of the pixel position before the offset corresponding to the first pixel position in the second image can also be read, that is, the second depth information corresponding to the third pixel position is determined, denoted as depth0.

[0118] Step 106: Based on the first depth information and the second depth information, determine the second color reference image corresponding to the non-transparent object in the intermediate offset frame in the first image and the second image.

[0119] In the embodiments of this application, after determining the first depth information corresponding to the second pixel position and the second depth information corresponding to the third pixel position, a second color reference image corresponding to the non-transparent object in the intermediate offset frame can be further determined in the first image and the second image based on the first depth information and the second depth information.

[0120] It should be noted that, in the embodiments of this application, the second color reference image can be used to determine the color of non-transparent objects in the intermediate offset frame.

[0121] Furthermore, in the embodiments of this application, when determining the second color reference image corresponding to the non-transparent object in the intermediate offset frame in the first image and the second image based on the first depth information and the second depth information, mathematical operations can be performed based on the first depth information and the second depth information, and then the second color reference image corresponding to the non-transparent object in the intermediate offset frame can be further determined based on the operation result.

[0122] It should be noted that, in the embodiments of this application, the specific form of mathematical operation based on the first depth information and the second depth information is not limited. For example, the difference operation can be performed on the first depth information and the second depth information, or the division operation can be performed on the first depth information and the second depth information. This application does not impose any specific limitations.

[0123] For example, in some embodiments, when performing mathematical operations based on the first depth information and the second depth information, one possible time-limiting method is to calculate the difference between the second depth information and the first depth information, and then compare the difference between the two with a preset difference threshold.

[0124] It should be noted that, in the embodiments of this application, the preset difference threshold can be used to determine the degree of difference between depth values. The value of the difference threshold can be any value greater than 0, and this application does not impose any specific limitation.

[0125] Accordingly, in the embodiments of this application, if the difference is less than the difference threshold, then it can be considered that the depth difference before and after the offset is not significant, and thus it can be determined that the data of the second image is relatively accurate. Therefore, the second image can be selected as the second color reference image.

[0126] Accordingly, in the embodiments of this application, if the difference is greater than or equal to the difference threshold, it can be considered that the depth difference before and after the offset is large, and thus it can be determined that the data of the second image is not accurate. Therefore, the first image can be selected as the second color reference image.

[0127] Step 107: Determine the second color information corresponding to the non-transparent object based on the second color reference image.

[0128] In the embodiments of this application, after determining the second color reference image corresponding to the non-transparent object in the intermediate offset frame in the first image and the second image based on the first depth information and the second depth information, the second color information corresponding to the non-transparent object can be further determined based on the second color reference image.

[0129] It should be noted that, in the embodiments of this application, after determining the second color reference image to be referenced, the second color reference image can be further sampled using the offset parameter, so as to obtain the color information corresponding to each pixel position of the non-transparent object, thereby determining the second color information corresponding to the non-transparent object.

[0130] It should be noted that, in the embodiments of this application, after determining the first color information corresponding to the semi-transparent object and the second color information corresponding to the non-transparent object, the first color information and the second color information can be combined to further determine the interpolated image between the first image and the second image.

[0131] Furthermore, in the embodiments of this application, when determining the interpolated image between the first image and the second image based on the first color information, for any pixel position in the intermediate offset frame, the composite color information corresponding to the pixel position is determined based on the transparency parameter corresponding to the pixel position, the first color information and the second color information corresponding to the pixel position; all pixel positions in the intermediate offset frame are traversed in the above manner, and the interpolated image is determined based on all the composite color information corresponding to all pixel positions.

[0132] It should be noted that, in the embodiments of this application, for any pixel position, the transparency parameter corresponding to the pixel position can be determined first, and then the transparency parameter can be used to synthesize the first color information and the second color information corresponding to the pixel position. That is, the first color information corresponding to the semi-transparent object and the second color information corresponding to the non-transparent object are fused based on the transparency parameter, so as to determine the final synthesized color information corresponding to the pixel position.

[0133] It is understood that, in the embodiments of this application, by using the transparency parameter corresponding to each pixel position to synthesize the first color information and the second color information corresponding to each pixel position in the manner described above, and obtaining the synthesized color information corresponding to each pixel position, the interpolated image between the first image and the second image can be determined.

[0134] It should be noted that, in the embodiments of this application, when using the transparency parameter to synthesize the first color information and the second color information, various methods can be used to perform mathematical operations on the transparency parameter, the first color information, and the second color information. The embodiments of this application do not specifically limit the operation method.

[0135] For example, in some embodiments, assuming the transparency parameter is alpha, the first color information corresponding to a semi-transparent object is color1, and the second color information corresponding to a non-transparent object is color1, the corresponding composite color information finalcolor can be determined according to the following formula:

[0136] finalcolor=color1×alpha+color2×(1-alpha) (3)

[0137] It should be noted that in the embodiments of the present application, the transparency parameter corresponding to a pixel position can be determined according to the transparency (alpha) corresponding to the pixel position. Among them, preprocessing of the transparency can be selected to obtain the corresponding transparency parameter.

[0138] Furthermore, in the embodiments of the present application, for any pixel position in the intermediate offset frame, the transparency corresponding to the pixel position can be determined according to the semi-transparent texture map; then the transparency is preprocessed to determine the transparency parameter corresponding to the pixel position.

[0139] It should be noted that in the embodiments of the present application, the specific method of transparency preprocessing is not limited.

[0140] Exemplarily, in some embodiments, when preprocessing the transparency, an achievable method is to complete the preprocessing through the rule alpha = alpha < c? 0.0 : (alpha - c) / (1 - c), that is, if alpha < c, then the value of alpha can be set to 0, otherwise, the value of alpha is set to (alpha - c) / (1 - c). Among them, c can be a pre-set transparency parameter with different values. If the value range of the transparency is [0, 1], then c is a value greater than 0 and less than 1.

[0141] In summary, through the image interpolation method proposed in the above steps 101 to 107, during the generation of the interpolated image, for the semi-transparent object lacking depth information and motion vectors, the color information of the semi-transparent object can be determined by using the semi-transparent texture map corresponding to the second image, so as to fully utilize the color information corresponding to the semi-transparent object to determine the interpolated image, so that the semi-transparent object will not produce incorrect jitter, enabling the algorithm to be applied in actual projects without unacceptable semi-transparent object defects, and greatly improving the interpolation effect.

[0142] It should be noted that the image interpolation method proposed in the embodiments of the present application can also be applied to other scenarios that require generating new image frames. For example, it can also be applied to super-resolution processing, so that super-resolution can correctly process semi-transparent images and obtain a higher super-resolution effect.

[0143] This application provides an image frame interpolation method, which determines motion vectors and dynamic object information between a first image and a second image, and determines depth information and a semi-transparent texture corresponding to the second image. The first image is a frame preceding the second image. An intermediate offset frame is constructed based on the motion vectors, dynamic object information, and depth information. A first color reference image is determined between the first image and the second image based on the semi-transparent texture. The first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame. First color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated frame image between the first image and the second image is determined based on the first color information. In other words, in this application, based on the construction of the intermediate offset frame between two frames, the color information corresponding to the semi-transparent object can be further determined by acquiring and using the semi-transparent texture of the subsequent frame image. This allows for full utilization of the color information corresponding to the semi-transparent object during the generation of the interpolated frame image, ensuring that the final interpolated frame image reflects the true state of the semi-transparent object. This solves the problem of the interpolated frame image not matching the actual motion, greatly improving the performance and effect of image frame interpolation.

[0144] Based on the above embodiments, another embodiment of this application proposes an image frame interpolation method. When generating frames using a motion vector-based 3D frame interpolation scheme, it can prevent semi-transparent objects from producing erroneous jitters if the MV and depth cannot be obtained. This allows the algorithm to be applied in actual projects without producing unacceptable defects in semi-transparent objects, greatly improving the frame interpolation effect.

[0145] It should be noted that, in the examples of this application, Figure 5 This is a schematic diagram illustrating the implementation of the motion vector-based frame interpolation method proposed in this application. Figure 1 ,like Figure 5 As shown, the main steps in image frame interpolation are as follows: correcting the MV, constructing an intermediate frame offset map (intermediate offset frame), generating color information sampling rules for the preceding and following frames, and using the rules to sample colors for the preceding and following frames.

[0146] For example, in some embodiments, MovingMask (dynamic object information) and motion vector MV can be identified first through shader label and RT label information.

[0147] For example, in some embodiments, when correcting the MV, the dynamic object part in the MV and the MovingMask can be compared. Only if the two are equal or match can the comparison pass, and the MV that matches the MovingMask is directly marked as the final correct MV_final. If the two are not equal or match, the MovingMask needs to be used to correct the MV to obtain the corrected MV_final.

[0148] For example, in some embodiments, when constructing the intermediate frame offset map, the data can be processed. Taking the pixel position (x0, y0) in the later frame image (second image) as an example, the offset parameter offset is calculated according to offset = MV_final / offset coefficient, and the data is packaged together with the depth information depth and Moving Mask. The data is compared with the priority of Moving Mask being higher than that of depth. Finally, the packaged data is stored at the position (x0-offset.x, y0-offset.y) of the intermediate offset frame.

[0149] For example, in some embodiments, when generating color information, the sampling rules for the frames before and after sampling are processed based on the constructed intermediate frame offset map. Taking the pixel position (x1, y1) in the intermediate frame offset map as an example, the offset and depth1 (first depth information) stored at the pixel position (x1, y1) in the intermediate frame offset map are sampled. At the same time, the original depth0 (second depth information) is collected in the second image with (x1+offset.x, y1+offset.y) as coordinates. The depth0 and depth1 are compared. If depth1-depth0 is less than a given threshold (difference threshold), it is determined that the color information of the frame after sampling (second image) is used as the second color information corresponding to the non-transparent object, that is, the second image is used as the second color reference image. Otherwise, the color information of the frame before sampling (first image) is determined as the second color information corresponding to the non-transparent object, that is, the first image is used as the second color reference image.

[0150] For example, in some embodiments, after determining the color reference image (previous frame or next frame) to be sampled, the corresponding color information can be further sampled to generate the interpolated image.

[0151] Furthermore, in the examples of this application, Figure 6 This is a schematic diagram illustrating the implementation of the motion vector-based frame interpolation method proposed in this application. Figure 2 ,like Figure 6As shown, when performing image interpolation processing, considering semi-transparent objects (semi-transparent objects) lacking motion vectors and depth information, the following parts of processing can be carried out: correcting the MV, constructing an intermediate frame offset map (intermediate offset frame), generating rules for sampling the front and back frames of color information, using the rules to perform color sampling on the front and back frames, determining semi-transparent information, generating rules for sampling the color information of semi-transparent objects, using the rules to perform color sampling on semi-transparent objects, and fusing color information.

[0152] Exemplarily, in some embodiments, the semi-transparent texture map of the back frame can also be obtained through the shader tag and RT tag information.

[0153] Exemplarily, in some embodiments, when determining semi-transparent information, processing is carried out based on the constructed intermediate frame offset map. Taking the pixel position (x1, y1) as an example, the semi-transparent information at the current position is sampled, and its color grayscale value is calculated through the RGB channels of the semi-transparent texture map. Whether this grayscale value is greater than the specified threshold (grayscale threshold) is stored as bLum1, and the transparency alpha of the semi-transparent texture map is saved. At the same time, the semi-transparent information at this position is collected with (x1 + offset.x, y1 + offset.y) as the coordinates, and its color grayscale value is calculated through the RGB channels of the semi-transparent texture map. Whether this grayscale value is greater than the specified threshold is saved as bLum2.

[0154] Exemplarily, in some embodiments, when generating rules for sampling the color information of semi-transparent objects, if bLum2 > bLum1, it means that if the intermediate frame (intermediate offset frame) samples the back frame information according to offset, the semi-transparent object that should not exist at this position will be wrongly sampled. Therefore, at this time, the front frame needs to be sampled according to offset, and the sampled color is recorded as color1. If bLum2 <= bLum1, the intermediate frame can sample the back frame information according to offset, and the sampled color is recorded as color1.

[0155] Exemplarily, in some embodiments, when performing color information fusion, the color information corresponding to the non-transparent object is color2. For the current pixel position (x1, y1) of the intermediate offset frame, the merged color information can be obtained according to finalcolor = color1 × alpha + color2 × (1 - alpha). Among them, the transparency can be preprocessed through the rule alpha = alpha < c? 0.0 : (alpha - c) / (1 - c).

[0156] Exemplarily, in some embodiments, after completing color fusion, it can be further synthesized with the separated UI to generate an interpolated image.

[0157] Therefore, it can be seen that when generating frames using the MV-based 3D frame interpolation scheme, this scheme handles semi-transparent objects for which MV and depth cannot be obtained, preventing semi-transparent objects from producing incorrect jitter. This allows the algorithm to be used in real-world projects without producing unacceptable defects in semi-transparent objects, greatly improving the frame interpolation effect.

[0158] Furthermore, in the embodiments of the application, Figure 7 This is a schematic diagram of the implementation process of the image frame interpolation method proposed in the embodiments of this application. Figure 4 ,like Figure 7 As shown, the image frame interpolation method may include the following steps:

[0159] Step 701: Obtain motion vectors, dynamic object information, depth information, and semi-transparent textures.

[0160] In the embodiments of this application, motion vectors and dynamic object information (Moving Mask) between the first image and the second image, as well as the depth information and semi-transparent texture corresponding to the second image, can be identified through shader tags and RT tag information.

[0161] Step 702: Construct intermediate offset frames based on motion vectors, dynamic object information, and depth information.

[0162] In the embodiments of this application, the intermediate offset frame can be understood as a frame of image between the first image and the second image that includes the pixel positions of dynamic objects. The intermediate offset frame is determined using motion vectors and dynamic object information between the first image and the second image.

[0163] In the embodiments of this application, when constructing an intermediate offset frame, the corrected motion vector can be determined first based on the motion vector and dynamic object information, then the offset parameter can be determined based on the corrected motion vector, and finally the intermediate offset frame can be constructed based on the offset parameter, dynamic object information and depth information.

[0164] In the embodiments of this application, during the construction of the intermediate offset frame, the offset parameters, depth information, and dynamic object information corresponding to any pixel position in the second image can be used as the offset parameters, depth information, and dynamic object information corresponding to the corresponding offset pixel position. For example, the offset parameters, depth information, and dynamic object information corresponding to any pixel position in the second image can be packaged into data, and then the packaged data can be stored as data of the offset pixel position determined based on the offset parameters.

[0165] Step 703: Determine the second color reference image corresponding to the non-transparent object in the first and second images based on the depth information.

[0166] In embodiments of this application, the second color reference image can be used to determine the color of non-transparent objects in the intermediate offset frame.

[0167] In embodiments of this application, first depth information corresponding to the second pixel position and second depth information corresponding to the third pixel position can be determined. Then, based on the first and second depth information, a second color reference image corresponding to the non-transparent object in the intermediate offset frame can be determined in the first and second images. Specifically, mathematical operations can be performed based on the first and second depth information, and the second color reference image corresponding to the non-transparent object in the intermediate offset frame can be further determined based on the operation result.

[0168] Step 704: Determine the second color information corresponding to the non-transparent object based on the second color reference image.

[0169] In the embodiments of this application, after determining the second color reference image that needs to be referenced, the second color reference image can be further sampled using the offset parameter, so as to obtain the color information corresponding to each pixel position of the non-transparent object, thereby determining the second color information corresponding to the non-transparent object.

[0170] Step 705: Determine the first color reference image corresponding to the semi-transparent object in the first image and the second image based on the semi-transparent texture.

[0171] In embodiments of this application, the first color reference image can be used to determine the color of a semi-transparent object in an intermediate offset frame.

[0172] In an embodiment of this application, when determining the first color reference image corresponding to the semi-transparent object in the intermediate offset frame based on the semi-transparent texture in the first image and the second image, the first grayscale indicator parameter corresponding to the second pixel position and the second grayscale indicator parameter corresponding to the third pixel position can be determined firstly based on the semi-transparent texture; wherein, the second pixel position is the pixel position before the offset corresponding to the first pixel position; and then the first color reference image is determined in the first image and the second image based on the first grayscale indicator parameter and the second grayscale indicator parameter.

[0173] Step 706: Determine the first color information corresponding to the semi-transparent object based on the first color reference image.

[0174] In the embodiments of this application, after determining the first color reference image that needs to be referenced, the first color reference image can be further sampled using the offset parameter, so as to obtain the color information corresponding to each pixel position of the semi-transparent object, thereby determining the first color information corresponding to the semi-transparent object.

[0175] Step 707: Perform a synthesis process based on the first color information and the second color information to obtain the interpolated image.

[0176] In the embodiments of this application, when determining the interpolated image between the first image and the second image based on the first color information, for any pixel position in the intermediate offset frame, the composite color information corresponding to the pixel position is determined based on the transparency parameter corresponding to the pixel position, the first color information and the second color information corresponding to the pixel position; all pixel positions in the intermediate offset frame are traversed in the above manner, and the interpolated image is determined based on all the composite color information corresponding to all pixel positions.

[0177] In the embodiments of this application, for any pixel position, the transparency parameter corresponding to the pixel position can be determined first, and then the transparency parameter can be used to synthesize the first color information and the second color information corresponding to the pixel position. That is, the first color information corresponding to the semi-transparent object and the second color information corresponding to the non-transparent object are fused based on the transparency parameter, so as to determine the final synthesized color information corresponding to the pixel position.

[0178] In the embodiments of this application, the first color information and the second color information corresponding to each pixel position are synthesized using the transparency parameter corresponding to each pixel position in the manner described above, so as to obtain the synthesized color information corresponding to each pixel position, and the interpolated image between the first image and the second image can be determined.

[0179] In summary, the image interpolation method proposed in this application, during the generation of interpolated images, can determine the color information of semi-transparent objects lacking depth information and motion vectors by utilizing the semi-transparent texture corresponding to the second image. This fully utilizes the color information corresponding to the semi-transparent object to determine the interpolated image, preventing the semi-transparent object from producing erroneous jitter. This allows the algorithm to be applied in actual projects without producing unacceptable defects in the semi-transparent object, greatly improving the interpolation effect.

[0180] It should be noted that the image frame interpolation method proposed in this application embodiment can also be applied to other scenarios that require the generation of new image frames. For example, it can also be applied to super-resolution processing, thereby enabling super-resolution to correctly process semi-transparent images and obtain higher super-resolution effects.

[0181] This application provides an image interpolation method, which determines the motion vector and dynamic object information between a first image and a second image, and determines the depth information and semi-transparent texture corresponding to the second image. The first image is a frame preceding the second image. An intermediate offset frame is constructed based on the motion vector, dynamic object information, and depth information. A first color reference image is determined between the first and second images based on the semi-transparent texture. This first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame. The first color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated frame image between the first and second images is determined based on the first color information. In other words, in this application, based on the construction of the intermediate offset frame between two frames, the color information corresponding to the semi-transparent object can be further determined by acquiring and using the semi-transparent texture of the subsequent frame image. This allows for full utilization of the color information corresponding to the semi-transparent object during the generation of the interpolated frame image, ensuring that the final interpolated frame image reflects the true state of the semi-transparent object. This solves the problem of the interpolated frame image not matching the actual motion, greatly improving the performance and effect of image interpolation processing.

[0182] Based on the above embodiments, in another embodiment of this application... Figure 8 This is a schematic diagram of the composition structure of the image frame interpolation device proposed in the embodiments of this application, as shown below. Figure 8 As shown, the image frame interpolation device 80 proposed in this application embodiment may include a determining unit 801 and a constructing unit 802.

[0183] The determining unit 801 is used to determine the motion vector and dynamic object information between the first image and the second image, and to determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image;

[0184] The construction unit 802 is used to construct an intermediate offset frame based on the motion vector, the dynamic object information, and the depth information;

[0185] The determining unit 801 is further configured to determine a first color reference image in the first image and the second image based on the semi-transparent texture; wherein the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame; determine the first color information corresponding to the semi-transparent object based on the first color reference image, and determine the interpolated frame image between the first image and the second image based on the first color information.

[0186] In the embodiments of this application, further, Figure 9 This is a schematic diagram of the composition structure of the terminal device proposed in the embodiments of this application, such as... Figure 9 As shown, the terminal device 90 proposed in this application embodiment includes a processor 901 and a memory 902 storing instructions executable by the processor 901. Further, the terminal device 900 may also include a communication interface 903 and a bus 904 for connecting the processor 901, the memory 902 and the communication interface 903.

[0187] In the embodiments of this application, the processor 901 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this application embodiment does not specifically limit this. The terminal device 900 may also include a memory 902, which can be connected to the processor 901. The memory 902 is used to store executable program code, which includes computer operation instructions. The memory 902 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0188] In embodiments of this application, bus 904 is used to connect communication interface 903, processor 901, and memory 902, as well as the mutual communication between these devices.

[0189] In embodiments of this application, memory 902 is used to store instructions and data.

[0190] Furthermore, in embodiments of this application, the processor 901 is used for:

[0191] Determine the motion vectors and dynamic object information between the first image and the second image, and determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image;

[0192] Based on the motion vector, the dynamic object information, and the depth information, an intermediate offset frame is constructed;

[0193] Based on the semi-transparent texture, a first color reference image is determined in the first image and the second image; wherein, the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame;

[0194] The first color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated image between the first image and the second image is determined based on the first color information.

[0195] In practical applications, the aforementioned memory 902 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 901.

[0196] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0197] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] This application provides an image frame interpolation apparatus and a terminal device. The apparatus determines motion vectors and dynamic object information between a first image and a second image, and determines depth information and a semi-transparent texture corresponding to the second image. The first image is a frame preceding the second image. An intermediate offset frame is constructed based on the motion vectors, dynamic object information, and depth information. A first color reference image is determined between the first and second images based on the semi-transparent texture. This first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame. First color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated frame image between the first and second images is determined based on the first color information. In other words, in this application, based on the construction of the intermediate offset frame between two frames, the color information corresponding to the semi-transparent object can be further determined by acquiring and using the semi-transparent texture of the subsequent frame image. This allows for full utilization of the color information corresponding to the semi-transparent object during the generation of the interpolated frame image, ensuring that the final interpolated frame image reflects the true state of the semi-transparent object. This solves the problem of the interpolated frame image not matching the actual motion, greatly improving the performance and effect of image frame interpolation processing.

[0199] Specifically, the program instructions corresponding to an image frame interpolation method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to an image frame interpolation method in the storage media are read or executed by an electronic device, the following steps are included:

[0200] Determine the motion vectors and dynamic object information between the first image and the second image, and determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image;

[0201] Based on the motion vector, the dynamic object information, and the depth information, an intermediate offset frame is constructed;

[0202] Based on the semi-transparent texture, a first color reference image is determined in the first image and the second image; wherein, the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame;

[0203] The first color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated image between the first image and the second image is determined based on the first color information.

[0204] This application also provides a computer program product.

[0205] In some embodiments, the computer program product may include a computer program or instructions.

[0206] In some embodiments, the computer program product can be applied to the terminal device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the terminal device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0207] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0208] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0209] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0210] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0211] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An image frame interpolation method, characterized in that, The method includes: Determine the motion vectors and dynamic object information between the first image and the second image, and determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image; Based on the motion vector, the dynamic object information, and the depth information, an intermediate offset frame is constructed; Based on the semi-transparent texture, a first color reference image is determined in the first image and the second image; wherein, the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame; The first color information corresponding to the semi-transparent object is determined based on the first color reference image, and the interpolated image between the first image and the second image is determined based on the first color information.

2. The method according to claim 1, characterized in that, The step of determining a first color reference image between the first image and the second image based on the semi-transparent texture includes: For the first pixel position in the intermediate offset frame, the first grayscale indicator parameter corresponding to the second pixel position in the second image and the second grayscale indicator parameter corresponding to the third pixel position in the second image are determined according to the semi-transparent texture; wherein, the coordinate position of the second pixel position in the second image is the same as the coordinate position of the first pixel position in the intermediate offset frame, and the second pixel position is the pixel position corresponding to the third pixel position after offsetting according to the offset parameter; The first color reference image is determined in the first image and the second image based on the first grayscale indicator parameter and the second grayscale indicator parameter.

3. The method according to claim 2, characterized in that, The step of determining the first grayscale indicator parameter corresponding to the second pixel position in the second image and the second grayscale indicator parameter corresponding to the third pixel position in the second image based on the semi-transparent texture includes: The grayscale value corresponding to the second pixel position is determined based on the semi-transparent texture, and the first grayscale indicator parameter is determined based on the first grayscale value; The grayscale value corresponding to the third pixel position is determined based on the semi-transparent texture, and the second grayscale indicator parameter is determined based on the second grayscale value.

4. The method according to claim 2, characterized in that, Determining the first color reference image in the first image and the second image based on the first grayscale indicator parameter and the second grayscale indicator parameter includes: If the first grayscale indicator parameter is less than the second grayscale indicator parameter, the first image is determined as the first color reference image; If the first grayscale indicator parameter is greater than or equal to the second grayscale indicator parameter, the second image is determined as the first color reference image.

5. The method according to any one of claims 2-4, characterized in that, After constructing the intermediate offset frame based on the motion vector, the dynamic object information, and the depth information, the method further includes: Determine the first depth information corresponding to the first pixel position and the second depth information corresponding to the third pixel position; Based on the first depth information and the second depth information, a second color reference image corresponding to the non-transparent object in the intermediate offset frame is determined in the first image and the second image; The second color information corresponding to the non-transparent object is determined based on the second color reference image.

6. The method according to claim 5, characterized in that, Determining the interpolated image between the first image and the second image based on the first color information includes: For any pixel position in the intermediate offset frame, the composite color information corresponding to the pixel position is determined based on the transparency parameter corresponding to the pixel position, the first color information and the second color information corresponding to the pixel position; Traverse all pixel positions in the intermediate offset frame, and determine the interpolated image based on all the synthesized color information corresponding to all pixel positions.

7. The method according to claim 6, characterized in that, The method further includes: For any pixel position in the intermediate offset frame, the transparency corresponding to the pixel position is determined according to the semi-transparent texture. The transparency is preprocessed to determine the transparency parameter corresponding to the pixel position.

8. The method according to any one of claims 2-4, 6, and 7, characterized in that, The step of constructing an intermediate offset frame based on the motion vector, the dynamic object information, and the depth information includes: The corrected motion vector is determined based on the motion vector and the dynamic object information; The offset parameter is determined based on the corrected motion vector; An intermediate offset frame is constructed based on the offset parameter, the dynamic object information, and the depth information.

9. An image frame interpolation device, characterized in that, The image interpolation device includes: a determining unit and a constructing unit. The determining unit is used to determine the motion vector and dynamic object information between the first image and the second image, and to determine the depth information and semi-transparent texture corresponding to the second image; wherein, the first image is a frame image preceding the second image; The construction unit is used to construct an intermediate offset frame based on the motion vector, the dynamic object information, and the depth information; The determining unit is further configured to determine a first color reference image in the first image and the second image based on the semi-transparent texture; wherein the first color reference image is used to sample the color information of the semi-transparent object in the intermediate offset frame; determine the first color information corresponding to the semi-transparent object based on the first color reference image, and determine the interpolated frame image between the first image and the second image based on the first color information.

10. A terminal device, characterized in that, The terminal device includes: a processor and a memory; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1 to 8 when running the computer program.

11. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1 to 8.

Images