Frame insertion processing method and device, electronic equipment, chip and medium

Abstract

The invention relates to a frame insertion processing method and device, electronic equipment, a chip and a medium, and the method comprises the steps: responding to a display instruction of a first image frame, according to the image contents, belonging to a corresponding semantic region, in the first image frame and an adjacent second image frame, and according to a scene to which the semantic region belongs, carrying out frame insertion processing on the first image frame; inserting a transition image frame between the first image frame and the second image frame; wherein the transition image frame is determined according to the image content belonging to the corresponding semantic region in the first image frame and the second image frame and the scene to which the semantic region belongs, and the characteristics of different semantic regions can be considered, so that the generation of the transition image frame of which the edge of the object is torn or distorted is avoided, and the quality of the generated transition image frame is improved; and the display efficiency of the image frame is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an interpolation processing method, apparatus, electronic device, chip, and medium. Background Art

[0002] The current interpolation processing method mainly determines the target value of a pixel point based on the values of corresponding pixel points in a first image frame and an adjacent second image frame, and then generates an intermediate image frame. Among them, tearing or distortion and other phenomena are likely to occur at the edges of objects in the generated intermediate image frame, affecting the quality of the intermediate image frame. Summary of the Invention

[0003] The present disclosure provides an interpolation processing method, apparatus, electronic device, chip, and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an interpolation processing method, the method including: in response to a display instruction of a first image frame, inserting an intermediate image frame between the first image frame and the second image frame according to the image content belonging to the corresponding semantic region in the first image frame and the adjacent second image frame, and according to the scene to which the semantic region belongs.

[0005] In an embodiment of the present disclosure, the applications to which the first image frame and the second image frame belong include interactive multimedia applications; the interactive multimedia applications include at least one of game applications and live broadcast applications.

[0006] In an embodiment of the present disclosure, the method further includes: determining whether an interpolation switch in the interactive multimedia application to which the first image frame and the second image frame belong is in an on state; in a case where the interpolation switch is in the on state, performing response processing on the display instruction; in a case where the interpolation switch is in an off state, stopping performing response processing on the display instruction.

[0007] In an embodiment of the present disclosure, the image content includes regional texture data; the inserting an intermediate image frame between the first image frame and the second image frame according to the image content belonging to the corresponding semantic region in the first image frame and the adjacent second image frame, and according to the scene to which the semantic region belongs includes: determining regional optical flow data of a first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of a second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs; determining the intermediate image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame; inserting the intermediate image frame between the first image frame and the second image frame.

[0008] In one embodiment of the present disclosure, the method further includes: obtaining texture data of the first image frame and texture data of the second image frame; determining regional texture data of the first semantic region in the first image frame according to the texture data of the first image frame; and determining regional texture data of the second semantic region in the second image frame according to the texture data of the second image frame.

[0009] In one embodiment of the present disclosure, the texture data includes: first texture data for region division and second texture data for determining optical flow; the determining regional texture data of the first semantic region in the first image frame according to the texture data of the first image frame includes: determining each first semantic region in the first image frame according to the first texture data; and for the first semantic region, determining part of the texture data located within the first semantic region in the second texture data as the regional texture data of the first semantic region.

[0010] In one embodiment of the present disclosure, the first texture data includes depth template texture data and motion vector texture data; the first semantic region includes a sky region, a background region, and a person region; the determining each first semantic region in the first image frame according to the first texture data includes: determining the sky region in the first image frame according to the depth template texture data; determining the person region in the first image frame according to the motion vector texture data; and determining the region in the first image frame other than the sky region and the person region as the background region in the first image frame.

[0011] In one embodiment of the present disclosure, the depth template texture data includes depth values of each pixel point in the first image frame; the determining the sky region in the first image frame according to the depth template texture data includes: for a pixel point in the first image frame, determining the pixel point as a sky pixel point when the depth value of the pixel point is greater than or equal to a depth threshold; and determining the sky region in the first image frame according to the sky pixel points in the first image frame.

[0012] In one embodiment of the present disclosure, the motion vector texture data includes motion degree values of each pixel point in the first image frame; the determining the person region in the first image frame according to the motion vector texture data includes: for a pixel point in the first image frame, determining the pixel point as a person-related pixel point when the motion degree value of the pixel point is greater than or equal to a value threshold; and determining the person region in the first image frame according to the person-related pixel points in the first image frame.

[0013] In one embodiment of the present disclosure, the second texture data includes at least one of the following: normal texture data, reflectivity texture data, and specular reflection texture data.

[0014] In one embodiment of the present disclosure, determining the regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs includes: determining a regional interpolation model matching the first semantic region according to the scene to which the first semantic region belongs; inputting the regional texture data of the first semantic region and the regional texture data of the second semantic region corresponding to the first semantic region into the regional interpolation model, and obtaining the regional optical flow data of the first semantic region output by the regional interpolation model.

[0015] In one embodiment of the present disclosure, the scene to which the first semantic region belongs indicates the pixel movement ability of the first semantic region; the pixel movement ability of the first semantic region is positively correlated with the computing power resource amount of the regional interpolation model matching the first semantic region.

[0016] In one embodiment of the present disclosure, the first semantic region includes a sky region, a background region, and a human region; the computing power resource amount of the regional interpolation model matching the sky region is less than the computing power resource amount of the regional interpolation model matching the human region; the computing power resource amount of the regional interpolation model matching the background region is less than the computing power resource amount of the regional interpolation model matching the human region.

[0017] In one embodiment of the present disclosure, the method further includes: determining whether there is an optical flow adjustment strategy for the first semantic region according to the scene to which the first semantic region belongs; and when there is an optical flow adjustment strategy for the first semantic region, performing adjustment processing on the regional optical flow data of the first semantic region according to the optical flow adjustment strategy.

[0018] In an embodiment of the present disclosure, a first semantic region having an optical flow adjustment strategy includes a background region; the regional optical flow data of the background region includes optical flow sub-data of background objects in the background region; the adjustment process of the regional optical flow data of the first semantic region according to the optical flow adjustment strategy includes: determining a first coordinate transformation matrix from the world coordinate system to a first image coordinate system where the first image frame is located, and determining a second coordinate transformation matrix from the world coordinate system to a second image coordinate system where the second image frame is located; determining a position offset of the background object according to the reference position information of the background object in the world coordinate system, the first coordinate transformation matrix, and the second coordinate transformation matrix; and adjusting the optical flow sub-data of the background object according to the position offset.

[0019] In an embodiment of the present disclosure, the determining the position offset of the background object according to the reference position information of the background object in the world coordinate system, the first coordinate transformation matrix, and the second coordinate transformation matrix includes: determining first position information of the background object in the first image coordinate system according to the reference position information and the first coordinate transformation matrix; determining second position information of the background object in the second image coordinate system according to the reference position information and the second coordinate transformation matrix; and determining the position offset of the background object according to the first position information and the second position information.

[0020] In an embodiment of the present disclosure, the optical flow sub-data of the background object includes optical flow vectors of each object pixel point in the region where the background object is located; the adjusting the optical flow sub-data of the background object according to the position offset includes: for an object pixel point, determining a magnitude of the optical flow vector according to the optical flow vector of the object pixel point; determining an offset amplitude according to the position offset; and when the magnitude is greater than the offset amplitude, adjusting the optical flow vector of the object pixel point until the magnitude of the optical flow vector is less than or equal to the offset amplitude.

[0021] In an embodiment of the present disclosure, the determining the transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame includes: performing a splicing process on the regional optical flow data of each first semantic region in the first image frame to obtain optical flow data of the first image frame; and when the optical flow data includes optical flow vectors of all pixel points in the first image frame, determining the transition image frame according to the optical flow data and the first image frame.

[0022] In one embodiment of the present disclosure, determining the transitional image frame based on the regional optical flow data of the first semantic region in the first image frame and the first image frame further includes: when the optical flow vector of the first pixel point in the first image frame is not included in the optical flow data, determining the surrounding pixel points of the first pixel point in the first image frame; determining the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points; and adding the optical flow vector of the first pixel point to the optical flow data.

[0023] In one embodiment of the present disclosure, determining the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points includes: determining the weight of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the regional boundary of the first semantic region, and / or according to the angle between the optical flow vector of the surrounding pixel points and the normal line of the first point in the regional boundary; the first point is the point corresponding to the minimum distance; and performing weighted summation processing on the optical flow vectors of the surrounding pixel points according to the weights of the surrounding pixel points to obtain the optical flow vector of the first pixel point.

[0024] In one embodiment of the present disclosure, determining the weight of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the regional boundary of the first semantic region, and the angle between the optical flow vector of the surrounding pixel points and the normal line of the first point in the regional boundary includes: determining the reciprocal value of the minimum distance and the cosine value of the angle; and determining the weight of the surrounding pixel points according to the reciprocal value and the cosine value.

[0025] In one embodiment of the present disclosure, determining the transitional image frame based on the optical flow data and the first image frame includes: performing a 0.5-fold scaling process on the optical flow vectors of each pixel point in the optical flow data to obtain the processed optical flow data; and determining the transitional image frame according to the processed optical flow data and the first image frame.

[0026] According to the second aspect of the embodiments of the present disclosure, there is also provided an interpolation processing device, including: an insertion module, configured to respond to a display instruction of a first image frame, and insert a transitional image frame between the first image frame and a second adjacent image frame according to the image content of the corresponding semantic region in the first image frame and the second image frame, and according to the scene to which the semantic region belongs.

[0027] In one embodiment of the present disclosure, the applications to which the first image frame and the second image frame belong include interactive multimedia applications; the interactive multimedia applications include at least one of game applications and live broadcast applications.

[0028] In one embodiment of the present disclosure, the device further includes: a first determination module and a response processing module; the first determination module is configured to determine whether an interpolation switch in an interactive multimedia application to which the first image frame and the second image frame belong is in an on state; the response processing module is configured to perform response processing on the display instruction when the interpolation switch is in the on state; the response processing module is further configured to stop performing response processing on the display instruction when the interpolation switch is in an off state.

[0029] In one embodiment of the present disclosure, the image content includes regional texture data; the insertion module includes: a first determination unit, a second determination unit, and an insertion unit; the first determination unit is configured to determine regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs; the second determination unit is configured to determine the transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame; the insertion unit is configured to insert the transition image frame between the first image frame and the second image frame.

[0030] In one embodiment of the present disclosure, the device further includes: an acquisition module, a second determination module, and a third determination module; the acquisition module is configured to acquire the texture data of the first image frame and the texture data of the second image frame; the second determination module is configured to determine the regional texture data of the first semantic region in the first image frame according to the texture data of the first image frame; the third determination module is configured to determine the regional texture data of the second semantic region in the second image frame according to the texture data of the second image frame.

[0031] In one embodiment of the present disclosure, the texture data includes: first texture data for regional division and second texture data for determining optical flow; the second determination module includes: a third determination unit and a fourth determination unit; the third determination unit is configured to determine each first semantic region in the first image frame according to the first texture data; the fourth determination unit is configured to, for the first semantic region, determine the partial texture data located within the first semantic region in the second texture data as the regional texture data of the first semantic region.

[0032] In an embodiment of the present disclosure, the first texture data includes depth template texture data and motion vector texture data; the first semantic region includes a sky region, a background region, and a person region; specifically, the third determination unit is configured to determine the sky region in the first image frame according to the depth template texture data; determine the person region in the first image frame according to the motion vector texture data; and determine the region other than the sky region and the person region in the first image frame as the background region in the first image frame.

[0033] In an embodiment of the present disclosure, the depth template texture data includes the depth values of each pixel point in the first image frame; specifically, the third determination unit is further configured to, for a pixel point in the first image frame, determine the pixel point as a sky pixel point when the depth value of the pixel point is greater than or equal to a depth threshold; and determine the sky region in the first image frame according to the sky pixel points in the first image frame.

[0034] In an embodiment of the present disclosure, the motion vector texture data includes the motion degree values of each pixel point in the first image frame; specifically, the third determination unit is further configured to, for a pixel point in the first image frame, determine the pixel point as a person-related pixel point when the motion degree value of the pixel point is greater than or equal to a value threshold; and determine the person region in the first image frame according to the person-related pixel points in the first image frame.

[0035] In an embodiment of the present disclosure, the second texture data includes at least one of the following: normal texture data, reflectivity texture data, and specular reflection texture data.

[0036] In an embodiment of the present disclosure, the first determination unit is specifically configured to determine a region interpolation model matching the first semantic region according to the scene to which the first semantic region belongs; input the region texture data of the first semantic region and the region texture data of the second semantic region corresponding to the first semantic region into the region interpolation model, and obtain the region optical flow data of the first semantic region output by the region interpolation model.

[0037] In an embodiment of the present disclosure, the scene to which the first semantic region belongs indicates the motion ability of the pixel points in the first semantic region; the motion ability of the pixel points in the first semantic region is positively correlated with the computing power resource amount of the region interpolation model matching the first semantic region.

[0038] In an embodiment of the present disclosure, the first semantic region includes a sky region, a background region, and a person region; the computing power resource amount of the region interpolation model matching the sky region is less than that of the region interpolation model matching the person region; the computing power resource amount of the region interpolation model matching the background region is less than that of the region interpolation model matching the person region.

[0039] In an embodiment of the present disclosure, the device further includes: a fourth determination module and an adjustment processing module; the fourth determination module is configured to determine whether there is an optical flow adjustment strategy for the first semantic region according to the scene to which the first semantic region belongs; the adjustment processing module is configured to, when there is an optical flow adjustment strategy for the first semantic region, perform adjustment processing on the regional optical flow data of the first semantic region according to the optical flow adjustment strategy.

[0040] In an embodiment of the present disclosure, the first semantic region with an optical flow adjustment strategy includes a background region; the regional optical flow data of the background region includes optical flow sub-data of background objects in the background region; the adjustment processing module includes: a fifth determination unit, a sixth determination unit, and an adjustment processing unit; the fifth determination unit is configured to determine a first coordinate transformation matrix from the world coordinate system to the first image coordinate system where the first image frame is located, and determine a second coordinate transformation matrix from the world coordinate system to the second image coordinate system where the second image frame is located; the sixth determination unit is configured to determine a position offset of the background object according to the reference position information of the background object in the world coordinate system, the first coordinate transformation matrix, and the second coordinate transformation matrix; the adjustment processing unit is configured to perform adjustment processing on the optical flow sub-data of the background object according to the position offset.

[0041] In an embodiment of the present disclosure, the sixth determination unit is specifically configured to determine first position information of the background object in the first image coordinate system according to the reference position information and the first coordinate transformation matrix; determine second position information of the background object in the second image coordinate system according to the reference position information and the second coordinate transformation matrix; determine the position offset of the background object according to the first position information and the second position information.

[0042] In an embodiment of the present disclosure, the optical flow data of the background object includes the optical flow vectors of each object pixel point in the area where the background object is located; specifically, the adjustment processing unit is configured to, for an object pixel point, determine the magnitude of the optical flow vector according to the optical flow vector of the object pixel point; determine the offset amplitude according to the position offset; and in the case where the magnitude is greater than the offset amplitude, perform adjustment processing on the optical flow vector of the object pixel point until the magnitude of the optical flow vector is less than or equal to the offset amplitude.

[0043] In an embodiment of the present disclosure, the second determination unit is specifically configured to splice the regional optical flow data of each of the first semantic regions in the first image frame to obtain the optical flow data of the first image frame; and in the case where the optical flow data includes the optical flow vectors of all pixel points in the first image frame, determine the transition image frame according to the optical flow data and the first image frame.

[0044] In an embodiment of the present disclosure, the second determination unit is further specifically configured to, in the case where the optical flow data does not include the optical flow vector of the first pixel point in the first image frame, determine the surrounding pixel points of the first pixel point in the first image frame; determine the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points; and add the optical flow vector of the first pixel point to the optical flow data.

[0045] In an embodiment of the present disclosure, the second determination unit is further specifically configured to determine the weight of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the regional boundary of the first semantic region, and / or according to the angle between the optical flow vector of the surrounding pixel points and the normal line of the first point in the regional boundary; the first point is the point corresponding to the minimum distance; and perform weighted summation processing on the optical flow vectors of the surrounding pixel points according to the weights of the surrounding pixel points to obtain the optical flow vector of the first pixel point.

[0046] In an embodiment of the present disclosure, the second determination unit is further specifically configured to determine the reciprocal value of the minimum distance and the cosine value of the angle; and determine the weight of the surrounding pixel points according to the reciprocal value and the cosine value.

[0047] In an embodiment of the present disclosure, the second determination unit is further specifically configured to perform a 0.5-fold scaling process on the optical flow vectors of each pixel point in the optical flow data to obtain the processed optical flow data; and determine the transition image frame according to the processed optical flow data and the first image frame.

[0048] According to a third aspect of the embodiments of the present disclosure, there is also provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to implement the steps of the frame interpolation processing method as described above.

[0049] According to a fourth aspect of the embodiments of the present disclosure, there is also provided a non-transitory computer-readable storage medium, which, when the instructions stored in the storage medium are executed by a processor, enables the processor to execute the frame interpolation processing method as described above.

[0050] According to a fifth aspect of the embodiments of the present disclosure, there is also provided a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive a signal, the signal includes computer instructions, and when the processor executes the computer instructions, the chip executes the frame interpolation processing method as described above.

[0051] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0052] By responding to the display instruction of the first image frame, inserting an intermediate image frame between the first image frame and the adjacent second image frame according to the image content of the corresponding semantic regions in the first image frame and the second image frame, and according to the scene to which the semantic region belongs; wherein, the intermediate image frame is determined according to the image content of the corresponding semantic regions in the first image frame and the second image frame and the scene to which the semantic region belongs, which can consider the characteristics of different semantic regions, thereby avoiding generating intermediate image frames with torn or distorted object edges, improving the quality of the generated intermediate image frames, and further improving the display efficiency of the image frames.

[0053] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation to the present disclosure.

[0055] Figure 1 It is a flowchart of the frame interpolation processing method according to an embodiment of the present disclosure;

[0056] Figure 2 It is a flowchart of the frame interpolation processing method according to another embodiment of the present disclosure;

[0057] Figure 3 It is a flowchart of the frame interpolation processing method according to another embodiment of the present disclosure;

[0058] Figure 4 It is a schematic diagram of depth template texture data;

[0059] Figure 5 It is a schematic diagram of the sky area;

[0060] Figure 6 It is a schematic diagram of the human area;

[0061] Figure 7 It is a flowchart of the frame interpolation processing method according to another embodiment of the present disclosure;

[0062] Figure 8 It is a schematic diagram of frame interpolation processing;

[0063] Figure 9 It is a schematic structural diagram of the frame interpolation processing device according to an embodiment of the present disclosure;

[0064] Figure 10 It is a structural block diagram of an electronic device shown according to an exemplary embodiment of the present disclosure;

[0065] Figure 11 It is a schematic structural diagram of a chip according to an embodiment of the present disclosure. Detailed implementation manners

[0066] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0067] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data may be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0068] Figure 1 It is a flowchart of the frame interpolation processing method according to an embodiment of the present disclosure. Among them, it should be noted that the frame interpolation processing method of this embodiment can be applied to a frame interpolation processing device, and this device can be configured in an electronic device or a chip so that the electronic device or the chip can perform the frame interpolation processing function.

[0069] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a terminal device, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc., which are hardware devices with various operating systems, touch screens, and / or display screens.

[0070] In addition, the frame interpolation processing device can also be software or hardware in the electronic device. Among them, the software can be, for example, frame interpolation processing software, etc. The hardware can be, for example, at least one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). Among them, when the frame interpolation processing device is obtained by combining multiple processors, each processor can execute some steps of the frame interpolation processing method. Among them, in the following embodiments, the execution subject is taken as an electronic device for illustration.

[0071] As Figure 1 shown, the method includes the following steps:

[0072] Step 101, in response to a display instruction of a first image frame, insert an intermediate image frame between the first image frame and a second adjacent image frame according to the image content of the corresponding semantic regions in the first image frame and the second image frame, and according to the scene to which the semantic region belongs.

[0073] In the embodiments of the present disclosure, the applications to which the first image frame and the second image frame belong may include interactive multimedia applications; the interactive multimedia applications may include at least one of game applications and live broadcast applications.

[0074] Among them, in an interactive multimedia application such as a game application or a live broadcast application, the frame interpolation processing timing may be after the first image frame and the second image frame are rendered. At this time, the terminal device can obtain the image content generated during the rendering process of the first image frame and the image content generated during the rendering process of the second image frame for frame interpolation processing. After the first image frame is rendered, the first image frame can be subjected to display processing. During the display processing of the first image frame, the electronic device can perform the rendering operation of the second image frame. Therefore, in order to be able to display the intermediate image frame in a timely manner after the first image frame is displayed, in response to the display instruction of the first image frame, frame interpolation processing can be performed by combining the image content of the first image frame and the image content of the second image frame.

[0075] In the embodiments of the present disclosure, the first image frame and the second image frame may be two consecutive image frames in an interactive multimedia application. That is to say, the first image frame and the second image frame are adjacent, and the second image frame is displayed after the first image frame.

[0076] In the embodiments of the present disclosure, the first image frame and the second image frame may be two image frames with a preset number of frames in between in an interactive multimedia application. For example, there are two frames between the first image frame and the second image frame. That is to say, the first image frame and the second image frame are not adjacent, and the second image frame is displayed two frames after the first image frame.

[0077] In the embodiments of the present disclosure, the determination methods of the semantic regions in the first image frame and the second image frame may be the same. Taking the first image frame as an example, the semantic region in the first image frame may be obtained by dividing according to the objects or object features in the first image frame. Among them, object features include, for example, object categories, object motion capabilities, etc. For example, the semantic region in the first image frame may be obtained by dividing according to the object motion capabilities, so that the pixel motion capabilities of the pixel points in the same semantic region are consistent.

[0078] Among them, the scene to which the semantic region belongs may indicate the pixel motion capabilities of the semantic region. The pixel motion capabilities indicated by different scenes may be different. Among them, when the semantic region in the first image frame is obtained by dividing according to the objects or object features other than the object motion capabilities, the scenes to which different semantic regions in the first image frame belong may be the same or different. When the semantic region in the first image frame is obtained by dividing according to the object motion capabilities, the scenes to which different semantic regions in the first image frame belong may be different.

[0079] In the embodiments of the present disclosure, it may be assumed that the semantic region in the first image frame is the first semantic region, and the semantic region in the second image frame is the second semantic region. Among them, the image content in the first image frame and the second image frame that belongs to the corresponding semantic region refers to the image content of the first semantic region in the first image frame and the image content of the second semantic region corresponding to the first semantic region in the second image frame. Among them, the scene to which the second semantic region corresponding to the first semantic region belongs may be the same as the scene to which the first semantic region belongs; and the image content in the first semantic region and the image content in the second semantic region may be similar or consistent.

[0080] Among them, based on the above assumptions, the process of the electronic device executing step 101 can be, for example, in response to a display instruction of the first image frame, inserting an intermediate image frame between the first image frame and the second image frame according to the image content of the first semantic region in the first image frame, the image content of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs.

[0081] Among them, the semantic region can include, for example, at least one of the following: sky region, person region, and background region. Among them, the pixel motion ability of the sky region and the background region is relatively poor, while the pixel motion ability of the person region is relatively strong. The first semantic region and the corresponding second semantic region can both be the sky region, or both be the background region, or both be the person region.

[0082] In the embodiments of the present disclosure, the image content can include at least one of the following: texture data, RGB data, etc. Among them, the texture data can include at least one of the following: depth template texture data, motion vector texture data, normal texture data, reflectivity texture data, and specular reflection texture data, etc. Taking the first image frame as an example, the depth template texture data can include the depth values of each pixel in the first image frame. The motion vector texture data can include the motion degree values of each pixel in the first image frame. The normal texture data can include the surface normal information of each pixel in the first image frame. The reflectivity texture data can include the surface reflection attributes of each pixel in the first image frame. The specular reflection texture data can include the specular reflection intensity information of each pixel in the first image frame.

[0083] In the embodiments of the present disclosure, to facilitate the flexible management of the object for the frame interpolation process, that is, the object can determine whether to perform the frame interpolation process according to actual needs. For example, it is determined whether to perform the frame interpolation process according to the performance of the electronic device, the remaining traffic, etc. An interpolation switch can be set in the interactive multimedia application, and the interpolation switch is displayed on the interface of the interactive multimedia application to facilitate the object to adjust the on / off state of the interpolation switch. Correspondingly, the electronic device can also execute the following process: determining whether the interpolation switch in the interactive multimedia application to which the first image frame and the second image frame belong is in the on state; in the case where the interpolation switch is in the on state, performing a response process on the display instruction; in the case where the interpolation switch is in the off state, stopping the response process on the display instruction.

[0084] Among them, in the case where an interpolation switch is set in the interactive multimedia application, the electronic device can execute step 101 when the interpolation switch is in the on state; and stop executing step 101 when the interpolation switch is in the off state.

[0085] In the frame interpolation method according to an embodiment of the present disclosure, in response to a display instruction of a first image frame, an intermediate image frame is inserted between the first image frame and an adjacent second image frame according to the image content of the corresponding semantic regions in the first image frame and the second image frame, and according to the scene to which the semantic region belongs; wherein, the intermediate image frame is determined according to the image content of the corresponding semantic regions in the first image frame and the second image frame and the scene to which the semantic region belongs, and can take into account the characteristics of different semantic regions, thereby avoiding generating an intermediate image frame with torn or distorted object edges, improving the quality of the generated intermediate image frame, and further improving the display efficiency of the image frame.

[0086] Figure 2 The flowchart of the frame interpolation method according to another embodiment of the present disclosure is shown. It should be noted that the frame interpolation method of this embodiment can be applied to a frame interpolation device, which can be configured in an electronic device or a chip, so that the electronic device or the chip can perform the frame interpolation function.

[0087] Among them, the electronic device can be any device with computing power, such as a personal computer (PC), a mobile terminal, a terminal device, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc., which are hardware devices with various operating systems, touch screens, and / or display screens.

[0088] In addition, the frame interpolation device can also be software or hardware in the electronic device. Among them, the software can be, for example, frame interpolation software. The hardware can be, for example, at least one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). Among them, when the frame interpolation device is obtained by combining multiple processors, each processor can execute part of the steps of the frame interpolation method. In the following embodiments, the execution subject is taken as an electronic device for illustration.

[0089] As Figure 2 shown, the method includes the following steps:

[0090] Step 201, in response to a display instruction of a first image frame, determine the regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs; the first image frame and the second image frame are adjacent.

[0091] In an embodiment of the present disclosure, the process of the electronic device executing step 201 may be, for example, determining a region interpolation model that matches the first semantic region according to the scene to which the first semantic region belongs; inputting the region texture data of the first semantic region and the region texture data of the second semantic region corresponding to the first semantic region into the region interpolation model, and obtaining the region optical flow data of the first semantic region output by the region interpolation model.

[0092] Among them, the number of region interpolation models may be multiple. The region interpolation models corresponding to different first semantic regions may be different. For example, in the case where the first semantic region includes a sky region, a person region, and a background region, the region interpolation model corresponding to the sky region may be a sky interpolation model; the region interpolation model corresponding to the person region may be a person interpolation model; the region interpolation model corresponding to the background region may be a background interpolation model.

[0093] Among them, for different first semantic regions, different region interpolation models are used to determine the region optical flow data. Considering the accuracy of the region interpolation model, the accuracy of the determined region optical flow data can be improved.

[0094] In an embodiment of the present disclosure, it should be noted that for different semantic regions, the motion ability of pixel points is different, and the data processing amount when determining the region optical flow data is different. For example, for a semantic region with poor pixel point motion ability, the data processing amount is less; for a semantic region with strong pixel point motion ability, the data processing amount is more. Therefore, in order to reduce the data processing amount during the interpolation process, region interpolation models with different computing power resource amounts may be configured for semantic regions with different pixel point motion abilities. Correspondingly, when the scene to which the first semantic region belongs indicates the pixel point motion ability of the first semantic region, the pixel point motion ability of the first semantic region and the computing power resource amount of the region interpolation model that matches the first semantic region may be positively correlated; so that for semantic regions with different pixel point motion abilities, region interpolation models with different computing power resource amounts can be used to determine the region optical flow data.

[0095] Among them, the first semantic region includes a sky region, a background region, and a person region; the computing power resource amount of the region interpolation model that matches the sky region is less than the computing power resource amount of the region interpolation model that matches the person region; the computing power resource amount of the region interpolation model that matches the background region is less than the computing power resource amount of the region interpolation model that matches the person region. Among them, the computing power resource amount refers to the computing power required for training or inferring the region interpolation model.

[0096] In the embodiments of the present disclosure, the regional optical flow data of the first semantic region may include the optical flow vectors of each pixel point in the first semantic region; for the pixel points in the first semantic region, the pixel points are processed for movement based on the optical flow vectors of the pixel points, and can move to the position where a certain pixel point in the second semantic region in the second image frame is located; a certain pixel point and the pixel point in the second image frame that matches the pixel point.

[0097] Step 202: Determine a transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame.

[0098] In the embodiments of the present disclosure, the process of the electronic device executing step 202 may be, for example, performing splicing processing on the regional optical flow data of each first semantic region in the first image frame to obtain the optical flow data of the first image frame; determining whether the optical flow data includes the optical flow vectors of all pixel points in the first image frame; in the case where the optical flow data includes the optical flow vectors of all pixel points in the first image frame, determining a transition image frame according to the optical flow data and the first image frame; in the case where the optical flow data does not include the optical flow vector of the first pixel point in the first image frame, determining the surrounding pixel points of the first pixel point in the first image frame; determining the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points; adding the optical flow vector of the first pixel point to the optical flow data, and then repeating the above determination step.

[0099] Among them, the electronic device determines the optical flow vector for the first pixel point in the first image frame that does not have an optical flow vector set and adds it to the optical flow data, which can ensure that the optical flow data includes the optical flow vectors of all pixel points in the first image frame; generating a transition image frame according to the optical flow data can ensure that there are no hole points in the generated transition image frame, thereby further improving the quality of the generated transition image frame.

[0100] In the embodiments of the present disclosure, in order to ensure the accuracy of the optical flow vector of the first pixel point, the electronic device may determine the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points of the first pixel point. Correspondingly, the process of the electronic device determining the optical flow vector of the first pixel point in the first image frame may be, for example, determining the weights of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the regional boundary of the first semantic region, and / or according to the angle between the optical flow vectors of the surrounding pixel points and the normal line of the first point in the regional boundary; the first point is the point corresponding to the minimum distance; performing weighted summation processing on the optical flow vectors of the surrounding pixel points according to the weights of the surrounding pixel points to obtain the optical flow vector of the first pixel point.

[0101] In one example, the electronic device can determine the weight of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the region boundary of the first semantic region. Specifically, the electronic device can determine the weight of the surrounding pixel points according to the reciprocal value of the minimum distance. The smaller the minimum distance is, the greater the weight is, so that the electronic device can rely more on the surrounding pixel points closer to the region boundary to determine the optical flow vector of the first pixel point. Since the first pixel point is generally closer to the region boundary or located on the region boundary, relying more on the surrounding pixel points closer to the region boundary to determine the optical flow vector of the first pixel point can ensure the consistency between the optical flow vector of the first pixel point obtained and the optical flow vectors of the pixel points on the region boundary, further avoiding situations such as tearing and distortion, and further improving the quality of the transitional image frame.

[0102] In another example, the electronic device can determine the weight of the surrounding pixel points according to the angle between the optical flow vector of the surrounding pixel points and the normal line of the first point in the region boundary. Specifically, the electronic device can determine the weight of the surrounding pixel points according to the cosine value of the angle. The smaller the angle is, the greater the weight is, so as to ensure the consistency in the direction between the optical flow vector of the first pixel point and the optical flow vectors of the pixel points on the region boundary, and further avoid situations such as tearing and distortion.

[0103] In another example, the electronic device can determine the weight of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the region boundary of the first semantic region, and the angle between the optical flow vector of the surrounding pixel points and the normal line of the first point in the region boundary. Specifically, the electronic device can determine the reciprocal value of the minimum distance and the cosine value of the angle; and determine the weight of the surrounding pixel points according to the reciprocal value and the cosine value.

[0104] In this example, the determination formula of the optical flow vector of the first pixel point can be shown, for example, as the following formula (1).

[0105]

[0106] Wherein, F represents the optical flow vector of the first pixel point; x i represents the minimum distance; θ i represents the angle; represents the optical flow vector of the i-th surrounding pixel point.

[0107] In an embodiment of the present disclosure, the optical flow data of the first image frame is the optical flow data of the first image frame relative to the second image frame; the electronic device needs to first combine the optical flow data to determine the optical flow data of the first image frame relative to the transitional image frame, and then generate the transitional image frame. Correspondingly, the process of the electronic device determining the transitional image frame according to the optical flow data and the first image frame may be, for example, performing a 0.5-fold scaling process on the optical flow vectors of each pixel point in the optical flow data to obtain the processed optical flow data; determining the transitional image frame according to the processed optical flow data and the first image frame.

[0108] Step 203: Insert the transitional image frame between the first image frame and the second image frame.

[0109] It should be noted that for the detailed description of steps 201 to 203, reference can be made to Figure 1 Step 101 in the illustrated embodiment, and no detailed description will be given here.

[0110] In the frame interpolation processing method of the embodiment of the present disclosure, by responding to the display instruction of the first image frame, determining the regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs; the first image frame and the second image frame are adjacent; determining the transitional image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame; inserting the transitional image frame between the first image frame and the second image frame; wherein, combining the regional texture data of the corresponding semantic regions in the first image frame and the second image frame and the scene to which the first semantic region in the first image frame belongs, determining the regional optical flow data of the first semantic region in the first image frame, and then determining the transitional image frame; it is possible to consider the pixel motion ability of the semantic regions indicated by different scenes, thereby improving the accuracy of the determined regional optical flow data, further avoiding phenomena such as tearing and distortion, and further improving the accuracy of the generated transitional image frame.

[0111] Figure 3 It is a flowchart of the frame interpolation processing method according to another embodiment of the present disclosure. It should be noted that the frame interpolation processing method of this embodiment can be applied to a frame interpolation processing device, which can be configured in an electronic device or a chip so that the electronic device or the chip can perform the frame interpolation processing function.

[0112] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC for short), a mobile terminal, a terminal device, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc., which are hardware devices with various operating systems, touch screens, and / or display screens.

[0113] In addition, the frame interpolation processing device may also be software or hardware in an electronic device. Among them, software such as frame interpolation processing software, etc. Hardware such as at least one or a combination of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a Neural Processing Unit (NPU). Among them, when the frame interpolation processing device is obtained by combining multiple processors, each processor may execute some steps of the frame interpolation processing method. Among them, in the following embodiments, the electronic device is used as an example of the execution subject for illustration.

[0114] As Figure 3 shown, the method includes the following steps:

[0115] Step 301, in response to a display instruction of a first image frame, obtain texture data of the first image frame and texture data of a second image frame; the first image frame and the second image frame are adjacent.

[0116] In the embodiments of the present disclosure, the texture data may include: first texture data for region division and second texture data for determining optical flow. Among them, the first texture data may include: depth template texture data and motion vector texture data. The second texture data may include: normal texture data, reflectivity texture data, and specular reflection texture data.

[0117] Step 302, determine region texture data of a first semantic region in the first image frame according to the texture data of the first image frame.

[0118] In the embodiments of the present disclosure, the texture data may include: first texture data for region division and second texture data for determining optical flow. The process of the electronic device executing step 302 may be, for example, according to the first texture data of the first image frame, determine each first semantic region in the first image frame; for the first semantic region, determine the partial texture data located in the first semantic region in the second texture data of the first image frame as the region texture data of the first semantic region.

[0119] Among them, the first texture data includes: depth template texture data and motion vector texture data. Correspondingly, the process of the electronic device determining each first semantic region in the first image frame may be, for example, according to the depth template texture data, determine the sky region in the first image frame; according to the motion vector texture data, determine the human region in the first image frame; determine the region other than the sky region and the human region in the first image frame as the background region in the first image frame.

[0120] Among them, the depth template texture data may include the depth values of each pixel point in the first image frame. Correspondingly, for the pixel points in the first image frame, the electronic device may determine a pixel point as a sky pixel point when the depth value of the pixel point is greater than or equal to the depth threshold; determine the sky area in the first image frame according to the sky pixel points in the first image frame. Among them, the schematic diagram of the depth template texture data may be as shown in Figure 4 shown. In Figure 4 , the depth value of the person is small, and the depth value of the sky is large. Among them, the schematic diagram of the determined sky area may be as shown in Figure 5 shown. Figure 5 The gray area in

[0121] Among them, the motion vector texture data includes the motion degree values of each pixel point in the first image frame. Correspondingly, for the pixel points in the first image frame, the electronic device may determine a pixel point as a person-related pixel point when the motion degree value of the pixel point is greater than or equal to the value threshold; determine the person area in the first image frame according to the person-related pixel points in the first image frame. Among them, the schematic diagram of the person area may be as shown in Figure 6 shown. Figure 6 The white area in

[0122] In the embodiments of the present disclosure, a part of the texture data in the second texture data located in the first semantic area, that is, the second texture data of the target pixel points in the second texture data; the target pixel points are the pixel points located in the first semantic area.

[0123] Among them, by combining the division process of the first texture data for each first semantic area in the first image frame, the electronic device can adopt different optical flow data determination strategies for the regional texture data of different first semantic areas to determine the regional optical flow data of the first semantic area. For example, when determining the regional flow data of the first semantic area, consider the pixel motion ability of the first semantic area, thereby further improving the accuracy of the determined regional optical flow data.

[0124] Step 303, determine the regional texture data of the second semantic area in the second image frame according to the texture data of the second image frame.

[0125] In the embodiments of the present disclosure, the process of the electronic device executing step 303 may be, for example, to determine each second semantic area in the second image frame according to the first texture data of the second image frame; for the second semantic area, determine the part of the second texture data in the second image frame located in the second semantic area as the regional texture data of the second semantic area.

[0126] Among them, for the determination process of the second semantic region and the determination process of the regional texture data of the second semantic region, reference can be made to the determination process of the first semantic region and the regional texture data of the first semantic region in step 302, and no detailed description will be given here.

[0127] Step 304: Determine the regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs.

[0128] Step 305: Determine the transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame.

[0129] Step 306: Insert the transition image frame between the first image frame and the second image frame.

[0130] It should be noted that for the detailed description of steps 304 to 306, reference can be made to Figure 2 Steps 201 to 203 in the illustrated embodiment, and no detailed description will be given here.

[0131] In the frame interpolation processing method of the present disclosure embodiment, by responding to the display instruction of the first image frame, the texture data of the first image frame and the texture data of the second image frame are obtained; the first image frame and the second image frame are adjacent; according to the texture data of the first image frame, the regional texture data of the first semantic region in the first image frame is determined; according to the texture data of the second image frame, the regional texture data of the second semantic region in the second image frame is determined; according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs, the regional optical flow data of the first semantic region is determined; according to the regional optical flow data of the first semantic region in the first image frame and the first image frame, the transition image frame is determined; the transition image frame is inserted between the first image frame and the second image frame; among them, the division processing of each semantic region in the first image frame and the second image frame according to the first texture data enables the electronic device to adopt different optical flow data determination strategies for the regional texture data of different semantic regions to determine the regional optical flow data of the semantic region, thereby further improving the accuracy of the determined regional optical flow data and further improving the quality of the generated transition image frame.

[0132] Figure 7 It is a flowchart of the frame interpolation processing method of another embodiment of the present disclosure. It should be noted that the frame interpolation processing method of this embodiment can be applied to a frame interpolation processing device, and the device can be configured in an electronic device or a chip so that the electronic device or the chip can perform the frame interpolation processing function.

[0133] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a terminal device, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc., which are hardware devices with various operating systems, touch screens, and / or display screens.

[0134] In addition, the frame interpolation processing device can also be software or hardware in the electronic device, etc. Among them, the software can be, for example, frame interpolation processing software, etc. The hardware can be, for example, at least one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU), etc. Among them, when the frame interpolation processing device is obtained by combining multiple processors, each processor can execute some steps of the frame interpolation processing method. Among them, in the following embodiments, the execution subject is taken as an electronic device for illustration.

[0135] As Figure 7 shown, the method includes the following steps:

[0136] Step 701, in response to a display instruction of a first image frame, determine the regional optical flow data of a first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of a second semantic region corresponding to the first semantic region in a second image frame, and the scene to which the first semantic region belongs; the first image frame and the second image frame are adjacent.

[0137] Step 702, determine whether there is an optical flow adjustment strategy for the first semantic region according to the scene to which the first semantic region belongs.

[0138] In the embodiments of the present disclosure, for at least one scene, the electronic device can be pre-set with corresponding optical flow adjustment strategies. For example, for the scene to which the background region in the semantic region belongs, since the background region generally occupies a large area, and in the case of dense parallel lines (such as stairs), etc., the optical flow vectors of the pixel points in the background region are prone to dislocation. Therefore, in order to improve the accuracy of the regional optical flow data of the background region, an optical flow adjustment strategy can be set for the scene to which the background region belongs to adjust and process the regional optical flow data of the background region.

[0139] Among them, for the sky area in the semantic area, there is generally no situation of dense parallel lines in the sky area, and the optical flow vectors of the pixel points in the sky area are not likely to be misaligned, or the misalignment does not affect the display of the sky area. Therefore, for the scene to which the sky area belongs, an optical flow adjustment strategy may not be set. If an optical flow adjustment strategy needs to be set for the scene to which the sky area belongs, the set optical flow adjustment strategy may refer to the optical flow adjustment strategy set for the scene to which the background area belongs.

[0140] Step 703, in the case where there is an optical flow adjustment strategy for the first semantic area, adjust and process the regional optical flow data of the first semantic area according to the optical flow adjustment strategy.

[0141] In the embodiments of the present disclosure, the first semantic area with an optical flow adjustment strategy includes the background area; the regional optical flow data of the background area includes the optical flow sub-data of the background objects in the background area. Correspondingly, the process of the electronic device executing step 703 may be, for example, determining a first coordinate transformation matrix from the world coordinate system to the first image coordinate system where the first image frame is located, and determining a second coordinate transformation matrix from the world coordinate system to the second image coordinate system where the second image frame is located; determining the position offset of the background object according to the reference position information of the background object in the world coordinate system, the first coordinate transformation matrix, and the second coordinate transformation matrix; and adjusting and processing the optical flow sub-data of the background object according to the position offset.

[0142] Among them, the first image coordinate system can be determined according to the imaging device that captures the first image frame, that is, the location of the imaging device is the origin of the first image coordinate system. Among them, the first coordinate transformation matrix can be determined according to the position and attitude of the imaging device in the world coordinate system. Specifically, the electronic device can determine the homogeneous transformation matrix according to the position and attitude of the imaging device; perform an inverse process on the homogeneous transformation matrix, and determine the inverse process result as the first coordinate transformation matrix. The determination processes of the second image coordinate system and the second coordinate transformation matrix are similar.

[0143] Among them, the formula of the homogeneous transformation matrix can be, for example, as shown in the following formula (2).

[0144]

[0145] Among them, T represents the homogeneous transformation matrix. R represents the attitude of the imaging device in the world coordinate system, that is, a 3*3 rotation matrix. T represents the position of the imaging device in the world coordinate system, that is, a 3*1 translation vector.

[0146] In an embodiment of the present disclosure, the process by which an electronic device determines the position offset of a background object may be, for example, to determine the first position information of the background object in the first image coordinate system according to the reference position information and the first coordinate transformation matrix; to determine the second position information of the background object in the second image coordinate system according to the reference position information and the second coordinate transformation matrix; and to determine the position offset of the background object according to the first position information and the second position information.

[0147] Among them, the reference position of the background object in the world coordinate system may be, for example, as shown in the following formula (3).

[0148]

[0149] Among them, P world represents the reference position. x, y, and z respectively represent the X-axis value, Y-axis value, and Z-axis value in the reference position.

[0150] Among them, the calculation formula for the position offset may be, for example, as shown in the following formula (4).

[0151]

[0152] Among them, ΔP screen represents the position offset.

[0153] In an embodiment of the present disclosure, the process by which an electronic device adjusts the optical flow data of a background object according to the position offset may be, for example, for an object pixel point, to determine the modulus of the optical flow vector according to the optical flow vector of the object pixel point; to determine the offset amplitude according to the position offset; and in the case where the modulus is greater than the offset amplitude, to adjust the optical flow vector of the object pixel point until the modulus of the optical flow vector is less than or equal to the offset amplitude.

[0154] Among them, the electronic device may, in the case where the modulus of the optical flow vector is greater than the offset amplitude of the position offset, perform a reduction process on the numerical values of each dimension of the optical flow vector in the same proportion until the modulus of the optical flow vector is less than or equal to the offset amplitude.

[0155] Step 704: Determine a transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame.

[0156] Step 705: Insert the transition image frame between the first image frame and the second image frame.

[0157] It should be noted that for the detailed descriptions of step 701 and steps 704 to 705, reference may be made to Figure 1 Steps 201 to 203 in the embodiment shown, and no detailed description will be given here.

[0158] In the frame interpolation processing method according to the embodiments of the present disclosure, in response to a display instruction of a first image frame, region optical flow data of a first semantic region is determined according to region texture data of the first semantic region in the first image frame, region texture data of a second semantic region corresponding to the first semantic region in a second image frame, and a scene to which the first semantic region belongs; the first image frame and the second image frame are adjacent; whether there is an optical flow adjustment strategy for the first semantic region is determined according to the scene to which the first semantic region belongs; in the case where there is an optical flow adjustment strategy for the first semantic region, the region optical flow data of the first semantic region is adjusted according to the optical flow adjustment strategy; a transition image frame is determined according to the region optical flow data of the first semantic region in the first image frame and the first image frame; the transition image frame is inserted between the first image frame and the second image frame; wherein, the adjustment processing of the region optical flow data of the first semantic region with an optical flow adjustment strategy can avoid the problem of optical flow vector misalignment of the first semantic region, improve the accuracy of the region optical flow data of the first semantic region, and further improve the quality of the generated transition image frame.

[0159] The following is an example for illustration. As Figure 8 shown, it is a schematic diagram of frame interpolation processing. The following steps may be included.

[0160] Step 801, obtain texture data and coordinate transformation matrices of the first image frame and the second image frame at each processing stage of the first image frame and the second image frame.

[0161] Among them, the processing stages include: Geometry Pass, SmokeEffect Pass, motion blur pass, merge lighting pass. Among them, the texture data obtained in the Geometry Pass includes Depth Stencil Texture, Normals Texture, Albedo Texture, and Specular Texture. The texture data obtained in the motion blur pass includes Motion Vectors Texture. Among them, the SmokeEffect Pass obtains a first coordinate transformation matrix and a second coordinate transformation matrix (World To Camera Matrix). Among them, the merge lighting pass obtains the RGB data (Main Texture) in the image frame.

[0162] Step 802: Determine the depth texture based on the depth template texture data, and then determine the dynamic scaling depth texture to identify the sky regions in the first and second image frames. Determine the person regions based on the motion vector texture data, and then determine the background regions. Obtain the region texture data for each region.

[0163] Among them, taking the first image frame as an example, the area normal texture data of the sky region (Normals Sky Area), the area normal texture data of the background region (Normals Background Area), the area normal texture data of the person region (Normals People Area), the area albedo texture data of the sky region (Albedo Sky Area), the area albedo texture data of the background region (Albedo Background Area), the area albedo texture data of the person region (Albedo People Area), the area specular texture data of the sky region (Specular Sky Area), the area specular texture data of the background region (Specular Background Area), and the area specular texture data of the person region (Specular People Area) can be obtained.

[0164] Step 803: Determine the area optical flow data for each region based on the obtained region texture data and the region interpolation model.

[0165] Among them, combining the area normal texture data, the area albedo texture data, the area specular texture data of the sky region in the first and second image frames, and the sky interpolation model (FISky), determine the area optical flow data of the sky region (Sky Flow). Combining the area normal texture data, the area albedo texture data, the area specular texture data of the background region in the first and second image frames, and the background interpolation model (FISM), determine the area optical flow data of the background region (Background Flow). Combining the area normal texture data, the area albedo texture data, the area specular texture data of the person region in the first and second image frames, and the person interpolation model (FILM), determine the area optical flow data of the person region (People Flow).

[0166] Step 804: Adjust the area optical flow data of the background region in each region to obtain the processed area optical flow data of the background region (Clip Background Flow).

[0167] Step 805: Concatenate the processed regional optical flow data of the background area, the regional optical flow data of the sky area, and the regional optical flow data of the human area to obtain the optical flow data. Perform hole filling processing on the optical flow data to obtain the processed optical flow data (Inpainting Flow).

[0168] Step 806: Determine the RGB data (Interpolation Texture) of the transition frame image according to the processed optical flow data and the RGB data in the first image frame.

[0169] Figure 9 Schematic diagram of the structure of an interpolation processing device according to an embodiment of the present disclosure.

[0170] As Figure 9 shown, the interpolation processing device may include: an insertion module 901, configured to respond to a display instruction of the first image frame, and insert a transition image frame between the first image frame and the second image frame according to the image content belonging to the corresponding semantic area in the first image frame and the adjacent second image frame, and according to the scene to which the semantic area belongs.

[0171] In an embodiment of the present disclosure, the applications to which the first image frame and the second image frame belong include interactive multimedia applications; the interactive multimedia applications include at least one of game applications and live broadcast applications.

[0172] In an embodiment of the present disclosure, the device further includes: a first determination module and a response processing module; the first determination module is configured to determine whether the interpolation switch in the interactive multimedia application to which the first image frame and the second image frame belong is in an on state; the response processing module is configured to perform response processing on the display instruction when the interpolation switch is in the on state; the response processing module is further configured to stop performing response processing on the display instruction when the interpolation switch is in the off state.

[0173] In one embodiment of the present disclosure, the image content includes regional texture data; the insertion module 901 includes: a first determination unit, a second determination unit, and an insertion unit; the first determination unit is configured to determine the regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs; the second determination unit is configured to determine the transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame; the insertion unit is configured to insert the transition image frame between the first image frame and the second image frame.

[0174] In one embodiment of the present disclosure, the device further includes: an acquisition module, a second determination module, and a third determination module; the acquisition module is configured to acquire the texture data of the first image frame and the texture data of the second image frame; the second determination module is configured to determine the regional texture data of the first semantic region in the first image frame according to the texture data of the first image frame; the third determination module is configured to determine the regional texture data of the second semantic region in the second image frame according to the texture data of the second image frame.

[0175] In one embodiment of the present disclosure, the texture data includes: first texture data for regional division and second texture data for determining optical flow; the second determination module includes: a third determination unit and a fourth determination unit; the third determination unit is configured to determine each first semantic region in the first image frame according to the first texture data; the fourth determination unit is configured to, for the first semantic region, determine the partial texture data located within the first semantic region in the second texture data as the regional texture data of the first semantic region.

[0176] In one embodiment of the present disclosure, the first texture data includes depth template texture data and motion vector texture data; the first semantic region includes a sky region, a background region, and a human region; specifically, the third determination unit is configured to determine the sky region in the first image frame according to the depth template texture data; determine the human region in the first image frame according to the motion vector texture data; and determine the region in the first image frame other than the sky region and the human region as the background region in the first image frame.

[0177] In an embodiment of the present disclosure, the depth template texture data includes the depth values of each pixel point in the first image frame; specifically, the third determination unit is further configured to, for the pixel points in the first image frame, determine that the pixel point is a sky pixel point when the depth value of the pixel point is greater than or equal to a depth threshold; and determine the sky region in the first image frame according to the sky pixel points in the first image frame.

[0178] In an embodiment of the present disclosure, the motion vector texture data includes the motion degree values of each pixel point in the first image frame; specifically, the third determination unit is further configured to, for the pixel points in the first image frame, determine that the pixel point is a person-related pixel point when the motion degree value of the pixel point is greater than or equal to a value threshold; and determine the person region in the first image frame according to the person-related pixel points in the first image frame.

[0179] In an embodiment of the present disclosure, the second texture data includes at least one of the following: normal texture data, reflectivity texture data, and specular reflection texture data.

[0180] In an embodiment of the present disclosure, the first determination unit is specifically configured to determine a region interpolation model that matches the first semantic region according to the scene to which the first semantic region belongs; input the region texture data of the first semantic region and the region texture data of the second semantic region corresponding to the first semantic region into the region interpolation model, and obtain the region optical flow data of the first semantic region output by the region interpolation model.

[0181] In an embodiment of the present disclosure, the scene to which the first semantic region belongs indicates the pixel motion ability of the first semantic region; the pixel motion ability of the first semantic region is positively correlated with the computing power resource amount of the region interpolation model that matches the first semantic region.

[0182] In an embodiment of the present disclosure, the first semantic region includes a sky region, a background region, and a person region; the computing power resource amount of the region interpolation model that matches the sky region is less than the computing power resource amount of the region interpolation model that matches the person region; the computing power resource amount of the region interpolation model that matches the background region is less than the computing power resource amount of the region interpolation model that matches the person region.

[0183] In an embodiment of the present disclosure, the device further includes: a fourth determination module and an adjustment processing module; the fourth determination module is configured to determine whether there is an optical flow adjustment strategy for the first semantic region according to the scene to which the first semantic region belongs; the adjustment processing module is configured to, when there is an optical flow adjustment strategy for the first semantic region, perform adjustment processing on the regional optical flow data of the first semantic region according to the optical flow adjustment strategy.

[0184] In an embodiment of the present disclosure, the first semantic region with an optical flow adjustment strategy includes a background region; the regional optical flow data of the background region includes optical flow sub-data of background objects in the background region; the adjustment processing module includes: a fifth determination unit, a sixth determination unit, and an adjustment processing unit; the fifth determination unit is configured to determine a first coordinate transformation matrix from the world coordinate system to the first image coordinate system where the first image frame is located, and determine a second coordinate transformation matrix from the world coordinate system to the second image coordinate system where the second image frame is located; the sixth determination unit is configured to determine a position offset of the background object according to the reference position information of the background object in the world coordinate system, the first coordinate transformation matrix, and the second coordinate transformation matrix; the adjustment processing unit is configured to perform adjustment processing on the optical flow sub-data of the background object according to the position offset.

[0185] In an embodiment of the present disclosure, the sixth determination unit is specifically configured to determine first position information of the background object in the first image coordinate system according to the reference position information and the first coordinate transformation matrix; determine second position information of the background object in the second image coordinate system according to the reference position information and the second coordinate transformation matrix; and determine the position offset of the background object according to the first position information and the second position information.

[0186] In an embodiment of the present disclosure, the optical flow sub-data of the background object includes optical flow vectors of each object pixel point in the region where the background object is located; the adjustment processing unit is specifically configured to, for an object pixel point, determine a magnitude of the optical flow vector according to the optical flow vector of the object pixel point; determine an offset amplitude according to the position offset; and when the magnitude is greater than the offset amplitude, perform adjustment processing on the optical flow vector of the object pixel point until the magnitude of the optical flow vector is less than or equal to the offset amplitude.

[0187] In one embodiment of the present disclosure, the second determination unit is specifically configured to splice the regional optical flow data of each of the first semantic regions in the first image frame to obtain the optical flow data of the first image frame; when the optical flow data includes the optical flow vectors of all pixel points in the first image frame, determine the transitional image frame according to the optical flow data and the first image frame.

[0188] In one embodiment of the present disclosure, the second determination unit is further specifically configured to, when the optical flow data does not include the optical flow vector of a first pixel point in the first image frame, determine the surrounding pixel points of the first pixel point in the first image frame; determine the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points; and add the optical flow vector of the first pixel point to the optical flow data.

[0189] In one embodiment of the present disclosure, the second determination unit is further specifically configured to determine the weight of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the regional boundary of the first semantic region, and / or according to the included angle between the optical flow vector of the surrounding pixel points and the normal line of a first point in the regional boundary; the first point is the point corresponding to the minimum distance; and perform weighted summation processing on the optical flow vectors of the surrounding pixel points according to the weights of the surrounding pixel points to obtain the optical flow vector of the first pixel point.

[0190] In one embodiment of the present disclosure, the second determination unit is further specifically configured to determine the reciprocal value of the minimum distance and the cosine value of the included angle; and determine the weight of the surrounding pixel points according to the reciprocal value and the cosine value.

[0191] In one embodiment of the present disclosure, the second determination unit is further specifically configured to perform a 0.5-fold scaling process on the optical flow vectors of each pixel point in the optical flow data to obtain the processed optical flow data; and determine the transitional image frame according to the processed optical flow data and the first image frame.

[0192] In the interpolation processing device according to the embodiments of the present disclosure, by responding to a display instruction of a first image frame, an intermediate image frame is inserted between the first image frame and an adjacent second image frame according to the image content belonging to the corresponding semantic regions in the first image frame and the second image frame, and according to the scene to which the semantic region belongs; wherein, the intermediate image frame is determined according to the image content belonging to the corresponding semantic regions in the first image frame and the second image frame and the scene to which the semantic region belongs, and can consider the characteristics of different semantic regions, thereby avoiding generating an intermediate image frame with torn or distorted object edges, improving the quality of the generated intermediate image frame, and further improving the display efficiency of the image frame.

[0193] According to a third aspect of the embodiments of the present disclosure, an electronic device is further provided, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to: implement the steps of the frame interpolation processing method as described above.

[0194] To implement the above embodiments, the present disclosure also proposes a non-transitory computer-readable storage medium.

[0195] Wherein, when the instructions in the storage medium are executed by the processor, the processor is enabled to execute the frame interpolation processing method as described above.

[0196] To implement the above embodiments, the present disclosure also provides a computer program product.

[0197] Wherein, when the computer program product is executed by the processor of the electronic device, the electronic device is enabled to execute the method as described above.

[0198] Figure 10 It is a block diagram of the structure of an electronic device shown according to an exemplary embodiment. Figure 10 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0199] As Figure 10 shown, the electronic device 1000 includes a processor 111, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 112 or the program loaded from the memory 116 into the random access memory (RAM, Random Access Memory) 113. In the RAM 113, various programs and data required for the operation of the electronic device 1000 are also stored. The processor 111, the ROM 112, and the RAM 113 are connected to each other through a bus 114. The input / output (I / O, Input / Output) interface 115 is also connected to the bus 114.

[0200] The following components are connected to the I / O interface 115: a memory 116 including a hard disk, etc.; and a communication part 117 including a network interface card such as a local area network (LAN) card, a modem, etc., and the communication part 117 performs communication processing via a network such as the Internet; a drive 118 is also connected to the I / O interface 115 as needed.

[0201] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program carried on a computer-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 117. When the computer program is executed by the processor 111, the above-mentioned functions defined in the method of the present disclosure are performed.

[0202] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by the processor 111 of the electronic device 1000 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium can be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices, etc.

[0203] In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0204] Figure 11 Schematic diagram of the structure of a chip according to an embodiment of the present disclosure. As Figure 11 shown, the chip includes a processor 1101 and an interface circuit 1102. Among them, the number of processors 1101 can be one or more, and the number of interface circuits 1102 can be one or more.

[0205] Optionally, the interface circuit 1102 is used to receive signals, and the signals include computer instructions. When the processor 1101 executes the computer instructions, the chip executes the frame interpolation processing method described in the above embodiments of the present disclosure.

[0206] The collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0207] In addition, the term "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be understood as advantageous compared to other aspects or designs. Instead, the use of the term exemplary is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to mean any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then "X applies A or B" is satisfied in any of the foregoing instances. Additionally, unless otherwise specified or clear from the context indicating a singular form, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more".

[0208] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, after reading and understanding the specification and drawings, those skilled in the art will envision equivalent variations and modifications. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. Specifically with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. Additionally, although a particular feature of the present disclosure may have been disclosed with respect to only one of several implementations, such a feature may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application. Furthermore, with respect to the terms "comprising", "having", "including", "possessing", or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "including".

[0209] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0210] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A frame interpolation processing method, characterized in that, The method includes: In response to a display instruction of a first image frame, an intermediate image frame is inserted between the first image frame and an adjacent second image frame according to the image content of the corresponding semantic regions in the first image frame and the second image frame, and according to the scene to which the semantic region belongs.

2. The method according to claim 1, wherein The application to which the first image frame and the second image frame belong includes an interactive multimedia application; The interactive multimedia application includes at least one of a game application and a live broadcast application.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Determining whether an interpolation switch in the interactive multimedia application to which the first image frame and the second image frame belong is in an on state; In the case where the interpolation switch is in the on state, performing response processing on the display instruction; In the case where the interpolation switch is in the off state, stopping performing response processing on the display instruction.

4. The method according to claim 1, wherein The image content includes regional texture data; the inserting an intermediate image frame between the first image frame and the second image frame according to the image content of the corresponding semantic regions in the first image frame and the adjacent second image frame, and according to the scene to which the semantic region belongs, includes: Determining the regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs; Determining the intermediate image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame; Inserting the intermediate image frame between the first image frame and the second image frame.

5. The method according to claim 4, wherein The method further includes: Obtaining the texture data of the first image frame and the texture data of the second image frame; Determining the regional texture data of the first semantic region in the first image frame according to the texture data of the first image frame; Determining the regional texture data of the second semantic region in the second image frame according to the texture data of the second image frame.

6. The method according to claim 5, wherein The texture data includes: first texture data for regional division and second texture data for determining optical flow; the determining the regional texture data of the first semantic region in the first image frame according to the texture data of the first image frame includes: Determining each first semantic region in the first image frame according to the first texture data; For the first semantic region, determining the partial texture data located within the first semantic region in the second texture data as the regional texture data of the first semantic region.

7. The method according to claim 6, wherein The first texture data includes depth template texture data and motion vector texture data; the first semantic region includes a sky region, a background region, and a human region; the determining each first semantic region in the first image frame according to the first texture data includes: Determining the sky region in the first image frame according to the depth template texture data; Determining the human region in the first image frame according to the motion vector texture data; Determine the area other than the sky area and the human area in the first image frame as the background area in the first image frame.

8. The method according to claim 7, wherein The depth template texture data includes the depth values of each pixel point in the first image frame; determining the sky area in the first image frame according to the depth template texture data includes: For the pixel points in the first image frame, when the depth value of the pixel point is greater than or equal to the depth threshold, determine that the pixel point is a sky pixel point; Determine the sky area in the first image frame according to the sky pixel points in the first image frame.

9. The method according to claim 7, wherein The motion vector texture data includes the motion degree values of each pixel point in the first image frame; determining the human area in the first image frame according to the motion vector texture data includes: For the pixel points in the first image frame, when the motion degree value of the pixel point is greater than or equal to the value threshold, determine that the pixel point is a human-related pixel point; Determine the human area in the first image frame according to the human-related pixel points in the first image frame.

10. The method according to claim 6, wherein The second texture data includes at least one of the following: normal texture data, reflectivity texture data, and specular reflection texture data.

11. The method according to claim 4, wherein Determining the regional optical flow data of the first semantic region according to the regional texture data of the first semantic region in the first image frame, the regional texture data of the second semantic region corresponding to the first semantic region in the second image frame, and the scene to which the first semantic region belongs includes: Determine a regional interpolation model that matches the first semantic region according to the scene to which the first semantic region belongs; Input the regional texture data of the first semantic region and the regional texture data of the second semantic region corresponding to the first semantic region into the regional interpolation model, and obtain the regional optical flow data of the first semantic region output by the regional interpolation model.

12. The method according to claim 11, characterized in that, The scene to which the first semantic region belongs indicates the motion ability of the pixel points in the first semantic region; The motion ability of the pixel points in the first semantic region is positively correlated with the computing power resource amount of the regional interpolation model that matches the first semantic region.

13. The method according to claim 11 or 12, characterized in that The first semantic region includes a sky area, a background area, and a human area; The computing power resource amount of the regional interpolation model that matches the sky area is less than the computing power resource amount of the regional interpolation model that matches the human area; The computing power resource amount of the regional interpolation model that matches the background area is less than the computing power resource amount of the regional interpolation model that matches the human area.

14. The method according to claim 4, wherein The method further includes: Determine whether there is an optical flow adjustment strategy for the first semantic region according to the scene to which the first semantic region belongs; When there is an optical flow adjustment strategy for the first semantic region, perform adjustment processing on the regional optical flow data of the first semantic region according to the optical flow adjustment strategy.

15. The method according to claim 14, wherein The first semantic region with an optical flow adjustment strategy includes a background area; the regional optical flow data of the background area includes the optical flow sub-data of the background objects in the background area; Adjusting and processing the regional optical flow data of the first semantic region according to the optical flow adjustment strategy includes: Determining a first coordinate transformation matrix from the world coordinate system to the first image coordinate system where the first image frame is located, and determining a second coordinate transformation matrix from the world coordinate system to the second image coordinate system where the second image frame is located; Determining the position offset of the background object according to the reference position information of the background object in the world coordinate system, the first coordinate transformation matrix, and the second coordinate transformation matrix; Adjusting and processing the optical flow sub-data of the background object according to the position offset.

16. The method according to claim 15, wherein Determining the position offset of the background object according to the reference position information of the background object in the world coordinate system, the first coordinate transformation matrix, and the second coordinate transformation matrix includes: Determining the first position information of the background object in the first image coordinate system according to the reference position information and the first coordinate transformation matrix; Determining the second position information of the background object in the second image coordinate system according to the reference position information and the second coordinate transformation matrix; Determining the position offset of the background object according to the first position information and the second position information.

17. The method according to claim 15, characterized in that, The optical flow sub-data of the background object includes the optical flow vectors of each object pixel point in the area where the background object is located; adjusting and processing the optical flow sub-data of the background object according to the position offset includes: For an object pixel point, determining the magnitude of the optical flow vector according to the optical flow vector of the object pixel point; Determining the offset amplitude according to the position offset; When the magnitude is greater than the offset amplitude, adjusting the optical flow vector of the object pixel point until the magnitude of the optical flow vector is less than or equal to the offset amplitude.

18. The method according to claim 4, wherein Determining the transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame includes: Performing a splicing process on the regional optical flow data of each first semantic region in the first image frame to obtain the optical flow data of the first image frame; When the optical flow data includes the optical flow vectors of all pixel points in the first image frame, determining the transition image frame according to the optical flow data and the first image frame.

19. The method according to claim 18, wherein Determining the transition image frame according to the regional optical flow data of the first semantic region in the first image frame and the first image frame further includes: When the optical flow data does not include the optical flow vector of the first pixel point in the first image frame, determining the surrounding pixel points of the first pixel point in the first image frame; Determining the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points; Adding the optical flow vector of the first pixel point to the optical flow data.

20. The method according to claim 19, characterized in that, Determining the optical flow vector of the first pixel point according to the optical flow vectors of the surrounding pixel points includes: Determine the weight of the surrounding pixel points based on the minimum distance between the surrounding pixel points and the region boundary of the first semantic region, and / or based on the angle between the optical flow vector of the surrounding pixel points and the normal line of the first point in the region boundary; the first point is the point corresponding to the minimum distance; Perform weighted summation processing on the optical flow vectors of the surrounding pixel points according to the weights of the surrounding pixel points to obtain the optical flow vector of the first pixel point.

21. The method according to claim 20, wherein Determine the weight of the surrounding pixel points according to the minimum distance between the surrounding pixel points and the region boundary of the first semantic region, and the angle between the optical flow vector of the surrounding pixel points and the normal line of the first point in the region boundary, including: Determine the reciprocal value of the minimum distance and the cosine value of the angle; Determine the weight of the surrounding pixel points according to the reciprocal value and the cosine value.

22. The method according to claim 18, characterized in that, The determining the transition image frame according to the optical flow data and the first image frame includes: Perform a 0.5-fold scaling process on the optical flow vectors of each pixel point in the optical flow data to obtain the processed optical flow data; Determine the transition image frame according to the processed optical flow data and the first image frame.

23. An interpolation processing device, characterized in that, The device includes: An insertion module, configured to respond to a display instruction of a first image frame, and insert a transition image frame between the first image frame and the second image frame according to the image content belonging to the corresponding semantic region in the first image frame and the adjacent second image frame, and according to the scene to which the semantic region belongs.

24. An electronic device, characterized in that, including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to: Implement the steps of the frame interpolation processing method according to any one of claims 1 to 22.

25. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor, enable the processor to execute the frame interpolation processing method according to any one of claims 1 to 22.

26. A chip, characterized in that, including one or more interface circuits and one or more processors; the interface circuit is used to receive signals, the signals include computer instructions, and when the processor executes the computer instructions, enable the chip to execute the frame interpolation processing method according to any one of claims 1 to 22.

Images