Frame insertion method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a frame insertion method and device, electronic equipment and a readable storage medium, and belongs to the technical field of image processing. The frame insertion method provided by the embodiment of the invention comprises the following steps: acquiring a first picture frame corresponding to a first picture; generating, by a display processing unit (DPU), a second picture frame according to the first picture frame, the resolution of the second picture frame being smaller than the resolution of the first picture frame; determining a first frame insertion model based on the second picture frame; and performing frame insertion according to the first picture frame and the first frame insertion model.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a frame insertion method, device, electronic device and readable storage medium. Background Art

[0002] Typically, when a user is using an electronic device, the electronic device can obtain a picture frame of a display screen of the electronic device, generate an intermediate picture frame based on the picture frame, and use the intermediate picture frame to interpolate a picture frame corresponding to the display screen, thereby improving the frame rate of the display screen, and the user can watch a display screen with higher smoothness.

[0003] However, since the electronic device may need a long time and consume more power to generate the above-mentioned intermediate picture frame based on the above-mentioned picture frame, it may cause high power consumption for inserting the picture frame corresponding to the display picture, thus resulting in high power consumption of the electronic device. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a frame insertion method, device, electronic device and readable storage medium, which can solve the problem of high power consumption of electronic devices.

[0005] In a first aspect, an embodiment of the present application provides an interpolation method, the method comprising: obtaining a first picture frame corresponding to a first picture; generating a second picture frame according to the first picture frame through a display processing unit DPU, wherein a resolution of the second picture frame is smaller than a resolution of the first picture frame; determining a first interpolation model based on the second picture frame; and interpolating according to the first picture frame and the first interpolation model.

[0006] In a second aspect, an embodiment of the present application provides a frame insertion device, which includes: an acquisition module and a processing module. The acquisition module is used to acquire a first picture frame corresponding to the first picture. The processing module is used to generate a second picture frame through a display processing unit DPU according to the first picture frame acquired by the acquisition module, and the resolution of the second picture frame is smaller than the resolution of the first picture frame; and determine a first frame insertion model based on the second picture frame; and perform frame insertion according to the first picture frame and the first frame insertion model.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0011] In an embodiment of the present application, the electronic device can obtain a first picture frame corresponding to the first picture, and generate a second picture frame according to the first picture frame through the display processing unit DPU, and the resolution of the second picture frame is less than the resolution of the first picture frame; thus, the electronic device can determine the first interpolation model based on the second picture frame, and interpolate according to the first picture frame and the first interpolation model. Since the electronic device can quickly generate a second picture frame with a lower resolution through the DPU when obtaining the first picture frame corresponding to the first picture, without generating the second picture frame through the neural processing unit NPU, the time consumption of generating the second picture frame can be reduced, and the second picture frame does not need to be generated through the graphics processor GPU, which can reduce the power consumption of generating the second picture frame; in this way, the electronic device can quickly determine the first interpolation model based on the second picture frame, and quickly interpolate according to the first picture frame and the first interpolation model, so the time required for interpolation can be reduced and the power consumption required for interpolation can be reduced, so the power consumption of the electronic device can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is one of the flowcharts of the frame insertion method provided in the embodiment of the present application;

[0013] Figure 2 This is the second flowchart of the frame insertion method provided in the embodiment of the present application;

[0014] Figure 3 This is the third flowchart of the frame insertion method provided in the embodiment of the present application;

[0015] Figure 4 This is the fourth flowchart of the frame insertion method provided in the embodiment of the present application;

[0016] Figure 5 is a structural schematic diagram of a frame insertion device provided in an embodiment of the present application;

[0017] Figure 6 This is one of the hardware structure diagrams of the electronic device provided in the embodiment of the present application;

[0018] Figure 7 This is the second schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0020] The following is an explanation of the terms used in the embodiments of the present application.

[0021] 1. Scene classification

[0022] Generally, scene classification refers to the classification of scenes in an image, and the scene types of the scene classification may include beaches, streets, etc. The difference between scene classification and image classification is that image classification focuses on the foreground and subject of the image, while scene classification focuses on the background and environment of the image.

[0023] 2. Display Processing Unit (DPU)

[0024] Usually, a DPU is provided in an electronic device, and the DPU can be used for image rendering and display optimization, or screen refresh rate and power consumption management, or graphics acceleration and hardware acceleration, etc. For example, the DPU can combine multiple layers (such as the layer of the game screen, the layer of the video screen, and the layer of the status bar) into one layer, and then display it on the screen, thereby reducing the high power consumption caused by displaying multiple layers; or the DPU can perform image processing to enhance the display effect, and the image processing may include scaling, color enhancement, etc.

[0025] 3. Neural Processing Unit (NPU)

[0026] The neural processing unit, also known as a neural network processor, is an acceleration unit specially designed for artificial intelligence (AI). It excels at operators commonly used in deep learning, such as convolution kernels.

[0027] 4. Other terms

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0030] The following, in conjunction with the accompanying drawings, describes in detail the frame insertion method, device, electronic device, and readable storage medium provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0031] The frame insertion method provided in the embodiment of the present application can be applied to the process in which a user uses an electronic device to watch a video, and can also be applied to the process in which a user uses an electronic device to play a game.

[0032] In one scenario, assuming that a user wants to use an electronic device to watch a video, in order to improve the user's video viewing experience, the electronic device can first obtain a picture frame 1 in the video, and generate an intermediate picture frame based on the picture frame 1, and insert the intermediate picture frame after the picture frame 1 to interpolate the video, thereby increasing the frame rate of the video, and the user can watch the video with a higher picture smoothness. However, the applicant found in the actual application process that: since the picture quality of the video may be higher, the resolution of the picture frame 1 is higher at this time. In this way, in the process of the electronic device generating the intermediate picture frame based on the picture frame 1 with a higher resolution, it may take a longer time and consume more power, which may result in higher power consumption for interpolating the picture frames of the video.

[0033] However, in an embodiment of the present application, when the electronic device obtains the picture frame 1 in the video, it can first generate a picture frame 2 with a lower resolution according to the picture frame 1 through the DPU, and then determine the interpolation model 1 according to the picture frame 2, and interpolate according to the picture frame 1 and the interpolation model 1. It can be understood that since the electronic device can quickly generate a picture frame 2 with a lower resolution through the DPU when the picture frame 1 is obtained, compared with the use of other processing units to generate a picture frame 2 with a lower resolution according to the picture frame 1, for the NPU, since the speed at which the NPU generates a picture frame 2 with a lower resolution according to the picture frame 1 is very slow, it will cause a higher delay in interpolation. In the embodiment of the present application, the speed at which the DPU generates a picture frame 2 with a lower resolution according to the picture frame 1 is faster, so the delay in interpolation can be improved. For the GPU, when the GPU load is high, the power consumption of the GPU to generate a picture frame 2 with a lower resolution according to the picture frame 1 is as high as 20 mA, and when playing the video, the additional waking up of the GPU will bring an additional 10 mA power consumption, thus resulting in a higher power consumption of the electronic device. In the embodiment of the present application, the power consumption of using DPU to generate a lower-resolution picture frame 2 based on picture frame 1 is less than the power consumption of using GPU to generate a lower-resolution picture frame 2 based on picture frame 1. Therefore, the power consumption of the electronic device can be reduced.

[0034] In another scenario, assuming that a user wants to use an electronic device to play a game, in order to improve the user's gaming experience, the electronic device can first obtain a frame 1 corresponding to the game screen, and generate an intermediate frame based on the frame 1, and insert the intermediate frame into the frame 1 to interpolate the frame corresponding to the game screen, thereby increasing the frame rate of the game screen, and the user can watch a smoother game screen. However, the applicant found in the actual application process that: since the game screen may have a higher picture quality, the resolution of the frame 1 is higher at this time. In this way, in the process of the electronic device generating an intermediate frame based on the frame 1 with a higher resolution, it may take a longer time and consume more power, and therefore, it may cause the power consumption of interpolating the frame corresponding to the game screen to be higher.

[0035] However, in an embodiment of the present application, when the electronic device obtains the screen frame 1 corresponding to the game screen, it can first generate a screen frame 2 with a lower resolution according to the screen frame 1 through the DPU, and then determine the interpolation model 1 according to the screen frame 2, and interpolate according to the screen frame 1 and the interpolation model 1. Since the electronic device can quickly generate a screen frame 2 with a lower resolution through the DPU when it obtains the screen frame 1, compared with using other processing units to generate a screen frame 2 with a lower resolution according to the screen frame 1, for the NPU, since the speed at which the NPU generates a screen frame 2 with a lower resolution according to the screen frame 1 is very slow, it will cause a higher delay in interpolation. In the embodiment of the present application, the speed at which the DPU generates a screen frame 2 with a lower resolution according to the screen frame 1 is faster, so the delay in interpolation can be improved. For the GPU, when the GPU load is high, the power consumption of the GPU to generate a screen frame 2 with a lower resolution according to the screen frame 1 is as high as 20mA, and when playing a video, additionally waking up the GPU will bring an additional 10mA of power consumption, thus resulting in higher power consumption of the electronic device. In the embodiment of the present application, the power consumption of using DPU to generate a lower-resolution picture frame 2 based on picture frame 1 is less than the power consumption of using GPU to generate a lower-resolution picture frame 2 based on picture frame 1. Therefore, the power consumption of the electronic device can be reduced.

[0036] An embodiment of the present application provides a frame insertion method. Figure 1 FIG. 1 is a flow chart of a frame insertion method provided in an embodiment of the present application. Figure 1 As shown, the interpolation method provided in the embodiment of the present application may include the following steps 101 to 104.

[0037] Step 101: An electronic device obtains a first picture frame corresponding to a first picture.

[0038] In some embodiments of the present application, the first screen may be any one of the following: a screen of an application running in the foreground, or a screen of an application running in the background.

[0039] In the case where the first screen is a screen of an application running in the foreground, the first screen can be understood as a screen currently displayed by the electronic device.

[0040] The above-mentioned application programs may include at least one of the following: video application programs, chat application programs, game application programs, etc.

[0041] In some embodiments of the present application, the first picture frame may specifically be any one of the following: a picture frame currently displayed in the first picture, a picture frame currently being rendered in the first picture, or a picture frame to be rendered in the first picture.

[0042] The picture frame to be rendered in the first picture may be understood as: the next i picture frames of the picture frame currently being rendered in the first picture, where i is a positive integer.

[0043] In one scenario, when a video application is running in the foreground of an electronic device and a video is being played in the video application, the electronic device can send each frame of the video to the DPU through the central processor, so that the DPU can render each frame and display each rendered frame in sequence, so that the electronic device can directly obtain the first frame corresponding to the first frame through the DPU.

[0044] In another scenario, when a game application is running in the foreground of an electronic device, the electronic device can send each screen frame of the game application to the DPU through the central processor, so that the DPU can render each screen frame and display each rendered screen frame in sequence, so that the electronic device can directly obtain the first screen frame corresponding to the first screen through the DPU.

[0045] In some embodiments of the present application, the electronic device can acquire a picture frame corresponding to the first picture in real time during the process of displaying the first picture, and acquire a first picture frame corresponding to the first picture when determining that the resolution of the picture frame is greater than or equal to a preset resolution.

[0046] It can be understood that if the resolution of the picture frame is greater than or equal to the preset resolution, it can be considered that the power consumption of generating the intermediate picture frame based on the picture frame in the subsequent steps is relatively large, so the electronic device obtains the first picture frame. If the resolution of the picture frame is less than the preset resolution, it is considered that the power consumption of generating the intermediate picture frame based on the picture frame in the subsequent steps is relatively small, so the electronic device may not obtain the first picture frame to reduce the power consumption of the electronic device.

[0047] Step 102: The electronic device generates a second picture frame according to the first picture frame through the DPU.

[0048] In the embodiment of the present application, the resolution of the second picture frame is smaller than the resolution of the first picture frame.

[0049] In some embodiments of the present application, the electronic device may downsample the first picture frame through the DPU to obtain the second picture frame.

[0050] It can be understood that since the CPU of the electronic device will send the picture frame to the DPU for rendering when the electronic device displays the first picture, the second picture frame is generated by the DPU. On the one hand, since there is no need to send the picture frame to other units (such as NPU, GPU, etc.), the power consumption of generating the second picture frame can be reduced. On the other hand, since the DPU has a strong ability to perform image processing, the DPU can quickly generate the second picture frame, thereby reducing the delay in generating the second picture frame.

[0051] Of course, in order to further reduce the power consumption of the electronic device, the electronic device can also set the condition of generating the second picture frame according to the first picture frame through the DPU, so as to reduce the situation of generating the second picture frame in unnecessary scenes.

[0052] In some embodiments of the present application, Figure 1 ,like Figure 2 As shown, before the above step 102, the interpolation method provided in the embodiment of the present application further includes the following step 201, and the above step 102 can be specifically implemented by the following step 102a.

[0053] Step 201: The electronic device determines a picture type corresponding to a first picture frame.

[0054] In some embodiments of the present application, the above-mentioned picture type can be any of the following: a high-load picture type and a low-load picture type. The high-load picture type can be understood as: a type in which the load of the DPU is high during the process of rendering and displaying the picture frame, and the low-load picture type can be understood as: a type in which the load of the DPU is low during the process of rendering and displaying the picture frame.

[0055] Exemplarily, the above-mentioned high-load picture type picture frames may include at least one of the following: picture frames of a game lobby, picture frames of a two-dimensional animation. The above-mentioned low-load picture type picture frames may include at least one of the following: picture frames of a game screen, picture frames of a high-resolution three-dimensional video.

[0056] In some embodiments of the present application, the electronic device may perform picture recognition on the first picture frame to determine the picture type corresponding to the first picture frame.

[0057] Step 102a: When the picture type matches the preset picture type, the electronic device generates a second picture frame according to the first picture frame through the DPU.

[0058] In some embodiments of the present application, the preset picture type may specifically be a high-load picture type.

[0059] In an embodiment of the present application, if the picture type corresponding to the first picture frame matches the preset picture type, it can be considered that the picture type corresponding to the first picture frame is a high-load picture type. At this time, the user may have a higher demand for picture smoothness. Therefore, the electronic device can generate a second picture frame based on the first picture frame through the DPU, determine the first interpolation model based on the second picture frame, and use the first picture frame and the first interpolation model for interpolation, thereby improving the smoothness of the picture and meeting the needs of the user.

[0060] Thus, it can be seen that since the electronic device can generate the second picture frame through the DPU according to the first picture frame only when the picture type corresponding to the first picture frame matches the preset picture type, it can ensure that the electronic device generates the second picture frame through the DPU according to the first picture frame only when the user requires the smoothness of the picture, and determines the first interpolation model based on the second picture frame, and uses the first picture frame and the first interpolation model for interpolation, thereby ensuring that the power consumption of the electronic device is reduced while meeting the user's needs.

[0061] Step 103: The electronic device determines a first interpolation frame model based on the second picture frame.

[0062] In some embodiments of the present application, the electronic device can input the second picture frame into the AI ​​model in the NPU to obtain the scene type corresponding to the second picture frame output by the AI ​​model, so that the electronic device can determine the first interpolation model according to the scene type.

[0063] The following is an example of a specific solution for the electronic device to determine the first interpolation model.

[0064] In some embodiments of the present application, Figure 1 ,like Figure 3 As shown, the above step 103 can be specifically implemented through the following steps 103a and 103b.

[0065] Step 103a: The electronic device determines a first scene type corresponding to a second picture frame, and obtains at least one second interpolation frame model.

[0066] In some embodiments of the present application, the first scene type may be any one of the following: a real-world style type, an animation style type, a ball-containing type, or a special-effects-containing type.

[0067] The above-mentioned ball-type can be understood as: a type of picture frame containing balls. The above-mentioned special effect type can be understood as: a type of picture frame containing special effects.

[0068] In some embodiments of the present application, the electronic device may input the second picture frame into the AI ​​model in the NPU, and determine the first scene type corresponding to the second picture frame through the AI ​​model. It can be understood that the first scene type corresponding to the second picture frame can be quickly determined through the AI ​​model.

[0069] In some embodiments of the present application, the above step 103a can be specifically implemented by the following step 103a1 or step 103a2.

[0070] Step 103a1: When the first duration is less than a preset duration, the electronic device determines the second scene type as the first scene type.

[0071] In the embodiment of the present application, the first duration is: the duration from the moment when the scene type corresponding to the picture frame is last determined to the moment when the first picture frame is acquired, and the second scene type is the scene type determined last time.

[0072] In some embodiments of the present application, the above-mentioned preset duration may specifically be A milliseconds ms, where A is a positive integer.

[0073] In the embodiment of the present application, since the scene type corresponding to the picture frame of the screen does not change in a short period of time, the electronic device can directly determine the scene type corresponding to the last determined picture frame as the first scene type when the time since the scene type corresponding to the picture frame was last determined is less than a preset time length, that is, when the time since the scene type corresponding to the picture frame was last determined is short.

[0074] Thus, it can be seen that, since the time since the scene type corresponding to the picture frame was last determined is less than the preset time length, that is, when the time since the scene type corresponding to the picture frame was last determined is short, the electronic device can directly determine the scene type corresponding to the picture frame last determined as the first scene type. Therefore, the accuracy of the determined first scene type can be ensured while the computational load of the electronic device is reduced in the process of determining the first scene type.

[0075] Step 103a2: When the first duration is greater than or equal to the preset duration, the electronic device inputs the second picture frame into the preprocessing model, and determines the first scene type through the preprocessing model.

[0076] In the embodiment of the present application, the first duration is: the duration from the moment when the scene type corresponding to the picture frame is last determined to the moment when the first picture frame is acquired, and the second scene type is the scene type determined last time.

[0077] In an embodiment of the present application, the above-mentioned preprocessing model may be a model among multiple AI models in the NPU.

[0078] In some embodiments of the present application, the above-mentioned preset processing model can be a lightweight network on the mobile terminal, such as MobileNet V4. Of course, the preprocessing model can also be other models, which is not limited in the embodiments of the present application.

[0079] In the embodiment of the present application, since the scene type corresponding to the picture frame of the picture will not change in a short period of time, the electronic device can determine the first scene type corresponding to the second picture frame through a preprocessing model when the first duration is greater than or equal to the preset duration, that is, when the time since the scene type corresponding to the picture frame was last determined is long, that is, when the scene type corresponding to the picture frame may change.

[0080] It can be seen that since the electronic device can determine the first scene type corresponding to the second picture frame through the preprocessing model when the first duration is greater than or equal to the preset duration, that is, when the scene type corresponding to the picture frame may change, the accuracy and efficiency of the determined first scene type can be ensured.

[0081] In the embodiment of the present application, each of the at least one second interpolation frame model corresponds to a scene type.

[0082] It can be understood that each second interpolation frame model is used to process picture frames of a scene type.

[0083] In some embodiments of the present application, the at least one second interpolation model may be a model pre-trained in the electronic device or a model pre-trained in the server.

[0084] In some embodiments of the present application, before the above step 103a, the frame insertion method provided by the embodiment of the present application may further include the following steps 401 and 402.

[0085] Step 401: The electronic device obtains at least one training data set.

[0086] In the embodiment of the present application, each of the at least one training data set includes at least one picture frame corresponding to a scene type.

[0087] In some embodiments of the present application, the electronic device may obtain at least one training data set from an open source data set and / or a screen recording file of the electronic device and / or a picture rendering pipeline and / or a picture decoding framework.

[0088] For example, it is assumed that at least one scene type includes: real world style type, animation style type, ball type, and special effect type. Then the electronic device can obtain multiple picture frames from the open source data set and / or the screen recording file of the electronic device and / or the picture rendering pipeline and / or the picture decoding framework, and classify the multiple picture frames to obtain training data set 1 and training data set 2, wherein the training data set 1 includes at least one picture frame corresponding to the real world style type, and the training data set 2 includes at least one picture frame corresponding to the animation style type. Then, the electronic device can also specifically record multiple ball sports videos containing small balls, and obtain training data set 3 based on the multiple ball sports videos, wherein the training data set 3 includes at least one picture frame containing the ball type, and the picture frame is a picture frame in multiple ball sports videos. In addition, the electronic device can also specifically record multiple game videos containing special effects, and obtain training data set 4 based on the multiple game videos, wherein the training data set 4 includes at least one picture frame containing the special effect type, and the picture frame is a picture frame in multiple game videos.

[0089] Of course, the electronic device may also obtain the at least one training data set in other ways, which is not limited in the embodiments of the present application.

[0090] Step 402: The electronic device uses each training data set in at least one training data set to train a preset model to obtain each second interpolation frame model in at least one second interpolation frame model.

[0091] In some embodiments of the present application, the electronic device may first use a first training data set in at least one training data set to train a preset model to obtain a first model, and then use each training data set in at least one training data set except the first training data set to train the first model to obtain each training data set, thereby obtaining at least one training data set. The first training data set is a training data set corresponding to a preset scene type.

[0092] Among them, the above preset model can specifically be a lightweight network for mobile terminals, such as MobileNetV4.

[0093] For example, it is assumed that at least one training data set includes four training data sets, and the four training data sets include a first training data set, a second training data set, a third training data set, and a fourth training data set. The first training data set includes at least one picture frame corresponding to the real world style type, the second training data set includes at least one picture frame corresponding to the animation style type, the third training data set includes at least one picture frame corresponding to the ball type, and the fourth training data set includes at least one picture frame corresponding to the special effect type. Thus, the electronic device can first use the first training data set to train the preset model to obtain a first model (which can be called a general model), and then use the second training data set to train the first model to obtain a second interpolation model, which is used to process the picture frames corresponding to the animation style type, and use the third training data set to train the first model to obtain another second interpolation model, which is used to process the picture frames corresponding to the ball type, and use the fourth training data set to train the first model to obtain another second interpolation model, which is used to process the picture frames corresponding to the special effect type. In this way, four second interpolation frame models can be trained, namely the above-mentioned first model and three second interpolation frame models.

[0094] It should be noted that the electronic device may also be trained using other training methods, which is not limited in the embodiments of the present application.

[0095] It can be seen that since the electronic device can obtain different training data sets including picture frames corresponding to different scene types, and use different training data sets to train the second interpolation model suitable for different scene types, therefore, in the subsequent steps, the electronic device can select the second interpolation model that matches the scene type corresponding to the second picture frame for interpolation, thereby improving the smoothness of the picture after interpolation while ensuring the picture quality of the picture.

[0096] Step 103b: The electronic device determines a model whose corresponding scene type in at least one second interpolation frame model matches the first scene type as a first interpolation frame model.

[0097] In some embodiments of the present application, the electronic device may first determine a scene type that is the same as the first scene type from at least one scene type corresponding to at least one second interpolation model, and then determine a second interpolation model corresponding to the scene type as the first interpolation model.

[0098] It can be understood that the first interpolation model is used to process the picture frames of the first scene type, that is, using the first interpolation model in the picture frames of the first scene type, an accurate intermediate picture frame (such as the third picture frame in the following embodiment) can be output.

[0099] For example, when the first scene type is an anime-style type, the first interpolation model can be understood as a model dedicated to processing anime-style picture frames, so that the intermediate picture frames output after processing by the first interpolation model are less blurred, that is, the picture quality of the intermediate picture frames is higher, thereby improving the effect of interpolating the picture frames corresponding to the first picture.

[0100] For another example, when the first scene type includes a ball type, the first interpolation model can be understood as a model specifically used to process picture frames including ball types. In this way, the intermediate picture frames output after processing by the first interpolation model have fewer artifacts, that is, the picture quality of the intermediate picture frames is higher, thereby improving the effect of interpolating the picture frames corresponding to the first picture.

[0101] Thus, it can be seen that since the electronic device can determine the first scene type corresponding to the second picture frame and determine the first interpolation model suitable for the first scene type from at least one second interpolation model, it is possible to avoid artifacts or interference caused by interpolation, thereby improving the smoothness of the first picture while ensuring the picture quality of the first picture.

[0102] Step 104: The electronic device performs frame insertion according to the first picture frame and the first frame insertion model.

[0103] In some embodiments of the present application, the electronic device may first input the first picture frame into the first interpolation model, and generate an intermediate picture frame according to the first picture frame through the first interpolation model, so that the electronic device can insert the intermediate picture frame after the first picture frame in the picture frame corresponding to the first picture, or can insert the intermediate picture frame after (or before) the picture frame currently being rendered, so as to interpolate the picture frame corresponding to the first picture.

[0104] The specific scheme of frame insertion by the electronic device is described below by way of example.

[0105] In some embodiments of the present application, Figure 1 ,like Figure 4 As shown, the above step 104 can be specifically implemented through the following steps 104a and 104b.

[0106] Step 104a: The electronic device inputs the first picture frame into the first interpolation model, and generates a third picture frame according to the first picture frame through the first interpolation model.

[0107] In the embodiment of the present application, since the first interpolation model is dedicated to processing the picture frames of the first scene type, after the first interpolation model processes the second picture frame, an accurate third picture frame can be output.

[0108] Step 104b: The electronic device inserts the third picture frame between the first picture frame and the fourth picture frame.

[0109] In the embodiment of the present application, the fourth picture frame is the next picture frame of the first picture frame.

[0110] It can be understood that the fourth picture frame is a picture frame in the picture frames corresponding to the first picture, and the fourth picture frame is a picture frame next to the first picture frame.

[0111] It can be seen that since the electronic device can input the first picture frame into the first interpolation model to quickly generate an accurate third picture frame through the first interpolation model, and insert the third picture frame between the first picture frame and the fourth picture frame, it is possible to improve the smoothness of the first picture while ensuring the picture quality of the first picture.

[0112] In some embodiments of the present application, after using the third picture frame to interpolate the picture frame corresponding to the first picture, the electronic device can again obtain the picture frame corresponding to the first picture, and again generate a picture frame with a lower resolution based on the picture frame through the DPU, and again determine the interpolation model based on the picture frame with a lower resolution (or directly determine the first interpolation model), and perform interpolation based on the picture frame and the interpolation model (or the first interpolation model).

[0113] It can be understood that, in the process of displaying the first picture, the electronic device can execute the above steps 101 to 104 multiple times to insert multiple frames of the picture frames corresponding to the first picture, thereby improving the fluency of the first picture.

[0114] The embodiment of the present application provides a method for interpolation, wherein the electronic device can obtain a first picture frame corresponding to the first picture, and generate a second picture frame according to the first picture frame through the display processing unit DPU, wherein the resolution of the second picture frame is less than the resolution of the first picture frame; thus, the electronic device can determine the first interpolation model based on the second picture frame, and interpolate according to the first picture frame and the first interpolation model. Since the electronic device can quickly generate a second picture frame with a lower resolution through the DPU when obtaining the first picture frame corresponding to the first picture, without generating the second picture frame through the neural processing unit NPU, the time consumption for generating the second picture frame can be reduced, and the second picture frame does not need to be generated through the graphics processor GPU, which can reduce the power consumption for generating the second picture frame; thus, the electronic device can quickly determine the first interpolation model based on the second picture frame, and quickly interpolate according to the first picture frame and the first interpolation model, thus reducing the time required for interpolation and reducing the power consumption required for interpolation, thus reducing the power consumption of the electronic device.

[0115] Furthermore, during the data input stage, the electronic device scales down the display screen on the DPU to obtain a second screen frame, and then writes the second screen frame back to the preprocessing model in the NPU. There is no need to pass high-resolution images to the preprocessing model in the NPU, which can reduce bandwidth; and there is no need to scale down the image on the NPU or GPU, which reduces the NPU / GPU load and improves performance. When the resolution is very large, such as 2K reduced to 300 pixels, the benefits of DPU scaling are more obvious.

[0116] The following is a specific example to illustrate the complete process of the interpolation method provided in the embodiment of the present application.

[0117] Example 1:

[0118] The frame insertion method provided in the embodiment of the present application may include the following steps:

[0119] Step 1: Prepare the training dataset.

[0120] Game and video images can be obtained from open source data sets, mobile phone screen recording, game rendering pipeline or video decoding framework to obtain multiple picture frames. And two training data sets (such as the first training data set and the second training data set in the above embodiment) can be obtained according to the real world style type and the animation style type. Next, ball sports videos containing small balls and game picture frames containing special effects can be specially recorded separately to obtain two training data sets (such as the third training data set and the fourth training data set in the above embodiment). In this way, four training data sets can be obtained, such as the first training data set, the second training data set, the third training data set and the fourth training data set, the first training data set includes at least one picture frame corresponding to the real world style type, the second training data set includes at least one picture frame corresponding to the animation style type, the third training data set includes at least one picture frame corresponding to the ball type, and the fourth training data set includes at least one picture frame corresponding to the special effect type.

[0121] Step 2: Model training.

[0122] The electronic device may first obtain a preset interpolation model, the structure of which is obtained by lightweight RIFE algorithm, and firstly train on the first training data set to obtain a general model (such as the first model in the above embodiment). Then fine-tune on the second training data set, the third training data set, and the fourth training data set, that is, use the second training data set, the third training data set, and the fourth training data set to train the general model respectively to obtain three interpolation models. Thus, the electronic device may determine the first model and the three interpolation models as the second interpolation models to determine four second interpolation models.

[0123] Step 3: Preprocess scene classification.

[0124] When it is determined that frame insertion is to be performed, the electronic device can directly obtain the first picture frame corresponding to the first picture (for example, the currently displayed frame) from the DPU. Since the resolution of the first picture frame is very high, downsampling processing can be performed on the DPU first to obtain a low-resolution second picture frame, and the second picture frame is written back to the NPU.

[0125] Next, the preprocessing model can be used on the NPU to classify the image to determine the first scene type corresponding to the second picture frame, and the first scene type is sent to the CPU of the electronic device. The preprocessing model can be a common mobile lightweight network, such as MobileNetV4.

[0126] Step 4: Insert frames.

[0127] The CPU of the electronic device selects a second interpolation frame model (e.g., the first interpolation frame model in the above embodiment) from the four second interpolation frame models according to the scene classification result, and inputs the first picture frame into the first interpolation frame model, and outputs a third picture frame (intermediate picture frame) according to the first picture frame through the first interpolation frame model. Thus, the electronic device can insert the intermediate picture frame behind (or in front of) the current rendering frame, or behind the first picture frame.

[0128] It should be noted that although the picture displayed by the DPU at the same moment may be several frames later than the rendered frame, the interpolation style is generally consistent in several consecutive frames, so the intermediate picture frame can be inserted into the current rendered frame.

[0129] For Example 1, the embodiment of the present application can achieve real-time interpolation of 2K high resolution on mobile terminals. Using a dedicated model in anime scenes has better effects than general models, with less blur caused by interpolation. Using a dedicated model in scenes containing special effects and small balls can reduce artifacts caused by interpolation. It takes very little power consumption to pre-process scene classification and obtain better interpolation effects.

[0130] Example 2:

[0131] Different from the first embodiment, the pre-processing scene classification adds a sleep mechanism, which has lower power consumption. The interpolation method provided in the embodiment of the present application may include the following steps:

[0132] Step 1: Prepare the training dataset.

[0133] It should be noted that for the description of step 1 in Example 2, reference can be made to the specific description of step 1 in Example 1, and the embodiments of the present application will not be repeated here.

[0134] Step 2: Model training.

[0135] It should be noted that for the description of step 2 in Example 2, reference can be made to the specific description of step 2 in Example 1, and the embodiments of the present application will not be repeated here.

[0136] Step 3: Preprocess scene classification.

[0137] Given a sleep interval Ams, when the time from the last preprocessing scene classification is less than Ams, directly run step 4, that is, directly transmit the scene classification determined by the last preprocessing scene classification to the CPU, so that the CPU can select the corresponding interpolation model according to the scene classification.

[0138] Otherwise, the current moment is recorded as the moment of the most recent preprocessing scene classification.

[0139] The first picture frame is reduced on the DPU to obtain a second picture frame, and the second picture frame is written back to the NPU. The second picture frame is classified using the preprocessing model to obtain a first scene classification, and the first scene classification is transmitted to the CPU.

[0140] Step 4: Insert frames.

[0141] According to the first scene classification (or the scene classification determined by the last preprocessing scene classification), a corresponding second interpolation frame model (i.e., the first interpolation frame model) is selected from the four second interpolation frame models. The first picture frame is input into the first interpolation frame model, and the third picture frame (intermediate picture frame) is output according to the first picture frame through the first interpolation frame model. Thus, the electronic device can insert the intermediate picture frame behind (or in front of) the current rendering frame, or insert it behind the first picture frame.

[0142] For Example 2, in addition to achieving the effect in Example 1, the embodiment of the present application can further improve the power consumption of pre-processing scene classification. Pre-processing scene classification is only run once every several milliseconds (i.e., every several frames), which consumes less power, because the classification of the scene generally does not change every frame, so the accuracy is still guaranteed.

[0143] The frame insertion method provided in the embodiment of the present application may be executed by a frame insertion device. In the embodiment of the present application, the frame insertion device provided in the embodiment of the present application is described by taking the frame insertion method executed by the frame insertion device as an example.

[0144] Figure 5 FIG. 2 shows a schematic diagram of the structure of a frame insertion device provided in an embodiment of the present application. Figure 5 As shown, the frame insertion device 50 provided in the embodiment of the present application may include: an acquisition module 51 and a processing module 52.

[0145] The acquisition module 51 is used to acquire a first picture frame corresponding to the first picture. The processing module 52 is used to generate a second picture frame through the DPU according to the first picture frame acquired by the acquisition module 51, wherein the resolution of the second picture frame is smaller than the resolution of the first picture frame; and determine a first interpolation model based on the second picture frame; and interpolate according to the first picture frame and the first interpolation model.

[0146] The embodiment of the present application provides a frame insertion device. When the frame insertion device obtains a first picture frame corresponding to a first picture, it can first quickly generate a second picture frame with a lower resolution through a DPU, without generating the second picture frame through a neural processing unit NPU. This can reduce the time consumption of generating the second picture frame, and does not need to generate the second picture frame through a graphics processing unit GPU. This can reduce the power consumption of generating the second picture frame. In this way, the frame insertion device can quickly determine a first frame insertion model based on the second picture frame, and quickly perform frame insertion according to the first picture frame and the first frame insertion model. Therefore, the time required for frame insertion can be reduced and the power consumption required for frame insertion can be reduced. In this way, the power consumption of the frame insertion device can be reduced.

[0147] In a possible implementation, the processing module 52 is specifically used to determine the first scene type corresponding to the second picture frame, and obtain at least one second interpolation model, each second interpolation model corresponds to a scene type; and determine the model whose corresponding scene type in at least one second interpolation model matches the first scene type as the first interpolation model.

[0148] In a possible implementation, the acquisition module 51 is further used to acquire at least one training data set before the processing module 52 determines the first interpolation model whose corresponding scene type matches the first scene type from the at least one second interpolation model, wherein each training data set includes at least one picture frame corresponding to a scene type. The processing module 52 is further used to train the preset model using each training data set in the at least one training data set acquired by the acquisition module 51 to obtain each second interpolation model in the at least one second interpolation model.

[0149] In a possible implementation, the processing module 52 is specifically used to determine the second scene type as the first scene type when the first duration is less than a preset duration; or, when the first duration is greater than or equal to the preset duration, input the second picture frame into a preprocessing model to determine the first scene type through the preprocessing model; wherein the first duration is: the duration from the moment when the scene type corresponding to the picture frame was last determined to the moment when the first picture frame was obtained, and the second scene type is the scene type determined last time.

[0150] In a possible implementation, the processing module 52 is further configured to determine a picture type corresponding to the first picture frame before generating the second picture frame through the DPU according to the first picture frame. The processing module 52 is specifically configured to generate the second picture frame through the DPU according to the first picture frame when the picture type matches the preset picture type.

[0151] In a possible implementation, the processing module 52 is specifically used to input the first picture frame into the first interpolation model, generate a third picture frame according to the first picture frame through the first interpolation model; and insert the third picture frame between the first picture frame and the fourth picture frame, and the fourth picture frame is the next picture frame of the first picture frame.

[0152] The interpolation device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile internet device (mobile internet device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (network attached storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0153] The interpolation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0154] The frame insertion device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0155] In some embodiments of the present application, Figure 6 As shown, an embodiment of the present application further provides an electronic device 60, including a processor 61 and a memory 62, wherein the memory 62 stores programs or instructions that can be executed on the processor 61, and when the program or instructions are executed by the processor 61, the various process steps of the above-mentioned interpolation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0156] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0157] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0158] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0159] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0160] Among them, the processor 110 is used to obtain a first picture frame corresponding to the first picture; and generate a second picture frame according to the first picture frame through the DPU, and the resolution of the second picture frame is smaller than the resolution of the first picture frame; and determine a first interpolation model based on the second picture frame; interpolation is performed according to the first picture frame and the first interpolation model.

[0161] An embodiment of the present application provides an electronic device. When the electronic device obtains a first picture frame corresponding to a first picture, it can first quickly generate a second picture frame with a lower resolution through a DPU without generating the second picture frame through a neural processing unit NPU. This can reduce the time consumption of generating the second picture frame, and does not need to generate the second picture frame through a graphics processor GPU. This can reduce the power consumption of generating the second picture frame. In this way, the electronic device can quickly determine a first interpolation model based on the second picture frame, and quickly interpolate according to the first picture frame and the first interpolation model. Therefore, the time required for interpolation can be reduced and the power consumption required for interpolation can be reduced. In this way, the power consumption of the electronic device can be reduced.

[0162] In some embodiments of the present application, the processor 110 is specifically used to determine the first scene type corresponding to the second picture frame, and obtain at least one second interpolation model, each second interpolation model corresponds to a scene type; and determine the model whose corresponding scene type in at least one second interpolation model matches the first scene type as the first interpolation model.

[0163] In some embodiments of the present application, the processor 110 is further used to obtain at least one training data set before determining a first interpolation model whose corresponding scene type matches the first scene type from at least one second interpolation model, each training data set respectively including at least one picture frame corresponding to a scene type; and use each training data set in the at least one training data set to train the preset model respectively to obtain each second interpolation model in the at least one second interpolation model.

[0164] In some embodiments of the present application, the processor 110 is specifically used to determine the second scene type as the first scene type when the first duration is less than a preset duration; or, when the first duration is greater than or equal to the preset duration, input the second picture frame into a preprocessing model to determine the first scene type through the preprocessing model; wherein the above-mentioned first duration is: the duration from the moment when the scene type corresponding to the picture frame was last determined to the moment when the first picture frame was acquired, and the above-mentioned second scene type is the scene type determined last time.

[0165] In some embodiments of the present application, the processor 110 is further configured to determine a picture type corresponding to the first picture frame before generating a second picture frame according to the first picture frame through the DPU.

[0166] The processor 110 is specifically configured to generate a second picture frame according to the first picture frame through the DPU when the picture type matches the preset picture type.

[0167] In some embodiments of the present application, the processor 110 is specifically used to input the first picture frame into the first interpolation model, generate a third picture frame according to the first picture frame through the first interpolation model; and insert the third picture frame between the first picture frame and the fourth picture frame, and the fourth picture frame is the next picture frame of the first picture frame.

[0168] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0169] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0170] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0171] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned interpolation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0172] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0173] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned interpolation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0174] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0175] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned interpolation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0176] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0178] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A frame insertion method, characterized in that: include: Obtaining a first picture frame corresponding to the first picture; Generate a second picture frame according to the first picture frame by a display processing unit DPU, wherein the resolution of the second picture frame is smaller than the resolution of the first picture frame; Determine a first interpolation frame model based on the second picture frame; Frame interpolation is performed according to the first picture frame and the first frame interpolation model.

2. The method according to claim 1, characterized in that The determining a first interpolation frame model based on the second picture frame comprises: Determine a first scene type corresponding to the second picture frame, and obtain at least one second interpolation frame model, each second interpolation frame model corresponding to a scene type; A model whose corresponding scene type in the at least one second interpolation frame model matches the first scene type is determined as the first interpolation frame model.

3. The method according to claim 2, characterized in that The determining the first scene type corresponding to the second picture frame includes: When the first duration is less than the preset duration, determining the second scene type as the first scene type; or, When the first duration is greater than or equal to a preset duration, inputting the second picture frame into a preprocessing model, and determining the first scene type through the preprocessing model; The first duration is the duration from the moment when the scene type corresponding to the picture frame is determined last time to the moment when the first picture frame is acquired, and the second scene type is the scene type determined last time.

4. The method according to claim 1, characterized in that: Before generating the second picture frame according to the first picture frame by the DPU, the method further includes: Determining a picture type corresponding to the first picture frame; The generating a second picture frame according to the first picture frame by the DPU includes: In a case where the picture type matches a preset picture type, the second picture frame is generated according to the first picture frame by the DPU.

5. The method according to claim 1, characterized in that The performing frame interpolation according to the first picture frame and the first frame interpolation model includes: Inputting the first picture frame into the first interpolation frame model, and generating the third picture frame according to the first picture frame through the first interpolation frame model; The third picture frame is inserted between the first picture frame and a fourth picture frame, and the fourth picture frame is a next picture frame of the first picture frame.

6. A frame insertion device, characterized in that: The frame insertion device comprises: an acquisition module and a processing module; The acquisition module is used to acquire a first picture frame corresponding to the first picture; The processing module is used to generate a second picture frame through the DPU according to the first picture frame obtained by the acquisition module, wherein the resolution of the second picture frame is smaller than the resolution of the first picture frame; and determine a first interpolation model based on the second picture frame; and perform interpolation according to the first picture frame and the first interpolation model.

7. The device according to claim 6, characterized in that The processing module is specifically used to determine the first scene type corresponding to the second picture frame, and obtain at least one second interpolation model, each second interpolation model corresponds to a scene type; and determine the model whose corresponding scene type in the at least one second interpolation model matches the first scene type as the first interpolation model.

8. The device according to claim 7, characterized in that The processing module is specifically configured to determine the second scene type as the first scene type when the first duration is less than a preset duration; or, when the first duration is greater than or equal to a preset duration, input the second picture frame into a preprocessing model, and determine the first scene type through the preprocessing model; The first duration is the duration from the moment when the scene type corresponding to the picture frame is determined last time to the moment when the first picture frame is acquired, and the second scene type is the scene type determined last time.

9. The device according to claim 6, characterized in that The processing module is further configured to determine a picture type corresponding to the first picture frame before generating the second picture frame according to the first picture frame through the DPU; The processing module is specifically configured to generate the second picture frame according to the first picture frame through the DPU when the picture type matches the preset picture type.

10. The device according to claim 6, characterized in that The processing module is specifically used to input the first picture frame into the first interpolation model, generate the third picture frame according to the first picture frame through the first interpolation model; and insert the third picture frame between the first picture frame and the fourth picture frame, and the fourth picture frame is the next picture frame of the first picture frame.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the interpolation method according to any one of claims 1 to 5 are implemented.

Images