Video processing method, device, equipment and readable storage medium

The generation of intermediate frames through non-integer multiple interpolation and deep learning models has solved the problem of poor flexibility and artifact improvement in frame rate in the existing technology, and achieved efficient video fluency and visual quality improvement.

CN114938461BInactive Publication Date: 2024-11-01CHINANETCENT TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210349214.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-01
Publication Date
2024-11-01
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing video interpolation method can only achieve an integer multiple frame rate improvement, poor flexibility, high limitations, and has artifacts on high-speed motion videos, which cannot meet users' demand for non-integrated multiple frame rate.

Method used

Using non-integer multiple interpolation method, grouping and interpolation of image frames of the source video to generate target videos, supporting non-integer multiple improvements in frame rate, using deep learning models such as RIFE to generate intermediate frames to improve video fluency and vision quality.

Benefits of technology

It achieves flexible frame rate improvement without being restricted by integer multiples, enhances the smoothness and visual quality of the video, avoids artifact problems, and is suitable for a variety of video scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114938461B_ABST
    Figure CN114938461B_ABST
Patent Text Reader

Abstract

The present application discloses a video processing method, apparatus, device and readable storage medium. When the electronic device determines that the interpolation mode of the source video is a non-integer multiple interpolation mode, the image frames contained in the source video are grouped according to the first frame rate of the source video to obtain multiple groups. Then, the electronic device interpolates each group separately to obtain the target video and outputs it. With this solution, non-integer multiple interpolation modes are supported, and it is not limited to integer multiple interpolation. It has high flexibility and a wide range of applications, and achieves the purpose of increasing the frame rate and enhancing the visual quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, device, equipment and readable storage medium. Background Art

[0002] With the development of video technology and Internet technology, it has become a common phenomenon for users to watch videos on personal computers, mobile phones, tablets and other electronic devices, and the number of various videos is growing rapidly.

[0003] At the same time, users' demands for videos are becoming more and more diverse. Some videos need to be modified before they can be played. Image frame rate enhancement is a video post-processing method that converts low frame rate videos into high frame rate videos. This method generates a new intermediate frame based on the knowledge learned from the corresponding pixels of two adjacent image frames, and inserts the intermediate frame between the two adjacent image frames to achieve the purpose of increasing the frame rate. For example, the image frame rate is increased from 30 frames per second (FPS) to 60FPS.

[0004] However, the existing interpolation method can only achieve interpolation times the power of 2, which has poor flexibility, high limitations and poor effect. Summary of the invention

[0005] The present application provides a video processing method, apparatus, device and readable storage medium, which have high flexibility, wide application range and improved video processing effect by supporting non-integer multiples of interpolation frames.

[0006] In a first aspect, an embodiment of the present application provides a video processing method, including:

[0007] Determine the frame interpolation method of the source video;

[0008] When the interpolation mode is a non-integer multiple interpolation mode, the image frames included in the source video are grouped according to the first frame rate of the source video to obtain a plurality of groups, each of the plurality of groups includes a plurality of continuous image frames, and the non-integer multiple interpolation mode is used to indicate an interpolation mode of obtaining a target video by increasing the first frame rate by a non-integer multiple of 2;

[0009] Inserting frames into each of the plurality of groups to obtain the target video;

[0010] The target video is output.

[0011] In a second aspect, an embodiment of the present application provides a processing device, including:

[0012] A determination module, used to determine the frame insertion method of the source video;

[0013] a processing module, configured to group the image frames included in the source video according to the first frame rate of the source video to obtain a plurality of groups, each of the plurality of groups including a plurality of continuous image frames, when the interpolation mode is a non-integer multiple interpolation mode, and the non-integer multiple interpolation mode is used to indicate an interpolation mode of increasing the first frame rate by a non-integer multiple of 2 to obtain a target video;

[0014] A frame insertion module, used for inserting frames into each of the plurality of groups to obtain the target video;

[0015] An output module is used to output the target video.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in the first aspect or various possible implementation methods of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method described in the first aspect or various possible implementation methods of the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computing program, and when the computer program is executed by a processor, the method described in the first aspect or various possible implementation methods of the first aspect is implemented.

[0019] The video processing method, apparatus, device and readable storage medium provided by the embodiment of the present application, when the electronic device determines that the interpolation mode of the source video is a non-integer multiple interpolation mode, the image frames contained in the source video are grouped according to the first frame rate of the source video to obtain multiple groups. Then, the electronic device interpolates each group separately to obtain the target video and outputs it. With this solution, non-integer multiple interpolation modes are supported, which are not limited to integer multiple interpolation, have high flexibility and a wide range of applications, and achieve the purpose of increasing the frame rate and enhancing the visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1Schematic diagram of a network architecture applicable to the video processing method provided in the embodiment of the present application;

[0022] Figure 2 is a flow chart of a video processing method provided by an embodiment of the present application;

[0023] Figure 3 This is a schematic diagram of a process of non-integer multiple frame insertion in a video processing method provided in an embodiment of the present application;

[0024] Figure 4 This is a schematic diagram of a frame insertion process in a video processing method provided in an embodiment of the present application;

[0025] Figure 5 is another schematic diagram of a frame insertion process in the video processing method provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of the process of inserting frames of a target group in the video processing method provided in an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of a process of inserting a frame between a first frame and a second frame in a video processing method provided in an embodiment of the present application;

[0028] Figure 8 is another flow chart of the video processing method provided by an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of a video processing device provided in an embodiment of the present application;

[0030] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0032] With the rapid development of the Internet, there are more and more videos, and users have higher requirements for video quality, including clarity, smoothness, color saturation, etc. Frame rate improvement is a common method to improve video smoothness. High frame rate video greatly improves the user's viewing experience. In order to let users watch silky smooth videos, the frame rate of the camera video is improved from 25FPS to 60FPS, and then to 240FPS or even higher by improving the camera hardware. However, if the frame rate is improved only by hardware iteration, the cost is high. For this reason, video interpolation technology is applied. The purpose of video interpolation is to generate high frame rate video based on low frame rate video.

[0033] At present, there are more and more scenarios where the frame rate is improved by video interpolation technology. For example, users expect to convert some old, low-frame-rate videos that were not processed well at the beginning into high-frame-rate videos. For another example, under the condition of limited channel bandwidth, the low-frame-rate video sent by the encoder only transmits a part of the video content. The decoder restores the complete video content based on the video interpolation technology, thereby improving the utilization rate of the channel bandwidth while ensuring that the video quality meets the user's viewing needs. For another example, different media products support different frame rates. For example, the frame rate of a movie is 24FPS, which is suitable for movie playback. However, the frame rate of TV playback is 30FPS. In order to play the movie on TV and obtain high-quality visual quality, the movie must be interpolated to obtain a movie with a frame rate of 30FPS.

[0034] The existing video interpolation method is a 2-power interpolation method, which is also called an integer interpolation method, a double frame rate increase method, an integer frame rate increase, a 2-power interpolation method, etc. By increasing the frame rate by an integer multiple of 2 to obtain a high frame rate video, the purpose of improving visual quality and increasing video realism and smoothness is achieved. Assuming that the first frame rate of the source video is M, the second frame rate of the target video is N, and N / M=k, then k must be a multiple of 2.

[0035] However, sometimes the second frame rate of the target video that the user expects is not an integer multiple of 2 of the first frame rate of the source video. For example, the first frame rate of the source video is 24FPS, the second frame rate of the target video is 30FPS, and N / M = 5 / 4; for another example, the first frame rate of the source video is 25FPS, the second frame rate of the target video is 60FPS, and N / M = 12 / 5. In addition, the existing interpolation method has the disadvantages of slow speed and artifacts for high-speed motion videos.

[0036] Based on this, the embodiments of the present application provide a video processing method, apparatus, device and readable storage medium, which have high flexibility, wide application range and improved video processing effect by supporting non-integer multiples of interpolation frames.

[0037] Figure 1 This is a schematic diagram of a network architecture applicable to the video processing method provided in the embodiment of the present application. Figure 1 The network architecture includes: a server 11 and a terminal device 12. A network connection is established between the terminal device 12 and the server 11. The network connection can be a wired communication connection, a wireless communication connection, or an optical fiber cable.

[0038] The server 11 has huge computing power, storage capacity, etc., and can provide services to the terminal device 12. The server 11 can be hardware or software. When the server 11 is hardware, the server 11 is a single server or a distributed server cluster composed of multiple servers. When the server 11 is software, it can be multiple software modules or a single software module, etc., which is not limited by the embodiments of the present application.

[0039] The server 11 can group the adaptation frames contained in the source video, generate intermediate frames for each group, and interpolate the groups. Video Frame Interpolation (VFI) aims to synthesize intermediate frames between two consecutive image frames, and is widely used to improve frame rate and enhance visual quality. The server 11 deploys interpolation models based on deep learning, such as the Depth-Aware Video Frame Interpolation (DAIN) model, Super-Slomo, Quadratic model, Real-Time Intermediate Flow Estimation for Video Frame Interpolation (RIFE) model, etc., inputs two adjacent video frames into the interpolation model, and the interpolation model learns the corresponding pixels of the two frames of video to generate an intermediate frame. When the interpolation model is a RIFE model, the speed of generating intermediate frames is fast.

[0040] The terminal device 12 can be hardware or software. When the terminal device 12 is hardware, the terminal device 12 is, for example, a mobile phone, a tablet computer, an e-book reader, a laptop computer, a desktop computer, etc. When the terminal device 12 is software, it can be installed in the hardware devices listed above. In this case, the terminal device 12 is, for example, multiple software modules or a single software module, etc., which is not limited in the embodiments of the present application.

[0041] During the video processing, the user selects a source video on the terminal device 12 and sends an indication message to the server 12, where the indication message indicates the source video and also indicates the interpolation multiple or the second frame rate of the target video. After receiving the indication message, the server 11 obtains the source video and performs interpolation processing on the source video to obtain the target video.

[0042] In the embodiments of the present application, in addition to converting a low frame rate video into a high frame rate video by interpolating frames, interpolating frames can also be used to generate slow motion, video compression, generate training data, etc. Among them, the training data is used to train a deblurring model. After the deblurring model is trained, a clear motion video is obtained by inputting a blurred motion video; or, a blurred image is input into the deblurring model to obtain a clear image. The video processing method provided in the embodiments of the present application runs on a high-resolution video at real-time speed, enables the player to play videos with a higher frame rate, and provides video editing services to users.

[0043] It should be noted that although Figure 1 In the illustrated architecture, the server 11 performs video interpolation on the source video. However, the embodiment of the present application is not limited to this. In other feasible implementations, the interpolation model and the like may also be deployed on the terminal device 12. The terminal device 12 does not need to indicate the source video to the server 11, and does not need to indicate the interpolation multiple or the second frame rate of the target video to the server 12. Instead, the source video of the first frame rate is interpolated locally to obtain the target video of the second frame rate according to the indication information input by the user.

[0044] Below, based on Figure 1 The video processing method described in the embodiment of the present application is described in detail with reference to the architecture shown. Figure 2 , Figure 2 is a flow chart of a video processing method provided by an embodiment of the present application. The execution subject of this embodiment is an electronic device, which may be Figure 1 The terminal device or server in this embodiment includes:

[0045] 201. Determine a frame insertion method for the source video.

[0046] In an embodiment of the present application, the interpolation method includes a non-integer interpolation method and an integer interpolation method. The non-integer interpolation method refers to: the first frame rate is increased by a non-integer multiple of 2 to obtain an interpolation method of the target video. For example, the first frame rate of the source video is 24FPS, and the second frame rate of the target video is 30FPS, which means that the first frame rate of the source video is increased by 5 / 4 times to obtain the target video. For another example, the first frame rate of the source video is 25FPS, and the second frame rate of the target video is 60FPS, which means that the first frame rate of the source video is increased by 12 / 5 times to obtain the target video. For another example, the first frame rate of the source video is 25FPS, and the second frame rate of the target video is 75FPS, which means that the first frame rate of the source video is increased by 3 times to obtain the target video.

[0047] The integer multiple interpolation method refers to: the first frame rate is increased by an integer multiple of 2 to obtain the interpolation method of the target video. For example, the first frame rate of the source video is 10FPS, and the second frame rate of the target video is 20FPS, which means that the first frame rate of the source video is increased by 2 times to obtain the target video. For another example, the first frame rate of the source video is 25FPS, and the second frame rate of the target video is 100FPS, which means that the first frame rate of the source video is increased by 4 times to obtain the target video.

[0048] In an embodiment of the present application, the electronic device can flexibly determine the interpolation mode. In one embodiment, a default interpolation mode is pre-set, and the electronic device determines the interpolation mode according to the default value. In another embodiment, when the electronic device is a terminal device, the terminal device displays an interface, and the user enters the interpolation multiple or the second frame rate on the user interface.

[0049] When the user inputs the interpolation multiple, the terminal device determines the interpolation method as integer multiple interpolation. For example, if the user inputs 2, it means to interpolate the source video by 2 times. During the interpolation process, the electronic device generates an intermediate frame based on every two adjacent image frames of the source video, and inserts the intermediate frame between the two adjacent image frames. Taking a video with a first frame rate of 24PFS and a length of 1 hour as an example, the source video contains 86400 (24×60×60=86400) image frames. By inserting 1 intermediate frame between every two adjacent image frames, the target video is obtained, and the second frame rate of the target video is 48FPS. It should be noted that since 86399 image frames can be generated according to the source video, the target video has one less frame. However, since the number of image frames is relatively large, the target video can be ignored for having one less image frame.

[0050] For another example, the user inputs 4, which means to perform 4x interpolation on the source video. During the interpolation process, the electronic device generates an intermediate frame based on every two adjacent image frames of the source video, thereby obtaining three consecutive image frames, and then generates and inserts an intermediate frame based on every two adjacent image frames in the three consecutive image frames.

[0051] When the user inputs the second frame rate, the terminal device determines that the interpolation method is non-integer interpolation. For example, if the first frame rate is 24FPS and the second frame rate input by the user is 60FPS, it means non-integer interpolation. For another example, if the first frame rate is 25FPS and the second frame rate input by the user is 75FPS, it means non-integer interpolation. For convenience, the first frame rate → second frame rate is used below to indicate that the source video with the first frame rate is interpolated at a non-integer multiple to obtain the target video with the second frame rate. For example, 24FPS → 50FPS means that the source video with the first frame rate of 24FPS is interpolated at a non-integer multiple to obtain the target video with the second frame rate of 50FPS.

[0052] In another embodiment, the electronic device is the server described above, and the user inputs the interpolation multiple or the second frame rate on the user interface of the terminal device. The terminal device sends an indication message to the server. Correspondingly, the server receives the indication message, and when the indication message indicates the second frame rate, the server determines that the interpolation mode is a non-integer multiple interpolation mode; when the indication message indicates the interpolation multiple, the server determines that the interpolation mode is an integer multiple interpolation mode. This solution is adopted to achieve the purpose of flexibly specifying the interpolation mode.

[0053] 202. When the interpolation mode is a non-integer interpolation mode, the image frames contained in the source video are grouped according to the first frame rate of the source video to obtain a plurality of groups, each of the plurality of groups contains a plurality of continuous image frames, and the non-integer interpolation mode is used to indicate an interpolation mode of increasing the first frame rate by an integer multiple of 2 to obtain a target video.

[0054] In the embodiment of the present application, an image frame is the smallest unit of a video, and an image frame is a still picture. Based on the visual persistence effect of the human eye, multiple image frames are played quickly in sequence to form a video. The electronic device uses video processing software to cut the source video into multiple image frames.

[0055] After the electronic device cuts the source video into multiple image frames, it groups these image frames to obtain multiple groups, each of which contains multiple continuous image frames. For example, if the first frame rate of the source video is 24FPS and the length is 1 hour, the source video contains 86,400 image frames. In the first grouping method, the electronic device groups every 4 image frames into a group, thereby obtaining 21,600 groups. In this grouping method, any two groups have no intersection, that is, there are no image frames that belong to two groups at the same time.

[0056] In the second grouping method, two adjacent groups in the multiple groups are the first group and the second group, respectively, and the last image frame of the first group is the first image frame of the second group. In other words, there is an intersection between two adjacent groups. For example, the first frame rate of the source video is 24FPS, and the first 25 frames are numbered from frame 0 to frame 24, then there are a total of 8 groups, the first group is frame 0-frame 3, the second group is frame 3-frame 6, the third group is frame 6-frame 9, the fourth group is frame 9-frame 12, the fifth group is frame 12-frame 15, the sixth group is frame 15-frame 18, the seventh group is frame 18-frame 21, and the eighth group is frame 21-frame 24. With this solution, there are common image frames between two adjacent groups, which can ensure that the target video obtained after interpolation processing has a higher degree of fluency.

[0057] 203. Perform frame insertion on each of the multiple groups to obtain the target video.

[0058] Exemplarily, the electronic device is based on methods such as deep learning

[0059] For each group, the electronic device determines the first number of frames to be inserted in the group, the position of the inserted frames, etc., and then generates intermediate frames according to the first number, and inserts the intermediate frames into appropriate positions, thereby obtaining the target video. In the process of generating the intermediate frames, the electronic device learns and inserts the intermediate frames based on the two adjacent image frames for each insertion position, that is, between two adjacent image frames, based on a deep learning method.

[0060] 204. Output the target video.

[0061] In the embodiment of the present application, the second frame rate is higher than the first frame rate. The electronic device generates the target video and then outputs it for the user to click to play, etc., so that the user can watch the target video with higher fluency.

[0062] In the video processing method provided in the embodiment of the present application, when the electronic device determines that the interpolation mode of the source video is a non-integer multiple interpolation mode, the image frames contained in the source video are grouped according to the first frame rate of the source video to obtain multiple groups. Then, the electronic device interpolates each group separately to obtain the target video and outputs it. This solution supports non-integer multiple interpolation mode, is not limited to integer multiple interpolation, has high flexibility and a wide range of application, and achieves the purpose of increasing the frame rate and enhancing the visual quality.

[0063] Optionally, in the above embodiment, when the interpolation mode is a non-integer multiple interpolation mode, if the first frame rate is 24FPS, the electronic device groups the image frames contained in the source video into groups of every 4 frames to obtain multiple groups. For example, in scenarios such as 24FPS→30FSP, 24FPS→60FPS, 24FPS→120FPS, the electronic device groups the image frames contained in the source video into groups of every 4 frames to obtain multiple groups.

[0064] If the first frame rate is 25 FPS, the electronic device groups the image frames included in the source video into groups of every 5 frames to obtain multiple groups. For example, in scenarios such as 25 FPS→30 FPS, 25 FPS→60 FPS, and 50 FPS→60 FPS, the electronic device groups the image frames included in the source video into groups of every 5 frames to obtain multiple groups.

[0065] This solution can achieve accurate frame insertion by reasonably grouping the source videos.

[0066] Figure 3 It is a schematic diagram of the process of non-integer frame insertion in the video processing method provided in an embodiment of the present application.

[0067] Please refer to Figure 3, this embodiment includes:

[0068] 301. The electronic device determines a first number of frames that need to be inserted into each of the multiple packets.

[0069] Exemplarily, the electronic device determines the number of interpolated frames required in each group according to the number of groups, the first frame rate, the second frame rate, etc. For example, if the first frame rate is 24FPS and the second frame rate is 60FPS, then for the first second of the source video, the number of interpolated frames required is 36 (60-24=36) frames. Assuming that the first grouping method described above is adopted, each group is composed of 4 image frames, and there is no intersection between two adjacent groups, and a total of 6 groups are divided, then each group needs to insert 6 intermediate frames.

[0070] Assuming that the second grouping method mentioned above is adopted, each group consists of 4 image frames, and there is an intersection between two adjacent groups, a total of 8 groups, among which the last group has only 3 image frames, which are filled with image frames in the source video at the 2nd second. Therefore, 5 intermediate frames need to be inserted into each group.

[0071] 302. The electronic device determines, according to the first number, a second number of interpolated frames required between every two adjacent image frames in each group.

[0072] Wherein, the second number is at least two, each grouping includes a first subgroup and a second subgroup, the first subgroup includes two adjacent image frames, the second subgroup includes another two adjacent image frames, and the second number corresponding to the first subgroup is greater than the second number corresponding to the second subgroup.

[0073] Exemplarily, due to the non-integer multiple frame insertion, the number of intermediate frames inserted in the first subgroup and the number of intermediate frames inserted in the second subgroup are different.

[0074] 303. Perform frame insertion on each of the multiple groups according to the second number to obtain the target video.

[0075] Figure 4 It is a schematic diagram of a frame insertion process in the video processing method provided in an embodiment of the present application. Figure 4 In the example, the source videos are grouped based on the first grouping method.

[0076] Please refer to Figure 4 , assuming that the 24 image frames are frame 0 to frame 23, the first group includes frame 0, frame 1, frame 2, and frame 3. Figure 4 Only the first group is shown in the figure. During the frame insertion process, the electronic device determines that 6 intermediate frames are to be inserted into each group. In the first group, the first subgroup is the first frame and the second frame, and the second number is 3. The second subgroup has two, namely the 0th frame and the 1st frame, and the 2nd frame and the 3rd frame, and the second numbers are 2 and 1 respectively.

[0077] Initially, the first group only contains the image frames filled with white in the figure. The electronic device generates the 0.5th frame based on the 0th frame and the 1st frame; generates the 1.5th frame based on the 1st frame and the 2nd frame; and generates the 2.5th frame based on the 2nd frame and the 3rd frame, as shown in the middle frame filled with diagonal lines in the figure.

[0078] For the 0th frame and the 1st frame, since the second number corresponding to the second subgroup is 2, the electronic device generates the 0.75th frame according to the 0.5th frame and the 1st frame, or generates the 0.25th frame according to the 0th frame and the 0.5th frame, thereby inserting two intermediate frames between the 0th frame and the 1st frame, as shown by the intermediate frames filled with vertical lines in the figure.

[0079] For the 1st frame and the 2nd frame, since the pair of adjacent frames is the first subgroup and the second number is 3, the electronic device generates the 1.25th frame according to the 1st frame and the 1.5th frame, and generates the 1.75th frame according to the 1.5th frame and the 2nd frame, thereby inserting 3 intermediate frames between the 1st frame and the 2nd frame, as shown by the intermediate frames filled with vertical lines in the figure.

[0080] Figure 5 This is another schematic diagram of the frame insertion process in the video processing method provided in an embodiment of the present application. Figure 5 In the method, the source videos are grouped based on the second grouping method.

[0081] Please refer to Figure 5 , continue to use the 24FPS→60FPS example, based on Figure 5 Only the first group is shown. During the frame insertion process, the electronic device determines that 5 intermediate frames are to be inserted into each group. In the first group, the first subgroup is the first frame and the second frame, and the second number is 3. The second subgroup has two, namely, the 0th frame and the 1st frame, and the 2nd frame and the 3rd frame, and the second number is 1.

[0082] Initially, the first group only contains the image frames filled with white in the figure. The electronic device generates the 0.5th frame based on the 0th frame and the 1st frame, generates the 1.5th frame based on the 1st frame and the 2nd frame, and generates the 2.5th frame based on the 2nd frame and the 3rd frame.

[0083] For frame 1 and frame 2, since the pair of adjacent frames is the first subgroup and the second number is 3, the electronic device generates frame 1.25 according to frame 1 and frame 1.5, and generates frame 1.75 according to frame 1.5 and frame 2, thereby inserting 3 intermediate frames between frame 1 and frame 2.

[0084] With this solution, the electronic device determines the first number of frames required to be inserted in each group and the second number of frames required to be inserted between each two adjacent image frames in each group, and then inserts frames in the group, thereby achieving the most uniform insertion and improving the quality of the target video.

[0085] Optionally, in the above embodiment, the electronic device may specify the position of the first subgroup in each group. For example, the second and third images in each group are taken as the first subgroup, that is, Figure 5 The first and second frames in the image are interpolated four times between the first and second frames, and two times at the remaining positions. Alternatively, the electronic device randomly determines two adjacent image frames from each group to obtain the first subgroup, and uses the subgroup formed by the image frames other than the first subgroup in each group as the second subgroup. For example, in the first group, the first subgroup includes the first and second frames; the second group includes the third, fourth, fifth and sixth frames, and the fifth and sixth frames are randomly used as the first subgroup, and four times interpolation is performed between the fifth and sixth frames, and two times interpolation is performed at the remaining positions.

[0086] By adopting this solution, a first group is randomly generated in each group, that is, a special position is randomly generated, and one more round of frame insertion is performed at the special position, so as to achieve the purpose of fast and accurate non-integer multiple frame insertion.

[0087] Optionally, in the above embodiment, when interpolating the first subgroup, the second number corresponding to the first subgroup is 2 n -1, n≥2 and is an integer. When the first subgroup is interpolated 4 times, n=2, and the second number is 3; when the first subgroup is interpolated 8 times, n=3, and the second number is 7... and so on. During the interpolation process, the electronic device generates an intermediate frame based on two adjacent video frames in the first subgroup. After obtaining three consecutive image frames, the electronic device continues to interpolate between every two consecutive image frames in the three image frames until the number of intermediate frames in the first subgroup reaches the second number. With this solution, the quality of the target video is improved by evenly generating and inserting intermediate frames between two adjacent video frames.

[0088] Figure 6 This is a schematic diagram of the process of inserting frames into a target group in the video processing method provided in the embodiment of the present application. Figure 6 , this embodiment includes:

[0089] 601. For a target group among multiple groups, an electronic device reads and caches a first image frame in the target group, wherein the target group is a group among the multiple groups that is currently inserting frames.

[0090] Optical flow-based interpolation models, such as the RIFE model, are deployed on electronic devices. The model structure and acceleration tools are used to compress the processing time of image frames, thereby speeding up the processing without affecting the interpolation effect.

[0091] In the embodiment of the present application, the target group is any one of the multiple groups, and any two adjacent image frames in the target group are referred to as the first image frame and the second image frame, and the first image frame is located before the second image frame.

[0092] When the electronic device interpolates the first image frame and the second image frame, the electronic device first reads the first image frame, and after reading the first image frame, the interpolation model is not called, but the first image frame is cached. Then, the electronic device reads the second image frame.

[0093] 602. The electronic device reads a second image frame in the target group, where the first image frame and the second image frame are two adjacent image frames in the target group, and the first image frame is located before the second image frame.

[0094] 603. The electronic device interpolates the first image frame and the second image frame according to a second number corresponding to the first image frame and the second image frame.

[0095] After the electronic device reads the second image frame, it determines the second number corresponding to the first image frame and the second image frame, and then determines how many times to insert frames between the first image frame and the second image frame. When the second number is 3, 4 times of insert frames are made between the first image frame and the second image frame, and 3 rounds of models are called to generate 3 new intermediate frames.

[0096] Please refer to Figure 5 , assuming that the first image frame is frame 1 and the second image frame is frame 2, when the interpolation model is called in the first round, frame 1 and frame 2 are input to obtain frame 1.5. When the interpolation model is called in the second round, frame 1 and frame 1.5 are output to obtain frame 1.25. When the interpolation model is called in the third round, frame 1.5 and frame 2 are input to obtain frame 1.75. Afterwards, frame 1.25, frame 1.5, and frame 1.75 are sequentially inserted between frame 1 and frame 2, thereby completing 4x interpolation between frame 1 and frame 2.

[0097] With this solution, the electronic device performs frame insertion by reading and processing at the same time, thereby improving the efficiency of frame insertion.

[0098] Optionally, in the above embodiment, when the electronic device executes the above step 603, the first image frame and the second number of intermediate frames are sequentially output, and the second number of intermediate frames are image frames generated by interpolating the first image frame and the second image frame. After that, the second image frame is cached, and then the image frame adjacent to the second image frame and located after the second image frame is read, and the first image frame and the second number of intermediate frames are encoded while continuing to generate intermediate frames.

[0099] Continuing with the above example, the electronic device outputs the 1st frame, the 1.25th frame, the 1.5th frame, and the 1.75th frame. Afterwards, the 2nd frame is cached, the 3rd frame is read, and the number of times of interpolation between the 2nd frame and the 3rd frame is determined based on the second number. For example, if the second number is 1 and n=2, and 2 times of interpolation is performed between the 2nd frame and the 3rd frame, the 2nd frame and the 3rd frame are input into the interpolation model to obtain the 2.5th frame, the 2nd frame and the 2.5th frame are output, the 3rd frame is cached, and the 4th frame is read continuously until the interpolation is completed. While caching the 3rd frame, the 2nd frame and the 2.5th frame are encoded.

[0100] With this solution, electronic devices insert frames by reading, processing, caching and encoding at the same time, which improves the efficiency of frame insertion.

[0101] Optionally, in the above embodiment, when the interpolation mode is an integer multiple interpolation mode, the electronic device interpolates frames between every two adjacent frames of the source video according to the interpolation multiple, and the integer multiple interpolation mode is used to indicate the interpolation mode of increasing the first frame rate by an integer multiple of 2 to obtain the target video.

[0102] For example, assuming that the user indicates 2x interpolation, the electronic device generates an intermediate frame based on each two adjacent graphic frames and inserts the intermediate frame. If the user indicates 4x interpolation, 3 intermediate frames are inserted between each two adjacent image frames, and the electronic device generates an intermediate frame based on each two adjacent graphic frames to obtain 3 consecutive image frames, and then generates and inserts an intermediate frame for each two image frames. Similarly, when interpolating 8x, 7 intermediate frames are inserted between each two adjacent image frames.

[0103] With the first frame rate being 24 FPS, the user indicates 4x interpolation. Assuming that the 24 image frames are frame 0 to frame 23 in sequence, the electronic device inserts 3 intermediate frames between every two adjacent frames, and finally obtains the target video. Figure 7 It is a schematic diagram of the process of integer multiple frame insertion in the video processing method provided in an embodiment of the present application. Figure 7 It is a schematic diagram of the process of inserting frames between the first frame and the second frame in the video processing method provided in an embodiment of the present application.

[0104] Please refer to Figure 7, initially, there are only image frames filled with white, namely, the 1st and 2nd frames. After the electronic device reads the 1st frame, it does not call the interpolation model for the time being, but caches the 1st frame, reads the 2nd frame, calls the interpolation model, inputs the 1st and 2nd frames into the interpolation model to obtain the 1.5th frame, as shown in the image frame filled with oblique lines in the figure. Next, the electronic device calls the interpolation model for the second round, inputs the 1st and 1.5th frames, and obtains the 1.25th frame. As shown in the vertical line filling part in the figure. Afterwards, the electronic device calls the interpolation model for the third round, inputs the 1.5th frame and the 2nd frame, and obtains the 1.75th frame. As shown in the vertical line filling part in the figure. Finally, the 1st frame, the 1.25th frame, the 1.5th frame, and the 1.75th frame are output, and these image frames are encoded. At the same time, the electronic device caches the 2nd frame and reads the 3rd frame until all adjacent frames of the source video are interpolated.

[0105] It can be seen that the video processing method provided in the embodiment of the present application not only supports integer multiple interpolation, but also supports interpolation of common target frame rates, which solves the artifact problem. At the same time, during the integer multiple interpolation process, reading, processing, caching and encoding are performed simultaneously, thereby achieving the purpose of improving the efficiency of integer multiple interpolation.

[0106] Figure 8 is another flow chart of the video processing method provided by the embodiment of the present application. This embodiment includes:

[0107] 801. The user selects a source video and enters instruction information.

[0108] Exemplarily, the electronic device provides a user interface, and the user selects a source video and inputs instruction information through the user interface.

[0109] 802. The electronic device determines a first frame rate of a source video.

[0110] 803. The electronic device determines a frame interpolation mode of the source video. When the frame interpolation mode is an integer multiple frame interpolation mode, step 804 is executed; when the frame interpolation mode is a non-integer multiple frame interpolation mode, step 805 is executed.

[0111] 804, for 2 n Multiple interpolation, interpolation is performed based on the integer multiple of n.

[0112] For example, when n=1, an intermediate frame is sequentially inserted between each two adjacent image frames, and these two image frames are original image frames. When n>1 and is an integer, an intermediate frame is sequentially inserted between each two adjacent image frames to obtain three consecutive frames of images, and then an intermediate frame is inserted between each two frames... until 2 intermediate frames are inserted between the original two image frames. n - 1 intermediate frame.

[0113] 805. The electronic device groups the image frames included in the source video, and there is a special position in each group. One more round of frame insertion is performed at the special position than at other positions.

[0114] For example, from 24FPS to 60FPS, every 4 frames in the source video are grouped into a group, and there are 3 interpolation positions in the group. The first group contains the 0th frame, the 1st frame, the 2nd frame, and the 3rd frame. The 3 interpolation positions are between the 0th and 1st frames, between the 1st and 2nd frames, and between the 2nd and 3rd frames. Among these three positions, one position is interpolated 4 times, and the other two positions are interpolated 2 times, so that 5 intermediate frames are inserted into the group. For details, see Figure 5 , I will not go into details here.

[0115] 806. The electronic device generates intermediate frames while encoding and outputting the frames in sequence, thereby obtaining a target video.

[0116] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0117] Fig. 9 A schematic diagram of a video processing device provided in an embodiment of the present application. The video processing device 900 includes: a determination module 91 , a processing module 92 , a frame insertion module 93 and an output module 93 .

[0118] A determination module 91, used to determine a frame insertion method of a source video;

[0119] A processing module 92 is used for grouping the image frames included in the source video according to the first frame rate of the source video to obtain a plurality of groups, each of the plurality of groups including a plurality of continuous image frames, when the interpolation mode is a non-integer multiple interpolation mode, and the non-integer multiple interpolation mode is used to indicate an interpolation mode of increasing the first frame rate by a non-integer multiple of 2 to obtain a target video;

[0120] A frame insertion module 93, configured to insert a frame into each of the plurality of groups to obtain the target video;

[0121] The output module 94 is used to output the target video.

[0122] In a feasible implementation, the interpolation module 93 is used to determine a first number of interpolated frames required for each of the multiple groups, and determine a second number of interpolated frames required between each two adjacent image frames in each group based on the first number, wherein the second number is at least two, and each group includes a first subgroup and a second subgroup, the first subgroup includes two adjacent image frames, the second subgroup includes another two adjacent image frames, and the second number corresponding to the first subgroup is greater than the second number corresponding to the second subgroup; based on the second number, interpolate each of the multiple groups to obtain the target video.

[0123] In a feasible implementation, before the interpolation module 93 determines the second number of interpolated frames required between each two adjacent image frames in each group based on the first number, it is also used to randomly determine two adjacent image frames from each group to obtain the first subgroup, and use the subgroup formed by the image frames in each group except the first subgroup as the second subgroup.

[0124] In a feasible implementation, the interpolation module 93 interpolates frames for each of the plurality of groups according to the second number to obtain the target video, and is used for a first subgroup in the plurality of groups, when the second number corresponding to the first subgroup is 2 n -1, when n≥2 and is an integer, an intermediate frame is generated according to two adjacent videos in the first subgroup, and after obtaining three consecutive image frames, interpolation is continued between every two consecutive image frames in the three image frames until the number of intermediate frames in the first subgroup reaches the second number.

[0125] In a feasible implementation manner, two adjacent groups among the multiple groups are respectively a first group and a second group, and the last image frame of the first group is the first image frame of the second group.

[0126] In a feasible implementation, the interpolation module 93 interpolates corresponding groups among the multiple groups according to the second number to obtain the target video, and is used to read and cache the first image frame in the target group among the multiple groups, where the target group is the group currently being interpolated among the multiple groups; read the second image frame in the target group, where the first image frame and the second image frame are two adjacent image frames in the target group, and the first image frame is located before the second image frame; and interpolate the first image frame and the second image frame according to the second number corresponding to the first image frame and the second image frame.

[0127] In a feasible implementation, the processing module 92 is further used to sequentially output the first image frame and the second number of intermediate frames after the interpolation module 93 interpolates the first image frame and the second image frame according to the second number corresponding to the first image frame and the second image frame, wherein the intermediate frame is an image frame generated by interpolating the first image frame and the second image frame; cache the second image frame; and encode the first image frame and the second number of intermediate frames.

[0128] In a feasible implementation, the determination module 91 is used to identify indication information; when the indication information indicates the second frame rate of the target video, the interpolation mode is determined to be a non-integer multiple interpolation mode; when the indication information indicates an interpolation multiple, the interpolation mode is determined to be an integer multiple interpolation mode.

[0129] In a feasible implementation, the processing module 92 is used to, when the first frame rate is 24FPS, group the image frames contained in the source video into groups of every 4 frames to obtain multiple groups; when the first frame rate is 25FPS, group the image frames contained in the source video into groups of every 5 frames to obtain multiple groups.

[0130] In a feasible implementation, the interpolation module 93 is also used to interpolate between every two adjacent frames of the source video according to the interpolation multiple when the interpolation mode is an integer multiple interpolation mode, and the integer multiple interpolation mode is used to indicate the interpolation mode of increasing the first frame rate by an integer multiple of 2 to obtain the target video.

[0131] The video processing device provided in the embodiment of the present application can execute the actions of the electronic device in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0132] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device 1000 includes:

[0133] Processor 101 and memory 102;

[0134] The memory 102 stores computer instructions;

[0135] The processor 101 executes the computer instructions stored in the memory 102, so that the processor 101 performs the video processing method as described above.

[0136] The specific implementation process of the processor 101 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0137] Optionally, the electronic device 1000 further includes a communication component 103 , wherein the processor 101 , the memory 102 and the communication component 103 may be connected via a bus 104 .

[0138] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the video processing method described above.

[0139] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the video processing method described above is implemented.

[0140] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0141] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: include: Determine the frame interpolation method of the source video; When the interpolation mode is a non-integer multiple interpolation mode, the image frames contained in each second of the source video are grouped according to the first frame rate of the source video to obtain multiple groups for each second of the video source, each of the multiple groups contains multiple continuous image frames, the non-integer multiple interpolation mode is used to indicate an interpolation mode of obtaining a target video by increasing the first frame rate by a non-integer multiple of 2, two adjacent groups in the multiple groups are respectively a first group and a second group, and the last image frame of the first group is the first image frame of the second group; Inserting frames into each of the plurality of groups to obtain the target video; Outputting the target video; The interpolating frames of each of the plurality of groups to obtain the target video includes: Determine a first number of interpolated frames required for each of the plurality of packets; Determine, according to the first number, a second number of frames to be interpolated between every two adjacent image frames in each group, the second number being at least two, each group comprising a first subgroup and a second subgroup, the first subgroup comprising two adjacent image frames, the second subgroup comprising another two adjacent image frames, and the second number corresponding to the first subgroup being greater than the second number corresponding to the second subgroup; According to the second number, interpolate frames for each of the multiple groups to obtain the target video.

2. The method according to claim 1, characterized in that Before determining the second number of interpolated frames required between every two adjacent image frames in each group according to the first number, the method further includes: Two adjacent image frames are randomly determined from each group to obtain the first subgroup, and a subgroup formed by image frames in each group except the first subgroup is used as a second subgroup.

3. The method according to claim 1, characterized in that The step of inserting frames into each of the plurality of groups according to the second number to obtain the target video includes: For a first subgroup in the plurality of groups, when the second number corresponding to the first subgroup is 2 n -1, when n≥2 and is an integer, an intermediate frame is generated according to two adjacent videos in the first subgroup. After obtaining three consecutive image frames, interpolation is continued between every two consecutive image frames in the three image frames until the number of intermediate frames in the first subgroup reaches the second number.

4. The method according to claim 1, characterized in that: The interpolating frames of corresponding groups in the plurality of groups according to the second number to obtain the target video includes: For a target group among the multiple groups, read and cache the first image frame in the target group, wherein the target group is a group among the multiple groups that is currently being inserted; Reading a second image frame in the target group, wherein the first image frame and the second image frame are two adjacent image frames in the target group, and the first image frame is located before the second image frame; The first image frame and the second image frame are interpolated according to a second number corresponding to the first image frame and the second image frame.

5. The method according to claim 4, characterized in that After interpolating the first image frame and the second image frame according to the second number corresponding to the first image frame and the second image frame, the method further includes: outputting the first image frame and the second number of intermediate frames in sequence, wherein the intermediate frames are image frames generated by interpolating the first image frame and the second image frame; caching the second image frame; The first image frame and the second number of intermediate frames are encoded.

6. The method according to any one of claims 1 to 5, characterized in that: The determining of the frame insertion mode of the source video includes: Identify instruction information; When the indication information indicates the second frame rate of the target video, determining that the interpolation mode is a non-integer multiple interpolation mode; When the indication information indicates the interpolation multiple, the interpolation mode is determined to be an integer multiple interpolation mode.

7. The method according to any one of claims 1 to 5, characterized in that: When the interpolation mode is a non-integer multiple interpolation mode, grouping the image frames included in the source video according to the first frame rate of the source video to obtain a plurality of groups, including: When the first frame rate is 24 FPS, grouping the image frames included in the source video into a group of every 4 frames to obtain a plurality of groups; When the first frame rate is 25 FPS, the image frames included in the source video are grouped into groups of every 5 frames to obtain a plurality of groups.

8. The method according to any one of claims 1 to 5, characterized in that: Also includes: When the interpolation mode is an integer multiple interpolation mode, interpolation is performed between every two adjacent frames of the source video according to the interpolation multiple. The integer multiple interpolation mode is used to indicate the interpolation mode of increasing the first frame rate by an integer multiple of 2 to obtain the target video.

9. A video processing device, characterized in that: include: A determination module, used to determine the frame insertion method of the source video; a processing module, configured to group the image frames contained in each second of the source video according to the first frame rate of the source video, so as to obtain a plurality of groups for each second of the video source, when the interpolation mode is a non-integer multiple interpolation mode, so as to obtain a plurality of continuous image frames for each second of the video source, wherein each group of the plurality of groups contains a plurality of continuous image frames, and the non-integer multiple interpolation mode is used to indicate an interpolation mode of obtaining a target video by increasing the first frame rate by a non-integer multiple of 2, and two adjacent groups in the plurality of groups are respectively a first group and a second group, and the last image frame of the first group is a first image frame of the second group; A frame insertion module, used for inserting frames into each of the plurality of groups to obtain the target video; An output module, used for outputting the target video; The interpolation module is used to determine a first number of interpolations required for each of the multiple groups; A second number of frames that need to be interpolated between every two adjacent image frames in each group is determined based on the first number, the second number is at least two, each group includes a first subgroup and a second subgroup, the first subgroup includes two adjacent image frames, the second subgroup includes another two adjacent image frames, and the second number corresponding to the first subgroup is greater than the second number corresponding to the second subgroup; based on the second number, interpolate frames for each of the multiple groups to obtain the target video.

10. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.