Video processing methods, devices and electronic equipment

Abstract

This application provides a video processing method, apparatus, and electronic device, relating to the field of video image processing technology. First, it determines whether a video frame sequence contains repeating or non-repeating frames. Next, it identifies valid and invalid repeating frames within the video frame sequence. Finally, it generates transition frames based on the nearest adjacent valid frames in the video frame sequence to the invalid repeating frame, and replaces the invalid repeating frame with the transition frame. Compared to existing technologies, replacing invalid repeating frames with generated transition frames reduces the number of invalid repeating frames in the target video, increases the effective frame rate of the target video, alleviates video stuttering caused by a low effective frame rate, and improves the viewer's viewing experience.

Description

Technical Field

[0001] This application relates to the field of video image processing technology, and more specifically, to a video processing method, apparatus, and electronic device. Background Technology

[0002] Videos may contain a large number of invalid duplicate frames due to reasons such as secondary recording or sampling, or poor video shooting quality. This reduces the frame rate of the effective frames in the video, causing video stuttering and seriously affecting the viewing experience. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a video processing method, apparatus, and electronic device.

[0004] In a first aspect, embodiments of this application provide a video processing method, the method comprising:

[0005] Obtain the video frame sequence of the target video to be processed;

[0006] Determine which frames are repeating and which are not repeating in the video frame sequence;

[0007] Based on the repeating and non-repeating frames in the video frame sequence, determine the valid and invalid repeating frames in the video frame sequence;

[0008] A transition frame is generated based on the nearest valid frame before and after the invalid duplicate frame in the video frame sequence, and the invalid duplicate frame is replaced by the transition frame. The valid frames include non-duplicate frames and valid duplicate frames.

[0009] In one possible implementation, after the step of generating a transition frame based on the nearest adjacent valid frames in the video frame sequence to the invalid repeating frame, and replacing the invalid repeating frame with the transition frame, the method further includes:

[0010] Based on the timestamps of each frame in the video frame sequence of the target video and the new frame rate of the target video, the number of video frames to be inserted between adjacent valid frames is calculated, and the inserted frames generated based on the adjacent valid frames are inserted between the adjacent valid frames.

[0011] In one possible implementation, the target video is a live video, and the step of determining the valid and invalid repeating frames in the video frame sequence based on repeating and non-repeating frames in the video frame sequence includes:

[0012] A first sliding window with a first preset sliding width slides from the first frame of the video frame sequence along the growth direction of the video frame sequence, wherein the first preset sliding width is 5 to 10 frames;

[0013] For video frames located within the first sliding window, a preset video frame judgment rule is used to determine the valid and invalid duplicate frames in the video frame sequence, so that at least two valid frames are included among the multiple video frames located in any of the first sliding windows in the video frame sequence.

[0014] In one possible implementation, the preset video frame determination rule includes:

[0015] The second video frame in the first sliding window is taken as the current video frame to be judged;

[0016] Detect whether the current video frame is a duplicate frame;

[0017] When the current video frame is a repeating frame, it is detected whether there is a non-repeating frame in the video frames of the first sliding window after the current video frame;

[0018] If there is a non-repeating frame in the video frames of the first sliding window after the current video frame, the current video frame is determined to be an invalid repeating frame; if there is no non-repeating frame in the video frames of the first sliding window after the current video frame, the current video frame is determined to be a valid repeating frame.

[0019] In one possible implementation, the step of generating a transition frame based on the nearest adjacent valid frames in the video frame sequence to the invalid duplicate frame, and replacing the invalid duplicate frame with the transition frame, includes:

[0020] A second sliding window with a second preset sliding width slides from the first frame of the video frame sequence along the growth direction of the video frame sequence, wherein, in the growth direction of the video frame sequence, the second sliding window lags behind the first sliding window, and the second sliding window overlaps with the first sliding window only partially in video frames that have been distinguished as non-repeating frames, invalid repeating frames, or valid repeating frames.

[0021] In the second sliding window, the nearest valid frame before the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame are found, and a transition frame for replacing the invalid duplicate frame is calculated based on the nearest valid frame before the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame.

[0022] The invalid repeating frame is replaced with the transition frame.

[0023] In one possible implementation, the step of calculating a transition frame for replacing the invalid duplicate frame based on the most recent valid frame before the invalid duplicate frame and the most recent valid frame after the invalid duplicate frame includes:

[0024] Calculate the first number of video frames between the invalid duplicate frame and the nearest valid frame before the invalid duplicate frame, and calculate the second number of video frames between the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame;

[0025] Based on the first frame number, the second frame number, and the sum of the first frame number and the second frame number, the first weight parameter of the nearest valid frame before the invalid duplicate frame and the second weight parameter of the nearest valid frame after the invalid duplicate frame are obtained.

[0026] A transition frame for replacing the invalid duplicate frame is calculated based on the most recent valid frame before the invalid duplicate frame, the most recent valid frame after the invalid duplicate frame, the first weight parameter, and the second weight parameter.

[0027] In one possible implementation, the step of calculating the number of video frames to be inserted between adjacent valid frames based on the timestamps of each frame in the video frame sequence of the target video and the new frame rate of the target video, and inserting the inserted frames generated based on the adjacent valid frames between the adjacent valid frames, includes:

[0028] Obtain the number of existing video frames between adjacent valid frames in the second sliding window;

[0029] Based on the timestamps of the video frames in the second sliding window and the new frame rate of the target video, calculate the number of video frames that need to be inserted between adjacent valid frames in the second sliding window and the corresponding insertion positions.

[0030] The insertion frame is calculated based on the difference in video frame count between the position to be inserted and the adjacent valid frames in the second sliding window, and the adjacent valid frames in the second sliding window, and then the insertion frame is inserted into the position to be inserted.

[0031] In one possible implementation, the step of determining duplicate and non-duplicate frames in the video frame sequence includes:

[0032] For two consecutive video frames in the video frame sequence, calculate the pixel difference between the two consecutive video frames;

[0033] The peak signal-to-noise ratio is calculated based on the pixel difference between the two consecutive video frames.

[0034] The peak signal-to-noise ratio is compared with a preset threshold. If the peak signal-to-noise ratio is less than the preset threshold, the later video frame in the two consecutive video frames is determined to be a repeating frame. If the peak signal-to-noise ratio is not less than the preset threshold, the later video frame in the two consecutive video frames is determined to be a non-repeating frame.

[0035] Secondly, embodiments of this application also provide a video processing apparatus, the apparatus comprising:

[0036] The acquisition module is used to acquire the video frame sequence of the target video to be processed;

[0037] The judgment module is used to determine the repeating frames and non-repeating frames in the video frame sequence;

[0038] The determining module is used to determine the valid and invalid repeating frames in the video frame sequence based on the repeating and non-repeating frames in the video frame sequence;

[0039] The replacement module is used to generate a transition frame based on the nearest adjacent valid frames in the video frame sequence that are the invalid duplicate frame, and to replace the invalid duplicate frame with the transition frame. The valid frames include non-duplicate frames and valid duplicate frames.

[0040] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a computer-readable storage medium, the processor and the computer-readable storage medium being connected via a bus, the computer-readable storage medium being used to store programs, instructions or code, and the processor being used to execute the programs, instructions or code in the computer-readable storage medium to implement the video processing method in any possible implementation of the first aspect.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed, cause an electronic device to perform the video processing method described in the first aspect or any possible implementation thereof.

[0042] Based on any of the above aspects, the video processing method, apparatus, and electronic device provided in this application first determine whether a video frame sequence contains repeating or non-repeating frames; then, they identify valid and invalid repeating frames in the video frame sequence; finally, they generate transition frames based on the nearest adjacent valid frames in the video frame sequence to the invalid repeating frame, and replace the invalid repeating frame with the transition frame. Compared to the prior art, replacing invalid repeating frames with generated transition frames can reduce the number of invalid repeating frames in the target video, increase the effective frame rate of the target video, improve video stuttering caused by a low effective frame rate in the target video, and enhance the viewing experience for the audience. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic flowchart of the video processing method provided in this embodiment;

[0045] Figure 2 This is another flowchart illustrating the video processing method provided in this embodiment;

[0046] Figure 3 for Figure 2 A flowchart illustrating the sub-steps of step S12;

[0047] Figure 4 for Figure 2 A flowchart illustrating the sub-steps of step S13;

[0048] Figure 5 This is a schematic table illustrating the video frame determination rules provided in this embodiment;

[0049] Figure 6 for Figure 2 A flowchart illustrating the sub-steps of step S14;

[0050] Figure 7 This is a schematic diagram showing the positional relationship between the first sliding window and the second sliding window in the video sequence frames provided in this embodiment;

[0051] Figure 8 for Figure 2 A flowchart illustrating the sub-steps of step S15.

[0052] Figure 9 This is a schematic diagram of the functional modules of the video processing apparatus provided in the embodiments of this application;

[0053] Figure 10 A schematic diagram of a possible structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0055] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0056] To address the technical problems mentioned in the background section, this application provides a video processing method, please refer to... Figure 1 , Figure 1 A flowchart illustrating the video processing method provided in this application embodiment is shown. The order of some steps in the video processing method of this application embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of the video processing method are described below.

[0057] Step S11: Obtain the video frame sequence of the target video to be processed.

[0058] In this embodiment, the target video to be processed can originate from the electronic device itself, for example, by recording video using an electronic device, or by obtaining it from other electronic devices through data interaction.

[0059] Step S12: Determine whether the video frame sequence contains repeating frames or non-repeating frames.

[0060] A repeating frame is a video frame that is identical to its preceding adjacent video frame, while a non-repeating frame is a video frame that is different from its preceding adjacent video frame. In this step, repeating and non-repeating frames in the video frame sequence can be determined by comparing the differences between adjacent video frames.

[0061] Step S13: Based on the repeating and non-repeating frames in the video frame sequence, determine the valid and invalid repeating frames in the video frame sequence.

[0062] Valid duplicate frames are duplicate frames in a video frame sequence that should be retained, such as normal consecutive duplicate frames appearing in a static scene. Invalid duplicate frames are duplicate frames in a video frame sequence that should be deleted or replaced.

[0063] Step S14: Generate a transition frame based on the nearest valid frame before and after the invalid duplicate frame in the video frame sequence, and replace the invalid duplicate frame with the transition frame.

[0064] In this embodiment, valid frames include non-repeating frames and valid repeating frames. The adjacent valid frame preceding an invalid repeating frame can be either a non-repeating frame or a valid repeating frame; similarly, the adjacent valid frame following an invalid repeating frame can also be either a non-repeating frame or a valid repeating frame. For example, when the adjacent valid frame preceding an invalid repeating frame is a non-repeating frame, and the adjacent valid frame following an invalid repeating frame is a valid repeating frame, a transition frame can be generated based on the non-repeating frame preceding the invalid repeating frame and the valid repeating frame following the invalid repeating frame.

[0065] The above-described solution provided in this application can replace invalid repeating frames with generated transition frames, reduce invalid repeating frames in the target video, increase the effective frame rate of the target video, improve video stuttering caused by the low effective frame rate of the target video, and enhance the viewing experience for the audience.

[0066] In some cases, due to the low frame rate of the target video itself, even without many invalid duplicate frames, the frame rate of the effective frames remains low, and viewers will still perceive the video as choppy. To solve this problem, please refer to... Figure 2 Furthermore, after step S14, the video processing method provided in this application embodiment may also include step S15.

[0067] Step S15: Calculate the number of video frames to be inserted between adjacent valid frames based on the timestamps of each frame in the video frame sequence of the target video and the new frame rate of the target video, and insert the inserted frames generated based on the adjacent valid frames between the adjacent valid frames.

[0068] In the above method, an insertion frame is generated from adjacent valid frames and then inserted into the adjacent valid frames. Since the insertion frame is generated based on the adjacent valid frames, the image of the insertion frame has a transition function. After inserting the insertion frame into the adjacent valid frames, the number of valid frames can be increased and the frame rate of the target video frame can be improved, thereby improving the smoothness of the target video and further enhancing the viewing experience of the audience.

[0069] In this embodiment, please refer to Figure 3Step S12 can be achieved through the following sub-steps.

[0070] Sub-step S121: For two consecutive video frames in the video frame sequence, calculate the pixel difference between the two consecutive video frames.

[0071] In this embodiment, the pixel difference between two consecutive video frames can be achieved as follows: For two consecutive frames I and K, the variance (i.e., MSE) of each pixel in the preceding frame I and the following frame K is calculated first.

[0072]

[0073] Where m and n are the number of row pixels and column pixels in the frame image, respectively.

[0074] Sub-step S122: Calculate the peak signal-to-noise ratio based on the pixel difference between the two consecutive video frames.

[0075] In this embodiment, the formula for calculating the peak signal-to-noise ratio (PSNR) can be as follows:

[0076]

[0077] Among them, MAX I It is the largest pixel value in the frame image.

[0078] Sub-step S123: Compare the peak signal-to-noise ratio with a preset threshold. If the peak signal-to-noise ratio is less than the preset threshold, determine that the later video frame in the two consecutive video frames is a duplicate frame. If the peak signal-to-noise ratio is not less than the preset threshold, determine that the later video frame in the two consecutive video frames is a non-duplicate frame.

[0079] In live video scenarios, due to the immediacy of video playback, the processed video needs to be presented instantly (with short latency). To address this issue, in this embodiment, the target video can be a live video. Please refer to... Figure 4 Step S13 can be achieved through the following sub-steps.

[0080] Sub-step S131: Using a first sliding window with a first preset sliding width, slide along the growth direction of the video frame sequence starting from the first frame.

[0081] To ensure the real-time nature of video playback, it is necessary to quickly determine the type of video frame within the first sliding window. When the first sliding window is used to slide through the video frame sequence, its corresponding sliding width cannot be too long. In this embodiment, the sliding width of the first sliding window can be 5 to 10 frames.

[0082] Sub-step S132: For video frames located within the first sliding window, a preset video frame judgment rule is used to determine the valid and invalid duplicate frames in the video frame sequence, so that at least two valid frames are included among the multiple video frames located in any first sliding window in the video frame sequence.

[0083] To facilitate the explanation of the preset video frame judgment rules, the frames in the video frame sequence are labeled: U represents a non-repeating frame or a repeating frame (uncertain frame), N represents a non-repeating frame, and R represents a repeating frame. v Indicates a valid frame, R I This indicates an invalid duplicate frame; the current video frame to be judged is marked with the subscript C.

[0084] Please refer to Figure 5 With the first sliding window W a Taking a sliding width of 5 frames as an example, the preset video frame judgment rule can be implemented in the following way.

[0085] First, the second video frame in the first sliding window Wa is taken as the current video frame to be judged.

[0086] Next, it checks whether the current video frame is a duplicate frame. If the current video frame is detected to be a non-duplicate frame, it is recorded as a non-duplicate frame (e.g., ...). Figure 5 (The first case in the text), and the next video frame adjacent to the current video frame is taken as the new current video frame.

[0087] When the current video frame is a repeating frame, check if there are any non-repeating frames in the first sliding window Wa after the current video frame; if there are non-repeating frames in the first sliding window Wa after the current video frame, then the current video frame is determined to be an invalid repeating frame (e.g., Figure 5 The second scenario); if there are no non-repeating frames in the first sliding window Wa after the current video frame, the current video frame can be considered a normal continuous repeating frame appearing in a static scene, and is therefore determined to be a valid repeating frame (e.g. Figure 5 The third case in the text).

[0088] Furthermore, in this embodiment, please refer to Figure 6 Step S14 can be achieved through the following sub-steps.

[0089] Sub-step S141: Using a second sliding window with a second preset sliding width, slide along the growth direction of the video frame sequence starting from the first frame of the video frame sequence.

[0090] In this embodiment, in the direction of video frame sequence growth, the second sliding window lags behind the first sliding window, and the second sliding window and the first sliding window overlap only partially with video frames that have been classified as non-repeating frames, invalid repeating frames, or valid repeating frames. For example, please refer to... Figure 7 The second sliding window W b The sliding width and the first sliding window W a The sliding widths can be the same, for example, both 5 frames, and the second sliding window W b And the first sliding window W a Overlap one frame, second sliding window W b Lagging behind the first sliding window W a 4 frames.

[0091] In sub-step S142, in the second sliding window, the nearest valid frame before the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame are found, and a transition frame for replacing the invalid duplicate frame is calculated based on the nearest valid frame before the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame.

[0092] In this embodiment, sub-step S142 can calculate the transition frame in the following manner.

[0093] First, calculate the number of video frames between the invalid duplicate frame and the nearest valid frame before the invalid duplicate frame, and then calculate the number of video frames between the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame.

[0094] Next, based on the first frame number, the second frame number, and the sum of the first and second frame numbers, the first weight parameter of the nearest valid frame before the invalid duplicate frame and the second weight parameter of the nearest valid frame after the invalid duplicate frame are obtained.

[0095] Then, a transition frame for replacing the invalid duplicate frame is calculated based on the most recent valid frame before the invalid duplicate frame, the most recent valid frame after the invalid duplicate frame, the first weight parameter, and the second weight parameter.

[0096] For example, please refer to again Figure 7 In order to Figure 7 The second sliding window W b Taking the video frames in the example, in the second sliding window Wb, there are 3 invalid duplicate frames R. I If the original frame rate is maintained, then for each invalid duplicate frame R I First, find the nearest valid frame (either a non-duplicate frame N or a valid duplicate frame R). V Then find the nearest valid frame (either a non-duplicate frame N or a valid duplicate frame R). V ), respectively, are taken as adjacent valid frames. Then, based on the invalid duplicate frame R I The distance to adjacent valid frames is used as a reference weight to calculate a completely new transition frame (for example, using SVPflow frame interpolation). The closer the transition frame is to the previous valid frame, the greater its weight. Figure 7 The first invalid duplicate frame R in IIts and the preceding valid frame R V If the distance is 1 frame and the distance to the following valid frame N is 3 frames, then when using svpflow for frame interpolation calculation, the preceding valid frame R... V The weight is 1 - 1 / 4 = 3 / 4, and the weight of the subsequent effective frame N is 1 - 3 / 4 = 1 / 4.

[0097] Sub-step 143: Replace the invalid duplicate frame with a transition frame.

[0098] In this embodiment, please refer to Figure 8 Step S15 can be achieved through the following sub-steps.

[0099] Sub-step S151: Obtain the number of existing video frames between adjacent valid frames in the second sliding window.

[0100] Sub-step S152: Based on the timestamps of the video frames in the second sliding window and the new frame rate of the target video, calculate the number of video frames to be inserted between adjacent valid frames in the second sliding window and the corresponding insertion positions.

[0101] Sub-step S153: Calculate the insertion frame based on the difference in video frame count between the position to be inserted and the adjacent valid frames in the second sliding window, and the adjacent valid frames in the second sliding window, and insert the insertion frame into the position to be inserted.

[0102] In this embodiment, the method of calculating the inserted frame in sub-step S153 is similar to the method of calculating the transition frame in sub-step S142, and will not be described again here.

[0103] Please refer to Figure 9 , Figure 9 This is a functional module diagram of a video processing apparatus 100 provided in an embodiment of this application. The video processing apparatus 100 can be divided into functional modules according to the method embodiments executed by the server. That is, the following functional modules corresponding to the video processing apparatus 100 can be used to execute the aforementioned method embodiments. The video processing apparatus 100 may include an acquisition module 110, a judgment module 120, a determination module 130, and a replacement module 140. The functions of each functional module of the video processing apparatus 100 will be described in detail below.

[0104] The acquisition module 110 is used to acquire the video frame sequence of the target video to be processed.

[0105] In this embodiment, the target video to be processed can originate from the electronic device itself, for example, by recording video using an electronic device, or by obtaining it from other electronic devices through data interaction.

[0106] In this embodiment, the acquisition module 110 can be used to perform the above-described step S11. For a detailed description of the implementation of the acquisition module 110, please refer to the above-described description of step S11.

[0107] The judgment module 120 is used to determine the repeating frames and non-repeating frames in the video frame sequence.

[0108] A repeating frame is a video frame that is identical to its preceding adjacent video frame, while a non-repeating frame is a video frame that is different from its preceding adjacent video frame. In this embodiment, repeating and non-repeating frames in a video frame sequence can be determined by comparing the differences between adjacent video frames.

[0109] In this embodiment, the judgment module 120 can be used to execute the above-described step S12. For a detailed description of the implementation of the judgment module 120, please refer to the above-described description of step S12.

[0110] The determination module 130 is used to determine the valid and invalid repeating frames in the video frame sequence based on the repeating and non-repeating frames in the video frame sequence.

[0111] Valid duplicate frames are duplicate frames in a video frame sequence that should be retained, such as normal consecutive duplicate frames appearing in a static scene. Invalid duplicate frames are duplicate frames in a video frame sequence that should be deleted or replaced.

[0112] In this embodiment, the determining module 130 can be used to perform the above-described step S13. For a detailed description of the implementation of the determining module 130, please refer to the above-described detailed description of step S13.

[0113] The replacement module 140 is used to generate a transition frame based on the nearest valid frame before and after the invalid duplicate frame in the video frame sequence, and replace the invalid duplicate frame with the transition frame.

[0114] In this embodiment, valid frames include non-repeating frames and valid repeating frames. The adjacent valid frame preceding an invalid repeating frame can be either a non-repeating frame or a valid repeating frame; similarly, the adjacent valid frame following an invalid repeating frame can also be either a non-repeating frame or a valid repeating frame. For example, when the adjacent valid frame preceding an invalid repeating frame is a non-repeating frame, and the adjacent valid frame following an invalid repeating frame is a valid repeating frame, a transition frame can be generated based on the non-repeating frame preceding the invalid repeating frame and the valid repeating frame following the invalid repeating frame.

[0115] In this embodiment, the replacement module 140 can be used to perform the above-described step S14. For a detailed description of the implementation of the replacement module 140, please refer to the above-described description of step S14.

[0116] Furthermore, please refer to again Figure 9The video processing device 100 provided in this embodiment may further include an insertion module 150.

[0117] The insertion module 150 is used to calculate the number of video frames to be inserted between adjacent valid frames based on the timestamps of each frame in the video frame sequence of the target video and the new frame rate of the target video, and to insert the insertion frames generated based on the adjacent valid frames between the adjacent valid frames.

[0118] In this embodiment, the insertion module 150 can be used to perform the above-described step S15. For a detailed description of the implementation of the insertion module 150, please refer to the above-described detailed description of step S15.

[0119] It should be noted that the division of modules in the above-described device or system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software (e.g., open-source software) via processor calls; they can also be implemented entirely in hardware; or some modules can be implemented via processor calls to software, while others are implemented in hardware. As an example, replacement module 140 can be implemented by a separate processor and can be stored as program code in the memory of the above-described device or system. A processor in the device or system can then call and execute the functions of replacement module 140. The implementation of other modules is similar and will not be elaborated further here. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processor described here can be an integrated circuit with signal processing capabilities. During implementation, each step or module in the above technical solution can be completed through integrated logic circuits in the processor or by executing software programs.

[0120] Please refer to Figure 10 , Figure 10 A schematic diagram of the hardware structure of an electronic device 10 for implementing the video processing method described above, provided in an embodiment of this disclosure, is shown. Figure 10 As shown, the electronic device 10 may include a processor 11, a computer-readable storage medium 12, and a bus 13.

[0121] In a specific implementation, the processor 11 executes computer-executable instructions (e.g., instructions stored in the computer-readable storage medium 12) stored in the computer-readable storage medium 12. Figure 9 The various modules in the video processing apparatus 100 shown above enable the processor 11 to execute the video processing method as described in the above method embodiment, wherein the processor 11 and the computer-readable storage medium 12 can be connected via a bus 13.

[0122] The specific implementation process of processor 11 can be found in the various method embodiments executed by the above-mentioned electronic device 10. The implementation principle and technical effect are similar, and will not be repeated here in the embodiments of this application.

[0123] The computer-readable storage medium 12 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 11 is used to store programs or data.

[0124] Bus 13 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the buses in the accompanying drawings of this application are not limited to only one bus or one type of bus.

[0125] Furthermore, this application embodiment also provides a readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video processing method described above.

[0126] In summary, the video processing method, apparatus, and electronic device provided in this application first determine whether a video frame sequence contains repeating or non-repeating frames; then, they identify valid and invalid repeating frames within the video frame sequence; finally, they generate transition frames based on the nearest adjacent valid frames in the video frame sequence that are adjacent to the invalid repeating frame, and replace the invalid repeating frame with the transition frame. Compared to existing technologies, replacing invalid repeating frames with generated transition frames reduces the number of invalid repeating frames in the target video, increases the effective frame rate of the target video, alleviates video stuttering caused by a low effective frame rate, and improves the viewer's viewing experience.

[0127] The embodiments described above are merely some, not all, of the embodiments of this application. The components of the embodiments of this application typically described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of protection of this application, but merely to illustrate selected embodiments of this application. Based on this, the scope of protection of this application should be determined by the scope of the claims. Furthermore, all other embodiments that can be obtained by those skilled in the art based on the embodiments of this application without inventive effort should fall within the scope of protection of this application.

Claims

1. A video processing method, characterized in that the method comprises: Obtain the video frame sequence of the target video to be processed; Determine which frames are repeating and which are not repeating in the video frame sequence; A first sliding window with a first preset sliding width slides from the first frame of the video frame sequence along the growth direction of the video frame sequence. For video frames located within the first sliding window, a preset video frame judgment rule is used to determine valid and invalid duplicate frames in the video frame sequence, ensuring that at least two valid frames are included among the multiple video frames located in any of the first sliding windows in the video frame sequence. The preset video frame judgment rule includes: taking the second video frame in the first sliding window as the current video frame to be judged; detecting whether the current video frame is a duplicate frame; if the current video frame is a duplicate frame, detecting whether there are non-duplicate frames in the video frames of the first sliding window after the current video frame; if there are non-duplicate frames in the video frames of the first sliding window after the current video frame, then the current video frame is determined to be an invalid duplicate frame; if there are no non-duplicate frames in the video frames of the first sliding window after the current video frame, then the current video frame is determined to be a valid duplicate frame. A transition frame is generated based on the nearest valid frame before and after the invalid duplicate frame in the video frame sequence, and the invalid duplicate frame is replaced by the transition frame. The valid frames include non-duplicate frames and valid duplicate frames.

2. The video processing method as described in claim 1, characterized in that, After the step of generating a transition frame based on the nearest adjacent valid frames in the video frame sequence to the invalid duplicate frame, and replacing the invalid duplicate frame with the transition frame, the method further includes: Based on the timestamps of each frame in the video frame sequence of the target video and the new frame rate of the target video, the number of video frames to be inserted between adjacent valid frames is calculated, and the inserted frames generated based on the adjacent valid frames are inserted between the adjacent valid frames.

3. The video processing method as described in claim 1, characterized in that, The step of generating a transition frame based on the nearest adjacent valid frames in the video frame sequence to the invalid duplicate frame, and replacing the invalid duplicate frame with the transition frame, includes: A second sliding window with a second preset sliding width slides from the first frame of the video frame sequence along the growth direction of the video frame sequence, wherein, in the growth direction of the video frame sequence, the second sliding window lags behind the first sliding window, and the second sliding window overlaps with the first sliding window only partially in video frames that have been distinguished as non-repeating frames, invalid repeating frames, or valid repeating frames. In the second sliding window, the nearest valid frame before the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame are found, and a transition frame for replacing the invalid duplicate frame is calculated based on the nearest valid frame before the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame. The invalid repeating frame is replaced with the transition frame.

4. The video processing method as described in claim 3, characterized in that, The step of calculating a transition frame to replace the invalid duplicate frame based on the most recent valid frame before the invalid duplicate frame and the most recent valid frame after the invalid duplicate frame includes: Calculate the first number of video frames between the invalid duplicate frame and the nearest valid frame before the invalid duplicate frame, and calculate the second number of video frames between the invalid duplicate frame and the nearest valid frame after the invalid duplicate frame; Based on the first frame number, the second frame number, and the sum of the first frame number and the second frame number, the first weight parameter of the nearest valid frame before the invalid duplicate frame and the second weight parameter of the nearest valid frame after the invalid duplicate frame are obtained. A transition frame for replacing the invalid duplicate frame is calculated based on the most recent valid frame before the invalid duplicate frame, the most recent valid frame after the invalid duplicate frame, the first weight parameter, and the second weight parameter.

5. The video processing method as described in claim 4, characterized in that, The step of calculating the number of video frames to be inserted between adjacent valid frames based on the timestamps of each frame in the video frame sequence of the target video and the new frame rate of the target video, and inserting the inserted frames generated based on the adjacent valid frames between the adjacent valid frames, includes: Obtain the number of existing video frames between adjacent valid frames in the second sliding window; Based on the timestamps of the video frames in the second sliding window and the new frame rate of the target video, calculate the number of video frames that need to be inserted between adjacent valid frames in the second sliding window and the corresponding insertion positions. The insertion frame is calculated based on the difference in video frame count between the position to be inserted and the adjacent valid frames in the second sliding window, and the adjacent valid frames in the second sliding window, and then the insertion frame is inserted into the position to be inserted.

6. The video processing method according to any one of claims 1-5, characterized in that, The first preset sliding width is 5 to 10 frames.

7. The video processing method according to any one of claims 1-5, characterized in that, The step of determining duplicate and non-duplicate frames in the video frame sequence includes: For two consecutive video frames in the video frame sequence, calculate the pixel difference between the two consecutive video frames; The peak signal-to-noise ratio is calculated based on the pixel difference between the two consecutive video frames. The peak signal-to-noise ratio is compared with a preset threshold. If the peak signal-to-noise ratio is less than the preset threshold, the later video frame in the two consecutive video frames is determined to be a repeating frame. If the peak signal-to-noise ratio is not less than the preset threshold, the later video frame in the two consecutive video frames is determined to be a non-repeating frame.

8. A video processing apparatus, characterized in that the apparatus comprises: The acquisition module is used to acquire the video frame sequence of the target video to be processed; The judgment module is used to determine the repeating frames and non-repeating frames in the video frame sequence; A determining module is configured to slide a first sliding window with a first preset sliding width from the first frame of the video frame sequence along the growth direction of the video frame sequence; determine valid and invalid duplicate frames in the video frame sequence for video frames located within the first sliding window using a preset video frame judgment rule, so that at least two valid frames are included among the multiple video frames located in any of the first sliding windows in the video frame sequence. The preset video frame judgment rule includes: taking the second video frame in the first sliding window as the current video frame to be judged; detecting whether the current video frame is a duplicate frame; if the current video frame is a duplicate frame, detecting whether there are non-duplicate frames in the video frames of the first sliding window after the current video frame; if there are non-duplicate frames in the video frames of the first sliding window after the current video frame, then determining the current video frame as an invalid duplicate frame; if there are no non-duplicate frames in the video frames of the first sliding window after the current video frame, then determining the current video frame as a valid duplicate frame. The replacement module is used to generate a transition frame based on the nearest adjacent valid frames in the video frame sequence that are the invalid duplicate frame, and to replace the invalid duplicate frame with the transition frame. The valid frames include non-duplicate frames and valid duplicate frames.

9. An electronic device, characterized in that, The electronic device includes a processor and a computer-readable storage medium connected via a bus. The computer-readable storage medium is used to store programs, instructions, or code. The processor is used to execute the programs, instructions, or code in the computer-readable storage medium to implement the video processing method according to any one of claims 1-7.

Images