Video processing method and related device

Abstract

The invention discloses a video processing method and a related device, and the method comprises the steps: obtaining the attribute information of each to-be-decoded frame in a to-be-processed video code stream, enabling the attribute information to be used for indicating the frame type of each to-be-decoded frame, determining the to-be-decoded frame of which the frame type is a non-reference frame from the to-be-processed video code stream based on the attribute information, and removing the to-be-decoded frame. And obtaining a recombined code stream. Frame extraction processing is carried out from the recombined code stream based on the preset frame number to obtain a plurality of sampling frames, the extracted sampling frames are more likely to be key frames, the sampling number of depended frames to be decoded can be reduced, the total number of the extracted sampling frames is further reduced, and when decoding processing is carried out on the plurality of sampling frames to obtain a decoded video, the decoding efficiency of the decoded video is improved. The decoding speed can be greatly improved, and the video frame cutting efficiency is improved. In a word, the total number of the extracted sampling frames can be reduced, the decoding speed is improved, the video frame cutting efficiency is improved, the decoding process of the sampling frames is not affected, and smooth video frame cutting is guaranteed.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular to a video processing method and related apparatus. Background Technology

[0002] Video frame extraction is the process of extracting a portion of frames from a video. For example, if a video consists of 100 frames, video frame extraction can produce a video composed of 50 frames. Subsequent processing, such as motion recognition, can then be performed on this 50-frame video.

[0003] Current video frame extraction techniques generally employ a uniform frame extraction method, which extracts video frames from the video bitstream at equal time intervals or equal frame intervals. This method does not consider the type of video frame and treats all video frames "equally".

[0004] However, when the extracted video frames depend on other video frames for decoding, such as when the extracted video frames are non-critical frames, then the dependent video frames also need to be decoded. Assuming the required number of video frames is n, uniform frame cutting often requires decoding x frames, where x > n. This means the number of video frames to be decoded is too large, especially when the video stream contains a large number of non-critical frames. In this case, the number of non-critical frames extracted is likely to be quite high, and the final number of decoded frames x will be much greater than the required number of frames n. This results in a larger final number of decoded video frames, leading to slower overall decoding speed and lower efficiency in video frame cutting. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a video processing method and related apparatus, which can improve the efficiency of video frame extraction. The embodiments of this application disclose the following technical solutions:

[0006] On one hand, this application provides a video processing method, the method comprising:

[0007] Obtain the attribute information of each frame to be decoded in the video stream to be processed; the attribute information is used to indicate the frame type of each frame to be decoded, the frame type includes key frames and non-key frames, and the non-key frames include reference frames and non-reference frames.

[0008] Based on the attribute information, frames to be decoded that are non-reference frames are removed from the video bitstream to be processed to obtain the reconstructed bitstream;

[0009] Based on a preset number of frames, frame extraction processing is performed on the reconstructed bitstream to obtain multiple sampled frames;

[0010] The multiple sampled frames are decoded to obtain the decoded video.

[0011] In another aspect, this application provides a video processing apparatus, the apparatus comprising:

[0012] An acquisition unit is used to acquire attribute information of each frame to be decoded in the video stream to be processed; the attribute information is used to indicate the frame type of each frame to be decoded, the frame type includes key frames and non-key frames, and the non-key frames include reference frames and non-reference frames.

[0013] The elimination unit is used to eliminate frames to be decoded that are non-reference frames from the video bitstream to be processed based on the attribute information, so as to obtain the reconstructed bitstream;

[0014] The extraction unit is used to perform frame extraction processing from the reconstructed bitstream based on a preset number of frames to obtain multiple sampled frames;

[0015] The decoding unit is used to decode multiple sampled frames to obtain a decoded video.

[0016] In another aspect, this application provides a computer device, which includes a processor and a memory:

[0017] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0018] The processor is configured to execute the method according to the computer program.

[0019] In another aspect, this application provides a computer-readable storage medium for storing a computer program that, when executed by a computer device, implements the method described thereon.

[0020] In another aspect, this application provides a computer program product including a computer program that, when run on a computer device, causes the computer device to perform the method.

[0021] As can be seen from the above technical solution, in the process of obtaining decoded video through video frame extraction, in order to improve the efficiency of video frame extraction, the attribute information of each frame to be decoded in the video bitstream to be processed can be obtained. The attribute information is used to indicate the frame type of each frame to be decoded. The frame type includes keyframes and non-keyframes. Non-keyframes include reference frames and non-reference frames. That is, the attribute information can distinguish the frame type of the frame to be decoded. Then, based on the attribute information, the frames to be decoded that are non-reference frames can be identified from the video bitstream to be processed and removed, while keyframes and reference frames are retained to obtain the reconstructed bitstream. Since non-reference frames are frames to be decoded that are not depended on by other frames to be decoded during the decoding process, removing them will not affect the decoding process of other frames to be decoded, and the frames to be decoded in the reconstructed bitstream can still be decoded normally. Furthermore, by removing non-reference frames that are not key frames, the number of non-key frames can be reduced, thereby increasing the proportion of key frames in the reconstructed bitstream. This makes it more likely that the extracted sampled frames are key frames when performing frame extraction processing from the reconstructed bitstream based on a preset number of frames. Since key frames do not depend on other frames to be decoded during decoding, the number of samples of those dependent frames to be decoded can be reduced, thus reducing the total number of extracted sampled frames. When decoding multiple sampled frames to obtain the decoded video, this significantly improves decoding speed and video frame extraction efficiency. In summary, this application can reduce the total number of extracted sampled frames, improve decoding speed, and improve video frame extraction efficiency without affecting the decoding process of sampled frames, ensuring smooth video frame extraction. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of a video processing method provided in an embodiment of this application;

[0024] Figure 2 A flowchart illustrating a video processing method provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram illustrating the removal of a non-reference frame as provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a frame extraction process provided in an embodiment of this application;

[0027] Figure 5A schematic diagram of memory consumption provided in an embodiment of this application;

[0028] Figure 6 A schematic diagram illustrating a process of repeatedly removing non-reference frames, provided in an embodiment of this application;

[0029] Figure 7 A schematic diagram illustrating the extraction of a sampling frame as provided in an embodiment of this application;

[0030] Figure 8 A schematic diagram of a video frame sequence provided in an embodiment of this application;

[0031] Figure 9 A schematic diagram illustrating the determination of a second redundant frame, provided as an embodiment of this application;

[0032] Figure 10 A schematic diagram of a deduplication model provided in an embodiment of this application;

[0033] Figure 11 This application provides a schematic diagram of a video processing process.

[0034] Figure 12 A schematic diagram of a video processing apparatus provided in an embodiment of this application;

[0035] Figure 13 A structural diagram of a terminal device provided in an embodiment of this application;

[0036] Figure 14 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application will now be described with reference to the accompanying drawings.

[0038] As described in the background section, when the extracted video frames require other video frames for decoding—for example, if the extracted video frames are non-keyframes—then the dependent video frames also need to be decoded. This results in a large number of video frames being decoded, leading to slower overall decoding speed and lower efficiency in video frame extraction.

[0039] This application provides a video processing method and related apparatus. By removing non-reference frames, the total number of extracted sample frames can be reduced, the decoding speed can be improved, and the efficiency of video frame extraction can be increased without affecting the decoding process of the sample frames, thus ensuring the smooth execution of video frame extraction.

[0040] The video processing method provided in this application can be implemented using computer equipment, which can be a terminal device or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0041] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0042] refer to Figure 1 The image shown is a schematic diagram of a video processing method provided in an embodiment of this application. This video processing method can be applied to a server or terminal device. Figure 1 The above-mentioned computer equipment is illustrated using a server as an example.

[0043] The video stream to be processed can be a segment of video stream that needs to be processed. This stream includes multiple frames to be decoded, which are the undecoded video frames. The server can obtain the attribute information of each frame to be decoded within the video stream. This attribute information indicates the frame type of each frame. Frame types include key (Intra-coded frame, I) frames and non-key frames. Non-key frames include reference frames and non-reference frames. Key frames can be understood as independent frames that do not depend on other frames to be decoded during decoding. Non-key frames depend on other frames to be decoded during decoding. Reference frames are frames that are depended upon by other frames to be decoded, and non-reference frames are frames that are not depended upon by other frames to be decoded.

[0044] As an example, the video stream to be processed includes N frames to be decoded. Frame 1 to be decoded is a keyframe and does not depend on any other frame. Frame 2 to be decoded is a non-reference frame, that is, it is not depended on by any other frame. Frame 3 to be decoded is a reference frame, that is, it needs to be depended on by other frames. Frame N-1 to be decoded is a reference frame, and frame N to be decoded is a non-reference frame.

[0045] Since attribute information can distinguish the frame type of each frame to be decoded, and non-reference frames are frames that are not depended upon by other frames to be decoded during the decoding process, removing them will not affect the decoding process of other frames to be decoded. For example, removing non-reference frames 2 and N will not affect the decoding process of the remaining frames to be decoded. Therefore, the server can determine the frames to be decoded with non-reference frame type from the video stream to be processed based on attribute information. For example, it can select and remove frames to be decoded 2 and N to obtain the reconstructed bitstream. In short, the reconstructed bitstream includes keyframes and reference frames. As an example, the reconstructed bitstream includes at least keyframe 1, reference frame 3, and reference frame N-1.

[0046] The preset frame count can be understood as the expected number of frames to be extracted, providing a quantitative basis for frame extraction processing. Based on the preset frame count, the server can perform frame extraction processing from the reconstructed bitstream to obtain multiple sampled frames. Of course, since the extracted sampled frames may be reference frames, these reference frames will be extracted along with their dependent frames to be decoded, so the final total number of extracted sampled frames may be greater than the preset frame count. For example, the final extracted sampled frames include at least sampled frame 1, sampled frame 3, and sampled frame 15. Even so, by removing non-reference frames that are not key frames, the number of non-key frames can be reduced, thereby increasing the proportion of key frames in the reconstructed bitstream. This makes the extracted sampled frames more likely to be key frames. Since key frames do not depend on other frames to be decoded during decoding, the number of samples of those dependent frames to be decoded can be reduced, thus reducing the total number of extracted sampled frames. This significantly improves decoding speed and video frame extraction efficiency during decoding.

[0047] The server decodes multiple sampled frames to obtain the decoded video, which is the video after frame-by-frame processing. For example, the decoded video includes frames a and b. In summary, this application reduces the total number of extracted sampled frames, increases decoding speed, and improves video frame-by-frame efficiency without affecting the decoding process, ensuring smooth video frame-by-frame processing.

[0048] refer to Figure 2 The diagram shown is a flowchart illustrating a video processing method provided in an embodiment of this application. This method can be executed by a computer device. In this embodiment, the computer device is a server as an example for explanation.

[0049] Step 201: Obtain the attribute information of each frame to be decoded in the video stream to be processed.

[0050] The video stream to be processed can be understood as a segment of video stream that needs to be processed; for example, it could be the video stream of a specific video. The frame to be decoded can be a video frame within the video stream to be processed; since this frame has not yet undergone decoding, it is denoted as the frame to be decoded.

[0051] The video stream to be processed is obtained by encoding the video. Since the frame type of each frame to be decoded is determined during the encoding process and represented by attribute information, the server can obtain the attribute information of each frame to be decoded in the video stream to determine the frame type of each frame. Specifically, the server can perform video parsing on the video to be processed to obtain frame data packets for each video frame in the video stream. The frame data packets contain the attribute information of each frame to be decoded, such as the frame type.

[0052] The attribute information indicates the frame type of each frame to be decoded, thus distinguishing frames of different types. Frame types can include keyframes and non-keyframes, and non-keyframes can include reference frames and non-reference frames.

[0053] Specifically, the frame to be decoded can be a keyframe (I-frame), a predicted frame (P-frame), or a bidirectionally predicted frame (B-frame). A keyframe is an independent frame with complete image data. An I-frame does not depend on other frames to be decoded, therefore it can be decoded independently. A P-frame contains image difference data, i.e., the changes between the current frame and the preceding frame; therefore, a P-frame needs to rely on the preceding I-frame or P-frame for decoding. A B-frame contains the changes between the current frame and the frames before and after it; therefore, a B-frame needs to rely on both the preceding and following frames for decoding.

[0054] I-frames, P-frames, and B-frames can be categorized into key frames and non-key frames. Non-key frames can be further classified into reference frames and non-reference frames. Specifically, based on whether decoding depends on other frames to be decoded, frames can be divided into key frames and non-key frames. Key frames are I-frames, while non-key frames can include P-frames and B-frames; that is, both P-frames and B-frames require their dependent frames for decoding. A dependent frame of a frame to be decoded can be understood as the frame that the frame to be decoded depends on during decoding.

[0055] In some technologies, to improve decoding speed, all B-frames are discarded. However, in some newer video encoding methods, P-frames still depend on B-frames. Discarding B-frames can cause decoding artifacts and result in poor video display quality after frame segmentation. Analysis of various frame types reveals a category of frames that are not relied upon by other frames during decoding; they are not used as reference frames. These frames can be classified as non-reference frames, and discarding them has no impact on the decoding of other frames.

[0056] Therefore, using whether a frame to be decoded is depended upon by other frames to be decoded as another classification criterion, reference frames and non-reference frames can be distinguished from non-key frames. Reference frames can be understood as frames that are depended upon by other frames to be decoded during decoding, while non-reference frames are frames that are not depended upon by other frames to be decoded during decoding. In short, reference frames can be dependent on other frames to be decoded, while non-reference frames cannot. Whether a frame is a non-reference frame is not determined by whether it is a P-frame or a B-frame, but by whether it is depended upon by other frames to be decoded. However, a identified reference frame can be either a P-frame or a B-frame, and a non-reference frame can be either a P-frame or a B-frame.

[0057] In practical applications, the attribute information can be the value of the parameter `nal_ref_idc`. `nal_ref_idc` is an important parameter in the H.264 / AVC video coding standard. Specifically, `nal_ref_idc` is a crucial component of the Network Abstraction Layer Unit (NALU). The NALU Reference IdentifierContext is a basic data unit in the H.264 / AVC video coding standard, used to segment video data into small blocks for transmission and storage. `nal_ref_idc` indicates the importance and reference value of the NALU, and its value ranges from 0 to 3. A value of 0 indicates that the video frame corresponding to that NALU is a non-reference frame.

[0058] For the video stream to be processed, it can be the video stream corresponding to the video to be processed. For example, for an input video segment to be processed, the server can encode it to obtain the corresponding video stream, which can then be used as the video stream to be processed. Alternatively, the video stream to be processed can also be a small portion of the video stream extracted from a larger video stream.

[0059] In one possible implementation, before obtaining the attribute information of each frame to be decoded in the video stream to be processed in step 201, the server can also obtain the video stream of the video to be processed, determine multiple frames to be decoded from the video stream according to a preset frame interception strategy, and construct the video stream to be processed based on the multiple frames to be decoded.

[0060] Specifically, the video to be processed can be a segment of video that requires frame-by-frame processing, a video obtained from the internet, or a video captured by a mobile phone or other terminal device. The video content and duration are not specifically limited here.

[0061] The server can obtain the video bitstream of the video to be processed. This video bitstream can specifically be the video bitstream obtained by encoding the video to be processed, meaning the video bitstream corresponds to the entire video to be processed. To reduce the computational load of subsequent processes such as non-reference frame removal, a portion of frames can be determined in advance.

[0062] The server can determine a subset of frames from the video stream of the video to be processed according to a preset frame extraction strategy, thus obtaining multiple frames to be decoded. The preset frame extraction strategy can be a pre-defined method for selecting frames, such as selecting frames at certain intervals or according to a certain number of frames, to determine multiple frames to be decoded. Of course, at this point, the multiple frames to be decoded are only selected and determined; they are not extracted from the video stream, i.e., they are not sampled. As an example, the determined multiple frames to be decoded can be marked to indicate that they were obtained through selection. All the frames to be decoded can be used to form the video stream to be processed.

[0063] In short, by pre-determining a portion of the frames in the video stream of the video to be processed as the frames to be decoded, subsequent operations such as frame culling can be performed from a small number of frames to be decoded. This reduces the number of frames in the video stream, thereby reducing the computational load of frame culling and other steps, facilitating subsequent calculations and processing, and improving frame extraction efficiency.

[0064] Step 202: Based on the attribute information, remove the frames to be decoded that are non-reference frames from the video bitstream to be processed, and obtain the reconstructed bitstream.

[0065] Since the attribute information of the frame to be decoded can identify its frame type, such as whether it is a keyframe, reference frame, or non-reference frame, the server can determine the non-reference frame type from the video stream to be processed based on this attribute information. Furthermore, because non-reference frames are not depended upon by any other frames to be decoded, removing them will not affect other frames. Therefore, non-reference frames can be removed from the video stream to be processed, retaining keyframes and reference frames to obtain the reconstructed stream. In the reconstructed stream, the frame type of the frame to be decoded can be either a keyframe or a reference frame.

[0066] refer to Figure 3 The diagram illustrates a method for removing non-reference frames according to an embodiment of this application. The video stream to be processed includes N frames to be decoded. Based on attribute information, frame 1 to be decoded is determined to be a keyframe, frame 2 to be decoded is a non-reference frame, frames 3 and 4 to be decoded are both reference frames, frame 5 to be decoded is a keyframe, frame N-1 to be decoded is a reference frame, and frame N to be decoded is a non-reference frame. Non-reference frames 2 and N are discarded. Non-reference frames 2 and N are removed from the video stream to be processed to obtain a reconstructed stream. The reconstructed stream includes keyframe 1, reference frame 3, reference frame 4, keyframe 5, and reference frame N-1. The frames to be decoded between keyframe 5 and reference frame N-1 are not shown individually. Furthermore, this diagram does not limit the value of N; N only needs to be greater than 1.

[0067] More specifically, in the process of obtaining the reconstructed bitstream, after removing the non-reference frames from the video bitstream to be processed, the decoding order of the remaining frames to be decoded needs to be reordered, i.e., bitstream reordering, to obtain the reconstructed bitstream. In other words, bitstream reordering associates consecutive frames among the remaining frames to be decoded, establishing the decoding order between two adjacent frames. For example, if the decoding order of the video bitstream to be processed is frame 1 → frame 2 → frame 3, after removing the non-reference frames, we obtain frame 1 and frame 3 to be decoded. Bitstream reordering yields the reconstructed bitstream, with the decoding order being frame 1 → frame 3. That is, bitstream reordering places frame 3 after frame 1 and associates frame 1 and frame 3.

[0068] Compared to related technologies that remove all B-frames to reduce the total number of frames to be decoded, the different video encoding methods can lead to B-frames being dependent on P-frames, causing P-frames to fail to decode properly after B-frame removal, resulting in decoding failure. In this application, since non-reference frames are not dependent on by other frames to be decoded, removing them will not affect the decoding process of the remaining frames to be decoded. That is, each frame to be decoded in the reconstructed bitstream can still be decoded normally, ensuring that the decoding process can proceed smoothly, the picture can be displayed normally, and avoiding problems such as screen tearing.

[0069] It's worth noting that non-reference frames are a type of non-keyframe. During decoding, non-keyframes require the decoding of their dependent frames as well. This means that while a non-reference frame is not depended upon by other frames to be decoded, it may itself depend on other frames. In other words, a non-reference frame may not be a dependent frame for other frames, but it can have dependent frames. Therefore, removing non-reference frames reduces the decoding of their dependent frames, significantly reducing the extra time spent decoding dependent frames and thus improving decoding speed. Furthermore, since non-reference frames are usually quite similar to their neighboring frames to be decoded, removing them can reduce the redundancy of the final decoded video footage to some extent, minimizing the appearance of extremely similar frames.

[0070] Step 203: Extract frames from the reconstructed bitstream based on a preset number of frames to obtain multiple sampled frames.

[0071] The preset frame count can be understood as the expected number of frames to be extracted, that is, the number of frames in the decoded video obtained from frame extraction. For example, if the final video needs to include 10 frames, then the preset frame count is 10. The preset frame count provides a certain quantitative basis for frame extraction processing. Based on this preset frame count, the server can extract the frames to be decoded from the reconstructed bitstream, and record the extracted frames to be decoded as sample frames. Thus, multiple sample frames can be obtained through frame extraction processing.

[0072] The frame type of the sampled frames is not specifically limited here; they can be keyframes or reference frames. The frame type of each sampled frame affects the total number of sampled frames. When all sampled frames are keyframes, since keyframes do not depend on other frames, the total number of sampled frames equals the preset number of frames. For example, when the preset number of frames is 10, 10 keyframes can be sampled as 10 sampled frames. When at least one of the sampled frames is a reference frame, the frame to be decoded that the reference frame depends on also needs to be extracted to ensure that the reference frame can be decoded correctly. Therefore, the total number of sampled frames can be greater than the preset number of frames. For example, when the preset number of frames is 10, 12 sampled frames are obtained through frame extraction processing, including 4 keyframes and 8 reference frames. In short, the total number of sampled frames can be greater than or equal to the preset number of frames.

[0073] refer to Figure 4 The diagram shown is a schematic of a frame extraction process provided in an embodiment of this application. The reconstructed bitstream includes key frame 1, reference frame 3, reference frame 4, key frame 5 and reference frame N-1. Through frame extraction processing, multiple sampled frames are obtained, including key frame 1, reference frame 3, key frame 5 and reference frame N-1.

[0074] Step 204: Decode multiple sampled frames to obtain the decoded video.

[0075] The server can decode each sampled frame to obtain the corresponding image. It can then decode multiple sampled frames and combine these images to obtain the decoded video. The decoded video is the video obtained after frame segmentation. Frame segmentation is the aforementioned frame extraction process and the decoding of the extracted frames.

[0076] In summary, because the reconstructed bitstream contains fewer non-keyframes compared to the original video bitstream, the proportion of non-keyframes in the reconstructed bitstream is reduced, while the proportion of keyframes is increased. This leads to a higher probability of extracting keyframes when extracting sample frames from the reconstructed bitstream. Consequently, the total number of keyframes and non-keyframes in the sampled frames is greater, and the number of dependent frames associated with non-keyframes is also reduced. This decreases the number of dependent frames to decode, significantly reducing the additional time required for decoding dependent frames and thus improving decoding speed. In other words, because this application is more likely to capture keyframes as sample frames, the number of non-keyframes in the total sampled frames is reduced, resulting in fewer dependent frames. This reduces the total number of sampled frames, meaning fewer sampled frames need to be decoded, thereby greatly improving decoding speed and video frame extraction efficiency.

[0077] In practical applications, the decoding process for a single sampled frame may include the following: The intermediate results generated after decoding the sampled frame are cached in memory. These intermediate results can be in YUV format, where YUV is a color encoding method, where Y represents luminance and U and V represent chrominance. Next, the intermediate results undergo color space conversion to obtain an RGB format image, where R represents red, G represents green, and B represents blue.

[0078] When decoding a sampled frame, the intermediate results of that sampled frame and the intermediate results of its dependent frames need to be cached in memory. The memory can only be released after color space conversion. Before releasing the memory, a large number of dependent frames need to be cached, which greatly increases memory consumption and can easily lead to memory overflow (OOM).

[0079] refer to Figure 5 The diagram shown is a schematic of memory consumption according to an embodiment of this application. There are four sampling frames in the memory, namely I-frame, B-frame and two P-frames. The P-frame in the dashed box is the dependent frame of other frames. The remaining three frames in the solid box are specific frames, that is, the frames that need to display the corresponding screen. The color space is converted for these three frames to obtain RGB format frames. Then, the memory contains RGB format data of I-frame, B-frame and P-frame, but does not contain RGB format data of the P-frame that is the dependent frame, and the memory is released.

[0080] Therefore, by eliminating non-reference frames, this application reduces the total number of sampled frames, meaning fewer frames are cached in memory, thus reducing memory usage and lowering memory consumption. Furthermore, this application reduces the total number of extracted sampled frames, improving decoding speed and video frame extraction efficiency without affecting the decoding process, ensuring smooth video frame extraction.

[0081] After removing non-reference frames to obtain the reconstructed bitstream, the dependencies between frames in the reconstructed bitstream may change, which may lead to changes in frame type. For example, the frame type of the frame to be decoded may change from a reference frame to a non-reference frame, resulting in new non-reference frames appearing again in the reconstructed bitstream. In order to further improve the decoding speed, the new non-reference frames can be removed again.

[0082] In one possible implementation, step 202 removes non-reference frames from the video bitstream to be processed to obtain the reconstructed bitstream, which may specifically include steps 2021-2023 (not shown in the figures).

[0083] Step 2021: Based on the attribute information, remove the frames to be decoded that are non-reference frames from the video bitstream to be processed to obtain the first intermediate bitstream.

[0084] Specifically, the bitstream obtained after removing non-reference frames from the video bitstream to be processed based on attribute information can be denoted as the first intermediate bitstream.

[0085] Step 2022: Based on the decoding dependency relationship between the frames to be decoded, adjust the frame type of each frame to be decoded in the first intermediate bitstream to obtain the second intermediate bitstream.

[0086] The decoding dependency between frames to be decoded refers to the relationship between a frame to be decoded and other frames to be decoded during decoding; in other words, whether a frame depends on other frames to be decoded in order to perform decoding. If a frame to be decoded depends on another frame to be decoded, then there is a decoding dependency between them; otherwise, there is no decoding dependency between them.

[0087] Furthermore, the decoding dependencies between frames to be decoded change as frame culling proceeds. Since a culled non-reference frame may depend on other frames to be decoded, if the frame to be decoded that was depended on only serves as a dependency frame of the culled non-reference frame, then after the non-reference frame is removed, it means that in the first intermediate bitstream, no other frames to be decoded need to depend on this previously depended-on frame. Therefore, the frame that the previously culled non-reference frame depended on will become a new non-reference frame.

[0088] As an example, if non-reference frame 2 depends on reference frame 3, and reference frame 3 is only depended on by non-reference frame 2, then after removing non-reference frame 2, no other frame depends on reference frame 3, and the frame type of reference frame 3 changes to non-reference frame.

[0089] Therefore, based on the decoding dependencies between the frames to be decoded, the server can adjust the frame type of each frame to be decoded in the first intermediate bitstream. For example, the frame type of the frame to be decoded can be changed from a reference frame to a non-reference frame. The adjusted bitstream is recorded as the second intermediate bitstream. In short, the second intermediate bitstream contains the correct frame type of each frame to be decoded under the latest decoding dependencies.

[0090] refer to Figure 6 The diagram illustrates a process of multiple non-reference frame removal according to an embodiment of this application. The video stream to be processed includes six frames to be decoded: keyframe 1, non-reference frame 2, reference frame 3, reference frame 4, reference frame 5, and non-reference frame 6. After the first non-reference frame removal, a first intermediate stream is obtained. This first intermediate stream includes four frames to be decoded: keyframe 1, reference frame 3, reference frame 4, and reference frame 5. The arrows in the diagram indicate the decoding dependencies between frames: non-reference frame 2 depends on keyframe 1 and reference frame 3; reference frame 4 depends on keyframe 1; reference frame 5 depends on reference frame 4; and non-reference frame 6 depends on reference frame 5.

[0091] After removing non-reference frame 2, since no other frames to be decoded depend on reference frame 3 in the first intermediate bitstream, the frame type of reference frame 3 is adjusted, and reference frame 3 becomes a new non-reference frame. Similarly, after removing non-reference frame 6, since no other frames to be decoded depend on reference frame 5, reference frame 5 is adjusted to a new non-reference frame. That is, the second intermediate bitstream obtained after frame type adjustment includes four frames to be decoded: keyframe 1, non-reference frame 3, reference frame 4, and non-reference frame 5.

[0092] Step 2023: When the second intermediate bitstream contains a frame to be decoded that is a non-reference frame, the second intermediate bitstream is used as the video bitstream to be processed. The process of removing the non-reference frame to be decoded from the video bitstream to be processed based on attribute information is returned to obtain the first intermediate bitstream. This process continues until the stopping condition is met, resulting in a reconstructed bitstream. The stopping condition is not specifically limited and can be a certain number of iterations.

[0093] If, in the second intermediate bitstream, the frame type of the frame to be decoded is a non-reference frame, in order to further improve the decoding speed, the non-reference frame can be removed again. That is, the second intermediate bitstream is used as the video bitstream to be processed in the next round of removal, and steps 2021-2023 are repeated until the stopping condition is met. The second intermediate bitstream in the last round of removal is then determined as the reconstructed bitstream. Figure 6 As shown, non-reference frames are removed from the second intermediate bitstream again to obtain the reconstructed bitstream, which includes two frames to be decoded, namely key frame 1 and reference frame 4.

[0094] In short, after one round of non-reference frame removal, new non-reference frames will appear in the first intermediate bitstream. Therefore, multiple non-reference frame removal processes can be performed based on actual needs, thereby further improving the decoding speed.

[0095] In summary, by performing multiple rounds of non-reference frame removal on the video stream to be processed, the proportion of non-key frames in the stream can be reduced in each round of removal, while the proportion of key frames in the stream can be gradually increased. This can further reduce the total number of extracted sampled frames, thereby improving the decoding speed and video frame extraction efficiency.

[0096] In step 203, during the process of obtaining multiple sampled frames through frame extraction, in order to further reduce the total number of sampled frames, key frames can be extracted first to avoid sampling reference frames that require dependent frames.

[0097] In one possible implementation, step 203, which involves extracting frames from the reconstructed bitstream based on a preset number of frames to obtain multiple sampled frames, can specifically involve determining the total number of key frames in the reconstructed bitstream; and when the total number of key frames is greater than or equal to the preset number of frames, extracting frames from the key frames based on the preset number of frames to obtain multiple sampled frames.

[0098] The server can determine the total number of keyframes in the reconstructed bitstream to judge whether the total number of keyframes meets the preset frame count requirement; that is, it can determine whether the total number of keyframes is greater than or equal to the preset frame count. If it is greater, it means that the total number of keyframes is sufficient to meet the sampling requirement of the preset frame count. Furthermore, in order to minimize the number of sampled frames while meeting the preset frame count, the server can use the preset frame count as the sampling number to extract frames from the keyframes, using the sampled keyframes as sampled frames, thus obtaining multiple sampled frames. Each sampled frame is a keyframe.

[0099] Furthermore, since keyframes do not depend on other frames during decoding, there is no need to sample other frames during sampling. Therefore, when each sampled frame is a keyframe, the total number of sampled frames equals the preset number of frames. In other words, each sampled frame can correspond to a display image after decoding, so the total number of sampled frames is consistent with the preset number of frames used to identify the number of images.

[0100] As an example, if there are 10 frames in the decoded video, it means that the preset number of frames is 10. Assuming that the total number of keyframes in the reconstructed bitstream is 18, then 10 keyframes can be selected as sampling frames. The total number of sampling frames is equal to the preset number of frames, which is 10.

[0101] Of course, when the total number of keyframes in the reconstructed bitstream equals the preset number of frames, all keyframes can be used as sampling frames. That is, frame extraction processing is performed on all keyframes to meet the requirement of sampling the preset number of frames. As an example, if there are 10 frames in the decoded video, it means that the preset number of frames is 10. Assuming that the total number of keyframes in the reconstructed bitstream is also 10, all keyframes can be extracted, thus obtaining 10 sampling frames.

[0102] In summary, when the number of keyframes in the reconstructed bitstream is sufficient, sample frames can be extracted from these keyframes. This allows for normal decoding while minimizing the total number of sample frames, ensuring the total number of sample frames equals the preset frame count. This significantly reduces the total number of sample frames, greatly improving decoding speed and frame extraction efficiency. Furthermore, it minimizes the memory overhead of sample frame buffering, storing only the minimum number of sample frames.

[0103] When extracting sample frames from keyframes, sampling can be performed at certain intervals or randomly. In random sampling, keyframes can be arbitrarily selected as sample frames, resulting in all sample frames distributed at any position in the reconstructed bitstream. For example, assuming the total number of keyframes in the reconstructed bitstream is 8 and the preset frame count is 4, then 4 keyframes can be randomly selected from the 8, for example, the 1st, 2nd, 3rd, and 8th keyframes can be extracted as sample frames. Alternatively, sample frames can be obtained by uniformly sampling the keyframes.

[0104] In one possible implementation, frame extraction processing is performed from keyframes based on a preset number of frames to obtain multiple sample frames. Specifically, frame extraction processing is performed uniformly from keyframes based on a preset number of frames to obtain multiple sample frames.

[0105] In other words, the server can extract frames evenly from keyframes according to a preset number of frames, thereby obtaining multiple sampled frames evenly distributed in the reconstructed bitstream. Furthermore, since the interval between keyframes in a video is usually fixed—that is, the same number of other frames are between every two keyframes—this number is typically determined by the encoder settings and remains constant throughout the video. In other words, keyframes are usually evenly distributed throughout the video. By evenly extracting frames from evenly distributed keyframes, the resulting sampled frames also have a relatively even distribution, meaning the images corresponding to the sampled frames are evenly distributed throughout the video.

[0106] As an example, assuming the total number of keyframes in the reconstructed bitstream is 8 and the preset number of frames is 4, then 4 can be extracted evenly from the 8, and the 1st, 3rd, 5th, and 7th keyframes can be extracted as sampling frames, or the 2nd, 4th, 6th, and 8th keyframes can be extracted as sampling frames.

[0107] In summary, by uniformly extracting frames from keyframes, multiple evenly distributed sampled frames can be extracted, resulting in frame truncation effects that are as excellent as existing uniform frame truncation. Subsequent decoding of each sampled frame yields images that are evenly distributed within the video before truncation, allowing the truncation video to more accurately represent the content of the video before truncation. This avoids the situation where uneven distribution of sampled frames only reflects a portion of the video before truncation, which is detrimental to downstream models such as action recognition.

[0108] In step 203, during the process of extracting frames from the reconstructed bitstream to obtain multiple sampled frames, if the total number of key frames in the reconstructed bitstream is relatively small and cannot meet the preset frame quantity requirement, in order to further improve the decoding speed, sampled frames can be extracted from the key frames first, and the missing sampled frames can be extracted from the reference frames to fill them.

[0109] In one possible implementation, when the total number of keyframes is less than the preset number of frames, frame extraction processing is performed from the keyframes to obtain a first sampled frame of a first sampled number; based on a second sampled number, frame extraction processing is performed from the reference frame to obtain multiple second sampled frames; the multiple first sampled frames and the multiple second sampled frames are used as multiple sampled frames.

[0110] When the total number of keyframes in the reconstructed bitstream is less than the preset number of frames, it means that even if all keyframes are extracted as sampling frames, the preset number of frames cannot be met. Therefore, when performing frame extraction processing, reference frames in the reconstructed bitstream need to be considered, that is, frame extraction processing is performed from reference frames and keyframes.

[0111] To minimize the total number of sampled frames, frame extraction can be prioritized from keyframes. The extracted frames are designated as first sampled frames, and the total number of first sampled frames is represented as the first sampling count. Frame extraction can be performed according to a pre-defined frame segmentation method, such as fixed intervals of duration or fixed number of frames. For example, every three keyframes can be extracted as a first sampled frame. The total number of first sampled frames then yields the first sampling count. Alternatively, the number of frames to be extracted can be pre-defined, i.e., the first sampling count can be pre-set. This first sampling count serves as a limit on the number of frames extracted, resulting in multiple first sampled frames. For example, a first sampling count of 5 means extracting 5 frames from all keyframes as 5 first sampled frames. For clarity, the following explanation will assume a pre-defined first sampling count.

[0112] The first sampling number can be less than or equal to the total number of keyframes. When the first sampling number is less than the total number of keyframes, it means that some keyframes can be extracted from all keyframes to obtain multiple first sample frames with the first sampling number. Frame extraction can be performed uniformly to obtain evenly distributed first sample frames, or it can be performed arbitrarily. As an example, if the total number of keyframes in the reconstructed bitstream is 14 and the first sampling number is 10, then 10 can be sampled from the 14 keyframes to obtain 10 first sample frames. When the first sampling number is equal to the total number of keyframes, it means that all keyframes can be extracted to obtain a sufficient number of multiple first sample frames, reducing the total number of sample frames and improving decoding speed. As an example, if the total number of keyframes in the reconstructed bitstream is 14 and the first sampling number is also 14, then all 14 keyframes can be sampled to obtain 14 first sample frames.

[0113] Since the number of preset frames is relatively large, the total number of keyframes is relatively small, and the number of initial samples is also relatively small, the number of initial samples cannot meet the sampling requirements of the preset frame count. Therefore, the server can calculate the difference between the preset frame count and the number of initial samples, and record this difference as the second sampling count. In other words, the second sampling count is the preset frame count minus the first sampling count. The second sampling count can be understood as the number of frames that need to be extracted from the reference frames of the reconstructed bitstream.

[0114] The server can extract frames from the reference frames of the reconstructed bitstream and use a second sampling quantity as the limit for the number of samples in this sampling, thus obtaining multiple second-sampled frames. The second-sampled frame is the sampled frame obtained from the reference frame. Since each reference frame sampled as a second-sampled frame also requires sampling the dependent frames that the reference frame depends on, a total of two sampled frames are obtained. Therefore, the total number of multiple second-sampled frames is greater than or equal to the first sampling quantity.

[0115] refer to Figure 7 The diagram illustrates a method for extracting sampled frames according to an embodiment of this application. The reconstructed bitstream contains multiple frames to be decoded, namely key frame 1, reference frame 3, reference frame 4, key frame 5, and reference frame N-1. These frames are divided into a key frame sequence and a non-key frame sequence. The key frame sequence includes all key frames, as shown in the diagram (key frame 1, key frame 5, and key frame M). The non-key frame sequence includes all reference frames, as shown in the diagram (reference frame 3, reference frame 4, and reference frame N-1). Frame extraction is performed from the key frame sequence to obtain multiple first sampled frames, namely key frame 5 and key frame M. Frame extraction is performed from the non-key frame sequence to obtain multiple second sampled frames, namely reference frame 3 and reference frame N-1.

[0116] The server can use all the first and second sampled frames as the total number of sampled frames, thus obtaining multiple sampled frames that meet the preset frame count requirement. Since the sampled frames include keyframes and reference frames, the total number of sampled frames is greater than the preset frame count.

[0117] As an example, assuming the preset number of frames is 20, the total number of keyframes in the reconstructed bitstream is 14, and the first sampling number is 12, then 12 can be sampled from the 14 keyframes to obtain 12 first sampling frames. Based on the preset number of frames and the first sampling number, the second sampling number is 8, so the reference frames in the reconstructed bitstream can be sampled to obtain 10 second sampling frames. The 12 first sampling frames and the 10 second sampling frames make up 22 sampling frames.

[0118] In summary, when the total number of keyframes cannot meet the sampling requirement of the preset number of frames, the keyframes in the reconstructed bitstream can be extracted first to extract as many keyframes as possible as sampling frames, thereby reducing the total number of sampling frames and greatly reducing the memory overhead caused by cached dependent frames. In addition, in order to meet the sampling requirements, the reference frames in the reconstructed bitstream can be extracted to make up for the preset number of frames, thereby obtaining a sufficient number of sampling frames, which takes into account both the preset number of frames and the decoding speed.

[0119] By employing the aforementioned non-reference frame removal and keyframe extraction techniques, redundancy reduction in video can be achieved to a certain extent. The following section primarily describes the redundancy reduction process from a visual perspective. In some scenarios, visual redundancy still exists. For example, in videos where the overall visual changes are minimal, such as speeches or commentaries, the content repetition is high. Furthermore, when videos contain meaningless frames like transition frames or solid color frames, transition frames may have similar content to their adjacent frames, and solid color frames contain no visual content. Therefore, additional methods are needed to further remove redundant frames and achieve true video deduplication.

[0120] The following sections mainly introduce three methods for deduplication at the image level: image feature comparison deduplication, grayscale distribution difference deduplication, and deep deduplication model deduplication. Image feature comparison deduplication determines the similarity of images by comparing their features. Grayscale distribution difference deduplication is mainly used to remove meaningless frames using grayscale information. Deep deduplication model deduplication primarily uses neural network models for deduplication. The process of image feature comparison deduplication will be introduced first.

[0121] In one possible implementation, the server may also execute steps 301-304 (not shown in the figures).

[0122] Step 301: Extract features from each decoded frame in the decoded video to obtain the image features of each decoded frame.

[0123] Since all video frames in a decoded video have undergone the decoding process, each frame can be recorded as a decoded frame. Each decoded frame corresponds to a displayable image. The server can extract features from each decoded frame to obtain its corresponding image features. These image features reflect the characteristics of the decoded frame and can be used for image comparison. In practical applications, feature extraction can be performed on RGB format images to obtain image features.

[0124] As an example, image feature extraction algorithms can be used to extract features from decoded frames to obtain image features. Such image feature extraction algorithms can be, for example, Scale-invariant feature transform (SIFT) algorithm, Speeded-Up Robust Features (SURF) algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, etc.

[0125] Step 302: Calculate the first similarity between the image features of the two decoded frames.

[0126] Since each decoded frame has its own image features, for any two decoded frames, the server can calculate the similarity between their corresponding image features and record this similarity as the first similarity. The first similarity reflects the degree of image similarity between the two decoded frames; the higher the first similarity, the more similar the images of the two decoded frames are. For all decoded frames in the decoded video, the first similarity between any two decoded frames can be calculated, thereby achieving image deduplication across all decoded frames.

[0127] This application does not limit the representation of the first similarity in its embodiments. For example, the first similarity can be represented by the feature difference between two image features. In this case, the greater the feature difference, the smaller the first similarity. As an example, the magnitude of the feature difference can be measured by the cosine similarity between image features. The feature difference diff between two image features can be expressed as:

[0128] diff=cos(O(I t )-O(I t+1 ))

[0129] Among them, I t I represents the decoded frame at time t. t+1 This represents the decoded frame at time t+1, and O() represents the image feature extraction algorithm, O(I t ) and O(I t+1 The two decoded frames represent their respective image features, and cos() represents the cosine similarity calculation. The larger the feature difference (diff) between the two decoded frames, the smaller their initial similarity, indicating they are less similar.

[0130] Step 303: When the first similarity is greater than the first similarity threshold, determine the first redundant frame from the two decoded frames.

[0131] The first similarity threshold can be understood as a similarity threshold for determining the similarity between two image features, for example, it can be 70%. If the first similarity is greater than the first similarity threshold, it means that the similarity between the two image features is large enough to determine that the two decoded frames corresponding to these two image features are similar, and the two decoded frames are redundant to each other, so they need to be deduplicated. Therefore, one of these two decoded frames can be randomly selected as the first redundant frame. The first redundant frame can be understood as a redundant decoded frame in the decoded video.

[0132] As an example, if the first similarity between decoded frames A and B is 80%, which exceeds the first similarity threshold of 70%, then decoded frame A can be used as the first redundant frame. Of course, decoded frame B can also be used as the first redundant frame.

[0133] As an example, when measuring the first similarity by the feature difference between two image features, if the feature difference *diff* is greater than the difference threshold *θ*, it can be said that the first similarity is less than the first similarity threshold, indicating that I... t and I t+1 The less similar they are, the more similar they are. t Or I t+1 It belongs to the first redundant frame.

[0134] Step 304: Remove the first redundant frame from the decoded video to obtain the first deduplicated video.

[0135] The server can remove the first redundant frame from the decoded video and use the remaining video as the first deduplicated video.

[0136] In summary, by comparing image features, deduplication can be performed on decoded videos, removing frames with similar visuals. This results in a larger difference between the decoded frames in the first deduplicated video, reducing redundancy and improving the accuracy of subsequent video processing, such as enhancing action recognition accuracy.

[0137] In one possible implementation, the image features of two decoded frames can be the image features corresponding to two adjacent decoded frames in the decoded video.

[0138] In other words, the server can perform image comparison on two adjacent frames of a decoded video in a time series, comparing the image features corresponding to these two adjacent decoded frames to obtain a first similarity score. For example, I t and I t+1 The comparison is performed. The first similarity measure measures the degree of similarity between two adjacent images. Then, when the first similarity is greater than the first similarity threshold, one of the two adjacent images can be removed as the first redundant frame to obtain the first deduplicated video.

[0139] In this way, by deduplicating adjacent frames in the decoded video, we can avoid deduplicating two similar frames that are far apart in time sequence, which would cause the actions or changes in the frames to be removed. This application can ensure that the changing trend of the frames in the video is not erased, making the deduplicated video more consistent with the original video, and the changes in the content are preserved. This facilitates subsequent processing of the deduplicated video, such as identifying more accurate actions or having a more accurate understanding of the deduplicated video.

[0140] Next, we will explain the second deduplication method, namely, grayscale distribution difference deduplication. For meaningless frames such as transition frames and solid color frames, which result in transition or solid color scenes in the video, these scenes do not help the downstream model's processing. (Refer to...) Figure 8 The diagram shown illustrates a video frame sequence according to an embodiment of this application, including one transition frame, two normal frames, and one solid black frame. To further reduce image redundancy, these meaningless frames can be removed.

[0141] In one possible implementation, the server may also perform steps 401-403 (not shown in the figures).

[0142] Step 401: Determine the grayscale distribution information of each decoded frame in the decoded video.

[0143] Grayscale distribution information refers to the overall grayscale distribution of the image corresponding to the decoded frame. It reflects the grayscale distribution of pixels in the image. The grayscale value range can be, for example, 0-255, with white pixels having a grayscale value close to 255 and black pixels having a grayscale value close to 0. Through grayscale distribution information, we can know the brightness of the image corresponding to the decoded frame. For example, if there are many pixels with a grayscale value close to 255, it indicates that the image is generally bright, possibly a pure white image.

[0144] In practical applications, grayscale distribution information can be represented using histograms. The horizontal axis of the histogram represents the grayscale value, and the vertical axis represents the number of corresponding pixels, i.e., the frequency. For ease of statistical analysis, the grayscale value range can be divided into several smaller ranges, such as 0-40, 40-80, etc., and each smaller range can be represented on the horizontal axis. In this case, the vertical axis represents the frequency of pixels within each smaller range. To provide a more detailed representation of the statistical situation for each grayscale value, each grayscale value can also be represented on the horizontal axis. In this case, the vertical axis represents the frequency of pixels equal to that grayscale value. For example, when the grayscale value range is 0-255, the histogram will have 256 bars.

[0145] Step 402: Based on the grayscale distribution information, determine the second redundant frame from the decoded video.

[0146] Transition frames are used to connect two different frames. Generally, a transition frame has the same content as the frame before or after it, but the brightness is different. Therefore, transition frames are redundant frames that can be removed. Solid color frames contain only one color, such as white, black, or gray. These frames do not contain any image information and can be removed as redundant frames.

[0147] The server can identify a second redundant frame from the decoded video based on grayscale distribution information. This second redundant frame is the extra decoded frame that needs to be removed. In the grayscale distribution information of the second redundant frame, the number of pixels with grayscale values ​​within a preset grayscale range is greater than a preset number. In short, determining whether a frame is a second redundant frame requires comprehensive consideration of both the grayscale value and the number of pixels.

[0148] The preset grayscale range can be a pre-defined grayscale range, which can include one or more grayscale values. For example, if the preset grayscale range is greater than 125 and less than 127, then the range only includes grayscale values ​​equal to 126. Or, if the preset grayscale range is greater than 145 and less than 160, then the range includes multiple grayscale values.

[0149] When a preset grayscale range contains only one grayscale value, that grayscale value can be recorded as the preset grayscale value. The number of pixels whose grayscale value falls within the preset grayscale range is the same as the number of pixels whose grayscale value is the preset grayscale value. Consequently, in the second redundant frame, the number of pixels with the preset grayscale value is excessive, exceeding the preset number. For example, if the preset grayscale range is greater than 254 and less than 256 (i.e., the preset grayscale value is 255) and the preset number is 600, then if the total number of pixels with a grayscale value of 255 in a certain frame exceeds 600, that frame may be a pure white image, and thus it can be identified as a second redundant frame.

[0150] When a preset grayscale range includes multiple grayscale values, the number of pixels whose grayscale values ​​fall within that range is the total number of pixels whose grayscale values ​​fall within that range. Consequently, in the second redundant frame, there are too many pixels within that range, exceeding the preset number. For example, the preset grayscale range could be 200-255, and the preset number could be 1000. If, in a certain frame, the total number of pixels whose grayscale values ​​fall within that range exceeds 1000, then that frame is too bright and needs to be removed; thus, that frame can be identified as the second redundant frame.

[0151] refer to Figure 9The diagram shows a method for determining a second redundant frame according to an embodiment of this application. (a), (b), and (c) are grayscale distribution histograms. The horizontal axis represents the grayscale value, which is divided into 0-255 using 8 ranges. These 8 ranges are 0-31, 32-63, 64-95, 96-127, 128-159, 160-191, 192-223, and 224-255, corresponding to 8 bars. The vertical axis represents the number of corresponding pixels, i.e., the frequency. The diagram also shows a preset number. (d), (e), and (f) are the decoded frames corresponding to (a), (b), and (c), respectively. In (a), since the number of pixels with grayscale values ​​in the range of 0-31 exceeds the preset number, the video frame (d) corresponding to (a) can be determined as a transition frame, and then this transition frame is used as the second redundant frame. In (b), the number of pixels with grayscale values ​​between 128 and 159 exceeds the preset number. Therefore, the gray video frame (e) can be identified as a solid color frame and used as the second redundant frame. In (c), the pixel distribution of this frame is relatively uniform, meaning (f) is a normal frame and does not need to be discarded.

[0152] Step 403: Remove the second redundant frame from the decoded video to obtain the second deduplicated video.

[0153] After identifying the second redundant frame, the server can remove it from the decoded video and use the remaining video as the second deduplicated video.

[0154] In this way, by analyzing the grayscale distribution of the image, invalid frames such as transition frames and solid color frames are removed, thus filtering out these meaningless frames and further reducing the redundancy of the image. This ensures that each frame in the deduplicated video can better represent the video content before deduplication, making the processing results of the downstream model more accurate.

[0155] The third deduplication method, namely deep deduplication model, will be introduced in detail below. If the video contains text interference, picture-in-picture, or camera zoom, it is also necessary to deduplicat the images and remove similar images. For example, to deduplicat two images with similar content but different camera zoom levels, the third deduplication method can be used to achieve deep deduplication.

[0156] In one possible implementation, the server may also execute steps 501-505 (not shown in the figures).

[0157] Step 501: Input the decoded video into the deduplication model and extract the embedding vectors of each decoded frame in the decoded video.

[0158] The deduplication model can be a pre-trained neural network model capable of deduplicating images, such as a lightweight neural network model. The server can input the decoded video into the deduplication model, which can extract the embedding vectors of each decoded frame. The embedding vectors can represent the image characteristics of the decoded frames.

[0159] As examples, deduplication models can also include unsupervised momentum update contrastive learning schemes (MoCo), masked autoencoders (MAE), perceptual hash algorithms (Phash), etc.

[0160] Step 502: Calculate the second similarity between the embedding vectors of the two decoded frames.

[0161] For any two decoded frames in a decoded video, the deduplication model can calculate the similarity between the embedding vectors corresponding to these two decoded frames. This similarity can be denoted as the second similarity. The second similarity can represent the degree of similarity between the two frames at a deeper level.

[0162] Step 503: When the second similarity is greater than the second similarity threshold, determine the third redundant frame from the two decoded frames.

[0163] The second similarity threshold can be understood as a similarity value used to determine the similarity between two images. For example, it can be 66%. The deduplication model can determine whether the second similarity is greater than the second similarity threshold. If it is greater, it means that the second similarity is large enough to indicate that the two images are similar. Then, either of the two decoded frames corresponding to these two images can be used as the third redundant frame. The third redundant frame is the redundant decoded frame that needs to be removed.

[0164] In practical applications, the cosine similarity between two embedding vectors can be used to measure the second similarity. The smaller the cosine similarity, the smaller the difference between the two embedding vectors, and the greater their second similarity. Therefore, when the cosine similarity is less than the cosine similarity threshold, they are considered to be sufficiently similar, and the third redundant frame can be determined.

[0165] Step 504: Remove the third redundant frame from the decoded video and output the third deduplicated video.

[0166] The deduplication model can remove the identified third redundant frame from the decoded video, thereby outputting the third deduplicated video, which is the video obtained after deduplication by the deduplication model.

[0167] As an example, refer to Figure 10 The diagram shows a deduplication model provided in an embodiment of this application. The decoded video is input into the deduplication model. The decoded video includes 5 decoded frames, i.e. 5 scenes. The deduplication model extracts the embedding vector, calculates the second similarity, and determines the third redundant frame. Thus, it is determined that the character actions in frames 1, 2, and 3 are the same, only the character positions or scaling are different. Therefore, frames 2 and 3 can be used as the third redundant frames. The output third deduplicated video includes the first scene, the fourth scene, and the fifth scene.

[0168] In summary, by introducing a deduplication model for in-depth deduplication, we can eliminate text interference, picture-in-picture, and camera zoom, resulting in a significantly larger difference between the individual frames in the deduplicated video. This allows for a more accurate condensation of the original video with a limited number of frames, facilitating more accurate judgment and processing by downstream models and greatly reducing frame redundancy.

[0169] In addition, before using a deduplication model for image deduplication, the model needs to be pre-trained. For example, an initial model can be pre-trained based on an image self-supervised learning algorithm to obtain a deduplication model. During training, a large number of images can be obtained by data augmentation of the original image to construct positive samples, and other images can be used as negative samples. By comparing the embedding vectors of different samples, self-supervised learning can be achieved, enabling the model to learn the relevant knowledge of image deduplication.

[0170] Furthermore, this application can achieve image deduplication through image feature comparison deduplication, grayscale distribution difference deduplication, and deduplication model depth deduplication. One, two, or all three methods can be used. When two or three deduplication methods are needed, their execution order is not specifically limited. Taking the use of three methods as an example, for a decoded video, depth deduplication can be performed first, followed by image feature comparison deduplication and grayscale distribution difference deduplication. Since depth deduplication consumes more inference time than the other two methods, image feature comparison deduplication and grayscale distribution difference deduplication can be used first for preliminary deduplication, reducing the number of frames input to the deduplication model during depth deduplication, thereby accelerating the depth deduplication speed and improving frame extraction efficiency.

[0171] The video processing method provided in this application can be applied to various video-related scenarios, such as video understanding, action recognition, and video content description. Video understanding involves analyzing a video to determine its tags; for example, video understanding might identify a video as a modern drama. Action recognition involves identifying and classifying actions within a video. Video content description involves generating a summary of a video's main theme; for example, describing the story outline of a video. The following explanation uses action recognition as an example.

[0172] refer to Figure 11 The diagram illustrates a video processing procedure according to an embodiment of this application. The server acquires a video stream to be processed, which includes multiple frames to be decoded. Each frame has its own attribute information, indicating its frame type, such as whether it is a keyframe or a non-keyframe, or whether it belongs to a reference frame or a non-reference frame among non-keyframes. Based on the attribute information, the server can remove non-reference frames from the video stream to obtain a reconstructed stream. Then, based on a preset number of frames, frame extraction processing is performed on the reconstructed stream to obtain multiple sampled frames. Decoding these sampled frames yields the decoded video, thus completing the frame extraction process.

[0173] If new non-reference frames appear in the reconstructed bitstream, the process of removing non-reference frames can be repeated multiple times. When extracting sampled frames, if the total number of keyframes in the reconstructed bitstream is not less than the preset number of frames, frames can be extracted from the keyframes to obtain multiple sampled frames. Otherwise, frames can be extracted first from the keyframes, and then from the reference frames to obtain multiple sampled frames. Furthermore, frame extraction can be performed uniformly to obtain multiple evenly distributed sampled frames.

[0174] Next, the decoded video obtained from frame extraction can undergo redundancy removal, which can be accomplished in three ways. The first method is image feature comparison deduplication to remove similar frames. Specifically, the server can extract features from each decoded frame in the decoded video to obtain image features, compare adjacent image features pairwise, and when similarities are found, determine the first redundant frame to be removed and eliminate it from the decoded video to obtain the first deduplicated video.

[0175] The second method is grayscale distribution difference deduplication, which removes invalid frames such as transition frames and solid color frames. For the first deduplicated video, the grayscale distribution information of each decoded frame can be determined. Based on this grayscale distribution information, the second redundant frames are identified and removed. The second redundant frames are, for example, transition frames or solid color frames, thus obtaining the second deduplicated video.

[0176] The third method is deep deduplication using a deduplication model, achieving a deeper level of deduplication, such as addressing text interference or camera zoom issues in the video. The second deduplicated video is input into the deduplication model, which extracts the embedding vectors of each decoded frame. Based on the similarity between the embedding vectors, the model identifies and removes the third redundant frame, resulting in the third deduplicated video. Figure 11 As shown, since the second and third frames in the decoded video are similar, only one can be retained through redundancy removal, thus obtaining the third deduplicated video. This third deduplicated video is then input into the downstream action recognition model to identify the action in the video as playing with a yo-yo.

[0177] It should be noted that, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0178] This application also provides a video processing apparatus, see reference. Figure 12 The diagram shown is a schematic representation of a video processing apparatus provided in an embodiment of this application. The apparatus includes:

[0179] The acquisition unit 1201 is used to acquire attribute information of each frame to be decoded in the video bitstream to be processed; the attribute information is used to indicate the frame type of each frame to be decoded, the frame type includes key frames and non-key frames, and the non-key frames include reference frames and non-reference frames.

[0180] The elimination unit 1202 is used to eliminate frames to be decoded that are non-reference frames from the video bitstream to be processed based on the attribute information, so as to obtain a reconstructed bitstream.

[0181] Extraction unit 1203 is used to perform frame extraction processing from the reconstructed bitstream based on a preset number of frames to obtain multiple sampled frames;

[0182] The decoding unit 1204 is used to decode multiple sampled frames to obtain a decoded video.

[0183] Specifically, the rejection unit is used for:

[0184] Based on the attribute information, frames to be decoded that are non-reference frames are removed from the video bitstream to be processed to obtain the first intermediate bitstream;

[0185] Based on the decoding dependency relationship between the frames to be decoded, the frame type of each frame to be decoded in the first intermediate bitstream is adjusted to obtain the second intermediate bitstream.

[0186] When there are frames to be decoded in the second intermediate bitstream that are non-reference frames, the second intermediate bitstream is used as the video bitstream to be processed, and the step of removing frames to be decoded that are non-reference frames from the video bitstream to be processed based on the attribute information is returned to execute until the stopping condition is met, and the reconstructed bitstream is obtained.

[0187] Specifically, the extraction unit is used for:

[0188] Determine the total number of keyframes in the reconstructed bitstream;

[0189] When the total number of keyframes is greater than or equal to the preset number of frames, frame extraction processing is performed on the keyframes based on the preset number of frames to obtain multiple sampled frames; the total number of sampled frames is equal to the preset number of frames.

[0190] Specifically, the device further includes:

[0191] The first extraction unit is used to perform frame extraction processing from the key frames when the total number of key frames is less than the preset number of frames, to obtain the first sampled frames of the first sampling number.

[0192] The second extraction unit is used to perform frame extraction processing from the reference frame based on the second sampling quantity to obtain multiple second sampled frames; the second sampling quantity is the difference between the preset frame quantity and the first sampling quantity.

[0193] The first determining unit is configured to take multiple first sampling frames and multiple second sampling frames as multiple sampling frames; the total number of sampling frames is greater than the preset number of frames.

[0194] Specifically, the extraction unit is used for:

[0195] Based on the preset number of frames, frame extraction is performed uniformly from the key frames to obtain multiple sampled frames.

[0196] Specifically, the device further includes:

[0197] The third extraction unit is used to extract features from each decoded frame in the decoded video to obtain the image features of each decoded frame;

[0198] The first computing unit is used to calculate the first similarity between the image features of two decoded frames;

[0199] The second determining unit is used to determine a first redundant frame from the two decoded frames when the first similarity is greater than the first similarity threshold.

[0200] The first redundant frame is removed from the decoded video to obtain the first deduplicated video.

[0201] Specifically, the image features of the two decoded frames are the image features corresponding to each of the two adjacent decoded frames in the decoded video.

[0202] Specifically, the device further includes:

[0203] The third determining unit is used to determine the grayscale distribution information of each decoded frame in the decoded video;

[0204] The fourth determining unit is used to determine a second redundant frame from the decoded video based on the grayscale distribution information; in the grayscale distribution information of the second redundant frame, the number of pixels with grayscale values ​​within a preset grayscale range is greater than a preset number.

[0205] The first elimination unit is used to eliminate the second redundant frame from the decoded video to obtain the second deduplicated video.

[0206] Specifically, the device further includes:

[0207] The first extraction unit is used to input the decoded video into the deduplication model and extract the embedding vector of each decoded frame in the decoded video.

[0208] The second computing unit is used to calculate the second similarity between the embedding vectors of two decoded frames;

[0209] The fifth determining unit is used to determine a third redundant frame from the two decoded frames when the second similarity is greater than the second similarity threshold.

[0210] The second elimination unit is used to remove the third redundant frame from the decoded video and output the third deduplicated video.

[0211] Specifically, the device further includes:

[0212] The processing unit is used to acquire the video bitstream of the video to be processed;

[0213] The sixth determining unit is used to determine multiple frames to be decoded from the video stream according to a preset frame interception strategy;

[0214] A construction unit is used to construct the video stream to be processed based on multiple frames to be decoded.

[0215] This application also provides a computer device, which is the computer device described above, and may include a terminal device or a server. The aforementioned video processing apparatus may be configured in this computer device. The computer device will now be described in conjunction with the accompanying drawings.

[0216] If the computer device is a terminal device, please refer to Figure 13 As shown, this application provides a terminal device, taking a mobile phone as an example:

[0217] Figure 13 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 13 The mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a Wi-Fi module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that... Figure 13 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0218] The following is combined Figure 13 A detailed introduction to each component of a mobile phone:

[0219] The RF circuit 1410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1480; in addition, it transmits uplink data to the base station.

[0220] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0221] The input unit 1430 can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432.

[0222] The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0223] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0224] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone.

[0225] WiFi is a short-range wireless transmission technology. Through the WiFi module 1470, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access.

[0226] The processor 1480 is the control center of the mobile phone. It connects to various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1420 and calling data stored in the memory 1420.

[0227] The mobile phone also includes a power supply 1490 (such as a battery) that powers the various components.

[0228] In this embodiment, the processor 1480 included in the terminal device is also used to execute a video processing method.

[0229] If the computer device is a server, this application embodiment also provides a server; please refer to [link to relevant documentation]. Figure 14 As shown, Figure 14 This is a structural diagram of a server 1500 provided in an embodiment of this application. The server 1500 can vary significantly due to different configurations or performance. It may include one or more processors, such as a Central Processing Unit (CPU), and a memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations stored in the storage media 1530 on the server 1500.

[0230] Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server.TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0231] The steps performed by the server in the above embodiments can be based on Figure 14 The server structure shown.

[0232] In addition, this application embodiment also provides a storage medium for storing a computer program for executing the method provided in the above embodiment.

[0233] This application also provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method provided in the above embodiments.

[0234] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0235] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0236] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0237] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Moreover, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video processing method, characterized in that, The method includes: Obtain the attribute information of each frame to be decoded in the video stream to be processed; the attribute information is used to indicate the frame type of each frame to be decoded, the frame type includes key frames and non-key frames, and the non-key frames include reference frames and non-reference frames. Based on the attribute information, frames to be decoded that are non-reference frames are removed from the video bitstream to be processed to obtain the reconstructed bitstream; Based on a preset number of frames, frame extraction processing is performed on the reconstructed bitstream to obtain multiple sampled frames; The multiple sampled frames are decoded to obtain the decoded video.

2. The method according to claim 1, characterized in that, The step of removing frames of non-reference frame type from the video bitstream to be processed based on the attribute information to obtain the reconstructed bitstream includes: Based on the attribute information, frames to be decoded that are non-reference frames are removed from the video bitstream to be processed to obtain the first intermediate bitstream; Based on the decoding dependency relationship between the frames to be decoded, the frame type of each frame to be decoded in the first intermediate bitstream is adjusted to obtain the second intermediate bitstream. When there are frames to be decoded in the second intermediate bitstream that are non-reference frames, the second intermediate bitstream is used as the video bitstream to be processed, and the step of removing frames to be decoded that are non-reference frames from the video bitstream to be processed based on the attribute information is returned to execute until the stopping condition is met, and the reconstructed bitstream is obtained.

3. The method according to claim 1, characterized in that, The frame extraction process based on a preset number of frames from the reconstructed bitstream to obtain multiple sampled frames includes: Determine the total number of keyframes in the reconstructed bitstream; When the total number of keyframes is greater than or equal to the preset number of frames, frame extraction processing is performed on the keyframes based on the preset number of frames to obtain multiple sampled frames; the total number of sampled frames is equal to the preset number of frames.

4. The method according to claim 3, characterized in that, The method further includes: When the total number of keyframes is less than the preset number of frames, frame extraction processing is performed on the keyframes to obtain the first sampled frames of the first sampled number. Based on the second sampling quantity, frame extraction processing is performed from the reference frame to obtain multiple second sampling frames; the second sampling quantity is the difference between the preset frame quantity and the first sampling quantity. Multiple first sampling frames and multiple second sampling frames are used as multiple sampling frames; the total number of sampling frames is greater than the preset number of frames.

5. The method according to claim 3, characterized in that, The step of extracting frames from the keyframes based on the preset number of frames to obtain multiple sampled frames includes: Based on the preset number of frames, frame extraction is performed uniformly from the key frames to obtain multiple sampled frames.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Feature extraction is performed on each decoded frame in the decoded video to obtain the image features of each decoded frame; Calculate the first similarity between the image features of two decoded frames; When the first similarity is greater than the first similarity threshold, a first redundant frame is determined from the two decoded frames; The first redundant frame is removed from the decoded video to obtain the first deduplicated video.

7. The method according to claim 6, characterized in that, The image features of the two decoded frames are the image features corresponding to each of the two adjacent decoded frames in the decoded video.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: Determine the grayscale distribution information of each decoded frame in the decoded video; Based on the grayscale distribution information, a second redundant frame is determined from the decoded video; in the grayscale distribution information of the second redundant frame, the number of pixels with grayscale values ​​within a preset grayscale range is greater than a preset number. The second redundant frame is removed from the decoded video to obtain the second deduplicated video.

9. The method according to any one of claims 1-5, characterized in that, The method further includes: The decoded video is input into the deduplication model to extract the embedding vector of each decoded frame in the decoded video; Calculate the second similarity between the embedding vectors of two decoded frames; When the second similarity is greater than the second similarity threshold, a third redundant frame is determined from the two decoded frames; The third redundant frame is removed from the decoded video, and the third deduplicated video is output.

10. The method according to any one of claims 1-5, characterized in that, Before obtaining the attribute information of each frame to be decoded in the video stream to be processed, the method further includes: Obtain the video bitstream of the video to be processed; According to a preset frame-cutting strategy, multiple frames to be decoded are determined from the video stream; The video stream to be processed is constructed based on multiple frames to be decoded.

11. A video processing apparatus, characterized in that, The device includes: An acquisition unit is used to acquire attribute information of each frame to be decoded in the video stream to be processed; the attribute information is used to indicate the frame type of each frame to be decoded, the frame type includes key frames and non-key frames, and the non-key frames include reference frames and non-reference frames. The elimination unit is used to eliminate frames to be decoded that are non-reference frames from the video bitstream to be processed based on the attribute information, so as to obtain the reconstructed bitstream; The extraction unit is used to perform frame extraction processing from the reconstructed bitstream based on a preset number of frames to obtain multiple sampled frames; The decoding unit is used to decode multiple sampled frames to obtain a decoded video.

12. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to perform the method according to any one of claims 1-10 according to the computer program.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a computer device, performs the method described in any one of claims 1-10.

14. A computer program product comprising a computer program, which, when run on a computer device, causes the computer device to perform the method of any one of claims 1-10.

Images