Video processing method and device, computer equipment and storage medium

By deciding the image frames to discard based on the frame drop rate during the video decoding process and updating the frame drop rate, the problem of video playback is solved, the smoothness and audio and video synchronization are improved, and it is suitable for high-speed playback scenarios.

CN118945424APending Publication Date: 2024-11-12TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310520363.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art is prone to lag when playing videos, especially when playing at high speeds. Traditional methods improve smoothness by reducing the number of video frames but have limited effects.

Method used

During the video decoding process, the frame drop strategy is determined based on the current frame drop rate, and whether to discard discardable image frames, and update the frame drop rate to synchronize the audio and video, ensuring that the video will automatically discard frames before decoding, saving time-consuming and Computing resources.

Benefits of technology

It improves the smoothness of video playback and the degree of audio and video synchronization, reduces the phenomenon of lag during high-speed playback, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945424A_ABST
    Figure CN118945424A_ABST
Patent Text Reader

Abstract

The invention relates to a video processing method and device, computer equipment and a storage medium. Relates to the field of network media, and comprises the following steps: in a process of decoding a video, when a discardable image frame in the video is decoded, determining a frame loss strategy for the discardable image frame according to a current frame loss rate; under the condition that the frame loss strategy is not discarded, inputting the discardable image frame into a decoder to obtain an image decoding frame which is output by the decoder and corresponds to the discardable image frame; determining a difference value between the display time of the image decoding frame and the time recorded by the audio and video synchronous clock to obtain a synchronous time difference corresponding to the image decoding frame; and under the condition that the synchronization time difference represents that the audio and the picture of the video are asynchronous in the decoding process, updating the current frame loss rate so as to determine a frame loss strategy of subsequent discardable image frames according to the updated current frame loss rate when the subsequent discardable image frames exist in the to-be-decoded video. By adopting the method, the video playing fluency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of audio and video technology, and in particular to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of computer technology and the Internet, the application of video is becoming more and more widespread. Video includes audio and images. When playing videos, if the player performance is poor, the video is prone to freeze during playback, especially when playing at a speed greater than 1x.

[0003] In traditional technology, in order to reduce the jamming during video playback, the smoothness of video playback can be improved by reducing the number of video frames involved in rendering.

[0004] However, the effect of improving the smoothness of video playback by reducing the number of video frames involved in rendering needs to be improved. Summary of the invention

[0005] Based on this, it is necessary to provide a video processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the smoothness of video playback in response to the above technical problems.

[0006] On the one hand, the present application provides a video processing method. The method includes: in the process of decoding a video, when a discardable image frame in the video is to be decoded, determining a frame loss strategy for the discardable image frame according to a current frame loss rate; the frame loss strategy is used to indicate whether to discard the discardable image frame before inputting the discardable image frame into a decoder; when the frame loss strategy is not to discard, input the discardable image frame into a decoder to obtain an image decoding frame corresponding to the discardable image frame output by the decoder; determine the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock to obtain the synchronization time difference corresponding to the image decoding frame; when the synchronization time difference indicates that the audio and video of the video are not synchronized during the decoding process, update the current frame loss rate, so that when subsequent discardable image frames in the video are to be decoded, the frame loss strategy of the subsequent discardable image frames is determined according to the updated current frame loss rate.

[0007] On the other hand, the present application also provides a video processing device. The device includes: a frame loss strategy determination module, which is used to determine the frame loss strategy for the discardable image frame according to the current frame loss rate when the discardable image frame in the video is to be decoded during the process of decoding the video; the frame loss strategy is used to indicate whether to discard the discardable image frame before the discardable image frame is input to the decoder; a decoded frame acquisition module, which is used to input the discardable image frame into the decoder when the frame loss strategy is not to discard, and obtain the image decoding frame corresponding to the discardable image frame output by the decoder; a synchronization time difference acquisition module, which is used to determine the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock, and obtain the synchronization time difference corresponding to the image decoding frame; a current frame loss rate update module, which is used to update the current frame loss rate when the synchronization time difference indicates that the audio and video of the video are not synchronized during the decoding process, so as to determine the frame loss strategy of the subsequent discardable image frame in the video to be decoded according to the updated current frame loss rate.

[0008] In some embodiments, the device is also used to: when decoding a non-discardable image frame in the video, input the non-discardable image frame into the decoder to obtain an image decoding frame output by the decoder corresponding to the non-discardable image frame; the current frame loss rate update module is also used to determine the audio and video synchronization status of the video during the decoding process according to the synchronization time difference corresponding to each of a plurality of image decoding frames continuously output by the decoder; if the audio and video synchronization status is that the audio and video are not synchronized, then update the current frame loss rate.

[0009] In some embodiments, the current frame loss rate update module is also used to arrange the synchronization time differences corresponding to a preset number of image decoding frames continuously output by the decoder in order from small to large to obtain a synchronization time difference sequence; determine a first reference serial number according to a first preset proportion and the preset number; determine the synchronization time difference sorted as the first reference serial number from the synchronization time difference sequence in order from front to back to obtain a first reference time difference; when the first reference time difference is less than or equal to a lagging time difference threshold, determine that the audio and picture synchronization condition of the video during the decoding process is that the picture lags behind the audio.

[0010] In some embodiments, the current frame loss rate update module is also used to determine a second reference serial number based on a second preset proportion and the preset number; determine the synchronization time difference sorted as the second reference serial number from the synchronization time difference sequence in order from back to front, and obtain a second reference time difference; when the second reference time difference is greater than or equal to the advance time difference threshold, determine that the audio and video synchronization condition of the video during the decoding process is that the picture is ahead of the audio.

[0011] In some embodiments, the current frame loss rate update module is further used to increase the current frame loss rate when the audio and video synchronization condition of the video is that the picture lags behind the audio; and to reduce the current frame loss rate when the audio and video synchronization condition of the video is that the picture is ahead of the audio.

[0012] In some embodiments, the device is also used to decode the audio frames in the video to obtain audio decoded frames, and input the audio decoded frames into the audio and video synchronization module; the device also includes a clock update module for updating the time recorded by the audio and video synchronization clock, and the clock update module is used to determine the difference between the system times when the decoded frames are input into the audio and video synchronization module twice, and obtain the system time difference; wherein the decoded frames input into the audio and video synchronization module twice are respectively audio decoded frames or image decoded frames; the time recorded by the audio and video synchronization clock is updated based on the system time difference.

[0013] In some embodiments, the clock updating module is further used to increase the system time difference based on the time recorded by the audio and video synchronization clock if the decoded frame input to the audio and video synchronization module twice is an image decoded frame.

[0014] In some embodiments, the clock update module is also used to determine the sum of the time recorded by the audio and video synchronization clock and the system time difference to obtain a candidate clock time if the decoded frame input later among the two decoded frames input to the audio and video synchronization module is an audio decoded frame; and update the time recorded by the audio and video synchronization clock based on the display time of the decoded frame input later and the candidate clock time.

[0015] In some embodiments, the clock update module is further used to update the time recorded by the audio and video synchronization clock to the display time of the decoded frame input later when the difference between the display time of the audio decoded frame and the candidate clock time is greater than or equal to a preset time difference threshold; and to update the time recorded by the audio and video synchronization clock to the candidate clock time when the difference between the display time of the audio decoded frame and the candidate clock time is less than the preset time difference threshold.

[0016] In some embodiments, the current frame loss rate update module is also used to respond to a playback speed switching event of switching the current playback speed from a first playback speed to a second playback speed; search for a frame loss rate that has a mapping relationship with the second playback speed from a relationship mapping table; the relationship mapping table is used to store the mapping relationship between the frame loss rate and the playback speed; if found, update the current frame loss rate to the found frame loss rate; if not found, compare the first playback speed with the second playback speed, and update the current frame loss rate based on the comparison result.

[0017] In some embodiments, the current frame loss rate update module is also used to increase the current frame loss rate based on the maximum frame loss rate if the comparison result is that the first playback speed is less than the second playback speed; if the comparison result is that the first playback speed is greater than the second playback speed, then reduce the current frame loss rate based on the minimum frame loss rate.

[0018] In some embodiments, the frame loss strategy determination module is further used to search for frame loss strategy characterization information from a preset frame loss mapping table based on the frame sequence number and the current frame loss rate of the discardable image frame; the elements in the frame loss mapping table are the frame loss strategy characterization information; and the frame loss strategy represented by the found frame loss strategy characterization information is determined as the frame loss strategy for the discardable image frame.

[0019] On the other hand, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned video processing method when executing the computer program.

[0020] On the other hand, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps in the above-mentioned video processing method when executed by a processor.

[0021] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned video processing method are implemented.

[0022] The above-mentioned video processing method, device, computer equipment, storage medium and computer program product, in the process of decoding a video, when a discardable image frame in the video is to be decoded, a frame loss strategy for the discardable image frame is determined according to the current frame loss rate, and the frame loss strategy is used to indicate whether to discard the discardable image frame before the discardable image frame is input into the decoder. When the frame loss strategy is not to discard, the discardable image frame is input into the decoder to obtain an image decoded frame corresponding to the discarded image frame output by the decoder, and the difference between the display time of the image decoded frame and the time recorded by the audio and video synchronization clock is determined to obtain the synchronization time difference corresponding to the image decoded frame. When the synchronization time difference indicates that the audio and video of the video are not synchronized during the decoding process, the current frame loss rate is updated, so that when there are subsequent discardable image frames in the video to be decoded, the frame loss strategy for subsequent discardable image frames is determined according to the updated current frame loss rate. Since whether to discard the discardable image frames is determined according to the current frame loss rate before decoding, the discardable image frames can be adaptively discarded before decoding. By saving the decoding time and computing resource overhead of the discardable image frames, a higher video playback frame rate is maintained under limited decoding performance, thereby improving the smoothness of video playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A diagram of an application environment of a video processing method in some embodiments;

[0024] Figure 2 is a schematic flow chart of a video processing method in some embodiments;

[0025] Figure 3 is a schematic flow chart of a video processing method in some embodiments;

[0026] Figure 4 A framework diagram for implementing feedback adaptive frame loss in some embodiments;

[0027] Figure 5 A flowchart for implementing feedback adaptive frame loss in some embodiments;

[0028] Figure 6 This is a video playback effect diagram in some embodiments;

[0029] Figure 7 This is a video playback effect diagram in some embodiments;

[0030] Figure 8 is a structural block diagram of a video processing device in some embodiments;

[0031] Fig. 9 is an internal structure diagram of a computer device in some embodiments;

[0032] Fig.101 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] The video processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other servers. The server 104 can be a server that provides video download services. The terminal 102 can be any device with a video playback function.

[0035] Specifically, in response to the operation of playing the video, the terminal 102 downloads the data packets of the video from the server 104, and caches the data packets of the video, wherein the data packets of the video include audio data packets and image data packets. The terminal 102 decodes the cached data packets to obtain audio decoding frames and image decoding frames, inputs the audio decoding frames and image decoding frames into the audio-visual synchronization module, and then renders the audio decoding frames and image decoding frames output by the audio-visual synchronization module before playing. In order to improve the smoothness of the video in the scenario of low decoding performance, high-speed or high-frame rate video playback, the image frames can be discarded and the proportion of discarded image frames can be adjusted. For example, in the process of decoding the video, when the discardable image frames in the video are to be decoded, the terminal 102 determines the frame loss strategy for the discardable image frames according to the current frame loss rate. The frame loss strategy is used to indicate whether to discard the discardable image frames before inputting the discardable image frames into the decoder. When the frame loss strategy is not to drop, the terminal 102 inputs the discarded image frame into the decoder to obtain the decoded image frame corresponding to the discarded image frame output by the decoder. The terminal 102 determines the difference between the display time of the decoded image frame and the time recorded by the audio and video synchronization clock to obtain the synchronization time difference corresponding to the decoded image frame. When the synchronization time difference indicates that the audio and video of the video are not synchronized during the decoding process, the current frame loss rate is updated to determine the subsequent discarded image frame frame The frame loss strategy is determined according to the updated current frame loss rate when there are subsequent discarded image frames in the video to be decoded.

[0036] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0037] In some embodiments, Figure 2 As shown, a video processing method is provided, which can be executed by a terminal or a server, or can be executed by a terminal and a server together. Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0038] Step 202, during the process of decoding the video, when a discardable image frame in the video is to be decoded, a frame loss strategy for the discardable image frame is determined according to the current frame loss rate; the frame loss strategy is used to indicate whether to discard the discardable image frame before inputting the discardable image frame into the decoder.

[0039] The video includes image frames and audio frames, and the image frames and audio frames included in the video are obtained through encoding. Image frames include non-discardable image frames and discardable image frames. The process of decoding a video may occur in the scene of playing a video. For example, in order to play a video, the video needs to be decoded before playing. Image frames can also be called video frames.

[0040] In the field of video encoding and decoding, since there is a pixel reference relationship between image frames, the loss of an image frame will cause other image frames that refer to the image frame to fail to be successfully decoded. This image frame that is not referenced by other image frames is called a discardable image frame. Even if the discardable image frame is discarded, it will not affect the decoding of other image frames. The image frame referenced by other image frames is called a non-discardable image frame. If the non-discardable image frame is discarded, it will affect the decoding of the image frame that references it for decoding. It should be noted that audio frames are all non-lost. Image frames are actually stored in the form of data packets, so image frames can also be called image data packets. Similarly, audio frames are actually stored in the form of data packets, so audio frames can also be called audio data packets.

[0041] The frame loss strategy includes discard and no discard. The current frame loss rate refers to the frame loss rate used at the current moment. The frame loss rate reflects the frequency of discarding discardable image frames, indicating how much proportion of discardable image frames need to be discarded. The larger the frame loss rate, the greater the frequency of discarding discardable image frames. The value range of the frame loss rate can be 0%-100%, and the value range of the frame loss rate can also be a set of discrete values. For example, the value range of the frame loss rate can be [0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%]. The terminal has an audio and video player, and the decoder refers to the module in the player used to decode audio and video.

[0042] Specifically, in the process of playing a video, the process of decoding the image frames and audio frames in the video is parallel. The terminal can use the decoder to decode the image frames and audio frames in the video in parallel. Therefore, the picture and audio in the decoded video may be out of sync. For example, the picture may be ahead of the audio, or the picture may lag behind the audio. The degree of asynchrony between audio and video can be reduced by discardable image frames. In the case where the picture is ahead of the audio, more discardable image frames can be discarded without decoding. In the case where the picture lags behind the audio, fewer discardable image frames can be discarded without decoding, or all discardable image frames participate in the decoding process. For non-discardable image frames, all participate in the decoding process.

[0043] In some embodiments, the terminal can obtain image frames to be decoded from the video. For example, the image frames to be decoded can be obtained from the video in sequence according to the decoding order or the display order, and it is determined whether the image frames to be decoded are discardable image frames or non-discardable image frames. In the case where the image frames to be decoded are non-discardable image frames, the image frames to be decoded are input into the decoder for decoding. In the case where the image frames to be decoded are discardable image frames, the discarding strategy of the image frames to be decoded is determined. If the discarding strategy is discarding, the image frames to be decoded are discarded. If the discarding strategy is not discarding, the image frames to be decoded are input into the decoder for decoding. Each time the terminal obtains an image frame to be decoded from the video, it can determine whether to decode the image frame to be decoded according to the method of this embodiment.

[0044] In some embodiments, the image frame, i.e., the image data packet, stores information for identifying whether the image frame is discardable. For example, the terminal can determine whether the image frame is discardable or not through Table 1. In Table 1, H.264 / AVC and H.265 / HEVC are video coding protocols, and AVC is the abbreviation of Advanced Video Coding. HEVC is the abbreviation of High Efficiency Video Coding, which refers to high-efficiency video coding. nal_ref_idc and nal_unit_type are data stored in the image data packet. nal_ref_idc represents the priority. When nal_ref_idc=0, the image frame is judged to be discardable. nal_unit_type specifies the type of NALU (Network Abstract Layer Unit). When the nal_unit_type type is RASL_N or RASL_R, the image frame is judged to be discardable.

[0045] Table 1 Judgment basis of discardable image frames in video coding and decoding protocols

[0046] Video codec protocol Judgment basis H.264 / AVC If the value of nal_ref_idc is 0, it can be discarded H.265 / HEVC If the nal_unit_type type is RASL_N or RASL_R, it can be discarded

[0047] In some embodiments, the terminal can search for frame loss strategy characterization information from a preset frame loss mapping table according to the frame sequence number of the discardable image frame and the current frame loss rate, and the elements in the frame loss mapping table are frame loss strategy characterization information; the frame loss strategy represented by the found frame loss strategy characterization information is determined as the frame loss strategy for the discardable image frame. The frame sequence number can be determined according to the decoding order, for example, the earlier the decoding order, the smaller the frame sequence number. The frame sequence number can also be determined according to the display order, for example, the earlier the display order, the smaller the frame sequence number.

[0048] Step 204: when the frame discarding strategy is not to discard, the discardable image frame is input into the decoder to obtain an image decoded frame output by the decoder and corresponding to the discardable image frame.

[0049] Specifically, when the frame loss strategy is not to drop, the terminal inputs the discardable image frame into the decoder for decoding to obtain an image decoding frame corresponding to the discardable image frame. When the frame loss strategy is to drop, the discardable image frame is discarded, that is, the discardable image frame is not decoded.

[0050] Step 206, determining the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock, and obtaining the synchronization time difference of the image decoding frame.

[0051] Among them, it should be noted that the time recorded by the audio and video synchronization clock can be updated in real time. Each time the synchronization time difference is calculated, it is updated according to the latest time recorded by the audio and video synchronization clock. The initial value of the time recorded by the audio and video synchronization clock is 0. The display time can also be called the display timestamp (pts,: present timestamp), the unit is us (microseconds), and the audio frame and the image frame each have a display time. The display time of the image decoding frame refers to the display time of the image frame that obtains the image decoding frame. The display time of the audio decoding frame refers to the display time of the audio frame that obtains the audio decoding frame. The display time is used to clarify the moment of rendering the image decoding frame or the audio decoding frame, and is an important synchronization basis for the alignment of audio and video rendering time.

[0052] Specifically, the terminal can calculate the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock, and determine the difference as the synchronization time difference corresponding to the image decoding frame. For example, the synchronization time difference can be calculated using formula (1): ΔT = PTS v -T clock (1), the unit is microseconds. PTS in formula (1) v Represents the display time of the image decoding frame, T clock Represents the time recorded by the audio-visual synchronization clock. If ΔT>0, it means that the display time of the image decoding frame is ahead of the time recorded by the audio-visual synchronization clock. If ΔT<0, it means that the display time of the image decoding frame lags behind the time recorded by the audio-visual synchronization clock.

[0053] In some embodiments, the player of the terminal also includes an audio-visual synchronization module, which can also be called an audio-visual synchronizer. The audio-visual synchronization module is a module in the player that implements audio-visual synchronization. The time recorded by the audio-visual synchronization clock can be stored and updated in the audio-visual synchronization module, that is, the updating of the audio-visual synchronization clock can be implemented by the audio-visual synchronization module. The audio-visual synchronization module can also have the function of calculating the synchronization time difference, that is, the process of calculating the synchronization time difference of the image decoder can be performed by the audio-visual synchronization module. The terminal can input the image decoded frame output by the decoder into the audio-visual synchronization module.

[0054] In some embodiments, in response to inputting the image decoding frame into the audio and video synchronization module, the terminal records the system time when the image decoding frame is input, and updates the time recorded by the audio and video synchronization clock based on the system time. After the update, the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock is determined to obtain the synchronization time difference corresponding to the image decoding frame.

[0055] In some embodiments, the terminal may also decode the audio frame in the video to obtain an audio decoded frame, and input the audio decoded frame into the audio-visual synchronization module. Inputting the audio decoded frame and the image decoded frame into the audio-visual synchronization module is performed in parallel. In response to inputting the audio decoded frame into the audio-visual synchronization module, the terminal records the system time when the audio decoded frame is input, and updates the time recorded by the audio-visual synchronization clock based on the system time.

[0056] Step 208, when the audio and video of the video are not synchronized during the decoding process characterized by the synchronous time difference, the current frame loss rate is updated to determine the subsequent discardable image frames in the video to be decoded according to the updated current frame loss rate.

[0057] Specifically, during the entire process of playing the video, the terminal can perform real-time detection of the synchronization of the audio and video of the video. The terminal calculates the difference between the display time of the latest decoded image decoding frame and the display time of the latest decoded audio decoding frame. When the absolute value of the difference is greater than a specified value, it is determined that the audio and video are not synchronized, and when the difference is less than 0, it is determined that the picture lags behind the audio, and when the difference is greater than 0, it is determined that the picture is ahead of the audio. The specified value can be set based on experience, for example, 5000 microseconds. It can be understood that the process of updating the current frame loss rate is a parallel process to the decoding process, so that during the decoding process, the current frame loss rate based on which the decoding is based can be updated in real time.

[0058] In some embodiments, the terminal may also determine the audio and video synchronization of the video during the decoding process according to the synchronization time difference corresponding to each of the multiple image decoding frames continuously output by the decoder. The audio and video are not synchronized, including the picture ahead of the audio and the picture behind the audio. When the audio and video synchronization of the video is that the picture lags behind the audio, the terminal increases the current frame loss rate. When the audio and video synchronization of the video is that the picture is ahead of the audio, the terminal reduces the current frame loss rate.

[0059] In the above-mentioned video processing method, in the process of decoding the video, when the discardable image frames in the video are to be decoded, the frame loss strategy for the discardable image frames is determined according to the current frame loss rate, and the frame loss strategy is used to indicate whether to discard the discardable image frames before inputting the discardable image frames into the decoder. When the frame loss strategy is not to discard, the discardable image frames are input into the decoder to obtain the image decoded frames corresponding to the discarded image frames output by the decoder, and the difference between the display time of the image decoded frame and the time recorded by the audio and video synchronization clock is determined to obtain the synchronization time difference corresponding to the image decoded frame. When the synchronization time difference indicates that the audio and video of the video are not synchronized during the decoding process, the current frame loss rate is updated, so that when there are subsequent discardable image frames in the video to be decoded, the frame loss strategy for subsequent discardable image frames is determined according to the updated current frame loss rate. Because before decoding, it is determined whether to discard the discardable image frames based on the current frame loss rate, thus achieving adaptive discarding of the discardable image frames before decoding, by saving the decoding time and computing resource overhead of the discardable image frames, a higher video playback frame rate is maintained under limited decoding performance, thereby improving the smoothness of video playback and the degree of video audio and picture synchronization.

[0060] The video processing method provided in the present application can be applied to any video playback scenario, especially for the scenario of high-speed / high frame rate video playback. The discardable frames can be adaptively discarded before decoding. By saving the decoding time / computing resource overhead of the discardable frames, a larger video playback frame rate output can be maintained under the limitation of decoding performance, thereby improving the video smoothness and experience.

[0061] Speed ​​playback is one of the functions of video playback devices. The speed playback function can quickly play videos in a short time. However, speed playback of videos has high requirements on the decoding performance of terminal playback devices. For example, a 25fps video source only requires the decoder to decode 25 frames per second at one speed, while at three times the speed, the decoder is required to decode 75 frames per second. FPS (frame per second, the unit of frame rate) indicates how many video frames are decoded / rendered per second. Video frames are image frames. Due to the limited performance of the decoder and the limitations of the hardware and software performance of the video playback device, video freezes easily occur in the case of speed playback. Therefore, how to maintain the highest frame rate output under limited decoding computing resources has important practical significance. The video processing method provided by the present application reduces the video freezes under speed playback by saving the decoding time / computing resource overhead of discardable frames, and improves the smoothness of the video under speed playback.

[0062] In some embodiments, the method further includes: when decoding a non-discardable image frame in a video, inputting the non-discardable image frame into a decoder to obtain an image decoded frame output by the decoder corresponding to the non-discardable image frame; when the synchronization time difference indicates that the audio and picture of the video are not synchronized during the decoding process, updating the current frame loss rate includes: determining the audio and picture synchronization status of the video during the decoding process according to the synchronization time difference corresponding to each of multiple image decoded frames continuously output by the decoder; if the audio and picture synchronization status is that the audio and picture are not synchronized, updating the current frame loss rate.

[0063] The multiple image decoding frames may be at least two image decoding frames, for example, 15 image decoding frames. The audio and video synchronization situation includes two situations: audio and video are not synchronized and audio and video are synchronized. The audio and video are not synchronized, including two situations: the picture lags behind the audio and the picture is ahead of the audio.

[0064] Specifically, the terminal may arrange the synchronization time differences corresponding to the multiple image decoding frames continuously output by the decoder in ascending order to obtain a synchronization time difference sequence, determine the synchronization time difference with a specified sequence number from the synchronization time difference sequence, use the synchronization time difference with the specified sequence number as a reference synchronization time difference, and determine the audio and video synchronization situation according to the reference synchronization time difference. For example, when the reference synchronization time difference is less than or equal to the lagging time difference threshold, the audio and video synchronization situation is determined to be that the picture lags behind the audio. When the reference synchronization time difference is greater than or equal to the leading time difference threshold, the audio and video synchronization situation is determined to be that the picture is ahead of the audio. Among them, the designated serial number can be determined according to the number of synchronization time differences in the synchronization time difference sequence. For example, the terminal can multiply the preset coefficient by the number of synchronization time differences in the synchronization time difference sequence. The value range of the preset coefficient is 0-1, for example, it can be 0.6 or 0.5, etc. When the value obtained by multiplication is an integer, the value is used as the designated serial number. For example, the number of synchronization time differences in the synchronization time difference sequence is 15, and the preset coefficient is 0.6, then the designated number needs to be 9. When the value obtained by multiplication is a decimal, the value is rounded down to obtain the designated serial number. For example, the number of synchronization time differences in the synchronization time difference sequence is 15, and the preset coefficient is 0.5, then the designated serial number is Indicates rounding down.

[0065] In some embodiments, the terminal may determine the proportion of synchronization time differences greater than or equal to the leading time difference threshold in the synchronization time difference sequence to obtain a first proportion. When the first proportion exceeds the first proportion threshold, it is determined that the audio-visual synchronization situation is that the picture is ahead of the audio. The terminal may determine the second proportion of synchronization time differences less than or equal to the lagging time difference threshold in the synchronization time difference sequence. When the second proportion exceeds the second proportion threshold, it is determined that the audio-visual synchronization situation is that the picture is ahead of the audio. The leading time difference threshold and the lagging time difference threshold can be set according to experience. The leading time difference threshold is a value greater than 0, and the lagging time difference threshold is a value less than 0. For example, the leading time difference threshold is 5000us, and the lagging time difference threshold is -80000us, where us refers to microseconds. The value ranges of the first proportion threshold and the second proportion threshold are both 0%-100%, and the sum of the first proportion threshold and the second proportion threshold is 100%. For example, the first proportion threshold is 60% and the second proportion threshold is 40%.

[0066] In some embodiments, when the terminal determines that the audio and video synchronization of the video is that the picture lags behind the audio, the terminal increases the current frame loss rate; when the terminal determines that the audio and video synchronization of the video is that the picture is ahead of the audio, the terminal reduces the current frame loss rate.

[0067] In this embodiment, the audio and video synchronization of the video during the decoding process is determined based on the synchronization time difference corresponding to each of the multiple image decoding frames continuously output by the decoder, so that the audio and video synchronization can be judged more accurately through statistical means, further improving the accuracy of updating the current frame loss rate.

[0068] In some embodiments, determining the audio and video synchronization status of the video during the decoding process based on the synchronization time differences corresponding to each of a plurality of image decoding frames continuously output by the decoder includes: arranging the synchronization time differences of a preset number of image decoding frames continuously output by the decoder in ascending order to obtain a synchronization time difference sequence; determining a first reference sequence number based on a first preset proportion and a preset number; determining the synchronization time difference sorted as the first reference sequence number from the synchronization time difference sequence in a front-to-back order to obtain a first reference time difference; and determining, when the first reference time difference is less than or equal to a lagging time difference threshold, that the audio and video synchronization status of the video during the decoding process is that the picture lags behind the audio.

[0069] The preset number is at least two, for example, 15. The value range of the first preset proportion is 0%-100%, for example, 40%.

[0070] Specifically, the terminal can perform a multiplication operation on the first preset proportion and the preset number, and determine the first reference serial number according to the value obtained by the multiplication operation. When the value obtained by the multiplication operation is an integer, the terminal determines the value obtained by the multiplication operation as the first reference serial number. When the value obtained by the multiplication operation is a decimal, the terminal rounds down the value obtained by the multiplication operation, and determines the value obtained after rounding down as the first reference serial number. The terminal determines the synchronization time difference ranked as the first reference serial number from the synchronization time difference sequence in order from front to back, and obtains the first reference time difference. The terminal compares the first reference time difference with the lagging time difference threshold. If the first reference time difference is less than or equal to the lagging time difference threshold, it is determined that the audio and video synchronization situation is that the picture lags behind the audio.

[0071] For example, taking the preset number as 15 and the first preset proportion as 40%, the first reference serial number is 40%×15=6. The terminal can obtain the synchronization time difference corresponding to each of the 15 image decoding frames continuously output by the decoder, and arrange these 15 synchronization time differences in order from small to large to obtain a synchronization time difference sequence. The terminal determines the 6th synchronization time difference from the synchronization time difference sequence in order from front to back to obtain the first reference time difference. The terminal compares the first reference time difference with the lagging time difference threshold. If the first reference time difference is less than or equal to the lagging time difference threshold, it is determined that the audio and video synchronization situation is that the picture lags behind the audio.

[0072] In this embodiment, a first reference serial number is determined based on a first preset proportion and a preset number, and a synchronization time difference sorted as the first reference serial number is determined from the synchronization time difference sequence in order from front to back to obtain a first reference time difference. Then, the proportion of the synchronization time difference from the first synchronization time difference to the first reference time difference in the synchronization time difference sequence is close to the first preset proportion. Therefore, when the first reference time difference is less than or equal to the lagging time difference threshold, the audio and video synchronization condition of the video during the decoding process is determined to be that the picture lags behind the audio. That is, when the synchronization time difference close to the first preset proportion is less than or equal to the lagging time difference threshold, the audio and video synchronization condition of the video during the decoding process is determined to be that the picture lags behind the audio, thereby improving the accuracy of judging the audio and video synchronization condition.

[0073] In some embodiments, the method also includes: determining a second reference serial number based on a second preset proportion and a preset number; determining a synchronization time difference sorted as the second reference serial number from a synchronization time difference sequence in order from back to front to obtain a second reference time difference; when the second reference time difference is greater than or equal to an advance time difference threshold, determining that the audio and video synchronization condition of the video during the decoding process is that the picture is ahead of the audio.

[0074] The preset number is at least two, for example, 15. The second preset ratio ranges from 0% to 100%, for example, 60%. The sum of the first preset ratio and the second preset ratio can be 100%. The first preset ratio can be equal to, less than, or greater than the second preset ratio.

[0075] Specifically, the terminal can perform a multiplication operation on the second preset proportion and the preset number, and determine the second reference serial number according to the value obtained by the multiplication operation. When the value obtained by the multiplication operation is an integer, the terminal determines the value obtained by the multiplication operation as the second reference serial number. When the value obtained by the multiplication operation is a decimal, the terminal rounds down the value obtained by the multiplication operation, and determines the value obtained after rounding down as the second reference serial number. The terminal determines the synchronization time difference sorted as the second reference serial number from the synchronization time difference sequence in order from back to front, and obtains the second reference time difference. The terminal compares the second reference time difference with the leading time difference threshold. If the second reference time difference is greater than or equal to the leading time difference threshold, it is determined that the audio and video synchronization situation is that the picture is ahead of the audio.

[0076] For example, taking the preset number as 15 and the second preset proportion as 60%, the second reference serial number is 60%×15=9. The terminal can obtain the synchronization time difference corresponding to each of the 15 image decoding frames continuously output by the decoder, and arrange these 15 synchronization time differences in order from small to large to obtain a synchronization time difference sequence. The terminal determines the 9th synchronization time difference from the synchronization time difference sequence in order from back to front to obtain the second reference time difference. The terminal compares the second reference time difference with the advance time difference threshold. If the second reference time difference is greater than or equal to the advance time difference threshold, it is determined that the audio and video synchronization situation is that the picture is ahead of the audio.

[0077] In some embodiments, during the video decoding process, the terminal repeatedly executes the steps of: determining the audio and video synchronization of the video during the decoding process according to the synchronization time difference of a preset number of image decoding frames continuously output by the decoder. The multiple image decoding frames continuously output by the decoder can be understood as a group of image decoding frames, and a different group of image decoding frames is used each time. For example, if the preset number is N, during the entire decoding of the video or the entire playback process of the video, the decoder sequentially outputs M image decoding frames, and the sequence S = {ΔT1, ΔT2, ..., ΔT M} is a sequence of synchronization time differences of these M image decoding frames, where ΔT iis the synchronization time difference of the i-th image decoding frame output by the decoder, 1≤i≤M, then the first time the audio and video synchronization of the video during the decoding process is determined according to the synchronization time difference of a preset number of image decoding frames continuously output by the decoder, the synchronization time difference sequence used is the subsequence consisting of the 1st synchronization time difference to the Nth synchronization time difference in sequence S. The second time the audio and video synchronization of the video during the decoding process is determined according to the synchronization time difference of a preset number of image decoding frames continuously output by the decoder, the synchronization time difference sequence used is the subsequence consisting of the N+1th synchronization time difference to the 2Nth synchronization time difference in sequence S, and so on. Therefore, for sequence S, during the entire decoding process of the video or the entire playback process of the video, the terminal can use a total of Different synchronization time difference sequences are used to determine the audio and video synchronization situation, and a different synchronization time difference sequence is used each time. Each time in the process of determining the step of audio and video synchronization of the video during the decoding process according to the synchronization time difference of a preset number of image decoding frames continuously output by the decoder, a first reference time difference and a second reference time difference are determined. Each first reference time difference can be stored in a sequence, and each second reference time difference can also be stored in a sequence. For example, the sequence composed of each first reference time difference is The sequence composed of the second reference time differences is Where, ΔT upj represents the first reference time difference generated by the step of determining the audio and video synchronization of the video during the decoding process according to the synchronization time difference of the preset number of image decoding frames continuously output by the decoder for the jth time, ΔT downj represents the second reference time difference generated by the step of determining the audio and video synchronization of the video during the decoding process according to the synchronization time difference of a preset number of image decoding frames continuously output by the decoder for the jth time, Among them, the first reference time difference can also be understood as the lower quantile corresponding to the first preset proportion in the synchronized time difference sequence, and the second reference time difference can also be understood as the upper quantile corresponding to the second preset proportion in the synchronized time difference sequence. For example, if the first preset proportion is 40%, then the first reference time difference can be understood as the 40% lower quantile in the synchronized time difference sequence, and if the second preset proportion is 60%, then the second reference time difference can be understood as the 60% upper quantile in the synchronized time difference sequence.

[0078] In this embodiment, a second reference serial number is determined based on a second preset proportion and a preset number, and a synchronization time difference sorted as the second reference serial number is determined from the synchronization time difference sequence in a back-to-front order to obtain a second reference time difference. Then, the proportion of the synchronization time difference from the tail synchronization time difference to the second reference time difference in the synchronization time difference sequence is close to the second preset proportion in the synchronization time difference sequence. Therefore, when the second reference time difference is greater than or equal to the advance time difference threshold, it is determined that the audio and video synchronization situation of the video during the decoding process is that the picture is ahead of the audio. That is, when the synchronization time difference close to the second preset proportion is greater than or equal to the advance time difference threshold, it is determined that the audio and video synchronization situation of the video during the decoding process is that the picture is ahead of the audio, thereby improving the accuracy of judging the audio and video synchronization situation.

[0079] In some embodiments, if the audio and video synchronization situation is that the audio and video are not synchronized, updating the current frame loss rate includes: when the audio and video synchronization situation of the video is that the picture lags behind the audio, increasing the current frame loss rate; when the audio and video synchronization situation of the video is that the picture is ahead of the audio, reducing the current frame loss rate.

[0080] Specifically, when the audio and video synchronization situation of the video is that the picture lags behind the audio, the terminal compares the current frame loss rate with the maximum frame loss rate, the maximum frame loss rate is 100% or a value close to 100%, if the current frame loss rate is equal to the maximum frame loss rate, the terminal keeps the current frame loss rate unchanged, that is, does not update the current frame loss rate. If the current frame loss rate is less than the maximum frame loss rate, the terminal increases the current frame loss rate. When the audio and video synchronization situation of the video is that the picture is ahead of the audio, the terminal compares the current frame loss rate with the minimum frame loss rate, the minimum frame loss rate is 0% or a value close to 0%, if the current frame loss rate is equal to the minimum frame loss rate, the terminal keeps the current frame loss rate unchanged, that is, does not update the current frame loss rate. If the current frame loss rate is greater than the minimum frame loss rate, the terminal reduces the current frame loss rate.

[0081] In some embodiments, the terminal may determine the frame loss rate change amount, and update the current ground frame loss rate according to the frame loss rate change amount. Specifically, when the terminal determines that the current frame loss rate is to be increased, the frame loss rate change amount may be increased on the basis of the current ground frame loss rate. When the terminal determines that the current frame loss rate is to be reduced, if the current frame loss rate is greater than or equal to the frame loss rate change amount, the terminal may subtract the frame loss rate change amount from the current ground frame loss rate. For example, the terminal may reduce the current frame loss rate according to formula (2) and increase the current frame loss rate according to formula (3).

[0082]

[0083]

[0084] In formula (2) and formula (3), 10% refers to the change in frame loss rate. oldrefers to the current frame loss rate before the update. new Refers to the current frame loss rate after update. 0% refers to the minimum frame loss rate, and 100% refers to the maximum frame loss rate.

[0085] In some embodiments, when the audio and video synchronization situation of the video is that the picture lags behind the audio, the terminal compares the current ground frame loss rate with the maximum frame loss rate. If the current ground frame loss rate is equal to the maximum frame loss rate, the terminal keeps the current ground frame loss rate unchanged, that is, does not update the current ground frame loss rate. If the current ground frame loss rate is less than the maximum frame loss rate, the terminal can increase the current frame loss rate according to the maximum frame loss rate. Specifically, the terminal can calculate the average of the maximum frame loss rate and the current frame loss rate, and update the current frame loss rate to the calculated average. Or the terminal can amplify the current frame loss rate according to a preset amplification factor, so as to achieve the purpose of increasing the current frame loss rate. If the result of the amplification process is less than or equal to the maximum frame loss rate, the current frame loss rate is updated to the result of the amplification process. If the result of the amplification process is greater than the maximum frame loss rate, the current frame loss rate is updated to the maximum frame loss rate. The preset amplification factor has a value range of 0-1, for example, it can be 0.5 or 0.7.

[0086] In some embodiments, when the audio and video synchronization situation of the video is that the picture is ahead of the audio, the terminal compares the current ground frame loss rate with the minimum frame loss rate. If the current ground frame loss rate is equal to the minimum frame loss rate, the terminal keeps the current ground frame loss rate unchanged, that is, does not update the current ground frame loss rate. If the current ground frame loss rate is greater than the minimum frame loss rate, the terminal can reduce the current frame loss rate according to the preset reduction coefficient, thereby achieving the purpose of reducing the current frame loss rate. The preset reduction coefficient is a value between 0 and 1, for example, it can be 0.5 or 0.6. Alternatively, the terminal can calculate the minimum frame loss rate or the average of the current frame loss rate, and update the current frame loss rate to the calculated average.

[0087] In some embodiments, a relationship mapping table may be stored in the player of the terminal, and the relationship mapping table is used to store the mapping relationship between the frame loss rate and the playback speed. The playback speed refers to the speed of playing the video, and the playback speed includes but is not limited to at least one of 0.5 times, 1 times, 1.5 times or 2 times. After the terminal updates the current frame loss rate, if it is detected that the current frame loss rate has not changed within the preset time length, the terminal can determine the current playback speed. If there is no mapping relationship of the current playback speed in the relationship mapping table, the mapping relationship between the current playback speed and the current frame loss rate is added to the relationship mapping table. If there is already a mapping relationship of the current playback speed in the relationship mapping table, and the frame loss rate with a mapping relationship with the current playback speed is not the current frame loss rate, the mapping relationship of the current playback speed in the relationship mapping table is updated to the mapping relationship between the current playback speed and the current frame loss rate. Therefore, when the playback speed is switched, the terminal can update the current frame loss rate according to the relationship mapping table.

[0088] In this embodiment, when the audio-visual synchronization of the video is that the picture lags behind the audio, it means that the picture in the video lags behind the audio in time, thereby increasing the current frame loss rate, which can cause more discardable image frames to be discarded, thereby saving the decoding time of the image frames, so that the picture catches up with the audio in time, so that the picture and the audio are quickly synchronized in time, so as to improve the smoothness during the playback process. When the audio-visual synchronization of the video is that the picture is ahead of the audio, it means that the picture in the video is ahead of the audio in time, thereby reducing the current frame loss rate, and can reduce the number of discardable image frames that are discarded, thereby decoding more image frames, so that while ensuring the degree of flow during the playback process, the played picture effect can also be improved.

[0089] In some embodiments, the method also includes: decoding the audio frames in the video to obtain audio decoded frames, and inputting the audio decoded frames into the audio and video synchronization module; the step of updating the time recorded by the audio and video synchronization clock includes: determining the difference between the system times when the decoded frames are input into the audio and video synchronization module twice, and obtaining the system time difference; wherein the decoded frames input into the audio and video synchronization module twice are respectively audio decoded frames or image decoded frames; and updating the time recorded by the audio and video synchronization clock based on the system time difference.

[0090] The decoded frame may be a video decoded frame or an audio decoded frame.

[0091] Specifically, in the process of playing the video, the terminal can sequentially obtain each audio frame to be decoded from the video, and for each audio frame to be decoded, input the audio frame to be decoded into the decoder for decoding to obtain the corresponding audio decoded frame, thereby decoding to obtain the audio decoded frames corresponding to each audio frame in the video. The terminal can input the audio decoded frame output by the decoder into the audio and video synchronization module, and record the system time when the audio decoded frame is input into the audio and video synchronization module. Similarly, the terminal can input the image decoded frame output by the decoder into the audio and video synchronization module, and record the system time when the image decoded frame is input into the audio and video synchronization module. The audio decoded frame and the image decoded frame are input into the audio and video synchronization module in parallel. If in the entire process of playing the video, the number of audio decoded frames input into the audio and video synchronization module is N, and the number of image decoded frames input into the audio and video synchronization module is M, then if the system time is arranged in ascending order, the system time of each audio decoded frame and each image decoded frame when they are input into the audio and video synchronization module can be expressed as a sequence as {T sys1 ,T sys2 ,…,T sysM+N}, T sysk It represents the kth decoded frame input into the audio and video synchronization module. The kth decoded frame can be an audio decoded frame or an image decoded frame. 1≤k≤M+N.

[0092] In some embodiments, the difference between the system times when the decoded frames are input to the audio-visual synchronization module twice before and after is determined to obtain the system time difference. The initial value of the time recorded by the audio-visual synchronization clock is 0. Whenever a decoded frame is input to the audio-visual synchronization module, the terminal records the system time when the decoded frame is input to the audio-visual synchronization module this time (recorded as the first system time). If this is the first time that the decoded frame is input to the audio-visual synchronization module, the time recorded by the audio-visual synchronization clock is updated to the first system time. If this is not the first time that the decoded frame is input to the audio-visual synchronization module, the terminal determines the system time when the decoded frame was input to the audio-visual synchronization module last time (recorded as the second system time). Each decoded frame input can be one of an audio decoded frame or an image decoded frame. The difference between the second system time and the first system time is calculated to obtain the current system time difference, and the current system time difference is added to the time recorded by the audio-visual synchronization clock, thereby realizing the update of the time recorded by the audio-visual synchronization clock.

[0093] In this embodiment, the difference between the system times when the decoded frames are input into the audio and video synchronization module twice is determined to obtain the system time difference, and the time recorded by the audio and video synchronization clock is updated based on the system time difference. The granularity of the time for updating the audio and video synchronization clock record is refined, so that the accuracy of audio and video synchronization achieved through the time recorded by the audio and video synchronization clock is higher.

[0094] In some embodiments, updating the time recorded by the audio and video synchronization clock based on the system time difference includes: if the decoded frame input to the audio and video synchronization module twice is an image decoded frame, then adding the system time difference to the time recorded by the audio and video synchronization clock.

[0095] Among them, the decoded frames input into the audio-visual synchronization module twice refer to the decoded frames input into the audio-visual synchronization module twice in succession, for example, the decoded frames input into the audio-visual synchronization module twice are the first decoded frame and the second decoded frame respectively, and the first decoded frame is input into the audio-visual synchronization module before the second decoded frame, for example, the first decoded frame is the decoded frame input into the audio-visual synchronization module for the first time, and the second decoded frame is the decoded frame input into the audio-visual synchronization module for the second time. The first decoded frame can be an audio decoded frame or an image decoded frame, and the second decoded frame can be an audio decoded frame or an image decoded frame. In response to the second decoded frame being input into the audio-visual synchronization module, the terminal calculates the difference between the system time when the second decoded frame is input and the system time when the first decoded frame is input, and obtains the system time difference corresponding to the second decoded frame. When the second decoded frame is an image decoded frame, the system time is added to the time recorded by the audio-visual synchronization clock to update the time recorded by the audio-visual synchronization clock.

[0096] Specifically, whenever a new decoded frame is input into the audio and video synchronization module, the system time difference is calculated once, and the time recorded by the audio and video synchronization clock is updated. The terminal responds to the decoded frame input into the audio and video synchronization module. If this is the first time that a decoded frame is input into the audio and video synchronization module, and the decoded frame is an image decoded frame, the terminal updates the time recorded by the audio and video synchronization clock to the system time when the image decoded frame is input. If this is not the first time that a decoded frame is input into the audio and video synchronization module, and the decoded frame is an image decoded frame, the terminal calculates the system time difference this time, and adds the system time difference this time to the time recorded by the audio and video synchronization clock, thereby realizing the update of the time recorded by the audio and video synchronization clock. For example, the terminal can update the time recorded by the audio and video synchronization clock through formula (4).

[0097]

[0098] In formula (4), T clock The time recorded by the audio and video synchronization clock, is the system time difference for this time, The last system time difference.

[0099] In this embodiment, the time recorded by the audio and video synchronization clock is updated according to the system time difference calculated when the image decoding frame is input into the audio and video synchronization module, thereby improving the granularity of the time recorded by the audio and video synchronization clock, so that the accuracy of audio and video synchronization achieved by the time recorded by the audio and video synchronization clock is higher.

[0100] In some embodiments, updating the time recorded by the audio and video synchronization clock based on the system time difference includes: if the decoded frame input later into the audio and video synchronization module twice is an audio decoded frame, determining the sum of the time recorded by the audio and video synchronization clock and the system time difference to obtain a candidate clock time; updating the time recorded by the audio and video synchronization clock based on the display time of the decoded frame input later and the candidate clock time.

[0101] Specifically, the decoded frames input to the audio-visual synchronization module twice refer to the decoded frames input to the audio-visual synchronization module twice in succession. For example, the decoded frames input to the audio-visual synchronization module twice are the first decoded frame and the second decoded frame, respectively, and the first decoded frame is input to the audio-visual synchronization module before the second decoded frame. For example, the first decoded frame is the decoded frame input to the audio-visual synchronization module for the first time, and the second decoded frame is the decoded frame input to the audio-visual synchronization module for the second time. The first decoded frame can be an audio decoded frame or an image decoded frame, and the second decoded frame can be an audio decoded frame or an image decoded frame. In response to the second decoded frame being input to the audio-visual synchronization module, the terminal calculates the difference between the system time when the second decoded frame is input and the system time when the first decoded frame is input, and obtains the system time difference corresponding to the second decoded frame. The terminal determines the sum of the time recorded by the audio-visual synchronization clock and the system time difference to obtain the candidate clock time, and updates the time recorded by the audio-visual synchronization clock according to the display time of the second decoded frame and the candidate clock time.

[0102] In some embodiments, the terminal responds to the decoded frame input into the audio-visual synchronization module. If this is the first time that the decoded frame is input into the audio-visual synchronization module, and the decoded frame is an audio decoded frame, the terminal calculates the difference between the system time when the audio decoded frame is input and the display time of the audio decoded frame. If the absolute value of the difference is less than the preset time difference threshold, the time recorded by the audio-visual synchronization clock is updated to the system time when the audio decoded frame is input. If the absolute value of the difference is greater than or equal to the preset time difference threshold, the time recorded by the audio-visual synchronization clock is updated to the display time of the audio decoded frame. The preset time difference threshold can be set based on experience, for example, it can be 3 milliseconds or 5 milliseconds. Thereby, the time recorded by the audio-visual synchronization clock is basically consistent with the time of the audio, so as to better judge the audio-visual synchronization situation. If this is not the first time that a decoded frame is input to the audio and video synchronization module, and the decoded frame is an audio decoded frame, the terminal calculates the system time difference this time, and calculates the sum of the time recorded by the audio and video synchronization clock and the system time difference to obtain the candidate clock time, and calculates the difference between the candidate clock time and the display time of the audio decoded frame input this time. If the absolute value of the difference is greater than or equal to the preset time difference threshold, the terminal updates the time recorded by the audio and video synchronization clock to the display time of the audio decoded frame. If the absolute value of the difference is less than the preset time difference threshold, the time recorded by the audio and video synchronization clock is updated to the candidate clock time, thereby realizing the update of the time recorded by the audio and video synchronization clock. In this embodiment, the time recorded by the audio and video synchronization clock can be basically consistent with the time of the audio, so as to better judge the audio and video synchronization situation.

[0103] In this embodiment, when an audio decoded frame is input into the audio and video synchronization module, the sum of the time recorded by the audio and video synchronization clock and the system time difference is determined to obtain the candidate clock time, and the time recorded by the audio and video synchronization clock is updated based on the display time of the decoded frame input later and the candidate clock time, thereby improving the updating accuracy of the time recorded by the audio and video synchronization clock.

[0104] In some embodiments, based on the display time of the decoded frame input later and the candidate clock time, updating the time recorded by the audio and video synchronization clock includes: when the difference between the display time of the audio decoded frame and the candidate clock time is greater than or equal to a preset time difference threshold, updating the time recorded by the audio and video synchronization clock to the display time of the decoded frame input later; when the difference between the display time of the audio decoded frame and the candidate clock time is less than the preset time difference threshold, updating the time recorded by the audio and video synchronization clock to the candidate clock time.

[0105] Specifically, the decoded frames input to the audio-visual synchronization module twice refer to the decoded frames input to the audio-visual synchronization module twice in succession, for example, the decoded frames input to the audio-visual synchronization module twice are the first decoded frame and the second decoded frame, and the first decoded frame is input to the audio-visual synchronization module before the second decoded frame, for example, the first decoded frame is the decoded frame input to the audio-visual synchronization module for the first time, and the second decoded frame is the decoded frame input to the audio-visual synchronization module for the second time. The first decoded frame can be an audio decoded frame or an image decoded frame, and the second decoded frame can be an audio decoded frame or an image decoded frame. In response to the second decoded frame being input to the audio and video synchronization module, the terminal calculates the difference between the system time when the second decoded frame is input and the system time when the first decoded frame is input, and obtains the system time difference corresponding to the second decoded frame. When the second decoded frame is an audio decoded frame, the terminal determines the sum of the time recorded by the audio and video synchronization clock and the system time difference to obtain the candidate clock time. The terminal can calculate the difference between the candidate clock time and the display time of the second decoded frame. If the difference is greater than or equal to the preset time difference threshold, the terminal updates the time recorded by the audio and video synchronization clock to the display time of the second decoded frame. If the difference is less than the preset time difference threshold, the time recorded by the audio and video synchronization clock is updated to the candidate clock time, thereby realizing the update of the time recorded by the audio and video synchronization clock.

[0106] In this embodiment, when the difference between the display time of the audio decoding frame and the candidate clock time is greater than or equal to the preset time difference threshold, the time recorded by the audio and video synchronization clock is updated to the display time of the decoded frame input later; when the difference between the display time of the audio decoding frame and the candidate clock time is less than the preset time difference threshold, the time recorded by the audio and video synchronization clock is updated to the candidate clock time, so that the time recorded by the audio and video synchronization clock is basically consistent with the display time of the audio, thereby better judging the audio and video synchronization situation.

[0107] In some embodiments, the method also includes: responding to a playback speed switching event of switching the current playback speed from a first playback speed to a second playback speed; searching for a frame loss rate having a mapping relationship with the second playback speed from a relationship mapping table; the relationship mapping table is used to store the mapping relationship between the frame loss rate and the playback speed; if found, updating the current frame loss rate to the found frame loss rate; if not found, comparing the first playback speed with the second playback speed, and updating the current frame loss rate based on the comparison result.

[0108] Among them, the first playback speed is different from the second playback speed. The playback speed refers to the speed of playing the video, and the playback speed includes but is not limited to at least one of 0.5 times, 1 times, 1.5 times or 2 times. The relationship mapping table is used to store the mapping relationship between the frame loss rate and the playback speed. The mapping relationship between the frame loss rate and the playback speed stored in the relationship mapping table. The relationship mapping table can be maintained by the player, and the relationship mapping table can be represented by MAP, for example. The relationship mapping table MAP is used to store the frame loss rate in a stable state corresponding to the playback speed. The frame loss rate in a stable state refers to the frame loss rate that remains unchanged within a preset time length when the playback speed is fixed. The preset time length can be set according to experience, for example, it can be 60 seconds or 70 seconds, etc., which is not limited here. The relationship mapping table can be expressed as:

[0109] MAP = {<rate1,drop1> ,<rate2,drop2> ,…, <rate n ,drop n >}

[0110] Among them, rate q Indicates the qth playback speed, drop q Indicates playback speed rate q The frame loss rate used in the stable state is 1≤q≤n. By setting a function expression, the mapping relationship between the playback speed and the frame loss rate can be added to the relationship mapping table MAP or the mapping relationship between the playback speed and the frame loss rate can be updated. For example, the function expression is set: set(rate q )=drop q , this function expression is used to add the mapping relationship between playback speed and frame loss rate to the relationship mapping table MAP or update the mapping relationship between playback speed and frame loss rate. You can also set another function expression to express the corresponding frame loss rate of playback speed in the relationship mapping table. For example, set the function expression: map(rate q )=drop q , this function expression is used to express rate q Corresponding drop in the relationship mapping table MAPq For example, if <3.0, 40%> is recorded in the relationship mapping table MAP, then map(3.0)=40%.

[0111] Specifically, the mapping relationship in the relationship mapping table can be added to the relationship mapping table during the process of playing the video. For example, if the terminal detects that the current frame loss rate remains unchanged within a preset time length, the terminal can determine the current playback speed. If the mapping relationship of the current playback speed does not exist in the relationship mapping table, the mapping relationship between the current playback speed and the current frame loss rate is added to the relationship mapping table. If the mapping relationship of the current playback speed already exists in the relationship mapping table, and the frame loss rate with a mapping relationship with the current playback speed is not the current frame loss rate, the mapping relationship of the current playback speed in the relationship mapping table is updated to the mapping relationship between the current playback speed and the current frame loss rate.

[0112] In some embodiments, in response to a playback speed switching event in which the current playback speed is switched from a first playback speed to a second playback speed, the terminal searches for a frame loss rate having a mapping relationship with the second playback speed from a relationship mapping table, and if found, updates the current frame loss rate to the found frame loss rate. new For the player, the current playback speed is rate current The player plays at a higher speed. new Look up the relationship mapping table MAP, and if the corresponding frame loss rate drop is found new , then update the current frame loss rate to the queried frame loss rate drop new , where drop new =map(rate new ).

[0113] In some embodiments, if no content is found, the terminal compares the first playback speed with the second playback speed. If the comparison result is that the first playback speed is less than the second playback speed, the current frame loss rate is increased; if the comparison result is that the first playback speed is greater than the second playback speed, the current frame loss rate is reduced.

[0114] In this embodiment, when the frame loss rate is found, the current frame loss rate is updated to the found frame loss rate, so that when the playback speed is switched, the frame loss rate in the stable state of the historical record can be directly used to update the current frame loss rate, thereby improving the response speed of audio and video synchronization; when the frame loss rate is not found, the first playback speed is compared with the second playback speed, and the current frame loss rate is updated based on the comparison result, so that when the frame loss rate is not found, the current frame loss rate can still be accurately and quickly updated.

[0115] In some embodiments, updating the current frame loss rate based on the comparison result includes: if the comparison result is that the first playback speed is less than the second playback speed, increasing the current frame loss rate based on the maximum frame loss rate; if the comparison result is that the first playback speed is greater than the second playback speed, reducing the current frame loss rate based on the minimum frame loss rate.

[0116] Specifically, if the comparison result is that the first playback speed is less than the second playback speed, the terminal can select a frame loss rate from the frame loss rates between the current frame loss rate and the maximum frame loss rate. For example, if the maximum frame loss rate is 100% and the current frame loss rate is 70%, the selected frame loss rate can be 80% or 90%, etc. The terminal can update the current frame loss rate to the selected frame loss rate. Alternatively, the terminal can calculate the average of the maximum frame loss rate and the first playback speed (recorded as the first average), and update the current frame loss rate to the first average.

[0117] In some embodiments, if the comparison result is that the first playback speed is greater than the second playback speed, the terminal can select a frame loss rate from the frame loss rates between the current frame loss rate and the minimum frame loss rate. For example, if the minimum frame loss rate is 0% and the current frame loss rate is 50%, the selected frame loss rate can be 10%, 20%, 30% or 40%, etc. The terminal can update the current frame loss rate to the selected frame loss rate. Alternatively, the terminal can calculate the average of the minimum frame loss rate and the first playback speed (recorded as the second average), and update the current frame loss rate to the second average.

[0118] For example, if the corresponding frame loss rate is not found, the terminal can compare the rate new and rate current , and update the frame loss rate according to formula (5), and set (rate new )=drop new Update the relationship mapping table MAP. Among them, rate new Refers to the second playback speed, rate current refers to the first playback speed, 0% refers to the minimum frame loss rate, 100% refers to the maximum frame loss rate, drop old refers to the current frame loss rate before the update. new Refers to the updated current frame loss rate.

[0119]

[0120] In this embodiment, if the comparison result is that the first playback speed is less than the second playback speed, the current frame loss rate is increased based on the maximum frame loss rate. If the comparison result is that the first playback speed is greater than the second playback speed, the current frame loss rate is reduced based on the minimum frame loss rate. In this way, even if the current frame loss rate is not found, it is still possible to accurately and quickly update the current frame loss rate.

[0121] In some embodiments, determining a frame loss strategy for discardable image frames based on a current frame loss rate includes: searching for frame loss strategy characterization information from a preset frame loss mapping table based on a frame sequence number of the discardable image frame and a current frame loss rate; the elements in the frame loss mapping table are frame loss strategy characterization information; and determining the frame loss strategy represented by the found frame loss strategy characterization information as the frame loss strategy for the discardable image frame.

[0122] Among them, the frame number can be determined according to the decoding order, for example, the earlier the decoding order, the smaller the frame number. The frame number can also be determined according to the display order, for example, the earlier the display order, the smaller the frame number. The elements in the frame loss mapping table are frame loss strategy characterization information. The elements in the frame loss mapping table are frame loss strategy characterization information. There are two types of frame loss strategy characterization information, one type represents the frame loss strategy of discarding, and the other represents the frame loss strategy of not discarding. The row index of the frame loss mapping table is the frame loss rate. Taking the preset number of frames as 10 and the frame loss rate value range of [0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%] as an example, the frame loss mapping table can be, for example, Table 2:

[0123] Table 2 Frame loss mapping table

[0124] 0 1 2 3 4 5 6 7 8 9 0% N N N N N N N N N N 10% Y N N N N N N N N N 20% Y N N N N Y N N N N 30% Y N N N Y N N N Y N 40% Y N N Y N N Y N N Y 50% Y N Y N Y N Y N Y N 60% N Y Y N Y Y N Y Y N 70% N Y Y Y N Y Y Y N Y 80% N Y Y Y Y N Y Y Y Y 90% N Y Y Y Y Y Y Y Y Y 100% Y Y Y Y Y Y Y Y Y Y

[0125] In Table 2, there are two types of frame loss strategy representation information, namely Y and N. The frame loss strategy represented by Y is drop, and the frame loss strategy represented by N is not drop.

[0126] Specifically, the column index is determined based on the frame sequence number of the discardable image frame. The terminal can determine the column index according to the preset frame number and the frame sequence number of the discardable image frame. For example, the terminal can perform a modulo operation on the frame sequence number of the discardable image frame and the preset frame number, and determine the value obtained by the modulo operation as the column index. For example, if the frame sequence number of the discardable image frame is I and the preset frame number is 10, then the column index f=I%10. The terminal uses the current frame loss rate as the row index and the determined column index as the column index to search for the corresponding frame loss strategy representation information from the frame loss mapping table. For example, in Table 2, if the current frame loss rate is 10% and the determined column index is 0, then the found frame loss strategy representation information is Y.

[0127] In this embodiment, the frame loss strategy is quickly determined through the frame loss mapping table, thereby improving the efficiency of determining the frame loss strategy.

[0128] In some embodiments, Figure 3As shown, a video processing method is provided. The method can be executed by a terminal or a server, or by both a terminal and a server. The method is applied to a terminal as an example for explanation. The terminal includes a player for playing a video. The player includes a cache queue, a frame loss manager, a decoder, an audio-visual synchronization module, a synchronization status monitor, and a renderer. The audio-visual synchronization module can also be called an audio-visual synchronization machine. Figure 4 As shown in the figure, it shows the framework diagram of the terminal implementing adaptive frame loss through the player. Figure 4 In the decoder, the audio packet is read from the data packet cache queue, each audio packet represents an audio frame, and the decoder decodes the read audio frame to obtain an audio decoding frame. The frame loss manager reads the video packet from the data packet cache queue, each video packet represents an image frame, and the frame loss manager determines whether the read image frame is a discardable image frame. If not, the read image frame is input into the decoder for decoding to obtain the corresponding image decoding frame. If so, it is determined whether to discard the read image frame. If it is determined to be discarded, the image frame is discarded. If it is determined not to be discarded, the image frame is input into the decoder for decoding to obtain the corresponding image decoding frame. The decoder inputs the audio decoding frame and the image decoding frame into the audio and video synchronization machine. The audio and video synchronization machine calculates the synchronization time difference corresponding to the image decoding frame, and inputs the synchronization time difference corresponding to the image decoding frame into the synchronization status monitor. The synchronization status monitor determines the audio and video synchronization of the video during the decoding process according to the synchronization time differences corresponding to the multiple image decoding frames continuously output by the decoder, and feeds back the audio and video synchronization to the frame loss manager. In addition, the audio and video synchronization machine inputs the synchronized audio decoding frame and image decoding frame into the renderer for rendering. As Figure 5 As shown, it shows the flow chart of the terminal implementing feedback adaptive frame loss through the player. Figure 5 In the text, the underlined part is implemented by the frame loss manager, the italic text part is implemented and fed back by the synchronization detector, "response to set the playback speed" is implemented by the player, "decoding" is implemented by the decoder, "audio-visual synchronization" is implemented by the audio-visual synchronization module, and "rendering" is implemented by the renderer. The frame loss ratio refers to the frame loss rate, and the speed refers to the playback speed. Specifically, Figure 5In the above, the player responds to setting the speed playback rate, which means responding to switching the current playback speed from the first playback speed to the second playback speed. "Whether there is a record of the frame loss ratio under the current speed" means that there is a frame loss rate that has a mapping relationship with the second playback speed. "Using this record as the current frame loss rate" means using the frame loss rate that has a mapping relationship with the second playback speed as the current frame loss rate. "Whether the speed is higher than the current speed" means judging whether the second playback speed is greater than the first playback speed, and "Whether the speed is higher than the current speed" means judging whether the second playback speed is less than the first playback speed. "Discarding 'discardable frames' according to the frame loss rate" means that when the image frame to be decoded is a discardable image frame, judging whether the image frame to be decoded needs to be discarded according to the current frame loss rate. "Counting the time difference between the video frame and the main clock every 15 frames" means that the synchronization status monitor calculates the synchronization time difference of each of the 15 decoded image frames continuously output by the decoder, and judges whether the "picture is ahead of the sound" or "the picture lags behind the sound" according to the synchronization time difference of each of the 15 decoded image frames. In the case of "the picture is ahead of the sound", the frame loss rate is reduced by one level, and in the case of "the picture lags behind the sound", the frame loss rate is increased by one level. For example, the value range of the frame loss rate is {0%, 10%, 20%, ..., 90%, 100%}, with a total of 11 levels. Each percentage in the value range is a level. For example, 0% is the 1st level, and 100% is the 11th level. If the current frame loss rate is 20%, then the current frame loss rate becomes 10% after reducing it by one level, and becomes 20% after increasing it by one level. Among them, the data packet cache queue is used to store downloaded audio packets and video packets, and the video packets here refer to image data packets. The frame loss manager is used to determine the image frames to be discarded, the decoder is used to decode the input image frames and video frames, the audio and video synchronizer is used to achieve audio and video synchronization, the synchronization status monitor determines whether the audio and video are synchronized, and when it is detected that they are not synchronized, it feedbacks to the frame loss manager, and the frame loss manager adjusts the current frame loss rate according to the feedback. Figure 3 As shown, the method comprises the following steps:

[0129] Step 302: decode each audio frame in the video in sequence through a decoder to obtain an audio decoding frame corresponding to each audio frame, and input the audio decoding frame into the audio-video synchronization module in sequence.

[0130] Step 304: read the image frame to be decoded from the video through the frame loss manager.

[0131] The frame loss manager reads the image frames to be decoded from the video in sequence, for example, reads the image frames to be decoded from the video in a decoding order.

[0132] Step 306 , determining whether the image frame to be decoded is a discardable image frame through the frame loss manager, if not, executing step 312 , if yes, executing step 308 .

[0133] Step 308 , judging whether the image frame to be decoded needs to be discarded according to the current frame loss rate by the frame loss manager, if so, executing step 310 , if not, executing step 312 .

[0134] Step 310: discard the image frame to be decoded through the frame discard manager.

[0135] Step 312: Input the image frame to be decoded into the decoder through the frame loss manager.

[0136] For example, a video playback device downloads a data packet from the network and caches the data packet into a data packet cache queue. The data packet includes an audio packet and a video packet. The frame loss manager maintains a frame loss rate (drop can be used to represent the frame loss rate) and initializes it to 0%. The value range of the frame loss rate is {0%, 10%, 20%, ..., 90%, 100%}, with a total of 11 levels. The frame loss rate indicates how much proportion of discardable image frames need to be discarded. Initializing it to 0% means that the initial state is not to discard any discardable image frames. The frame loss manager reads the image frame from the data packet cache queue and determines whether the read image frame is discardable. If it is determined that the read image frame is not discardable, the read image frame is input into the decoder for decoding. If it is determined that the read image frame is discardable, the discard manager takes every 10 data packets as a cycle and controls whether to input the discardable image frame into the decoder for decoding according to the frame loss rate. For example, if the current frame loss rate drop=30%, for the i-th video data packet, first calculate the factor f=i%10, and then determine whether to discard the i-th image frame or input the i-th image frame to the decoder for decoding based on the current frame loss rate and the factor f. For example, Table 2 can be used to determine whether to discard the i-th image frame or input the i-th image frame to the decoder for decoding.

[0137] Step 314: decode the image frame input into the decoder to obtain a decoded image frame, and input the decoded image frame into the audio and video synchronization module.

[0138] Step 316, when the audio-visual synchronization module receives the image decoding frame, the audio synchronization module updates the time recorded by the audio-visual synchronization clock according to the system time of the audio-visual synchronization module input by the image decoding frame, and calculates the difference between the display time of the image decoding frame and the time recorded by the updated audio-visual synchronization clock to obtain the synchronization time difference corresponding to the image decoding frame. When the audio-visual synchronization module receives the audio decoding frame, the audio-visual synchronization module updates the time recorded by the audio-visual synchronization clock according to the system time of the audio frame input by the audio-visual synchronization module.

[0139] Among them, the audio and video synchronization module inputs the audio decoding frame and the image decoding frame whose synchronization time difference is greater than the lagging time difference threshold and less than the leading time difference threshold into the renderer for rendering, so as to synchronize the audio and video during the playback process.

[0140] Step 318, the synchronization status monitor determines the audio and video synchronization status of the video during the decoding process according to the synchronization time differences corresponding to the multiple image decoding frames continuously output by the decoder, and feeds back the audio and video synchronization status to the frame loss manager.

[0141] Step 320: When the video is in synchronization with the audio and video and the picture lags behind the audio, the frame loss manager increases the current frame loss rate; when the video is in synchronization with the audio and video and the picture is ahead of the audio, the frame loss manager decreases the current frame loss rate.

[0142] Step 322: The player responds to the playback speed switching event to switch the current playback speed from the first playback speed to the second playback speed.

[0143] Step 324: The player searches the relationship mapping table for a frame loss rate that has a mapping relationship with the second playback speed.

[0144] Step 326, the player determines whether the file is found. If not, execute step 328; if found, execute step 330.

[0145] Step 328: The player compares the first playback speed with the second playback speed, and updates the current frame loss rate based on the comparison result.

[0146] Step 330: The player updates the current frame loss rate to the found frame loss rate.

[0147] The current frame loss rate used in step 308 may be continuously updated, and steps 320, 328, and 330 are three situations in which the current frame loss rate is updated. The latest current frame loss rate is used each time step 308 is executed. For example, if the current frame loss rate is 10%, the frame loss manager performs the step of determining whether the image frame to be decoded is a discardable image frame according to the current frame loss rate of 10%. If the current frame loss rate is updated to 20%, the frame loss manager performs the step of determining whether the image frame to be decoded is a discardable image frame according to the current frame loss rate of 20%.

[0148] In this embodiment, through the feedback of the synchronization state monitor of the audio and video synchronization machine, the discardable frames are adaptively discarded before decoding, and the decoding time / computing resource overhead of the discardable frames is saved, so that in the scenario of high-speed / high-frame-rate video playback, a large video playback frame rate output can be maintained under the limitation of decoding performance. This improves the smoothness and experience of users watching videos.

[0149] like Figure 6, a frame rate data diagram presented by a video source with a frame rate of 25fps played at a double speed without applying the video processing method provided by the present application is shown, such as Figure 7 In the figure, a frame rate data graph presented by a video source with a frame rate of 25fps played at double speed is shown when the video processing method provided by the present application is applied. Figure 6 and Figure 7 At time = 0 and time = 56, the video playback device (such as a player) starts playing at one-time speed, and at time = 28 and time = 86, the video playback device (such as a player) starts playing at three-time speed. Figure 6 and Figure 7 In the example, when playing at 1x speed, due to the remaining decoding performance, the decoding speed can support the current playback frame rate, i.e., 25fps. When playing at 3x speed, the decoding performance is not enough to support the actual 3x frame rate, i.e., 75fps. Figure 6 In the process, many video frames are dropped because they lag behind the audio and video synchronization machine, resulting in the actual playback video frame rate being less than 5fps. Figure 7 In the present invention, due to the application of the video processing solution provided by the present application, whether to discard the data packet before decoding is adaptively adjusted through the feedback mechanism, thereby saving a lot of decoding overhead, so that although the actual playback video frame rate is less than 75fps, it is also stable at about 50fps.

[0150] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0151] Based on the same inventive concept, the embodiment of the present application also provides a video processing device for implementing the above-mentioned video processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more video processing device embodiments provided below can refer to the limitations on the video processing method above, and will not be repeated here.

[0152] In some embodiments, Figure 8As shown, a video processing device is provided, including: a frame loss strategy determination module 802, a decoded frame acquisition module 804, a synchronization time difference acquisition module 806 and a current frame loss rate update module 808, wherein:

[0153] The frame loss strategy determination module 802 is used to determine the frame loss strategy for the discarded image frames according to the current frame loss rate when decoding the video. The frame loss strategy is used to indicate whether to discard the discarded image frames before inputting the discarded image frames into the decoder.

[0154] The decoded frame obtaining module 804 is used to input the discardable image frame into the decoder when the frame discarding strategy is not to discard the image frame, and obtain the image decoded frame output by the decoder corresponding to the discardable image frame.

[0155] The synchronization time difference obtaining module 806 is used to determine the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock, and obtain the synchronization time difference corresponding to the image decoding frame.

[0156] The current frame loss rate update module 808 is used to update the current frame loss rate when the audio and picture of the video are not synchronized during the decoding process characterized by the synchronous time difference, so as to determine the frame loss strategy of the subsequent discardable image frames in the video to be decoded according to the updated current frame loss rate.

[0157] In some embodiments, the device is also used for: when decoding a non-discardable image frame in a video, inputting the non-discardable image frame into a decoder to obtain an image decoded frame output by the decoder corresponding to the non-discardable image frame; the current frame loss rate update module 808 is also used to determine the audio and video synchronization of the video during the decoding process according to the synchronization time difference corresponding to each of a plurality of image decoded frames continuously output by the decoder; if the audio and video synchronization situation is that the audio and video are not synchronized, then updating the current frame loss rate.

[0158] In some embodiments, the current frame loss rate update module 808 is further used to arrange the synchronization time differences corresponding to a preset number of image decoding frames continuously output by the decoder in ascending order to obtain a synchronization time difference sequence; determine a first reference serial number according to a first preset proportion and a preset number; determine the synchronization time difference sorted as the first reference serial number from the synchronization time difference sequence in a front-to-back order to obtain a first reference time difference; when the first reference time difference is less than or equal to a lagging time difference threshold, determine that the audio and video synchronization condition of the video during the decoding process is that the picture lags behind the audio.

[0159] In some embodiments, the current frame loss rate update module 808 is also used to determine a second reference serial number based on a second preset proportion and a preset number; determine the synchronization time difference sorted as the second reference serial number from the synchronization time difference sequence in a back-to-front order to obtain a second reference time difference; when the second reference time difference is greater than or equal to the advance time difference threshold, determine that the audio and video synchronization condition of the video during the decoding process is that the picture is ahead of the audio.

[0160] In some embodiments, the current frame loss rate update module 808 is also used to increase the current frame loss rate when the audio and video synchronization condition of the video is that the picture lags behind the audio; and to reduce the current frame loss rate when the audio and video synchronization condition of the video is that the picture is ahead of the audio.

[0161] In some embodiments, the device is also used to decode the audio frames in the video to obtain audio decoded frames, and input the audio decoded frames into the audio and video synchronization module; the device also includes a clock update module for updating the time recorded by the audio and video synchronization clock, and the clock update module is used to determine the difference between the system times when the decoded frames are input into the audio and video synchronization module twice, and obtain the system time difference; wherein the decoded frames input into the audio and video synchronization module twice are respectively audio decoded frames or image decoded frames; the time recorded by the audio and video synchronization clock is updated based on the system time difference.

[0162] In some embodiments, the clock updating module is further used to increase the system time difference based on the time recorded by the audio and video synchronization clock if the decoded frame input to the audio and video synchronization module twice is an image decoded frame.

[0163] In some embodiments, the clock update module is also used to determine the sum of the time recorded by the audio and video synchronization clock and the system time difference to obtain a candidate clock time if the decoded frame input later among the two decoded frames input to the audio and video synchronization module is an audio decoded frame; and update the time recorded by the audio and video synchronization clock based on the display time of the decoded frame input later and the candidate clock time.

[0164] In some embodiments, the clock update module is also used to update the time recorded by the audio and video synchronization clock to the display time of the decoded frame input later when the difference between the display time of the audio decoded frame and the candidate clock time is greater than or equal to a preset time difference threshold; and to update the time recorded by the audio and video synchronization clock to the candidate clock time when the difference between the display time of the audio decoded frame and the candidate clock time is less than a preset time difference threshold.

[0165] In some embodiments, the current frame loss rate update module 808 is also used to respond to a playback speed switching event of switching the current playback speed from a first playback speed to a second playback speed; search for a frame loss rate that has a mapping relationship with the second playback speed from a relationship mapping table; the relationship mapping table is used to store the mapping relationship between the frame loss rate and the playback speed; if found, update the current frame loss rate to the found frame loss rate; if not found, compare the first playback speed with the second playback speed, and update the current frame loss rate based on the comparison result.

[0166] In some embodiments, the current frame loss rate update module 808 is also used to increase the current frame loss rate based on the maximum frame loss rate if the comparison result is that the first playback speed is less than the second playback speed; if the comparison result is that the first playback speed is greater than the second playback speed, then reduce the current frame loss rate based on the minimum frame loss rate.

[0167] In some embodiments, the frame loss strategy determination module 802 is also used to search for frame loss strategy characterization information from a preset frame loss mapping table based on the frame sequence number of the discardable image frame and the current frame loss rate; the elements in the frame loss mapping table are the frame loss strategy characterization information; and the frame loss strategy represented by the found frame loss strategy characterization information is determined as the frame loss strategy for the discardable image frame.

[0168] Each module in the above video processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0169] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the video processing method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video processing method is implemented.

[0170] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0171] Those skilled in the art will understand that Fig. 9 and Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0172] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned video processing method when executing the computer program.

[0173] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned video processing method are implemented.

[0174] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned video processing method when executed by a processor.

[0175] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: In a process of decoding a video, when a discardable image frame in the video is to be decoded, a frame dropping strategy for the discardable image frame is determined according to a current frame dropping rate; the frame dropping strategy is used to indicate whether to discard the discardable image frame before inputting the discardable image frame into a decoder; When the frame discarding strategy is not discarding, inputting the discardable image frame into a decoder to obtain an image decoded frame output by the decoder and corresponding to the discardable image frame; Determine the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock to obtain the synchronization time difference corresponding to the image decoding frame; When the audio and picture of the video are not synchronized during the synchronous time difference characterization decoding process, the current frame loss rate is updated so that when subsequent discardable image frames in the video are to be decoded, the frame loss strategy of the subsequent discardable image frames is determined according to the updated current frame loss rate.

2. The method according to claim 1, characterized in that The method further comprises: When decoding a non-discardable image frame in the video, input the non-discardable image frame into the decoder to obtain an image decoded frame output by the decoder and corresponding to the non-discardable image frame; In the case where the audio and picture of the video are not synchronized during the synchronous time difference characterization decoding process, updating the current frame loss rate includes: Determining the audio and video synchronization of the video during the decoding process according to the synchronization time differences corresponding to the multiple image decoding frames continuously output by the decoder; If the audio and video synchronization condition is that the audio and video are not synchronized, the current frame loss rate is updated.

3. The method according to claim 2, characterized in that Determining the audio and video synchronization of the video during the decoding process according to the synchronization time differences corresponding to the multiple image decoding frames continuously output by the decoder includes: Arranging the synchronization time differences corresponding to the preset number of image decoding frames continuously output by the decoder in ascending order to obtain a synchronization time difference sequence; Determine a first reference serial number according to the first preset proportion and the preset quantity; Determine, in order from front to back, the synchronization time difference sequence sorted as the first reference sequence number to obtain a first reference time difference; When the first reference time difference is less than or equal to the lagging time difference threshold, it is determined that the audio and video synchronization condition of the video during the decoding process is that the picture lags behind the audio.

4. The method according to claim 3, characterized in that The method further comprises: Determine a second reference serial number according to the second preset proportion and the preset quantity; Determine, in order from back to front, the synchronization time difference sequence sorted as the second reference sequence number to obtain a second reference time difference; When the second reference time difference is greater than or equal to the leading time difference threshold, it is determined that the audio and video synchronization condition of the video during the decoding process is that the picture is ahead of the audio.

5. The method according to claim 2, characterized in that: If the audio and video synchronization situation is that the audio and video are not synchronized, updating the current frame loss rate includes: When the audio and video synchronization condition of the video is that the picture lags behind the audio, increasing the current frame loss rate; When the audio and video synchronization condition of the video is that the picture is ahead of the audio, the current frame loss rate is reduced.

6. The method according to claim 1, characterized in that The method further comprises: Decoding the audio frames in the video to obtain audio decoding frames, and inputting the audio decoding frames into the audio and video synchronization module; The step of updating the time recorded by the audio and video synchronization clock includes: Determine the difference between the system times when the decoded frames are input to the audio and video synchronization module twice, and obtain the system time difference; wherein the decoded frames input to the audio and video synchronization module twice are respectively audio decoded frames or image decoded frames; The time recorded by the audio and video synchronization clock is updated based on the system time difference.

7. The method according to claim 6, characterized in that The time recorded by the audio and video synchronization clock is updated based on the system time difference, including: If the decoded frame inputted later in the two decoded frames inputted to the audio and video synchronization module is an image decoded frame, the system time difference is added to the time recorded by the audio and video synchronization clock.

8. The method according to claim 6, characterized in that The time recorded by the audio and video synchronization clock is updated based on the system time difference, including: If the decoded frame inputted later in the two decoded frames inputted to the audio and video synchronization module is an audio decoded frame, then determining the sum of the time recorded by the audio and video synchronization clock and the system time difference to obtain a candidate clock time; The time recorded by the audio-visual synchronization clock is updated based on the display time of the decoded frame input later and the candidate clock time.

9. The method according to claim 8, characterized in that The updating of the time recorded by the audio-visual synchronization clock based on the display time of the decoded frame input later and the candidate clock time comprises: When the difference between the display time of the audio decoded frame and the candidate clock time is greater than or equal to a preset time difference threshold, updating the time recorded by the audio and video synchronization clock to the display time of the decoded frame input later; When the difference between the display time of the audio decoding frame and the candidate clock time is less than the preset time difference threshold, the time recorded by the audio and video synchronization clock is updated to the candidate clock time.

10. The method according to claim 1, characterized in that The method further comprises: In response to a playback speed switching event of switching the current playback speed from a first playback speed to a second playback speed; Searching for a frame loss rate having a mapping relationship with the second playback speed from a relationship mapping table; the relationship mapping table is used to store a mapping relationship between the frame loss rate and the playback speed; If found, the current frame loss rate is updated to the found frame loss rate; If the frame loss rate is not found, the first playback speed is compared with the second playback speed, and the current frame loss rate is updated based on the comparison result.

11. The method according to claim 10, characterized in that The updating of the current frame loss rate based on the comparison result includes: If the comparison result is that the first playback speed is less than the second playback speed, increasing the current frame loss rate based on the maximum frame loss rate; If the comparison result is that the first playback speed is greater than the second playback speed, the current frame loss rate is reduced based on the minimum frame loss rate.

12. The method according to any one of claims 1 to 11, characterized in that: The determining of the frame loss strategy for the discardable image frame according to the current frame loss rate includes: According to the frame sequence number of the discardable image frame and the current frame loss rate, searching for frame loss strategy representation information from a preset frame loss mapping table; the elements in the frame loss mapping table are frame loss strategy representation information; The frame dropping strategy represented by the found frame dropping strategy representation information is determined as the frame dropping strategy for the discardable image frame.

13. A video processing device, characterized in that: The device comprises: A frame loss strategy determination module is used to determine a frame loss strategy for a discardable image frame in the video according to a current frame loss rate when a discardable image frame in the video is to be decoded during the video decoding process; the frame loss strategy is used to indicate whether to discard the discardable image frame before inputting the discardable image frame into a decoder; A decoded frame obtaining module, used for inputting the discardable image frame into a decoder when the frame discarding strategy is not discarding, and obtaining an image decoded frame output by the decoder corresponding to the discardable image frame; A synchronization time difference obtaining module, used to determine the difference between the display time of the image decoding frame and the time recorded by the audio and video synchronization clock, and obtain the synchronization time difference corresponding to the image decoding frame; The current frame loss rate update module is used to update the current frame loss rate when the audio and picture of the video are not synchronized during the synchronous time difference characterization decoding process, so as to determine the frame loss strategy of the subsequent discardable image frames in the video to be decoded according to the updated current frame loss rate.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.