Video stream playback method, system, electronic device and storage medium
By sorting and frame splicing of video stream data, combined with dynamic sleep adjustment technology, the problem of unstable playback of video surveillance system under network jitter and high speed is solved, and smoother and more efficient video streaming playback is achieved.
Patent Information
- Application Number
- CN202211502762.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-11-28
AI Technical Summary
In the field of video surveillance, network jitter and high-speed playback, video streaming playback systems are difficult to deal with network packet loss, disordered order and jitter, resulting in stuttering video playback, screen loss and poor performance.
The video data to be played is sorted and framed through the preset data structure, the complete video frame is obtained, and the video playback strategy and frame drop strategy are decoded and played, and the dynamic sleep adjustment technology is used to adapt to network jitter.
It effectively solves the problem of video playback lag caused by network jitter, improves the performance and stability of high-speed playback, and ensures the smooth playback of video streams.
Smart Images

Figure CN115802102B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of video, and in particular to a method, system, electronic device and storage medium for playing video streams. Background Art
[0002] In the field of video surveillance, a video streaming server sends video media data packets to a monitoring client that sends a media request according to relevant media access protocols. After receiving the data packets, the client parses and extracts them, performs frame splicing, decoding, and playback operations. Due to the randomness of network transmission, the problems often encountered in actual processes are: (1) network packet loss, a certain data packet is not received; (2) packet disorder, the packet sequence size received by the receiving end does not necessarily increase in order. The degree of disorder can be as minor as disorder within the frame, or as severe as the packet of the first frame running to the third or fourth frame; (3) network jitter, the arrival time of the data packet is fast and slow and uneven on the time axis (visually, when the frame interval exceeds 100ms, the playback will show a stuttering effect, even if the data packet is complete and in order); (4) high-speed playback requirements: Different from the consumer video field, which only has a normal speed playback mode, in the surveillance video field, there is a strong application demand for 2x, 4x, 8x, 16x and even 32x playback. In the high-speed mode (the decoding end generally uses GPU for decoding acceleration), the processing performance of the network data packet receiving end is also a potential bottleneck. Taking 50 packets per frame, 25 frames per second, and 16 times as an example, there are nearly 50*25*16=20,000 packets per second. On an Intel i78-core machine, the simplest linked list structure is used to save 20,000 packets and insert sorting. If nothing else is done, it takes nearly 600ms. (Time complexity O(n2)), which is close to the upper limit of 1s. The above four problems make it difficult to design and implement a high-performance network video playback system that can adapt to and handle various complex network transmission conditions. It is easy to cause video playback to be distorted, stuck, jerky, uneven playback, and high-speed playback performance cannot keep up, which seriously affects the video viewing experience.
[0003] In the core playback control algorithm (how to control when to display the decoded data after decoding), the more common video playback control strategy is to establish a certain size playback cache, and then control the playback according to the timestamp carried by the video frame. When the actual arrival time is equal to or later than the timestamp + cache time, it will be played. When it arrives early, it will wait. There is a problem with this. For the two typical cases of network jitter: one is early and the other is lagging, it can only handle the early arrival case, and there is no good adjustment effect on the lagging. When the cache is consumed, the entire playback strategy begins to fail. Summary of the invention
[0004] The main purpose of the embodiments of the present invention is to provide a method, system, electronic device and storage medium for playing video streams, which effectively adapt to the problem of video playback freeze caused by network jitter.
[0005] In a first aspect, a method for playing a video stream is provided, the method comprising:
[0006] Sorting the video data to be played through a preset data structure, and obtaining the sorted video data to be played;
[0007] Performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and putting the video data to be played into a queue to be decoded;
[0008] Parsing the complete video data to be played to obtain video encoding data and timestamp information;
[0009] Obtaining a frame to be decoded from the video encoding data, decoding the frame to be decoded to obtain YUV decoded data, and recording a decoding time of the decoding;
[0010] The YUV decoded data is played according to a preset video playback strategy and a frame loss strategy, wherein the video playback strategy is related to the length of a queue for caching the frames to be decoded, the mean time difference of the timestamp information, a preset dynamic sleep adjustment time variable deltaTime, and the decoding time.
[0011] In a possible implementation, the step of sorting the video data to be played using a preset data structure and obtaining the sorted video data to be played includes:
[0012] Add a flow flag flow_sign to each video data packet in the video data to be played, if the sequence number of the video data packet is less than 65000, the flow flag is 1, if the sequence number of the video data packet is greater than 65000, the flow flag is 0;
[0013] Sorting according to the stream flag bit includes:
[0014] If the stream flags are the same, the video data packets are sorted according to the actual sequence numbers; and,
[0015] If the flow flags are different, then for the sequence number with a flow flag of 0, its value is assigned to a 32-bit integer variable x; for the sequence number with a flow flag of 1, if the sequence number is greater than a preset sequence number threshold, its value is assigned to a 32-bit integer variable y1, wherein y1=sequence number value+65536; if the sequence number is less than the sequence number threshold, its value is assigned to a 32-bit integer variable y2, wherein y2 is the sequence number value.
[0016] In another possible implementation, performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded includes:
[0017] When it is detected that the data packet of the sorted video data to be played contains a mark of the start or end of a frame, the sequence number of the data packet is recorded through an array;
[0018] When the number of the sequence numbers is greater than a preset sequence number threshold, the sequence numbers are sorted;
[0019] Obtain media information between two adjacent data packets with sequence numbers. If the media information is continuous without packet loss, remove the video encapsulation layer to obtain the raw code stream. If the media information is discontinuous, discard the discontinuous video frames and advance the sequence number by one.
[0020] The raw code stream is put into the queue to be decoded.
[0021] In another possible implementation, playing the YUV decoded data according to a preset video playback strategy and a frame loss strategy includes:
[0022] Obtaining a time difference mean of the timestamp information;
[0023] Get the number of video frames to be decoded in the queue to be decoded. If N is less than N1, sleep and do not decode the video frame. If N is greater than N6, the queue to be decoded discards the frames to be decoded until the next key frame, and re-detects N. The sleep time T of the sleep is T1-T0-deltaTime, T1 is the time difference mean of the timestamp information, T0 is the decoding time of the current frame, deltaTime is related to N, and
[0024] N1≤N <N2:deltaTime=2
[0025] N2≤N <N3:deltaTime=4
[0026] N3≤N <N4:deltaTime=8
[0027] N4≤N <N5:deltaTime=10
[0028] N5≤N:deltaTime=15
[0029] Wherein, N is the number of frames to be decoded in the queue to be decoded, N1=5, N2=2N1, N3=3N1, N4=4N1, N5=6N1, N6=10N1.
[0030] In a second aspect, a system for playing a video stream is provided, the system comprising:
[0031] A module for obtaining the sorted video data to be played, which is used to sort the video data to be played according to a preset data structure and obtain the sorted video data to be played;
[0032] A complete to-be-played video data acquisition module is used to perform a frame splicing operation on the sorted to-be-played video data by removing incomplete frames, to acquire complete to-be-played video data, and to put the to-be-played video data into a queue to be decoded;
[0033] A video encoding and timestamp information acquisition module, used to parse the complete video data to be played and acquire video encoding data and timestamp information;
[0034] A decoding time acquisition module, used to acquire a frame to be decoded from the video encoding data, decode the frame to be decoded to acquire YUV decoded data, and record the decoding time of the decoding;
[0035] A video playback module is used to play the YUV decoded data according to a preset video playback strategy and a frame loss strategy. The video playback strategy is related to the length of the queue for caching the frames to be decoded, the time difference mean of the timestamp information, the preset dynamic sleep adjustment time variable deltaTime, and the decoding time.
[0036] In a possible implementation, the step of sorting the video data to be played using a preset data structure and obtaining the sorted video data to be played includes:
[0037] Add a flow flag flow_sign to each video data packet in the video data to be played, if the sequence number of the video data packet is less than 65000, the flow flag is 1, if the sequence number of the video data packet is greater than 65000, the flow flag is 0;
[0038] Sorting according to the stream flag bit includes:
[0039] If the stream flags are the same, the video data packets are sorted according to the actual sequence numbers; and,
[0040] If the flow flags are different, then for the sequence number with a flow flag of 0, its value is assigned to a 32-bit integer variable x; for the sequence number with a flow flag of 1, if the sequence number is greater than a preset sequence number threshold, its value is assigned to a 32-bit integer variable y1, wherein y1=sequence number value+65536; if the sequence number is less than the sequence number threshold, its value is assigned to a 32-bit integer variable y2, wherein y2 is the sequence number value.
[0041] In another possible implementation, performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded includes:
[0042] When it is detected that the data packet of the sorted video data to be played contains a mark of the start or end of a frame, the sequence number of the data packet is recorded through an array;
[0043] When the number of the sequence numbers is greater than a preset sequence number threshold, the sequence numbers are sorted;
[0044] Obtain media information between two adjacent data packets with sequence numbers. If the media information is continuous without packet loss, remove the video encapsulation layer to obtain the raw code stream. If the media information is discontinuous, discard the discontinuous video frames and advance the sequence number by one.
[0045] The raw code stream is put into the queue to be decoded.
[0046] In another possible implementation, playing the YUV decoded data according to a preset video playback strategy and a frame loss strategy includes:
[0047] Obtaining a time difference mean of the timestamp information;
[0048] Get the number of video frames to be decoded in the queue to be decoded. If N is less than N1, sleep and do not decode the video frame. If N is greater than N6, the queue to be decoded discards the frames to be decoded until the next key frame, and re-detects N. The sleep time T of the sleep is T1-T0-deltaTime, T1 is the time difference mean of the timestamp information, T0 is the decoding time of the current frame, deltaTime is related to N, and
[0049] N1≤N <N2:deltaTime=2
[0050] N2≤N <N3:deltaTime=4
[0051] N3≤N <N4:deltaTime=8
[0052] N4≤N <N5:deltaTime=10
[0053] N5≤N:deltaTime=15
[0054] Wherein, N is the number of frames to be decoded in the queue to be decoded, N1=5, N2=2N1, N3=3N1, N4=4N1, N5=6N1, N6=10N1.
[0055] In a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for playing a video stream as provided in the first aspect is implemented.
[0056] In a fourth aspect, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for playing a video stream provided in the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.
[0058] Figure 1 A flowchart of a method for playing a video stream provided by an embodiment of the present invention;
[0059] Figure 2 A structural diagram of a system for playing video streams provided by an embodiment of the present invention;
[0060] Figure 3 The figure is a schematic diagram of the physical structure of an electronic device of the present invention.
[0061] Specific implementation method
[0062] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.
[0063] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, modules and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, modules, components and / or groups thereof. It should be understood that when we refer to a module as being "connected" or "coupled" to another module, it may be directly connected or coupled to the other modules, or there may be intermediate modules. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any modules and all combinations of one or more associated listed items.
[0064] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation method of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0065] The technical solution of the present application and how to solve the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0066] like Figure 1 FIG. 1 is a flow chart of a method for playing a video stream provided by an embodiment of the present invention, wherein the method comprises:
[0067] Step 101, sorting the video data to be played using a preset data structure, and obtaining the sorted video data to be played;
[0068] Step 102, performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and putting the video data to be played into a queue to be decoded;
[0069] Step 103, parsing the complete video data to be played to obtain video encoding data and timestamp information;
[0070] Step 104, obtaining a frame to be decoded from the video encoding data, decoding the frame to be decoded to obtain YUV decoded data, and recording the decoding time of the decoding;
[0071] Step 105, playing the YUV decoded data according to a preset video playback strategy and a frame loss strategy, wherein the video playback strategy is related to the length of the queue for caching the frames to be decoded, the mean time difference of the timestamp information, a preset dynamic sleep adjustment time variable deltaTime, and the decoding time.
[0072] In the embodiment of the present invention, an efficient data structure suitable for insertion, deletion, and sorting is selected, such as a red-black tree, with a time complexity of log(N) for insertion, log(N) for deletion, and N*log(N) for sorting, to access network data packets, such as set in C++, TreeSet in Java, etc.
[0073] The step of sorting the video data to be played by using a preset data structure to obtain the sorted video data to be played includes:
[0074] Add a flow flag flow_sign to each video data packet in the video data to be played, if the sequence number of the video data packet is less than 65000, the flow flag is 1, if the sequence number of the video data packet is greater than 65000, the flow flag is 0;
[0075] Sorting according to the stream flag bit includes:
[0076] If the stream flags are the same, the video data packets are sorted according to the actual sequence numbers; and,
[0077] If the flow flags are different, then for the sequence number with a flow flag of 0, its value is assigned to a 32-bit integer variable x; for the sequence number with a flow flag of 1, if the sequence number is greater than a preset sequence number threshold, its value is assigned to a 32-bit integer variable y1, wherein y1=sequence number value+65536; if the sequence number is less than the sequence number threshold, its value is assigned to a 32-bit integer variable y2, wherein y2 is the sequence number value.
[0078] In the embodiment of the present invention, the packet number in the streaming media protocol is only 2 bytes, and the maximum is 65535. If it exceeds, it will generally start from 0. In order to compare the size of the packet number, it is necessary to redefine the packet size comparison function. By setting a new stream flag, the size of the data packet can be sorted. After re-sorting, the video media data packet is automatically ordered in the memory structure, and the memory structure is recorded as DataSet.
[0079] The step of performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded includes:
[0080] When it is detected that the data packet of the sorted video data to be played contains a mark of the start or end of a frame, the sequence number of the data packet is recorded through an array;
[0081] When the number of the sequence numbers is greater than a preset sequence number threshold, the sequence numbers are sorted;
[0082] Obtain media information between two adjacent data packets with sequence numbers. If the media information is continuous without packet loss, remove the video encapsulation layer to obtain the raw code stream. If the media information is discontinuous, discard the discontinuous video frames and advance the sequence number by one.
[0083] The raw code stream is put into the queue to be decoded.
[0084] The frame splicing operation in this step can effectively solve the problem of screen distortion during video playback caused by packet loss and disorder.
[0085] The playing of the YUV decoded data according to the preset video playing strategy and frame dropping strategy includes:
[0086] Obtaining a time difference mean of the timestamp information;
[0087] Get the number of video frames to be decoded in the queue to be decoded. If N is less than N1, sleep and do not decode the video frame. If N is greater than N6, the queue to be decoded discards the frames to be decoded until the next key frame, and re-detects N. The sleep time T of the sleep is T1-T0-deltaTime, T1 is the time difference mean of the timestamp information, T0 is the decoding time of the current frame, deltaTime is related to N, and
[0088] N1≤N <N2:deltaTime=2
[0089] N2≤N <N3:deltaTime=4
[0090] N3≤N <N4:deltaTime=8
[0091] N4≤N <N5:deltaTime=10
[0092] N5≤N:deltaTime=15
[0093] Wherein, N is the number of frames to be decoded in the queue to be decoded, N1=5, N2=2N1, N3=3N1, N4=4N1, N5=6N1, N6=10N1.
[0094] In the embodiment of the present invention, the video data to be played is sorted by a preset data structure to obtain the sorted video data to be played; the incomplete frames are removed to perform frame splicing operation on the sorted video data to be played to obtain complete video data to be played, and the video data to be played is put into a queue to be decoded; the complete video data to be played is parsed to obtain video encoding data and timestamp information; the frames to be decoded are obtained from the video encoding data, the frames to be decoded are decoded to obtain YUV decoding data, and the decoding time of the decoding is recorded; the YUV decoding data is played according to the preset video playback strategy and frame loss strategy. It effectively adapts to the problem of video playback freeze caused by network jitter.
[0095] For example:
[0096] The system includes four parts: network receiver, frame extractor, decoding and rendering module, and playback control module. The decoding and rendering module is embedded in the playback control module.
[0097] Network receiver: Receives network data packets and sorts the packets according to the sequence numbers in the streaming protocol.
[0098] Frame extraction module: extracts, splices, and parses the raw code stream of data packets that meet the regulation in the network receiver. The extracted raw code stream frames are put into the decoding queue.
[0099] Decoding and rendering module: Decoding and rendering the specified frame.
[0100] Playback control module: controls the decoding queue, and performs frame output, decoding and rendering module calls, and sleep delay operations on the decoding queue according to the appropriate cache size control strategy.
[0101] The following is an explanation of the specific implementation process of each module:
[0102] Take GB28181 national standard flow (SIP+RTP) as an example
[0103] The network receiving module processes the following steps:
[0104] Use network IO model (socket / udp, etc.) to receive video stream data;
[0105] According to the serial number comparison rules described above, overload the comparison function of c++set; the data type stored in set is the RTP information structure pointer;
[0106] Parse the RTP information to obtain the RTP packet sequence number, RTP content (PAYLOAD) and other information. Add a flow attribute flag flow_sign to each packet. According to the sequence number value, >65000, flow_sign=0<65000, flow_sign=1;
[0107] Use the insert interface of set to add the information structure obtained by parsing in 3, and the set will automatically complete the orderly insertion;
[0108] The frame extraction module processing steps are as follows:
[0109] (1) Detect the frame boundary conditions of each received RTP data packet. Since GB28181 uses PS encapsulation and each frame starts with 000001BA, 000001BA can be detected as the start mark of a frame. After detecting the packets that meet the conditions, they are sequentially entered into the mark array, and the number of elements in the mark array is counted +1.
[0110] (2) When the number of elements in the marker array reaches a threshold of 3, the marker array is sorted according to the RTP sequence number, and the RTP sequence number values between the first element and the second element (N1 to N2, excluding N2) are recorded;
[0111] (3) Traverse the packets in the range of DataSet N1 to N2 in the network receiver and detect packet loss;
[0112] (4) If the frame is lost, it is discarded and the process starts again with the next key frame without packet loss.
[0113] (5) If there is no packet loss, the RTP payload is spliced, the obtained complete frame is parsed, and the encapsulation layer is removed to obtain the raw code stream data;
[0114] (6) Put the raw code stream data into the decoding queue.
[0115] Taking RTSP as an example, the mark bit of RTP can be selected as the marker (the last packet of a frame). The subsequent steps are basically the same and will not be repeated here.
[0116] Decoding and rendering are conventional operations in this field, and will not be described in detail in the specific embodiments.
[0117] The specific steps of the playback control module are as follows:
[0118] Detect the time difference between the two adjacent frames in the decoding queue, recorded as T1. The typical GB28181 timestamp difference is 3600 (25fps), which is converted into a time unit of 40ms (3600 / 90000*1000ms=40ms);
[0119] Set five cache queue size thresholds N1 to N6. The benchmark value of N1 can be adjusted according to actual conditions and needs. The larger the N1 value, the better the anti-network jitter effect, but also the longer the network playback delay caused.
[0120] Check the number of elements in the decoding queue N, if it is less than N1, sleep and wait;
[0121] If N is greater than N6, the decoding queue starts dropping frames until the next key frame, and N is rechecked at the same time;
[0122] Set a dynamic time adjustment variable value deltaTime, the value formula is as above;
[0123] A final actual sleep time T is set, T = T1-T0-deltaTime. If T is a positive value, after the decoder takes out the current frame for decoding and rendering, it sleeps for the corresponding T time.
[0124] Repeat steps 3 to 6 until the system asks you to exit.
[0125] like Figure 2 FIG. 1 is a structural diagram of a system for playing video streams provided by an embodiment of the present invention, wherein the system comprises:
[0126] The module 201 for obtaining the sorted video data to be played is used to sort the video data to be played according to a preset data structure and obtain the sorted video data to be played;
[0127] The complete video data acquisition module 202 is used to perform a frame splicing operation on the sorted video data to be played by removing incomplete frames, obtain the complete video data to be played, and put the video data to be played into a queue to be decoded;
[0128] The video encoding and timestamp information acquisition module 203 is used to parse the complete video data to be played and obtain video encoding data and timestamp information;
[0129] A decoding time acquisition module 204 is used to acquire a frame to be decoded from the video encoding data, decode the frame to be decoded to acquire YUV decoded data, and record the decoding time of the decoding;
[0130] The video playback module 205 is used to play the YUV decoded data according to a preset video playback strategy and frame loss strategy. The video playback strategy is related to the length of the queue for caching the frames to be decoded, the time difference mean of the timestamp information, the preset dynamic sleep adjustment time variable deltaTime, and the decoding time.
[0131] In the embodiment of the present invention, an efficient data structure suitable for insertion, deletion, and sorting is selected, such as a red-black tree, with a time complexity of log(N) for insertion, log(N) for deletion, and N*log(N) for sorting, to access network data packets, such as set in C++, TreeSet in Java, etc.
[0132] The step of sorting the video data to be played by using a preset data structure to obtain the sorted video data to be played includes:
[0133] Add a flow flag flow_sign to each video data packet in the video data to be played, if the sequence number of the video data packet is less than 65000, the flow flag is 1, if the sequence number of the video data packet is greater than 65000, the flow flag is 0;
[0134] Sorting according to the stream flag bit includes:
[0135] If the stream flags are the same, the video data packets are sorted according to the actual sequence numbers; and,
[0136] If the flow flags are different, then for the sequence number with a flow flag of 0, its value is assigned to a 32-bit integer variable x; for the sequence number with a flow flag of 1, if the sequence number is greater than a preset sequence number threshold, its value is assigned to a 32-bit integer variable y1, wherein y1=sequence number value+65536; if the sequence number is less than the sequence number threshold, its value is assigned to a 32-bit integer variable y2, wherein y2 is the sequence number value.
[0137] In the embodiment of the present invention, the packet number in the streaming media protocol is only 2 bytes, and the maximum is 65535. If it exceeds, it will generally start from 0. In order to compare the size of the packet number, it is necessary to redefine the packet size comparison function. By setting a new stream flag, the size of the data packet can be sorted. After re-sorting, the video media data packet is automatically ordered in the memory structure, and the memory structure is recorded as DataSet.
[0138] The step of performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded includes:
[0139] When it is detected that the data packet of the sorted video data to be played contains a mark of the start or end of a frame, the sequence number of the data packet is recorded through an array;
[0140] When the number of the sequence numbers is greater than a preset sequence number threshold, the sequence numbers are sorted;
[0141] Obtain media information between two adjacent data packets with sequence numbers. If the media information is continuous without packet loss, remove the video encapsulation layer to obtain the raw code stream. If the media information is discontinuous, discard the discontinuous video frames and advance the sequence number by one.
[0142] The raw code stream is put into the queue to be decoded.
[0143] The frame splicing operation in this step can effectively solve the problem of screen distortion during video playback caused by packet loss and disorder.
[0144] The playing of the YUV decoded data according to the preset video playing strategy and frame dropping strategy includes:
[0145] Obtaining a time difference mean of the timestamp information;
[0146] Get the number of video frames to be decoded in the queue to be decoded. If N is less than N1, sleep and do not decode the video frame. If N is greater than N6, the queue to be decoded discards the frames to be decoded until the next key frame, and re-detects N. The sleep time T of the sleep is T1-T0-deltaTime, T1 is the time difference mean of the timestamp information, T0 is the decoding time of the current frame, deltaTime is related to N, and
[0147] N1≤N <N2:deltaTime=2
[0148] N2≤N <N3:deltaTime=4
[0149] N3≤N <N4:deltaTime=8
[0150] N4≤N <N5:deltaTime=10
[0151] N5≤N:deltaTime=15
[0152] Wherein, N is the number of frames to be decoded in the queue to be decoded, N1=5, N2=2N1, N3=3N1, N4=4N1, N5=6N1, N6=10N1.
[0153] In the embodiment of the present invention, the video data to be played is sorted by a preset data structure to obtain the sorted video data to be played; the incomplete frames are removed to perform frame splicing operation on the sorted video data to be played to obtain complete video data to be played, and the video data to be played is put into a queue to be decoded; the complete video data to be played is parsed to obtain video encoding data and timestamp information; the frames to be decoded are obtained from the video encoding data, the frames to be decoded are decoded to obtain YUV decoding data, and the decoding time of the decoding is recorded; the YUV decoding data is played according to the preset video playback strategy and frame loss strategy. It effectively adapts to the problem of video playback freeze caused by network jitter.
[0154] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor (processor) 301, a communication interface (Communications Interface) 302, a memory (memory) 303 and a communication bus 304, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus. The processor can call the logic instructions in the memory to execute the method of video stream playback, which includes: sorting the video data to be played through a preset data structure to obtain the sorted video data to be played; performing frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and putting the video data to be played into a queue to be decoded; parsing the complete video data to be played to obtain video encoding data and timestamp information; obtaining the frame to be decoded from the video encoding data, decoding the frame to be decoded to obtain YUV decoding data, and recording the decoding time of the decoding; playing the YUV decoding data according to the preset video playback strategy and frame loss strategy, the video playback strategy is related to the length of the queue for caching the frame to be decoded, the time difference mean of the timestamp information, the preset dynamic sleep adjustment time variable deltaTime, and the decoding time.
[0155] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0156] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the video stream playback method provided by the above-mentioned method embodiments, the method including: sorting the video data to be played through a preset data structure to obtain the sorted video data to be played; performing frame splicing operations on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded; parsing the complete video data to be played to obtain video encoding data and timestamp information; obtaining frames to be decoded from the video encoding data, decoding the frames to be decoded to obtain YUV decoded data, and recording the decoding time of the decoding; playing the YUV decoded data according to a preset video playback strategy and frame loss strategy, and the video playback strategy is related to the length of the queue for caching the frames to be decoded, the time difference mean of the timestamp information, the preset dynamic sleep adjustment time variable deltaTime, and the decoding time.
[0157] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the video stream playback method provided by the above-mentioned embodiments, the method comprising: sorting the video data to be played through a preset data structure to obtain the sorted video data to be played; performing frame splicing operations on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded; parsing the complete video data to be played to obtain video encoding data and timestamp information; obtaining frames to be decoded from the video encoding data, decoding the frames to be decoded to obtain YUV decoded data, and recording the decoding time of the decoding; playing the YUV decoded data according to a preset video playback strategy and frame loss strategy, the video playback strategy being related to the length of the queue for caching the frames to be decoded, the time difference mean of the timestamp information, the preset dynamic sleep adjustment time variable deltaTime, and the decoding time.
[0158] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0159] The above is only a partial implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for playing a video stream, characterized in that: The method comprises: Sorting the video data to be played through a preset data structure, and obtaining the sorted video data to be played; Performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and putting the video data to be played into a queue to be decoded; Parsing the complete video data to be played to obtain video encoding data and timestamp information; Obtaining a frame to be decoded from the video encoding data, decoding the frame to be decoded to obtain YUV decoded data, and recording a decoding time of the decoding; Playing the YUV decoded data according to the preset video playback strategy and frame loss strategy specifically includes: Obtaining a time difference mean of the timestamp information; Get the number of video frames to be decoded in the queue to be decoded. If N is less than N1, sleep and do not decode the video frame. If N is greater than N6, the queue to be decoded discards the frames to be decoded until the next key frame, and re-detects N. The sleep time T of the sleep is T1-T0-deltaTime, T1 is the time difference mean of the timestamp information, T0 is the decoding time of the current frame, deltaTime is related to N, and N1≤N <N2:deltaTime=2 N2≤N <N3:deltaTime=4 N3≤N <N4:deltaTime=8 N4≤N <N5:deltaTime=10 N5≤N:deltaTime=15 Wherein, N is the number of frames to be decoded in the queue to be decoded, N1=5, N2=2N1, N3=3N1, N4=4N1, N5=6N1, N6=10N1.
2. The method according to claim 1, characterized in that The method of sorting the video data to be played by using a preset data structure and obtaining the sorted video data to be played includes: Add a flow flag flow_sign to each video data packet in the video data to be played, if the sequence number of the video data packet is less than 65000, the flow flag is 1, if the sequence number of the video data packet is greater than 65000, the flow flag is 0; Sorting according to the stream flag bit includes: If the stream flags are the same, the video data packets are sorted according to the actual sequence numbers; and, If the flow flags are different, then for the sequence number with a flow flag of 0, its value is assigned to a 32-bit integer variable x; for the sequence number with a flow flag of 1, if the sequence number is greater than a preset sequence number threshold, its value is assigned to a 32-bit integer variable y1, wherein y1=sequence number value+65536; if the sequence number is less than the sequence number threshold, its value is assigned to a 32-bit integer variable y2, wherein y2 is the sequence number value.
3. The method according to claim 1, characterized in that The step of performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded includes: When it is detected that the data packet of the sorted video data to be played contains a mark of the start or end of a frame, the sequence number of the data packet is recorded through an array; When the number of the sequence numbers is greater than a preset sequence number threshold, the sequence numbers are sorted; Obtain media information between two adjacent data packets with sequence numbers. If the media information is continuous without packet loss, remove the video encapsulation layer to obtain the raw code stream. If the media information is discontinuous, discard the discontinuous video frames and advance the sequence number by one. The raw code stream is put into the queue to be decoded.
4. A system for playing video streams, characterized in that: The system comprises: A module for obtaining the sorted video data to be played, which is used to sort the video data to be played according to a preset data structure and obtain the sorted video data to be played; A complete to-be-played video data acquisition module is used to perform a frame splicing operation on the sorted to-be-played video data by removing incomplete frames, to acquire complete to-be-played video data, and to put the to-be-played video data into a queue to be decoded; A video encoding and timestamp information acquisition module, used to parse the complete video data to be played and acquire video encoding data and timestamp information; A decoding time acquisition module, used to acquire a frame to be decoded from the video encoding data, decode the frame to be decoded to acquire YUV decoded data, and record the decoding time of the decoding; The video playback module is used to play the YUV decoded data according to the preset video playback strategy and frame loss strategy, specifically including: Obtaining a time difference mean of the timestamp information; Get the number of video frames to be decoded in the queue to be decoded. If N is less than N1, sleep and do not decode the video frame. If N is greater than N6, the queue to be decoded discards the frames to be decoded until the next key frame, and re-detects N. The sleep time T of the sleep is T1-T0-deltaTime, T1 is the time difference mean of the timestamp information, T0 is the decoding time of the current frame, deltaTime is related to N, and N1≤N <N2:deltaTime=2 N2≤N <N3:deltaTime=4 N3≤N <N4:deltaTime=8 N4≤N <N5:deltaTime=10 N5≤N:deltaTime=15 Wherein, N is the number of frames to be decoded in the queue to be decoded, N1=5, N2=2N1, N3=3N1, N4=4N1, N5=6N1, N6=10N1.
5. The system according to claim 4, characterized in that The method of sorting the video data to be played by using a preset data structure and obtaining the sorted video data to be played includes: Add a flow flag flow_sign to each video data packet in the video data to be played, if the sequence number of the video data packet is less than 65000, the flow flag is 1, if the sequence number of the video data packet is greater than 65000, the flow flag is 0; Sorting according to the stream flag bit includes: If the stream flags are the same, the video data packets are sorted according to the actual sequence numbers; and, If the flow flags are different, then for the sequence number with a flow flag of 0, its value is assigned to a 32-bit integer variable x; for the sequence number with a flow flag of 1, if the sequence number is greater than a preset sequence number threshold, its value is assigned to a 32-bit integer variable y1, wherein y1=sequence number value+65536; if the sequence number is less than the sequence number threshold, its value is assigned to a 32-bit integer variable y2, wherein y2 is the sequence number value.
6. The system according to claim 4, characterized in that The step of performing a frame splicing operation on the sorted video data to be played by removing incomplete frames to obtain complete video data to be played, and placing the video data to be played into a queue to be decoded includes: When it is detected that the data packet of the sorted video data to be played contains a mark of the start or end of a frame, the sequence number of the data packet is recorded through an array; When the number of the sequence numbers is greater than a preset sequence number threshold, the sequence numbers are sorted; Obtain media information between two adjacent data packets with sequence numbers. If the media information is continuous without packet loss, remove the video encapsulation layer to obtain the raw code stream. If the media information is discontinuous, discard the discontinuous video frames and advance the sequence number by one. The raw code stream is put into the queue to be decoded.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for playing a video stream as described in any one of claims 1 to 3 is implemented.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for playing a video stream as described in any one of claims 1 to 3 is implemented.
