Video stream anti-jitter method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a video stream anti-jitter method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, when the video gear switching of the server is detected, the video transcoding delay is determined according to the video stream data packet received from the server, and the target jitter buffer data length is determined according to the video transcoding delay and the current video jitter buffer data length; when the currently played video stream and the currently played audio stream meet the synchronization condition, the video jitter buffer data length and the audio jitter buffer data length can be adjusted according to the target jitter buffer data length, and when the video gear is switched, the synchronization of audio and video playing can be kept and the anti-jitter capability of video stream playing can be improved. The video playing fluency is improved, the video playing lagging is reduced, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video stream anti-jitter method, device, equipment, storage medium and product. Background Art

[0002] In real-time communication systems such as live video broadcasting, there are extremely high requirements for the stability and reliability of data transmission. Packet loss and jitter are inevitable in real-time communication, which will lead to data transmission loss or dislocation, affecting communication quality and user experience. In weak network scenarios, switching transcoding gears is a common operation, that is, the server dynamically adjusts the quality of the video stream according to the current network conditions to ensure smoothness.

[0003] However, when switching transcoding gears, it is usually accompanied by a high transcoding delay. The video jitter buffer of the user end receiving the video stream cannot receive valid data within a certain period of time, which makes the data buffered in the video jitter buffer shorter, resulting in a decrease in anti-jitter capability and even video freeze, affecting the user experience. Summary of the invention

[0004] The embodiments of the present application provide a video stream anti-jitter method, apparatus, device, storage medium and product to solve the technical problem in the related art that the anti-jitter capability of video stream playback decreases when the transcoding gear is switched, resulting in video freeze and affecting the user experience. The anti-jitter capability of video stream playback when the transcoding gear is switched can be effectively improved, the video freeze can be reduced, and the user experience can be improved.

[0005] In a first aspect, an embodiment of the present application provides a video stream anti-jitter method, comprising:

[0006] In the case where a video gear switching occurs on the server, determining a video transcoding delay according to a video stream data packet received from the server;

[0007] Determine a current video jitter buffer data length, and determine a target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length;

[0008] Determine the synchronization between the currently playing video stream and the currently playing audio stream;

[0009] When the synchronization condition satisfies the synchronization condition, the video jitter buffer data length and the audio jitter buffer data length are adjusted according to the target jitter buffer data length.

[0010] In a second aspect, an embodiment of the present application provides a video stream anti-jitter device, including a delay determination module, a buffer determination module, a synchronization determination module and a buffer adjustment module, wherein:

[0011] The delay determination module is configured to determine the video transcoding delay according to the video stream data packet received from the server when detecting that the video gear switching occurs in the server;

[0012] The buffer determination module is configured to determine a current video jitter buffer data length, and determine a target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length;

[0013] The synchronization determination module is configured to determine the synchronization status of the currently playing video stream and the currently playing audio stream;

[0014] The buffer adjustment module is configured to adjust the video jitter buffer data length and the audio jitter buffer data length according to the target jitter buffer data length when the synchronization condition satisfies the synchronization condition.

[0015] In a third aspect, an embodiment of the present application provides a video stream anti-shake device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the video stream anti-jitter method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the video stream anti-jitter method as described in the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the video stream anti-shake method as described in the first aspect.

[0020] The embodiment of the present application determines the video transcoding delay according to the video stream data packet received from the server when detecting that the video gear switch occurs on the server, and determines the target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length. When the currently playing video stream and the currently playing audio stream meet the synchronization condition, the video jitter buffer data length and the audio jitter buffer data length can be adjusted according to the target jitter buffer data length. When the video gear switch occurs, the synchronization of audio and video playback can be maintained and the anti-jitter capability of video stream playback can be improved, the smoothness of video playback can be improved, the video playback jam can be reduced, and the user experience can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a video stream anti-jitter method provided by an embodiment of the present application;

[0022] Figure 2 is a flow chart of another video stream anti-jitter method provided in an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of a video transcoding delay determination process provided by an embodiment of the present application;

[0024] Figure 4 It is a structural schematic diagram of a video stream anti-jitter device provided in an embodiment of the present application;

[0025] Figure 5 It is a structural schematic diagram of a video stream anti-jitter device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0027] The video stream anti-jitter method provided in this application can be applied to real-time communication scenarios such as live video broadcasting. It aims to maintain the synchronization of audio and video playback while lengthening the audio and video buffer data when the video gear is switched, thereby improving the anti-jitter capability of video stream playback and improving the smoothness of video playback.

[0028] In existing real-time communication systems, it is common to switch video gears in a weak network environment. When switching transcoding gears, since video transcoding is usually accompanied by a high transcoding delay, the video jitter buffer (JitterBuffer) used to cache the video stream in the user end (such as the audience end in live video broadcast) cannot receive valid data for a period of time. During this period, the consumption of buffered data by the user end will cause the data cached in the video jitter buffer to become shorter, resulting in a decrease in the jitter resistance of the video stream, and even causing video freezes, affecting the user experience. Based on this, a video stream anti-jitter method of an embodiment of the present application is provided to solve the technical problem that the anti-jitter ability of video stream playback decreases when the transcoding gear is switched in the existing real-time communication system, causing video freezes, and affecting the user experience.

[0029] Figure 1 A flowchart of a video stream anti-jitter method provided in an embodiment of the present application is given. The video stream anti-jitter method provided in an embodiment of the present application can be executed by a video stream anti-jitter device, which can be implemented by hardware and / or software and integrated in a video stream anti-jitter device (for example, the audience end in a live video broadcast, the video receiving end in a video call, which can be a mobile phone, tablet, computer, etc.).

[0030] The following description is made by taking the video stream anti-jitter device executing the video stream anti-jitter method as an example. Figure 1 , the video stream anti-shake method includes:

[0031] S110: When it is detected that a video gear switch occurs on the server, a video transcoding delay is determined according to a video stream data packet received from the server.

[0032] Exemplarily, the server provided by the present solution can be used to receive a source video stream sent by a sending end (for example, the host end in a live video broadcast, the video sending end in a video call), and send the source video stream to a video stream anti-jitter device, or transcode the source video stream (for example, different gears correspond to different bit rates and / or resolutions) to obtain a transcoded video stream, and then send the transcoded video stream to the video stream anti-jitter device.

[0033] In one embodiment, the server provided by the present solution may include a first server (e.g., a host-end server), a transcoding server, and a second server (e.g., an audience-end server). The first server may be used to receive a source video stream and an audio stream sent by a transmitter, and the first server may send the source video stream and the audio stream to the second server, so that the second server can send the source video stream and the audio stream to a video stream anti-shake device. When the first server needs to switch the video gear, it may stop sending the source video stream to the second server, and send the source video stream to the transcoding server, so that the transcoding server can transcode the source video stream to obtain a transcoded video stream and send it to the first server, and the first server may send the transcoded video stream and the audio stream to the second server, so that the second server can send the transcoded video stream and the audio stream to the video stream anti-shake device. Since the transcoding of the source video stream takes a certain amount of time, video transcoding delay will occur. There will be a certain time interval between the time when the source video stream is sent from the first server to the transcoding server for transcoding and the time when the transcoded video stream is received. As a result, after the first server stops sending the source video stream to the second server, it will take a certain amount of time to start sending the transcoded video stream to the video stream anti-shake device.

[0034] The video stream anti-jitter device provided by this solution is configured with a video jitter buffer for caching video streams and an audio jitter buffer for caching audio streams by users. The video stream (source stream video stream or transcoded video stream) sent by the server will be cached in the video jitter buffer, and the audio stream sent by the server will be cached in the audio jitter buffer. The player in the video stream anti-jitter device will obtain the video frames corresponding to the video stream from the video jitter buffer, and obtain the audio frames corresponding to the audio stream from the audio jitter buffer for playback. For example, the server sends the source stream video stream and audio stream to the video stream anti-jitter device, and the video stream anti-jitter device caches the latest received source stream video stream into the video jitter buffer, caches the audio stream into the audio jitter buffer, and extracts the previously received and cached source stream video stream from the video jitter buffer, and extracts the previously received and cached audio stream from the audio jitter buffer for playback. When the video gear is switched on the server, due to the delay of video transcoding on the server, there will be a certain time interval from the stopping of the source video stream to the sending of the transcoded video stream. During this waiting time, the video stream anti-jitter device will continue to extract the source video stream from the video jitter buffer for playback, which will cause the video stream data buffered in the video jitter buffer to become shorter. After receiving the transcoded video stream, since the extraction of the video stream from the video jitter buffer for playback is a continuous process, the video jitter buffer will be maintained at the shortened length, resulting in a decrease in the anti-jitter capability of the video playback, which is prone to cause the video playback to freeze. This solution restores the anti-jitter capability of the video playback by synchronously lengthening the length of the video stream data and the audio stream data in the video jitter buffer when the video gear is switched on the server, thereby reducing the video playback freeze and improving the user experience.

[0035] Exemplarily, when it is detected that a video gear switch occurs on the server, the video transcoding delay can be determined based on the video stream data packet received from the server. In one embodiment, when the server provided by the present solution switches the video gear from the video source stream to other gears, the video transcoding delay generated by transcoding the source stream video stream to the transcoded video stream can be recorded, and the video transcoding delay can be added to the data packet corresponding to the transcoded video stream (for example, the setting field of the video data packet (for example, the first data packet) of each transcoded video frame in the transcoded video stream), and the video stream anti-jitter device can determine the video transcoding delay caused by the video gear switch occurring on the server from the data packet corresponding to the received transcoded video stream.

[0036] S120: Determine the current video jitter buffer data length, and determine the target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length.

[0037] The data length provided by this solution can be understood as the time length of the corresponding data. Exemplarily, the time length of the video stream cached in the current video jitter buffer is determined, and the time length is used as the current video jitter buffer data length. Optionally, the maximum timestamp minus the timestamp corresponding to each video frame in the video jitter buffer can be subtracted, and the playback time corresponding to a video frame can be added to obtain the current video jitter buffer data length.

[0038] In one embodiment, after determining the current video jitter buffer data length, the target jitter buffer data length may be determined according to the video transcoding delay and the current video jitter buffer data length. Optionally, the video transcoding delay and the current video jitter buffer data length may be added, and the addition result may be used as the target jitter buffer data length, or the addition result and a preset proportional coefficient (e.g., a proportional coefficient less than 1) may be used as the target jitter buffer data length, or the difference between the addition result and a preset adjustment parameter coefficient may be used as the target jitter buffer data length.

[0039] S130: Determine the synchronization status of the currently playing video stream and the currently playing audio stream.

[0040] Exemplarily, determine the synchronization status of the player's currently playing video stream and audio stream, where the synchronization status can be represented by the time difference between the video stream playback and the audio stream playback, for example, subtracting the timestamp corresponding to the currently playing video frame from the timestamp corresponding to the audio frame, and taking the absolute value of the subtraction result as the synchronization status of the currently playing video stream and the currently playing audio stream.

[0041] S140: When the synchronization condition satisfies the synchronization condition, the video jitter buffer data length and the audio jitter buffer data length are adjusted according to the target jitter buffer data length.

[0042] Exemplarily, it is determined whether the synchronization status of the currently playing video stream and the currently playing audio stream meets the preset synchronization condition. For example, when the audio and video playback time difference corresponding to the synchronization status is within a preset time range, it is considered that the synchronization status meets the synchronization condition.

[0043] In one embodiment, when it is determined that the synchronization condition meets the synchronization condition, the video jitter buffer data length and the audio jitter buffer data length may be adjusted according to the target jitter buffer data length. For example, the video jitter buffer data length and the audio jitter buffer data length may be synchronously lengthened according to the target jitter buffer data length until the video jitter buffer data length is lengthened to be the same as the target jitter buffer data length.

[0044] Optionally, the adjustment of the video jitter buffer data length and the audio jitter buffer data length can be achieved by adjusting the video stream playback speed and the audio stream playback speed (the video stream playback speed and the audio stream playback speed are adjusted synchronously). For example, the video stream playback speed and the audio stream playback speed are reduced synchronously. In the process of reducing the video stream playback speed and the audio stream playback speed, the speed at which the player obtains video frames from the video jitter buffer and obtains audio frames from the audio jitter buffer for playback is slower than the speed at which the video jitter buffer caches new video streams and new audio streams. At this time, the length of the video jitter buffer data cached in the video jitter buffer and the length of the audio jitter buffer data cached in the audio jitter buffer will gradually lengthen.

[0045] In the above, when it is detected that the video gear is switched on the server, the video transcoding delay is determined according to the video stream data packet received from the server, and the target jitter buffer data length is determined according to the video transcoding delay and the current video jitter buffer data length. When the currently playing video stream and the currently playing audio stream meet the synchronization conditions, the video jitter buffer data length and the audio jitter buffer data length can be adjusted according to the target jitter buffer data length. When the video gear is switched, the synchronization of audio and video playback can be maintained and the anti-jitter capability of video stream playback can be improved, the smoothness of video playback can be improved, the video playback jam can be reduced, and the user experience can be improved.

[0046] Based on the above embodiments, Figure 2 A flowchart of another video stream anti-shake method provided in an embodiment of the present application is given. The video stream anti-shake method is a specific embodiment of the above-mentioned video stream anti-shake method. Figure 2 , the video stream anti-shake method includes:

[0047] S210: When it is detected that a video gear switch occurs on the server, a video transcoding delay is determined according to a video stream data packet received from the server.

[0048] In a possible embodiment, the video stream anti-jitter method provided by the present solution determines the video transcoding delay based on the video stream data packet received from the server, and the video transcoding delay can be determined based on the transcoding delay field of the video stream data packet received from the server, wherein the video transcoding delay is determined by the server based on the first addition time of the transcoded stream image group and the second addition time of the corresponding source stream image group, and recorded in the transcoding delay field of the video stream data packet corresponding to the transcoded stream image group.

[0049] Exemplarily, when a video gear switch is detected on the server, the video transcoding delay can be determined in a preset transcoding delay field of the video stream data packet. The video transcoding delay can be determined by the server when performing video stream transcoding and recorded in the transcoding delay field of the video stream data packet.

[0050] In one embodiment, when the server provided by the present solution transcodes a source stream video stream into a transcoded video stream, the server can determine the first addition time of the transcoded stream image group and the second addition time of the corresponding source stream image group, and determine the video transcoding delay based on the first addition time of the transcoded stream image group and the second addition time of the corresponding source stream image group, for example, taking the difference between the first addition time and the second addition time as the video transcoding delay.

[0051] Among them, the second adding time of the source stream image group can be understood as the time when the source stream image group corresponding to the source stream video stream is received (for example, the time when the source stream image group is added to the preset source stream image group list), and the first adding time of the transcoded stream image group can be understood as the time when the transcoded stream image group corresponding to the transcoded video stream is generated (for example, the time when the transcoded stream image group is added to the preset transcoded stream image group list).

[0052] Optionally, after determining the video transcoding delay corresponding to each transcoded stream image group, the video transcoding delay can be recorded in the transcoding delay field of the video stream data packet corresponding to the transcoded video stream and sent to the video stream anti-jitter. This solution determines the video transcoding delay and records it in the transcoding delay field of the video stream data packet based on the first addition time of the transcoded stream image group and the second addition time of the corresponding source stream image group by the server. It can accurately determine the time taken to transcode from the source stream video stream to the transcoded video stream, accurately adjust the video jitter buffer data length and the audio jitter buffer data length, timely and accurately restore the video stream anti-jitter capability, and improve the video stream playback effect.

[0053] In a possible embodiment, the video transcoding delay can be obtained by comparing the timestamp information of the corresponding group of pictures (GOP) in the source video stream and the transcoded video stream and calculating the transcoding time difference of each group of pictures. Figure 3 As shown in a schematic diagram of a video transcoding delay determination process provided, the server provided by the present solution determines the video transcoding delay according to the first addition time of the transcoded stream image group and the second addition time of the corresponding source stream image group, which can be:

[0054] S201: Traverse the transcoding stream image group list and the source stream image group list to determine matching transcoding stream image groups and source stream image groups.

[0055] S202: Determine a first adding time when the transcoded key frame in the transcoded stream image group is added to the transcoded video stream queue, and a second adding time when the source stream key frame in the source stream image group is added to the source stream video stream queue.

[0056] S203: Determine a video transcoding delay according to the first adding time and the second adding time.

[0057] Exemplarily, a transcoded stream image group list for storing transcoded stream image groups and related information in a transcoded video stream, and a source stream image group list for storing source stream image groups and related information in a source stream video stream are traversed (for example, two reverse iterators are used to traverse the GOP information of the transcoded stream image group list and the source stream image group list respectively), and matching transcoded stream image groups and source stream image groups are found between the transcoded stream image group list and the source stream image group list.

[0058] Optionally, the matching transcoded stream image group and source stream image group can be image groups with the same key frame (I frame) sequence number (the image group includes key frames and predicted frames (B frames / P frames)), that is, the frame sequence numbers of the transcoded key frames in the matching transcoded stream image group and the source stream key frames in the source stream image group are the same. Optionally, when the transcoded stream image group and the source stream image group have participated in the determination of the video transcoding delay, the transcoded stream image group and the source stream image group can be marked as processed (for example, marked as isFetchData), and when traversing the GOP data of the transcoded stream image group list and the source stream image group list, if the transcoded stream image group and the source stream image group are marked as processed, it means that the data has been processed and the corresponding GOP data can be skipped.

[0059] In one embodiment, when a matching transcoded stream image group and a source stream image group are found, a first adding time when a transcoded key frame in the transcoded stream image group is added to a transcoded video stream queue and a second adding time when a source stream key frame in the matching source stream image group is added to a source stream video stream queue can be determined, and a video transcoding delay can be determined based on the first adding time and the second adding time. Optionally, a video transcoding delay corresponding to a transcoded stream image group can be the difference between the first adding time and the second adding time.

[0060] Among them, when a matching transcoding stream image group and source stream image group are found, the difference between the corresponding first addition time and the second addition time can be directly used as the video transcoding delay, and when multiple matching transcoding stream image groups and source stream image groups are found, the average video transcoding delay corresponding to multiple sets of matching transcoding stream image groups and source stream image groups can be used as the video transcoding delay for the current video gear switching. Based on this, this scheme determines the video transcoding delay according to the first addition time and the second addition time, which can be: subtracting the first addition time corresponding to multiple transcoding stream image groups from the second addition time corresponding to the matching source stream image group, and using the subtraction result as the transcoding delay of the transcoding stream image group; determining the average value of multiple transcoding delays, and using the average value as the video transcoding delay.

[0061] Exemplarily, for each matching transcoded stream image group and source stream image group, the first addition time corresponding to the transcoded stream image group is subtracted from the second addition time corresponding to the matching source stream image group, and the subtraction result is used as the transcoding delay of the transcoded stream image group. After determining the transcoding delay corresponding to each matching transcoded stream image group and source stream image group, the average value of these multiple transcoding delays can be determined, and the average value is used as the video transcoding delay. This solution accurately determines the video transcoding delay by calculating the average transcoding delay corresponding to each matching transcoded stream image group and source stream image group, and can accurately determine the target jitter buffer data length, and accurately restore the video stream anti-jitter capability.

[0062] For example, the first server searches for the source stream image group list of the source stream video stream and the transcoded stream image group list of the transcoded video stream obtained after transcoding from the transcoded stream list and the source stream list, and uses two reverse iterators to traverse the image group information of the transcoded stream image group list and the source stream image group list respectively. For each image group information of the transcoded stream image group list, the image group information of the corresponding source stream image group list is searched, for example, by comparing the current key frame sequence numbers of the two to determine the matching image group information. When a matching key frame is found, the addition time addTime of the two is further compared, and the transcoding delay delay (i.e., the transcoded stream GOP addition time minus the original stream GOP addition time) is calculated:

[0063] delay=codecGopInfo.addTime-originGopInfo.addTime

[0064] Wherein, delay is the transcoding delay corresponding to a transcoding stream image group, codecGopInfo.addTime is the first adding time of the transcoding key frame added to the transcoding video stream queue, and originGopInfo.addTime is the second adding time of the source stream key frame added to the source stream video stream queue.

[0065] The transcoding delay of each pair of transcoded stream image groups and source stream image groups is accumulated to the transcoding delay accumulation information delayTotal, and the counter delayCount is increased. After the traversal is completed, if multiple image groups are matched, the average delay avgDelay = delayTotal / delayCount is calculated and used as the final video transcoding delay avgDelay:

[0066] avgDelay=delayTotal / delayCount

[0067] This solution determines the matching transcoded stream image group and source stream image group by traversing the transcoded stream image group list and the source stream image group list, and accurately determines the video transcoding delay based on the first addition time of the transcoded key frame in the transcoded stream image group to the transcoded video stream queue, and the second addition time of the source stream key frame in the source stream image group to the source stream video stream queue. It can effectively monitor the delay of different users' video streams after transcoding in real time, accurately determine the target jitter buffer data length, accurately restore the video stream anti-jitter capability, and improve the video stream playback effect.

[0068] In a possible embodiment, the video stream anti-jitter method provided by the present solution further includes: receiving a first video stream data packet from the server, and determining the first video stream type based on the code stream type information of the first video stream data packet before determining the video transcoding delay based on the video stream data packet received from the server; when the first video stream type changes relative to the second video stream type corresponding to the second video stream data packet received previously, determining that a video gear switch occurs on the server.

[0069] In one embodiment, a server (e.g., a first server or a transcoding server) may add stream type information in a preset field of a video stream data packet according to the video stream type (the video stream type includes a source stream video stream and a transcoded video stream), and may determine the video stream type of the video stream according to the stream type information recorded in the preset field of the video stream data packet.

[0070] Exemplarily, the video stream anti-jitter device receives a first video stream data packet from a server in real time, and determines the first video stream type according to the code stream type information recorded in a preset field of the first video stream data packet. The first video stream type is compared with the second video stream type corresponding to the second video stream data packet received previously to determine whether the first video stream type has changed relative to the second video stream type. When the first video stream type has changed relative to the second video stream type, it can be determined that the server has switched the video gear. The second video stream data packet can be a video stream data packet received on the first video stream data packet.

[0071] Optionally, the video gear switching of the server provided by the present solution may be a video gear switching from a source video stream to a video stream of another gear. In this case, when the video stream type corresponding to the first video stream type is a transcoded video stream, and the video stream type corresponding to the second video stream type is a source video stream, it can be determined that the video gear switching from the source video stream to a video stream of another gear occurs on the server. For example, the bitstream type information can be recorded in a gear field preset in a video stream data packet (e.g., a subSid field), wherein a first preset value (e.g., 0) indicates that the video stream type is a source video stream, and other preset values ​​(e.g., 1, 2, 3, etc.) indicate that the video stream type is a transcoded video stream, and when the value corresponding to the bitstream type information corresponding to the first video stream data is a non-first preset value, and the value corresponding to the bitstream type information corresponding to the second video stream data is a first preset value, it can be determined that the video gear switching from the source video stream to a video stream of another gear occurs on the server.

[0072] For example, the video stream data packet protocol contains multiple fields for describing the meta information and data of the video stream, wherein the frameType field can be used to mark the frame type, such as I frame (key frame), and the subSid field can indicate the video stream type, wherein subSid=0 indicates that the video stream type is the source stream video stream, and other values ​​indicate that the video stream type is the corresponding transcoded video stream. The transcodeDelay field can indicate the video transcoding delay, recording the delay of each video packet from the original stream to the transcoded stream. During the video stream transmission process, once the source stream is switched, the calculation of the transcoding delay will first be performed in real time on the first server (videoServer). After the first server sends the source stream video stream to the transcoding server (codecServer), the transcoding server will return the processed transcoded video stream. After the first server receives the transcoded video stream, it will first calculate the video transcoding delay avgDelay, and insert it into the video stream data packet of the returned transcoded video stream through the transcodeDelay field, and then send it to the video stream anti-shake device through the second server. When the video stream anti-jitter device receives a video stream data packet, if it detects that the subSid field switches from 0 to other values, that is, boolisTranscode = (subsid! = 0), it can be determined that a video gear switch has occurred. At this time, the video stream anti-jitter device will trigger the jitter buffer adjustment strategy for the video gear switch. Among them, the video transcoding delay field transcodeDelay can be read from the I frame video packet header. When transcodeDelay corresponds to the transcoding switching, the first server stops sending the source video stream (Time2) to the time interval when the videoServer sends the transcoded stream (Time3), that is, the length of the buffer data consumed by the video stream anti-jitter device. The delay field can provide accurate transcoding delay information, thereby accurately determining the target jitter buffer data length and restoring the video stream anti-jitter capability.

[0073] S220: Determine the current video jitter buffer data length, and determine the target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length.

[0074] In a possible embodiment, the video stream anti-jitter method provided by the present solution determines the target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length, which can be: adding the video transcoding delay and the current video jitter buffer data length, and determining the addition result as the initial jitter buffer data length; subtracting the initial jitter buffer data length from the preset jitter size, and determining the subtraction result as the target jitter buffer data length. For example, the target jitter buffer data length can be determined by the following formula:

[0075] targetjitter=originJitter-tempJitter

[0076] originJitter=newJitterLen+transcodeDelay

[0077] Wherein, targetJitter is the target jitter buffer data length, transcodeDelay is the video transcoding delay, new]itterLen is the current video jitter buffer data length, originjitter is the initial jitter buffer data length, and tempJitter is the preset jitter size. Optionally, the preset jitter size can range from 1% to 10%, and the preset jitter size value is 5%.

[0078] Among them, the preset jitter size can be used to compensate for the jitter of the buffer data length that will exist even when the transcoding gear is not switched. In the process of lengthening the video jitter buffer data length, the consumption of transcoded video stream buffer data caused by network jitter will exist even if the transcoding gear is not switched. Therefore, there is no need to compensate for this part of the consumed buffer data, and the target jitter buffer data length can be determined more accurately.

[0079] S230: Determine the video playing timestamp of the currently playing video stream and the audio playing timestamp of the currently playing audio stream.

[0080] S240: Determine the audio and video playback time difference according to the video playback timestamp and the audio playback timestamp.

[0081] S250: When the audio and video playback time difference is within a preset time interval, determining that the synchronization status of the currently playing video stream and the currently playing audio stream meets the synchronization condition.

[0082] Exemplarily, after the target jitter buffer data length is reached, it is necessary to determine the synchronization status of the currently playing video stream and the currently playing audio stream. The video playback timestamp of the currently playing video stream of the player and the audio playback timestamp of the currently playing audio stream can be determined, and the absolute value of the difference between the video playback timestamp and the audio playback timestamp is calculated, and the absolute value of the difference is determined as the audio and video playback time difference.

[0083] In one embodiment, the audio and video playback time difference is compared in a preset time interval. When the audio and video playback time difference is within the preset time interval, it can be determined that the synchronization of the currently playing video stream and the currently playing audio stream meets the synchronization condition, and the video jitter buffer data length and the audio jitter buffer data length can be lengthened. The audio and video jitter buffer data length is synchronously lengthened only when the audio and video are played synchronously, thereby achieving smooth switching in the case of transcoding delay and ensuring that the audio and video synchronization is not affected. Optionally, the preset time interval provided by this solution can be set in a time range where the user cannot visually perceive the audio and video difference. For example, the preset time interval can be an audio and video synchronization first gear interval (for example, [-25ms, +100ms]).

[0084] In one embodiment, when the audio and video playback time difference is outside the preset time interval, it can be determined that the synchronization of the currently playing video stream and the currently playing audio stream does not meet the synchronization condition, and it is necessary to wait for the currently playing video stream and the currently playing audio stream to be within the synchronization interval (for example, align the audio and video playback based on the preset audio and video synchronization strategy) before lengthening the video jitter buffer data length and the audio jitter buffer data length. Optionally, the video stream anti-jitter device can make different rate adjustments according to the time difference between audio and video (avdiff, i.e., the difference between the audio playback timestamp audioCapTimestamp and the video playback timestamp videoCapTimestamp) at different gears of audio and video synchronization to adapt to network jitter. In the first gear interval of audio and video synchronization, the jitter buffer data length adjustment based on the video transcoding delay will be triggered, and in other gear intervals, the audio and video synchronization will be adjusted to the first gear first.

[0085] This solution determines the audio and video playback time difference based on the video playback timestamp of the currently playing video stream and the audio playback timestamp of the currently playing audio stream, and determines the synchronization of the audio and video playback based on the audio and video playback time difference and the preset time interval, accurately determines the timing of lengthening the video jitter buffer data length and the audio jitter buffer data length, accurately restores the video stream's anti-jitter capability, and improves the video stream playback effect.

[0086] S260: When the synchronization condition satisfies the synchronization condition, the video playback speed and the audio playback speed are reduced until the current video buffer length reaches the target jitter buffer data length.

[0087] In one embodiment, when the synchronization of the currently playing video stream and the currently playing audio stream meets the synchronization condition, the video playing speed and the audio playing speed can be synchronously reduced according to a set adjustment ratio (e.g., 5%-20%), for example, the video playing speed and the audio playing speed are reduced by 10%, and the video playing speed and the audio playing speed are adjusted to 90%. For example, the adjusted video playing speed and the audio playing speed can be expressed as:

[0088] newVideoRatio=newAudioRatio=e_slowdown_a

[0089] Wherein, newVideoRatio is the adjusted video playback speed, newAudioRatio is the adjusted audio playback speed, and e_slowdown_a is a constant, indicating that the playback speed is slowed down by a%, for example, e_slowdown_10 indicates that the playback speed is slowed down by 10%.

[0090] While the video playback speed and the audio playback speed are reduced, the current video buffer length and the audio buffer length will increase synchronously. When the current video buffer length reaches the target jitter buffer data length, the video playback speed and the audio playback speed will be restored.

[0091] Optionally, to avoid excessively lengthening the audio and video buffer length, after the current video buffer length is lengthened to the target jitter buffer data length for the first time, the target jitter buffer data length can be marked as 0 (targetJitter=0), indicating that the buffer data length lengthening operation has been performed, and the buffer data length lengthening operation will not be performed again unless the file is switched again. When the synchronization of the currently playing video stream and the currently playing audio stream meets the synchronization condition, this solution lengthens the video jitter buffer data length and the audio jitter buffer data length by reducing the video playback speed and the audio playback speed, accurately and effectively restoring the video stream's anti-jitter capability and improving the video stream playback effect.

[0092] In the above, when the video gear switching of the server is detected, the video transcoding delay is determined according to the video stream data packet received from the server, and the target jitter buffer data length is determined according to the video transcoding delay and the current video jitter buffer data length. When the currently playing video stream and the currently playing audio stream meet the synchronization conditions, the video jitter buffer data length and the audio jitter buffer data length can be adjusted according to the target jitter buffer data length. When the video gear switching occurs, the synchronization of audio and video playback can be maintained and the anti-jitter capability of video stream playback can be improved, the smoothness of video playback can be improved, the video playback jamming can be reduced, and the user experience can be improved. At the same time, by reducing the video playback speed and the audio playback speed, the video jitter buffer data length and the audio jitter buffer data length can be lengthened, the anti-jitter capability of the video stream can be accurately and effectively restored, and the video stream playback effect can be improved. In the jitter environment after the transcoding gear switching, it can still maintain good smoothness and stability, reduce the jamming problem caused by the gear switching, and maintain audio and video synchronization during the gear switching process, thereby improving the overall audio-visual experience.

[0093] Figure 4 Schematic diagram of the structure of a video stream anti-shake device provided by an embodiment of the present application. Figure 4 The video stream anti-jitter device includes a delay determination module 41, a buffer determination module 42, a synchronization determination module 43 and a buffer adjustment module 44.

[0094] Among them, the delay determination module 41 is configured to determine the video transcoding delay according to the video stream data packet received from the server when it is detected that the video gear switching occurs in the server; the buffer determination module 42 is configured to determine the current video jitter buffer data length, and determine the target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length; the synchronization determination module 43 is configured to determine the synchronization status of the currently playing video stream and the currently playing audio stream; the buffer adjustment module 44 is configured to adjust the video jitter buffer data length and the audio jitter buffer data length according to the target jitter buffer data length when the synchronization status meets the synchronization conditions.

[0095] In the above, when it is detected that the video gear is switched on the server, the video transcoding delay is determined according to the video stream data packet received from the server, and the target jitter buffer data length is determined according to the video transcoding delay and the current video jitter buffer data length. When the currently playing video stream and the currently playing audio stream meet the synchronization conditions, the video jitter buffer data length and the audio jitter buffer data length can be adjusted according to the target jitter buffer data length. When the video gear is switched, the synchronization of audio and video playback can be maintained and the anti-jitter capability of video stream playback can be improved, the smoothness of video playback can be improved, the video playback jam can be reduced, and the user experience can be improved.

[0096] In a possible embodiment, the video stream anti-jitter system further includes a switching detection module, and the switching detection module is configured as follows:

[0097] Receive a first video stream data packet from a server, and determine the first video stream type according to the code stream type information of the first video stream data packet;

[0098] When the first video stream type changes relative to the second video stream type corresponding to the second video stream data packet received previously, it is determined that a video gear switch occurs on the server.

[0099] In a possible embodiment, the delay determination module 41 determines the video transcoding delay according to the video stream data packet received from the server, and is configured as follows:

[0100] The video transcoding delay is determined according to the transcoding delay field of the video stream data packet received from the server, wherein the video transcoding delay is determined by the server according to the first addition time of the transcoding stream image group and the second addition time of the corresponding source stream image group, and recorded in the transcoding delay field of the video stream data packet corresponding to the transcoding stream image group.

[0101] In a possible embodiment, the server determines the video transcoding delay according to the first adding time of the transcoded stream image group and the second adding time of the corresponding source stream image group, and is configured as follows:

[0102] Traversing the transcoding stream image group list and the source stream image group list to determine matching transcoding stream image groups and source stream image groups;

[0103] Determine a first adding time of adding a transcoded key frame in the transcoded stream image group to a transcoded video stream queue, and a second adding time of adding a source stream key frame in the source stream image group to a source stream video stream queue;

[0104] The video transcoding delay is determined according to the first adding time and the second adding time.

[0105] In a possible embodiment, the server determines the video transcoding delay according to the first adding time and the second adding time, and is configured as follows:

[0106] Subtracting first adding times corresponding to the plurality of transcoded stream image groups from second adding times corresponding to the matched source stream image groups, and using the subtraction results as transcoding delays of the transcoded stream image groups;

[0107] An average of multiple transcoding delays is determined, and the average is used as the video transcoding delay.

[0108] In a possible embodiment, the buffer determination module 42 determines the target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length, and is configured as follows:

[0109] Adding the video transcoding delay and the current video jitter buffer data length, and determining the addition result as the initial jitter buffer data length;

[0110] The initial jitter buffer data length is subtracted from the preset jitter size, and the subtraction result is determined as the target jitter buffer data length.

[0111] In a possible embodiment, the synchronization determination module 43 determines the synchronization status of the currently playing video stream and the currently playing audio stream, and is configured as follows:

[0112] Determine the video playback timestamp of the currently playing video stream and the audio playback timestamp of the currently playing audio stream;

[0113] Determine the audio and video playback time difference according to the video playback timestamp and the audio playback timestamp;

[0114] When the audio and video playback time difference is within a preset time interval, it is determined that the synchronization status of the currently playing video stream and the currently playing audio stream meets the synchronization condition.

[0115] In a possible embodiment, the buffer adjustment module 44 adjusts the video jitter buffer data length and the audio jitter buffer data length according to the target jitter buffer data length, and is configured to: reduce the video playback speed and the audio playback speed until the current video buffer length reaches the target jitter buffer data length.

[0116] It is worth noting that in the embodiment of the above-mentioned video stream anti-shake device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0117] The embodiment of the present application further provides a video stream anti-jitter device, and the video stream anti-jitter device can integrate the video stream anti-jitter apparatus provided in the embodiment of the present application. Figure 5 Schematic diagram of the structure of a video stream anti-shake device provided by an embodiment of the present application. Figure 5 The video stream anti-shake device includes: an input device 53, an output device 54, a memory 52 and one or more processors 51; the memory 52 is used to store one or more programs; when the one or more programs are executed by the one or more processors 51, the one or more processors 51 implement the video stream anti-shake method provided in the above embodiment. The video stream anti-shake device, equipment and computer provided above can be used to execute the video stream anti-shake method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0118] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the video stream anti-jitter method provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiments of the present application, whose computer executable instructions are not limited to the video stream anti-jitter method provided above, can also execute the related operations in the video stream anti-jitter method provided in any embodiment of the present application. The video stream anti-jitter apparatus, device and storage medium provided in the above embodiments can execute the video stream anti-jitter method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the video stream anti-jitter method provided in any embodiment of the present application.

[0119] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the video stream anti-jitter method provided in each embodiment of the present application.

Claims

1. A video stream anti-jitter method, characterized in that: include: In the case where a video gear switching occurs on the server, determining a video transcoding delay according to a video stream data packet received from the server; Determine a current video jitter buffer data length, and determine a target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length; Determine the synchronization between the currently playing video stream and the currently playing audio stream; When the synchronization condition satisfies the synchronization condition, the video jitter buffer data length and the audio jitter buffer data length are adjusted according to the target jitter buffer data length.

2. The video stream anti-jitter method according to claim 1, characterized in that: Before determining the video transcoding delay according to the video stream data packet received from the server, the method further includes: Receive a first video stream data packet from a server, and determine a first video stream type according to the stream type information of the first video stream data packet; When the first video stream type changes relative to the second video stream type corresponding to the second video stream data packet received previously, it is determined that the server has switched the video gear.

3. The video stream anti-shake method according to claim 1, characterized in that: The determining of the video transcoding delay according to the video stream data packet received from the server comprises: The video transcoding delay is determined according to the transcoding delay field of the video stream data packet received from the server, wherein the video transcoding delay is determined by the server according to the first addition time of the transcoded stream image group and the second addition time of the corresponding source stream image group, and recorded in the transcoding delay field of the video stream data packet corresponding to the transcoded stream image group.

4. The video stream anti-shake method according to claim 3, characterized in that: The server determines the video transcoding delay according to the first adding time of the transcoded stream image group and the second adding time of the corresponding source stream image group, including: Traversing the transcoding stream image group list and the source stream image group list to determine matching transcoding stream image groups and source stream image groups; Determine a first adding time when the transcoded key frame in the transcoded stream image group is added to the transcoded video stream queue, and a second adding time when the source stream key frame in the source stream image group is added to the source stream video stream queue; The video transcoding delay is determined according to the first adding time and the second adding time.

5. The video stream anti-shake method according to claim 4, characterized in that: The determining the video transcoding delay according to the first adding time and the second adding time comprises: Subtracting first adding times corresponding to a plurality of the transcoded stream image groups from second adding times corresponding to the matched source stream image groups, and using the subtraction result as the transcoding delay of the transcoded stream image group; An average value of the plurality of transcoding delays is determined, and the average value is used as the video transcoding delay.

6. The video stream anti-shake method according to claim 1, characterized in that: The determining a target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length includes: Adding the video transcoding delay and the current video jitter buffer data length, and determining the addition result as the initial jitter buffer data length; The initial jitter buffer data length is subtracted from a preset jitter size, and the subtraction result is determined as a target jitter buffer data length.

7. The video stream anti-shake method according to claim 1, characterized in that: The step of determining the synchronization status of the currently playing video stream and the currently playing audio stream includes: Determine the video playback timestamp of the currently playing video stream and the audio playback timestamp of the currently playing audio stream; Determine the audio and video playback time difference according to the video playback timestamp and the audio playback timestamp; When the audio and video playback time difference is within a preset time interval, it is determined that the synchronization status of the currently playing video stream and the currently playing audio stream meets the synchronization condition.

8. The video stream anti-shake method according to claim 1, characterized in that: The adjusting the video jitter buffer data length and the audio jitter buffer data length according to the target jitter buffer data length includes: The video playback speed and the audio playback speed are reduced until the current video buffer length reaches the target jitter buffer data length.

9. A video stream anti-shake device, characterized in that: It includes a delay determination module, a buffer determination module, a synchronization determination module and a buffer adjustment module, wherein: The delay determination module is configured to determine the video transcoding delay according to the video stream data packet received from the server when detecting that the video gear switching occurs in the server; The buffer determination module is configured to determine a current video jitter buffer data length, and determine a target jitter buffer data length according to the video transcoding delay and the current video jitter buffer data length; The synchronization determination module is configured to determine the synchronization status of the currently playing video stream and the currently playing audio stream; The buffer adjustment module is configured to adjust the video jitter buffer data length and the audio jitter buffer data length according to the target jitter buffer data length when the synchronization condition satisfies the synchronization condition.

10. A video stream anti-shake device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video stream anti-shake method as described in any one of claims 1-8.

11. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the video stream anti-shake method according to any one of claims 1 to 8 when executed by a computer processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video stream anti-shake method according to any one of claims 1 to 8 is implemented.

Images