A media data processing method and apparatus, a terminal device, and a storage medium

Abstract

The embodiment of the application provides a media data processing method and device, terminal equipment and storage medium, the method comprises the following steps: receiving the playing parameter change instruction sent by the server in the process that the terminal equipment plays the media stream with the first adjustment time; determining the second adjustment time of keeping the audio and video synchronization playing after switching according to the first playing parameter; determining the playing time required for playing the media stream data which is not played before the switching timestamp according to the first adjustment time and the switching timestamp; determining the adjustment transition strategy of the playing parameter according to the second adjustment time and the playing time required for playing the media stream data; playing the media stream data which is not played before the switching timestamp according to the adjustment transition strategy, and the terminal equipment compares the new adjustment time with the time required for playing the buffered audio and video data at the original audio and video frame rate to determine the adjustment transition strategy until the terminal equipment plays stably at the new audio and video frame rate.

Description

Technical Field

[0001] This invention relates to the field of video network technology, and in particular to a method, apparatus, terminal device, and storage medium for processing media data. Background Technology

[0002] VideoNet is a proprietary protocol technology under a private network. Currently, VideoNet aggregates a large amount of media resources, such as social environment monitoring resources, live streaming resources, conference connection resources, and terminal resources under its own protocol. The accumulated terminal point resources have formed a huge resource pool. However, at present, these media resources are limited by their own proprietary protocols, which makes it difficult to interface with public standard protocols, and the degree of integration is relatively weak.

[0003] To play media resources on internet web pages, you can use technologies such as Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), or the representation of points in cylindrical coordinates in the RGB color model (Hue, Saturation, Lightness, HSL) to convert them. However, the receiving end needs to have a responsive protocol to receive, decapsulate, and display the content. Alternatively, you can build your own website service to play video network monitoring resources, but the website needs additional plugin support.

[0004] Playing video network resources on a web browser of a video network device involves integrating video network and MSE (MediaSource Extensibility) technology frameworks to directly download and play video network format media streams to the webpage without any plugins. However, because MSE technology plays video frame by frame, meaning audio and video need to be played simultaneously, changes in the video frame rate or audio sampling rate during actual playback can cause playback distortion. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention are proposed to provide a method, apparatus, terminal device and storage medium for processing media data that overcomes or at least partially solves the above problems.

[0006] In a first aspect, embodiments of the present invention provide a method for processing media data, the method comprising:

[0007] During the process of playing a media stream on the terminal device at the first adjustment time, a playback parameter change instruction sent by the server is received, wherein the playback parameter change instruction includes at least the first playback parameter and the switching timestamp;

[0008] Based on the first playback parameters, determine the second adjustment time to maintain synchronized audio and video playback after switching;

[0009] Based on the first adjustment time and the switching timestamp, determine the playback time required to play the media stream data that has not been played before the switching timestamp;

[0010] Based on the second adjustment time and the playback time required to play all the media stream data, a transition strategy for adjusting playback parameters is determined;

[0011] According to the adjusted transition strategy, media stream data that has not been played before the switching timestamp is played.

[0012] Optionally, the adjustment transition strategy includes at least one of adjusting the playback rate or adjusting the data frame buffer time.

[0013] Optionally, playing media stream data that has not been played before the switching timestamp according to the transition strategy of adjusting the playback rate includes:

[0014] If the playback time and the second adjustment time meet the first preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is reduced to the first preset value;

[0015] If the playback time and the second adjustment time meet the second preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is accelerated to the second preset value.

[0016] Optionally, playing the media stream data according to the transition strategy of adjusting the data frame buffer time includes:

[0017] If the playback time and the second adjustment time meet the third preset condition, then the buffer time interval of the first playback parameter is increased, and the playback rate of the stored media stream data is accelerated to the second preset value according to the second preset value in the third preset condition.

[0018] If the playback time and the second adjustment time meet the fourth preset condition, then the buffer time interval of the first playback parameter is reduced, and according to the first preset value in the fourth preset condition, the playback rate of the stored media stream data is slowed down to the first preset value.

[0019] Optionally, the first playback parameters include at least the audio sampling rate and / or the video frame rate, and determining the second adjustment time for maintaining synchronized audio and video playback after switching based on the first playback parameters includes:

[0020] Based on the audio sampling rate, determine the first playback duration for each audio data packet;

[0021] Based on the video frame rate, determine the second playback duration for each video data packet;

[0022] The duration difference is determined based on the first playback duration and the second playback duration;

[0023] Based on the time difference and the preset audio-video synchronous playback time, a second adjustment time is determined under the condition of audio-video synchronous playback.

[0024] Optionally, determining the playback time required to play the media stream data that has not been played before the switching timestamp, based on the first adjustment time and the switching timestamp, includes:

[0025] Obtain the initial moment when media data playback begins within the first adjustment time;

[0026] Based on the difference between the received switching timestamp and the initial time, the playback time is determined without any change in the playback parameters;

[0027] Retrieve unplayed media stream data stored before the switching timestamp;

[0028] Based on the second playback parameters within the first adjustment time period, determine the time required to play all the unplayed media stream data;

[0029] The playback time required to play the media stream data is determined based on the already played time and the time required to play the unplayed media stream data.

[0030] Optionally, playing media stream data that has not been played before the switching timestamp according to the adjusted transition strategy includes:

[0031] Obtain the media stream data corresponding to the first playback parameter;

[0032] Based on the aforementioned transition adjustment strategy, determine the playback rate after speeding up or slowing down during the transition process;

[0033] Media stream data that has not been played before the switching timestamp within the playback time is played using the playback rate that has been accelerated or slowed down during the transition process.

[0034] Play the media stream data corresponding to the first playback parameter using the playback rate corresponding to the first playback parameter.

[0035] Secondly, embodiments of the present invention provide a media data processing apparatus, the apparatus comprising:

[0036] The receiving module is used to receive a playback parameter change instruction sent by the server during the process of the terminal device playing a media stream at a first adjustment time, wherein the playback parameter change instruction includes at least a first playback parameter and a switching timestamp.

[0037] The first determining module is used to determine the second adjustment time for maintaining synchronized audio and video playback after switching, based on the first playback parameters.

[0038] The second determining module is used to determine, based on the first adjustment time and the switching timestamp, the playback time required to play the media stream data that has not been played before the switching timestamp.

[0039] The adjustment module is used to determine the adjustment transition strategy of the playback parameters based on the second adjustment time and the playback time required to play all the media stream data.

[0040] The playback module is used to play media stream data that has not been played before the switching timestamp, according to the adjusted transition strategy.

[0041] Optionally, the adjustment transition strategy includes at least one of adjusting the playback rate or adjusting the data frame buffer time.

[0042] Optionally, the playback module is used for:

[0043] If the playback time and the second adjustment time meet the first preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is reduced to the first preset value;

[0044] If the playback time and the second adjustment time meet the second preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is accelerated to the second preset value.

[0045] Optionally, the playback module is used for:

[0046] If the playback time and the second adjustment time meet the third preset condition, the buffer time interval of the first playback parameter is increased, and the playback rate of the stored media stream data is accelerated to the second preset value according to the second preset value in the third preset condition.

[0047] If the playback time and the second adjustment time meet the fourth preset condition, then the buffer time interval of the first playback parameter is reduced, and according to the first preset value in the fourth preset condition, the playback rate of the stored media stream data is slowed down to the first preset value.

[0048] Optionally, the first playback parameters include at least the audio sampling rate and / or the video frame rate, and the first determining module is used to:

[0049] Based on the audio sampling rate, determine the first playback duration for each audio data packet;

[0050] Based on the video frame rate, determine the second playback duration for each video data packet;

[0051] The duration difference is determined based on the first playback duration and the second playback duration;

[0052] Based on the time difference and the preset audio-video synchronous playback time, a second adjustment time is determined under the condition of audio-video synchronous playback.

[0053] Optionally, the second determining module is used to:

[0054] Obtain the initial moment when media data playback begins within the first adjustment time;

[0055] Based on the difference between the received switching timestamp and the initial time, the playback time is determined without any change in the playback parameters;

[0056] Retrieve unplayed media stream data stored before the switching timestamp;

[0057] Based on the second playback parameters within the first adjustment time period, determine the time required to play all the unplayed media stream data;

[0058] The playback time required to play the media stream data is determined based on the already played time and the time required to play the unplayed media stream data.

[0059] Optionally, the playback module is used for:

[0060] Obtain the media stream data corresponding to the first playback parameter;

[0061] Based on the aforementioned transition adjustment strategy, determine the playback rate after speeding up or slowing down during the transition process;

[0062] Obtain the media stream data corresponding to the first playback parameter;

[0063] Media stream data that has not been played before the switching timestamp within the playback time is played using the playback rate that has been accelerated or slowed down during the transition process.

[0064] Play the media stream data corresponding to the first playback parameter using the playback rate corresponding to the first playback parameter.

[0065] Thirdly, embodiments of the present invention provide a terminal device, including: at least one processor and a memory;

[0066] The memory stores a computer program; the at least one processor executes the computer program stored in the memory to implement the media data processing method provided in the first aspect.

[0067] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed, implements the media data processing method provided in the first aspect.

[0068] The embodiments of the present invention have the following advantages:

[0069] The media data processing method, apparatus, terminal device, and storage medium provided in this invention, during the playback of a media stream on the terminal device at a first adjustment time, receive a playback parameter change instruction sent by a server, wherein the playback parameter change instruction includes at least a first playback parameter and a switching timestamp; determine a second adjustment time to maintain synchronized audio and video playback after switching based on the first playback parameter; determine the playback time required to play the media stream data that was not played before the switching timestamp based on the first adjustment time and the switching timestamp; determine an adjustment transition strategy for the playback parameters based on the second adjustment time and the playback time required to play the media stream data; and play the media stream data that was not played before the switching timestamp based on the adjustment transition strategy. According to the embodiments of the present invention, after receiving a playback parameter change instruction, the terminal device recalculates a new adjustment time to keep the audio and video synchronized. Then, it calculates the time required to play the cached audio and video data at the original audio and video frame rate within the current adjustment time. The new adjustment time is then compared with the time required to play the cached audio and video data at the original audio and video frame rate to determine an adjustment transition strategy until the terminal device plays smoothly at the new audio and video frame rate. In the embodiments of the present invention, by setting a transition strategy, the audio rendering playback time and video rendering playback time are periodically adjusted according to the adjustment time, ensuring that even if the user views video network audio and video on the terminal webpage for a long time, there will be no delay problem. Attached Figure Description

[0070] Figure 1 This is a flowchart illustrating the steps of an embodiment of a media data processing method according to the present invention;

[0071] Figure 2 This is a flowchart illustrating the steps of another embodiment of the media data processing method of the present invention;

[0072] Figure 3 This is a flowchart illustrating the steps of another embodiment of the media data processing method of the present invention;

[0073] Figure 4 This is a structural block diagram of an embodiment of a media data processing device according to the present invention;

[0074] Figure 5 This is a schematic diagram of the structure of a terminal device according to the present invention. Detailed Implementation

[0075] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] Currently, the VideoNet and MSE technology frameworks can directly download media stream data in VideoNet media format to web pages on internet terminals for playback without any plugins. During playback, audio and video synchronization is achieved by dynamically adjusting the playback duration of each audio and video frame. However, during media stream playback, the video frame rate or audio sampling rate may dynamically change. For example, when the video frame rate of a live broadcast dynamically changes from 25 pfs to a 5 pfs desktop stream, the audio frame rate remains unchanged. This can easily lead to audio and video playback desynchronization issues.

[0077] For example:

[0078] ① Assuming the video frame rate is 25 frames per second (fps), then each video frame lasts 40ms. The audio sampling rate is 32kHz, with 1024 samples per frame, meaning each frame takes 1000 * 1024 / 32000 = 31.25ms to play. It's clear that audio is sent more frequently than video, so there will inevitably be situations where only audio is played without video. In these cases, the audio must wait for the next video to arrive before it can play. Over time, this leads to audio-video desynchronization and audio lag.

[0079] ② Assuming the video frame rate is 5 frames per second (fps), then each video frame lasts 200ms. The audio sampling rate is 32kHz, with 1024 samples per frame. Therefore, each frame takes 1000 * 1024 / 32000 = 31.25ms to play. It can be seen that audio is sent more frequently than video. Therefore, there will inevitably be situations where only audio is played without video. In these cases, the audio must wait for the next video to arrive before it can play. Over time, this leads to audio and video desynchronization, resulting in audio lag.

[0080] ③ Assuming the video frame rate is 40 frames per second (fps), then each video frame lasts 25ms. With an audio sampling rate of 32kHz and 1024 samples per frame, each frame takes 1000 * 1024 / 32000 = 31.25ms to play. It's clear that video transmission is more frequent than audio transmission, so there will inevitably be situations where only video is played without audio. In these cases, we must wait for the next audio file to arrive before the current video can play. Over time, this leads to audio-video desynchronization, with the audio leading the video.

[0081] It is known that regardless of the change in audio or video frame rate, the magnitude of the change will cause audio-video asynchrony; the only difference is the speed at which this asynchrony occurs, depending on the frame rate difference. Frame rate changes also cause changes in the reference baseline. In ① and ②, there are more audio packets than video packets, so the video is used as the baseline. In ③, there are more video packets than audio packets, so the audio is used as the baseline. These all change according to the frame rate parameter. Furthermore, to avoid asynchrony caused by frame rate changes and image distortion at the moment of frame rate change, special methods are needed to dynamically eliminate this distortion.

[0082] MediaSource Extensibility (MSE) is an API that continuously feeds audio and video binary data to the video tag for playback.

[0083] To address the aforementioned problems, one embodiment of the present invention provides a media data processing method for adjusting the synchronous playback of media stream data. The executing entity in this embodiment is a media data processing device, installed on a terminal device, which can be a computer, tablet computer, or mobile phone terminal, etc.

[0084] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a media data processing method according to the present invention. The method may specifically include the following steps:

[0085] S101. During the process of the terminal device playing the media stream at the first adjustment time, a playback parameter change instruction sent by the server is received, wherein the playback parameter change instruction includes at least the first playback parameter and the switching timestamp.

[0086] Specifically, users play video network media stream data on their terminal device's browser. Since the terminal device has MSE installed, media stream data can be played without requiring additional browser plugins. Users select the media stream data to play in their browser; that is, the terminal device receives the user's media data request and sends it to the internet server. The internet server, based on the media data request, retrieves the corresponding media stream data from the protocol server. The media stream data includes at least audio data, video data, and second playback parameters, including at least the video frame rate and audio capture rate. Furthermore, the media stream data is binary media stream data.

[0087] The terminal device calculates the first adjustment time for synchronized audio and video playback based on the second playback parameters, and plays the media stream data at the first adjustment time. After receiving the binary media stream data returned by the protocol server, the internet server parses the binary media stream data and determines whether the parsing result contains audio sampling rate and video frame rate. If audio sampling rate and video frame rate are present, the binary media stream data is sent to the terminal device. The terminal device determines the duration of playing one frame of video and the duration of playing one frame of audio based on the audio sampling rate and video frame rate, respectively, calculates the time difference between the duration of playing one frame of video and the duration of playing one frame of audio, and obtains the preset synchronized audio and video playback time, which is the time during which the audio and video are not synchronized as perceived by the human eye. The preset synchronized audio and video playback time is divided by the time difference to obtain a quantity value, that is, how many differences exist in the preset synchronized audio and video playback time. This quantity value is then multiplied by the duration of playing one frame of video to determine the adjustment time under the condition of synchronized audio and video playback.

[0088] During playback, if the second playback parameter changes, such as a change in the video frame rate or an audio sampling rate, the server sends a playback parameter change instruction to the terminal device. This instruction includes the second playback parameter and a switching timestamp. The second playback parameter can be a change in the audio sampling rate or a change in the video frame rate.

[0089] S102. Based on the first playback parameters, determine the second adjustment time to maintain synchronized audio and video playback after switching;

[0090] The second playback parameter is the playback parameter before the switch, and the first playback parameter is the playback parameter after the switch, including the audio sampling rate and / or video frame rate after the switch.

[0091] Specifically, after receiving a playback parameter change instruction, the terminal device determines a second adjustment time to maintain synchronized audio and video playback based on the first playback parameter in the instruction (i.e., the switched audio sampling rate and / or video frame rate). In other words, the first playback parameter includes at least one or more of the audio sampling rate and video frame rate. If the video frame rate changes, the second adjustment time to maintain synchronized audio and video playback needs to be recalculated based on the changed video frame rate and the original audio sampling rate. If the audio sampling rate changes, the second adjustment time to maintain synchronized audio and video playback needs to be recalculated based on the changed audio sampling rate and the original video frame rate. If both the video frame rate and the audio sampling rate change, the second adjustment time to maintain synchronized audio and video playback needs to be recalculated based on the changed audio sampling rate and the changed video frame rate.

[0092] S103. Based on the first adjustment time and the switching timestamp, determine the playback time required to play the media stream data that has not been played before the switching timestamp;

[0093] Specifically, the terminal device has already cached a portion of the media stream data during the first adjustment period. This portion of the media stream data is divided into audio data and video data, and they are stored in different arrays, namely, audio data is stored in the audio array and video data is stored in the video array. Both are stored in the memory. During the playback of the stored media stream data, the received switching playback parameter change instruction is used. During this process, it is necessary to calculate the time that has been played before the switching timestamp and the time to play the remaining media stream data after the switching timestamp. The sum of the two is the playback time required to play the media stream data.

[0094] Specifically, playback is performed with the second playback parameter before the timestamp is switched, and with the first playback parameter after the timestamp is switched.

[0095] S104. Determine the transition strategy for adjusting playback parameters based on the second adjustment time and playback time;

[0096] Specifically, after obtaining the second adjustment time and the actual time required to play the media stream data, the terminal device determines the adjustment transition strategy for subsequent media stream data playback. That is, during the media stream data playback process, it first plays with the second playback parameters (parameters before switching). When the second playback parameters (parameters before switching) change to the first playback parameters (parameters after switching), audio and video may become out of sync during media stream data playback. The playback speed of the media stream data needs to be adjusted until the terminal device can play the media stream data smoothly when playing it with the first playback parameters. During this process, it is necessary to determine the adjustment transition strategy for the playback parameters from the second playback parameters to the first playback parameters for stable playback.

[0097] For example, if the playback time T3 required to play all the media stream data is less than or equal to the second adjustment time T11, such as T3<=T11&T3 / m>=T11, then the playback rate is slowed down to m times, where m is the first adjustment parameter for the playback rate, 0<m<1. That is to say, the original buffered media stream data is played at the original playback rate, but the new media stream data has not yet been stored, so the playback rate of the remaining media stream data needs to be slowed down.

[0098] If the ratio of the playback time T3 required to play the media stream data to the second adjustment time T11 is less than the first adjustment parameter m of the playback rate (i.e., the ratio of the first preset value to the playback rate); for example, T3 / m < T11, then reduce the new frame rate buffer time interval until the old data is played out at a multiple of m, and then input the new data of this time period into the label.

[0099] This allows sufficient time to cache media stream data within the new adjustment period, preventing users from experiencing desynchronization or caching issues while watching in their browsers. The playback speed cannot be reduced too much, otherwise the picture quality will be noticeable. Optionally, m = 0.8.

[0100] If the playback time T3 required to play all the media stream data is greater than the second adjustment time T11, for example, T3>T11&T3 / n<=T11, then the playback rate is increased to n times, where n is the second adjustment parameter for the playback rate, n>1. That is to say, the original buffered media stream data is played at the original playback rate, and the new media stream data is ready, so the playback rate of the remaining media stream data needs to be increased.

[0101] If the ratio of the playback time T3 required to play all the media stream data to the second adjustment time T11 is greater than the second adjustment parameter n of the playback rate (i.e., the ratio of the second preset value to the playback rate); for example, T3 / n>T11, then increase the new frame rate buffer time interval until the old data is played out at a multiple of n, and then input the new data of this time period into the label;

[0102] This allows the remaining media stream data to be played quickly before new media stream data is played. Users will not experience desynchronization issues when watching in their browsers. The playback rate should not be too low, and the picture quality should be noticeable. Optionally, n = 1.2.

[0103] S105. According to the adjustment transition strategy, play the media stream data that has not been played before the switching timestamp.

[0104] Specifically, after determining the transition strategy, the terminal device adjusts the playback rate according to the transition strategy, such as speeding up or slowing down the playback rate. When the server detects a change in playback parameters, such as a change in video frame rate or audio sampling rate, it sends a playback parameter change instruction to the terminal device. The terminal device determines different transition strategies based on the playback parameters in the playback parameter change instruction. In this way, during the transition from the second playback parameter to the first playback parameter, the audio and video data can be played synchronously. Then, it gradually transitions to playing the media stream data smoothly with the first playback parameter, so that the user watching the video will not notice that the playback parameters have changed.

[0105] Specifically, within the second adjustment time, for example, within 1 second, the terminal device counts the number of audio data packets and video data packets sent by the server, the first number of audio data packets received within 1 second, and the second number of video data packets received. Then, it calculates the audio rendering and playback time based on the first number and the video rendering and playback time based on the second number.

[0106] In other words, the terminal device counts the number of audio data packets and video data packets received during the second adjustment period, and then distributes this period evenly among each audio data packet for playback rendering or each video data packet for playback rendering.

[0107] For example, before the switch, if the terminal device receives 100 audio data packets within 1000ms, the playback rendering time for one audio data packet is 1000 / 100 = 10ms. If it receives 50 video data packets within 1000ms, the playback rendering time for one video data packet is 1000 / 50 = 20ms. That is, the playback rendering time for audio data packets is 10ms, and the playback rendering time for video data packets is 20ms. If the video frame rate changes, the terminal device needs to recalculate the second adjustment time based on the changed video frame rate and audio sampling rate, and calculate the playback time required to play all the stored media stream data. Then, based on the second adjustment time and the playback time required to play all the media stream data, the corresponding transition strategy is determined. For example, if the video frame rate changes from 25 frames / second to 5 frames / second, the playback of the stored media stream data needs to be accelerated. When new media stream data is received, it gradually transitions to a smooth playback at a video frame rate of 5 frames / second.

[0108] In this embodiment of the invention, the video data packet is a video without sound, such as monitoring data without sound.

[0109] Specifically, after determining the audio rendering playback time and the video rendering playback time, the terminal device retrieves audio data packets from the audio array based on the audio rendering playback time and video data packets from the video array based on the video rendering playback time. The audio and video data packets are then added to the encoder and played through the browser using the second playback parameter. In this way, if the playback parameters change when the user watches the media stream data through the browser, there will be no audio and video desynchronization.

[0110] The media data processing method provided in this embodiment of the invention involves receiving a playback parameter change instruction sent by a server during the playback of a media stream on a terminal device at a first adjustment time. The playback parameter change instruction includes at least a first playback parameter and a switching timestamp. Based on the first playback parameter, a second adjustment time is determined to maintain synchronized audio and video playback after the switch. Based on the first adjustment time and the switching timestamp, the playback time required to play the media stream data that was not played before the switching timestamp is determined. Based on the second adjustment time and the playback time required to play the media stream data, an adjustment transition strategy for the playback parameters is determined. Based on the adjustment transition strategy, the media stream data that was not played before the switching timestamp is played. In this embodiment, after receiving a playback parameter change instruction, the terminal device recalculates the new adjustment time for maintaining audio and video synchronization. It then calculates the time required to play the cached audio and video data at the original audio and video frame rate within the current adjustment time. The new adjustment time is compared with the time required to play the cached audio and video data at the original audio and video frame rate to determine an adjustment transition strategy. This process continues until the terminal device plays smoothly at the new audio and video frame rate. In this embodiment, by setting a transition strategy and periodically adjusting the audio and video rendering playback times according to the adjustment time, it is ensured that even if a user views video on a webpage for an extended period, there will be no latency issues.

[0111] Another embodiment of the present invention further supplements the description of the media data processing method provided in the above embodiments.

[0112] like Figure 2 The diagram illustrates a flowchart of another embodiment of a media data processing method according to the present invention, which includes:

[0113] S201. During the process of the terminal device playing the media stream at the first adjustment time, a playback parameter change instruction sent by the server is received, wherein the playback parameter change instruction includes at least the first playback parameter and the switching timestamp.

[0114] Specifically, if the playback parameters of the media stream data obtained by the server change, for example, if the original video frame rate was 50fps and the current video frame rate is 20fps, the server will send a playback parameter change instruction to the terminal device in advance. The playback parameter change instruction includes at least a first playback parameter and a switching timestamp.

[0115] S202. Based on the first playback parameters, determine the second adjustment time for maintaining synchronized audio and video playback after switching; the first playback parameters include at least the audio sampling rate and / or the video frame rate;

[0116] Specifically, it includes:

[0117] Step A1: Determine the first playback duration for each audio data packet based on the audio sampling rate;

[0118] If the audio sampling rate changes, the first playback duration for each audio data packet is calculated based on the changed audio sampling rate.

[0119] If the audio sampling rate remains unchanged, the original audio sampling rate is used to calculate the first playback duration for each audio data packet.

[0120] Step A2: Determine the second playback duration for each video data packet based on the video frame rate;

[0121] Specifically, if the video frame rate changes, the second playback duration for each video data packet is calculated based on the changed video frame rate.

[0122] If the video frame rate remains unchanged, the original video frame rate is used to calculate the second playback duration for each video data packet.

[0123] Step A3: Determine the duration difference based on the first playback duration and the second playback duration;

[0124] Step A4: Determine the second adjustment time under the condition of audio and video synchronous playback based on the time difference and the preset audio and video synchronous playback time.

[0125] For example, within an initial timeframe, even if audio and video data packets differ, the human eye perceives them as synchronized. For instance, a difference in audio and video timestamps fluctuating within 300-500ms is imperceptible to the human eye; a 300ms difference is considered synchronization. Assuming a video frame rate of 25fps (Frames Per Second), each video frame lasts 40ms. With an audio sampling rate of 32K, each frame contains 1024 samples, and each frame takes 1000*1024 / 32000 = 31.25ms to play. Since the number of audio frames is greater than the number of video frames, playback is based on video packets. A difference of 40 - 31.25 = 8.75ms per frame means that each video frame played delays the audio frame by 8.75ms. Approximately 34 video packets are needed within 300ms. 34*40 = 1360ms will then produce an asynchrony; this 1360ms is the adjustment time.

[0126] In this embodiment of the invention, the adjustment time can be the maximum difference or can be set as needed, i.e., less than 1360ms. Any value can be selected. Preferably, the adjustment time can be 1000ms, which is 1s.

[0127] S203. Obtain the initial moment when media data playback begins within the first adjustment time;

[0128] For example, during the first adjustment period, the terminal device acquires the initial time of the first adjustment period, for example, the first adjustment period is 1 second and the initial time is 10:00. This initial time is used to calculate the time that media stream data has been played.

[0129] S204. Based on the difference between the received switching timestamp and the initial time, determine the playback time that has been played without any change in the playback parameters;

[0130] Specifically, the terminal device obtains the initial time within the first time period and the switching timestamp. Based on the initial time and the switching timestamp, it calculates the time of the already played media stream data assuming that the playback parameters have not changed.

[0131] S205. Obtain the unplayed media stream data stored before the switching timestamp;

[0132] Specifically, the terminal device has already stored media stream data during the first adjustment period and then retrieves the remaining unplayed media stream data.

[0133] S206. Determine the time required to play all the unplayed media stream data based on the second playback parameters within the first adjustment time period;

[0134] Specifically, the terminal device plays the remaining media stream data according to the modified second playback parameters and calculates the time required for the remaining media stream data.

[0135] S207. Based on the time already played and the time required to play the unplayed media stream data, determine the playback time required to play the media stream data.

[0136] S208. Obtain the media stream data corresponding to the first playback parameter;

[0137] S209. Based on the second adjustment time and the playback time required to play all the media stream data, determine the adjustment transition strategy for the playback parameters, and based on the adjustment transition strategy, determine the playback rate after speeding up or slowing down during the transition process.

[0138] Among them, the over-adjustment transition strategy includes at least one of the following, specifically:

[0139] As an optional implementation, according to a transition strategy for adjusting the playback rate, media stream data that has not been played before the switching timestamp is played, including:

[0140] If the playback time and the second adjustment time meet the first preset condition, then the playback rate of the media stream data that has not been played before the switching timestamp is reduced to the first preset value according to the first preset condition.

[0141] If the playback time and the second adjustment time meet the second preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is accelerated to the second preset value.

[0142] Specifically, the first preset condition is that the playback time T3 is less than or equal to the second adjustment time T11, and the ratio of the playback time T3 to the second adjustment time T11 is greater than or equal to the first adjustment parameter m of the playback rate (i.e., the ratio of the first preset value to the playback rate); when the playback time and the second adjustment time meet the first preset condition, the playback rate is slowed down to the first preset value, which is the slowed-down playback rate.

[0143] The second preset condition is that the playback time T3 is greater than the second adjustment time T11, and the ratio of the playback time T3 to the second adjustment time T11 is less than or equal to the second adjustment parameter n of the playback rate (i.e., the ratio of the second preset value to the playback rate). When the playback time and the second adjustment time meet the second preset condition, the playback rate is accelerated to the second preset value, which is the accelerated playback rate.

[0144] For example, if T3 <= T11 & T3 / 0.8 >= T11, the playback rate v is slowed down to the first preset value 0.8v.

[0145] For example, if T3>T11&T3 / 1.2<=T11, the playback rate v is increased to the second preset value 1.2v.

[0146] As another optional implementation, media stream data is played according to a transition strategy that adjusts the data frame buffer time, including:

[0147] If the playback time and the second adjustment time meet the third preset condition, the buffer time interval of the first playback parameter is increased, and the playback rate of the stored media stream data is accelerated to the second preset value.

[0148] If the playback time and the second adjustment time meet the fourth preset condition, then the buffer time interval of the first playback parameter is reduced, and the playback rate of the stored media stream data is slowed down to the first preset value.

[0149] Specifically, the third preset condition is that the ratio of playback time T3 to the second adjustment time T11 is greater than the second adjustment parameter n of the playback rate (i.e., the ratio of the second preset value to the playback rate); where n is the second adjustment parameter of the playback rate, n>1; when the playback time and the second adjustment time meet the third preset condition, the new frame rate buffer time interval is increased until the old data is played out at a multiple of n, and then the new data of this time period is input into the label.

[0150] The fourth preset condition is that the ratio of playback time T3 to the second adjustment time T11 is less than the first adjustment parameter m of the playback rate (i.e., the ratio of the first preset value to the playback rate). When the playback time and the second adjustment time meet the fourth preset condition, the new frame rate buffer time interval is reduced until the old data is played out at multiples of m, and then the new data for this time period is entered into the label.

[0151] For example, if T3 / 1.2 > T11, increase the new frame rate buffer interval until the old data is played out at a multiple of 1.2, and then input the new data for this time period into the tag. Repeat the above process.

[0152] If T3 / 0.8 < T11, reduce the new frame rate buffer interval until the old data finishes playing at a multiple of 0.8, then input the new data for this time period into the tag. Repeat the above process.

[0153] S210. Play the media stream data that has not been played before the switching timestamp during the playback time using the playback rate that has been accelerated or slowed down during the transition process.

[0154] Specifically, the terminal device plays media stream data that has not been played before the switching timestamp during the playback time using the playback rate accelerated or slowed down during the transition process. At the same time, it acquires the media stream data corresponding to the first playback parameter. In the next adjustment time, i.e. the second adjustment time, it plays the media stream data corresponding to the first playback parameter using the playback rate corresponding to the first playback parameter.

[0155] S211. Play the media stream data corresponding to the first playback parameter using the playback rate corresponding to the first playback parameter.

[0156] Specifically, after playing the media stream data that has not been played before the switching timestamp within the playback time, the media stream data is played at a faster or slower playback speed during the second adjustment time period. By adjusting the transition strategy, the terminal device plays the media stream data smoothly at the playback rate corresponding to the first playback parameter.

[0157] Figure 3is a step flow chart of another embodiment of the method for processing media data according to the present invention. First, the sampling rate and frame rate of audio and video are acquired, and it is calculated that asynchronous effects may occur within the required time duration T. For example, audio and video are considered synchronized if the difference between their time stamps is within 300 ms. Assuming that the video frame rate is 25 fps, one frame of video lasts 40 ms; the audio sampling rate is 32K, one frame has 1024 samples, and the playback duration of one frame is 1000*1024 / 32000=31.25 ms. If the number of audio frames is greater than the number of video frames, playback is performed based on video packets. The difference for one frame is 40-31.25=8.75 ms, that is, the audio frame will be delayed by 8.75 ms every time a video frame is played. Asynchronization will occur when approximately 34 video packets are played within 300 ms, where 34*40=1360 ms.

[0158] There are fewer video data packets, based on video data, the numbers of audio and video data packets within an accumulated time T=1000 ms (1000 ms < 1360 ms) are counted separately. The rendering playback duration required for each audio and video packet within this time range is calculated and re-assigned to ensure data synchronization within 1000 ms. Since the calculation is based on video packet data, the playback rendering duration of each audio frame is mainly adjusted.

[0159] After dynamically configuring the playback duration of each frame of data within the time T, the media stream data and related parameters (video frame rate and audio sampling rate) are sent to a decoder for H264 / AAC depacketing and FMP4 packaging, and then rendered in the Video tag. At the same time, dynamic configuration for the next time T is performed, which ensures real-time data update and avoids cumulative parameter delay errors.

[0160] When switching sources (the frame rate or audio sampling rate changes), the server notifies the terminal device to end the current flow cumulative statistics. The terminal device subtracts the previous recorded time stamp from the current last packet time stamp to obtain T2, calculates the required rendering playback duration of audio and video packets within the last time interval T2 when the frame rate does not change based on T2, and sends them to the Video tag for rendering.

[0161] A new second T adjustment time and a new set T11 (T11 < T) are calculated according to the new frame rate. The received media stream data are respectively buffered in corresponding audio arrays or video arrays.

[0162] The time duration T3 required for the currently buffered data in the video tag is obtained, wherein T3=T2 plus the original unplayed data, and T3 is compared with T11 to implement a corresponding strategy transition.

[0163] ① If T3 ≤ T11 and T3 / m ≥ T11, reduce the playback rate to m times the original playback rate (the playback rate cannot be reduced too much, otherwise the change will be perceivable by users), preferably, m=0.8.

[0164] ② If T3>T11 and T3 / n ≤ T11, reduce the playback rate to n times the original playback rate (the playback rate cannot be reduced too much, otherwise the change will be perceivable in the image). Preferably, n=1.2.

[0165] ③ If T3 / n>T11, increase the new frame rate buffering time interval until the old data is completely played at n times the original playback rate, then input the new data of this time period into the tag. Repeat the above process again.

[0166] ④ If T3 / m<T11, reduce the new frame rate buffering time interval until the old data is completely played at m times the original playback rate, then input the new data of this time period into the tag. Repeat the above process again.

[0167] Specific processing of media data includes:

[0168] 1. Pull the corresponding data resource according to the request;

[0169] 2. Obtain the device audio and video SDP, audio and video format, sampling rate, frame rate and other information and return them to the terminal device (MSE);

[0170] 3. Initialize relevant parameters according to the audio sampling rate and video frame rate, and calculate the required duration T based on the benchmark. Asynchronous effects may occur, and a suitable dynamic adjustment interval T1 is selected according to T.

[0171] 4. The server pulls the video network media stream from the transcoding server, removes the video network header and converts it into Internet media stream data, which includes timestamps and data types.

[0172] 5. The server continuously sends buffer data according to the request link;

[0173] 6. The terminal device parses the timestamp and format type of the received media stream data, and fills the data into the corresponding array.

[0174] 7. When video data is received, after adding it to the array, determine the difference between the timestamp and the last dynamic modification of playback time.

[0175] 8. Calculate the rendering and playback time required for each audio and video data packet, dynamically update the time of all frames within the previous T1 interval, and address the cumulative error caused by the different time of each audio and video frame.

[0176] 9. When switching sources (the frame rate changes), the server sends a playback parameter change command to the terminal device.

[0177] 10. The terminal device ends the current cumulative statistics of the stream, subtracts the previously recorded timestamp from the timestamp of the current last packet to obtain T2, calculates the required rendering and playback time for the audio and video packets within the last T2 time interval when the frame rate remains unchanged by using T2, and sends the result to the Vedio tag for rendering.

[0178] 11. The terminal device calculates a new second T adjustment time and sets a new T11 (T11<T) according to the new frame rate, and caches the received media stream data in corresponding arrays respectively.

[0179] 12. The terminal device obtains the time T3 consumed by the data cached in the current video tag, where T3 = T2 plus the original unplayed data, compares T3 with T11, and implements a corresponding transition strategy.

[0180] 13. If T3 ≤ T11 and T3 / 0.8 ≥ T11, reduce the playback rate to 0.8 (the playback rate cannot be lowered too much, otherwise the change will be perceivable on the picture); if T3 > T11 and T3 / 1.2 ≤ T11, increase the playback rate to 1.2 (the playback rate cannot be increased too little, otherwise the change will be perceivable on the picture). If T3 / 1.2 > T11, increase the caching time interval of the new frame rate until the old data is completely played at 1.2× speed, then input the new data of this time period into the tag, and repeat the above process. If T3 / 0.8 < T11, reduce the caching time interval of the new frame rate until the old data is completely played at 0.8× speed, then input the new data of this time period into the tag, and repeat the above process. This process aims to eliminate the disorder caused by frame rate changes, gradually transition to the new cumulative process of T11, and keep the playback picture relatively stable after a new stable state is reached.

[0181] The embodiment of the present invention implements plug-in-free playback of audio and video pictures on web pages, and enables normal picture playback when switching picture scenes (where frame rate changes occur).

[0182] It should be noted that, for the purpose of simple description, the method embodiment is presented as a series of action combinations. However, those skilled in the art should know that the embodiment of the present invention is not limited by the described action order, because some steps may be performed in other orders or simultaneously according to the embodiment of the present invention. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the involved actions are not necessarily required by the embodiment of the present invention.

[0183] The media data processing method provided in this embodiment of the invention involves receiving a playback parameter change instruction sent by a server during the playback of a media stream on a terminal device at a first adjustment time. The playback parameter change instruction includes at least a first playback parameter and a switching timestamp. Based on the first playback parameter, a second adjustment time is determined to maintain synchronized audio and video playback after the switch. Based on the first adjustment time and the switching timestamp, the playback time required to play the media stream data that was not played before the switching timestamp is determined. Based on the second adjustment time and the playback time required to play the media stream data, an adjustment transition strategy for the playback parameters is determined. Based on the adjustment transition strategy, the media stream data that was not played before the switching timestamp is played. In this embodiment, after receiving a playback parameter change instruction, the terminal device recalculates the new adjustment time for maintaining audio and video synchronization. It then calculates the time required to play the cached audio and video data at the original audio and video frame rate within the current adjustment time. The new adjustment time is compared with the time required to play the cached audio and video data at the original audio and video frame rate to determine an adjustment transition strategy. This process continues until the terminal device plays smoothly at the new audio and video frame rate. In this embodiment, by setting a transition strategy and periodically adjusting the audio and video rendering playback times according to the adjustment time, it is ensured that even if a user views video on a webpage for an extended period, there will be no latency issues.

[0184] Another embodiment of the present invention provides a media data processing apparatus for executing the media data processing method provided in the above embodiments.

[0185] Reference Figure 4 The diagram illustrates a structural block diagram of an embodiment of a media data processing device according to the present invention. This device may specifically include the following modules: a receiving module 401, a first determining module 402, a second determining module 403, an adjusting module 404, and a playback module 405, wherein:

[0186] The receiving module 401 is used to receive a playback parameter change instruction sent by the server during the process of the terminal device playing the media stream at a first adjustment time. The playback parameter change instruction includes at least a first playback parameter and a switching timestamp.

[0187] The first determining module 402 is used to determine the second adjustment time for maintaining synchronized audio and video playback after switching, based on the first playback parameters;

[0188] The second determining module 403 is used to determine the playback time required to play the media stream data that has not been played before the switching timestamp, based on the first adjustment time and the switching timestamp.

[0189] The adjustment module 404 is used to determine the adjustment transition strategy of the playback parameters based on the second adjustment time and the playback time required to play all the media stream data.

[0190] The playback module 405 is used to play media stream data that has not been played before the switching timestamp, according to the adjusted transition strategy.

[0191] The media data processing apparatus provided in this embodiment of the invention receives a playback parameter change instruction sent by a server during the playback of a media stream on a terminal device at a first adjustment time. The playback parameter change instruction includes at least a first playback parameter and a switching timestamp. Based on the first playback parameter, a second adjustment time is determined to maintain synchronized audio and video playback after the switch. Based on the first adjustment time and the switching timestamp, the playback time required to play the media stream data that was not played before the switching timestamp is determined. Based on the second adjustment time and the playback time required to play the media stream data, an adjustment transition strategy for the playback parameters is determined. Based on the adjustment transition strategy, the media stream data that was not played before the switching timestamp is played. In this embodiment, after receiving a playback parameter change instruction, the terminal device recalculates the new adjustment time for maintaining audio and video synchronization. It then calculates the time required to play the cached audio and video data at the original audio and video frame rate within the current adjustment time. The new adjustment time is compared with the time required to play the cached audio and video data at the original audio and video frame rate to determine an adjustment transition strategy. This process continues until the terminal device plays smoothly at the new audio and video frame rate. In this embodiment, by setting a transition strategy and periodically adjusting the audio and video rendering playback times according to the adjustment time, it is ensured that even if a user views video on a webpage for an extended period, there will be no latency issues.

[0192] Another embodiment of the present invention further provides a description of the media data processing apparatus provided in the above embodiments.

[0193] Optionally, the transition adjustment strategy may include at least one of adjusting the playback rate or adjusting the data frame buffer time.

[0194] Optionally, the playback module is used for:

[0195] If the playback time and the second adjustment time meet the first preset condition, then the playback rate of the media stream data that has not been played before the switching timestamp is reduced to the first preset value according to the first preset condition.

[0196] If the playback time and the second adjustment time meet the second preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is accelerated to the second preset value.

[0197] Optionally, the playback module is used for:

[0198] If the playback time and the second adjustment time meet the third preset condition, then increase the buffer time interval of the first playback parameter, and accelerate the playback rate of the stored media stream data to the second preset value according to the second preset value in the third preset condition.

[0199] If the playback time and the second adjustment time meet the fourth preset condition, the buffer time interval of the first playback parameter is reduced, and the playback rate of the stored media stream data is slowed down to the first preset value according to the first preset value in the fourth preset condition.

[0200] Optionally, the first playback parameters include at least the audio sampling rate and / or the video frame rate, and the first determining module is used for:

[0201] The first playback duration for each audio data packet is determined based on the audio sampling rate.

[0202] Based on the video frame rate, determine the second playback duration for each video data packet;

[0203] Determine the duration difference based on the first playback duration and the second playback duration;

[0204] Based on the time difference and the preset audio and video synchronous playback time, determine the second adjustment time under the condition of audio and video synchronous playback.

[0205] Optionally, the second determining module is used for:

[0206] Obtain the initial moment when media data playback begins within the first adjustment time;

[0207] Based on the difference between the received switching timestamp and the initial time, determine the playback time that has been played without any change in the playback parameters;

[0208] Retrieve unplayed media stream data stored before the switching timestamp;

[0209] Based on the second playback parameters within the first adjustment time period, determine the time required to play all the unplayed media stream data;

[0210] Determine the playback time required to complete the playback of the media stream data based on the time already played and the time required to play the remaining unplayed media stream data.

[0211] Optionally, the playback module is used for:

[0212] Obtain the media stream data corresponding to the first playback parameter;

[0213] Based on the adjusted transition strategy, determine the playback rate after speeding up or slowing down during the transition process;

[0214] Media stream data that has not been played before the switching timestamp during the playback time will be played using the playback rate that has been accelerated or slowed down during the transition process.

[0215] Play the media stream data corresponding to the first playback parameter at the playback rate corresponding to the first playback parameter.

[0216] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0217] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0218] The media data processing apparatus provided in this embodiment of the invention receives a playback parameter change instruction sent by a server during the playback of a media stream on a terminal device at a first adjustment time. The playback parameter change instruction includes at least a first playback parameter and a switching timestamp. Based on the first playback parameter, a second adjustment time is determined to maintain synchronized audio and video playback after the switch. Based on the first adjustment time and the switching timestamp, the playback time required to play the media stream data that was not played before the switching timestamp is determined. Based on the second adjustment time and the playback time required to play the media stream data, an adjustment transition strategy for the playback parameters is determined. Based on the adjustment transition strategy, the media stream data that was not played before the switching timestamp is played. In this embodiment, after receiving a playback parameter change instruction, the terminal device recalculates the new adjustment time for maintaining audio and video synchronization. It then calculates the time required to play the cached audio and video data at the original audio and video frame rate within the current adjustment time. The new adjustment time is compared with the time required to play the cached audio and video data at the original audio and video frame rate to determine an adjustment transition strategy. This process continues until the terminal device plays smoothly at the new audio and video frame rate. In this embodiment, by setting a transition strategy and periodically adjusting the audio and video rendering playback times according to the adjustment time, it is ensured that even if a user views video on a webpage for an extended period, there will be no latency issues.

[0219] In another embodiment of the present invention, a terminal device is provided for executing the media data processing method provided in the above embodiments.

[0220] Figure 5 This is a schematic diagram of the structure of a terminal device according to the present invention, such as... Figure 5 As shown, the terminal device includes: at least one processor 501 and a memory 502;

[0221] The memory stores a computer program; at least one processor executes the computer program stored in the memory to implement the media data processing method provided in the above embodiments.

[0222] The terminal device provided in this embodiment receives a playback parameter change instruction sent by a server during the playback of a media stream at a first adjustment time. The playback parameter change instruction includes at least a first playback parameter and a switching timestamp. Based on the first playback parameter, a second adjustment time is determined to maintain synchronized audio and video playback after the switch. Based on the first adjustment time and the switching timestamp, the playback time required to play the media stream data that was not played before the switching timestamp is determined. Based on the second adjustment time and the playback time required to play the media stream data, an adjustment transition strategy for the playback parameters is determined. Based on the adjustment transition strategy, the media stream data that was not played before the switching timestamp is played. This is achieved through the embodiment of the present invention. After receiving a playback parameter change instruction, the terminal device recalculates the new adjustment time for maintaining audio and video synchronization. It then calculates the time required to play the cached audio and video data at the original frame rate within the current adjustment time. The new adjustment time is compared with the time required to play the cached audio and video data at the original frame rate to determine an adjustment transition strategy. This process continues until the terminal device plays smoothly at the new audio and video frame rate. In this embodiment of the invention, by setting a transition strategy and periodically adjusting the audio and video rendering playback times according to the adjustment time, it is possible to ensure that even if a user views video on a webpage for an extended period, there will be no latency issues.

[0223] Another embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed, implements the media data processing method provided in any of the above embodiments.

[0224] According to the computer-readable storage medium of this embodiment, during the playback of a media stream on a terminal device at a first adjustment time, a playback parameter change instruction sent by a server is received, wherein the playback parameter change instruction includes at least a first playback parameter and a switching timestamp; based on the first playback parameter, a second adjustment time for maintaining synchronized audio and video playback after switching is determined; based on the first adjustment time and the switching timestamp, the playback time required to play all media stream data not played before the switching timestamp is determined; based on the second adjustment time and the playback time required to play all media stream data, an adjustment transition strategy for the playback parameters is determined; based on the adjustment transition strategy, the media stream data not played before the switching timestamp is played, thereby achieving the present invention. In this embodiment, after receiving a playback parameter change instruction, the terminal device recalculates the new adjustment time for maintaining audio and video synchronization. It then calculates the time required to play the cached audio and video data at the original frame rate within the current adjustment time. The new adjustment time is compared with the time required to play the cached audio and video data at the original frame rate to determine an adjustment transition strategy. This process continues until the terminal device plays smoothly at the new audio and video frame rate. In this embodiment, by setting a transition strategy and periodically adjusting the audio and video rendering playback times according to the adjustment time, it is ensured that even if a user views video on a webpage for an extended period, there will be no latency issues.

[0225] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0226] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0227] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing electronic device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing electronic device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0228] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing electronic device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0229] These computer program instructions can also be loaded onto a computer or other programmable data processing electronic device to cause a series of operational steps to be performed on the computer or other programmable electronic device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable electronic device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0230] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0231] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or electronic device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or electronic device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or electronic device that includes said element.

[0232] The present invention has provided a detailed description of a media data processing method and a media data processing apparatus. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for processing media data, characterized in that, The method includes: During the process of playing a media stream on the terminal device at the first adjustment time, a playback parameter change instruction sent by the server is received, wherein the playback parameter change instruction includes at least the first playback parameter and the switching timestamp; Based on the first playback parameters, determine the second adjustment time to maintain synchronized audio and video playback after switching; Based on the first adjustment time and the switching timestamp, determine the playback time required to play the media stream data that has not been played before the switching timestamp; Based on the second adjustment time and the playback time, determine the transition strategy for adjusting the playback parameters; According to the adjusted transition strategy, media stream data that has not been played before the switching timestamp is played; determining the playback time required to play all the media stream data that has not been played before the switching timestamp, based on the first adjustment time and the switching timestamp, includes: Obtain the initial moment when media data playback begins within the first adjustment time; Based on the difference between the received switching timestamp and the initial time, the playback time is determined without any change in the playback parameters; Retrieve unplayed media stream data stored before the switching timestamp; Based on the second playback parameters within the first adjustment time period, determine the time required to play all the unplayed media stream data; The playback time required to play the media stream data is determined based on the already played time and the time required to play the unplayed media stream data.

2. The method according to claim 1, characterized in that, The adjustment transition strategy includes at least one of adjusting the playback rate or adjusting the data frame buffer time.

3. The method according to claim 2, characterized in that, The step of playing media stream data that has not been played before the switching timestamp according to the transition strategy of adjusting the playback rate includes: If the playback time and the second adjustment time meet the first preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is reduced to the first preset value; If the playback time and the second adjustment time meet the second preset condition, then according to the first preset condition, the playback rate of the media stream data that has not been played before the switching timestamp is accelerated to the second preset value.

4. The method according to claim 2, characterized in that, The step of playing the media stream data according to the transition strategy of adjusting the data frame buffer time includes: If the playback time and the second adjustment time meet the third preset condition, then the buffer time interval of the first playback parameter is increased, and the playback rate of the stored media stream data is accelerated to the second preset value. If the playback time and the second adjustment time meet the fourth preset condition, then the buffer time interval of the first playback parameter is reduced, and the playback rate of the stored media stream data is slowed down to the first preset value.

5. The method according to claim 1, characterized in that, The first playback parameters include at least the audio sampling rate and / or the video frame rate. The step of determining the second adjustment time for maintaining synchronized audio and video playback after switching, based on the first playback parameters, includes: Based on the audio sampling rate, determine the first playback duration for each audio data packet; Based on the video frame rate, determine the second playback duration for each video data packet; The duration difference is determined based on the first playback duration and the second playback duration; Based on the time difference and the preset audio-video synchronous playback time, a second adjustment time is determined under the condition of audio-video synchronous playback.

6. The method according to claim 1, characterized in that, Playing media stream data that has not been played before the switching timestamp according to the adjusted transition strategy includes: Obtain the media stream data corresponding to the first playback parameter; Based on the aforementioned transition adjustment strategy, determine the playback rate after speeding up or slowing down during the transition process; Media stream data that has not been played before the switching timestamp within the playback time is played using the playback rate that has been accelerated or slowed down during the transition process. The media stream data corresponding to the first playback parameter is played at the playback rate corresponding to the first playback parameter.

7. A media data processing apparatus, characterized in that, The device includes: The receiving module is used to receive a playback parameter change instruction sent by the server during the process of the terminal device playing a media stream at a first adjustment time, wherein the playback parameter change instruction includes at least a first playback parameter and a switching timestamp. The first determining module is used to determine the second adjustment time for maintaining synchronized audio and video playback after switching, based on the first playback parameters. The second determining module is used to determine, based on the first adjustment time and the switching timestamp, the playback time required to play the media stream data that has not been played before the switching timestamp. The adjustment module is used to determine the adjustment transition strategy of the playback parameters based on the second adjustment time and the playback time required to play all the media stream data. The playback module is used to play media stream data that has not been played before the switching timestamp, according to the adjusted transition strategy; the second determining module is used to: Obtain the initial moment when media data playback begins within the first adjustment time; Based on the difference between the received switching timestamp and the initial time, the playback time is determined without any change in the playback parameters; Retrieve unplayed media stream data stored before the switching timestamp; Based on the second playback parameters within the first adjustment time period, determine the time required to play all the unplayed media stream data; The playback time required to play the media stream data is determined based on the already played time and the time required to play the unplayed media stream data.

8. A terminal device, characterized in that, include: At least one processor and memory; The memory stores computer programs; The at least one processor executes the computer program stored in the memory to implement the media data processing method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method for processing media data according to any one of claims 1-6.

Images