Distributed streaming media playback system, method, device and storage medium

Abstract

The embodiment of the present application provides a distributed streaming media playback system, method, device and storage medium. In the embodiment of the present application, the front-end processing module and the back-end rendering module are decoupled, demultiplexing and decoding operations are performed in the front-end processing module, and the timestamps corresponding to the audio data and video data are incrementally corrected to provide a basis for subsequent synchronous rendering. The back-end rendering module adds the audio data and its timestamp after timestamp calibration, as well as the video data and its timestamp to the first queue and the second queue respectively to wait for rendering. It is only necessary to synchronize the timestamps of the audio data and video data in the first queue and the second queue on demand, which improves the rendering efficiency, reduces the situation of sound and picture being out of sync and picture interruptions, realizes seamless playback, and enhances the user experience.

Description

Technical Field

[0001] The present application relates to the field of audio and video processing technology, and in particular to a distributed streaming media playback system, method, device and storage medium. Background Art

[0002] Streaming media is a media format that uses streaming technology for continuous real-time playback over the Internet. The encapsulation format, video encoding method, and audio encoding method of the streaming data stream often change. For example, the streaming media transmission protocol (HTTP Live Streaming, HLS) has a discontinuity mode. For example, an HLS stream can include different content, such as the main video and advertising content embedded in the main video. Different content may use different encapsulation formats, different encoding methods, or different resolutions. When the streaming media switches between different content, the player may experience audio and video synchronization and picture interruptions during playback, affecting the user experience. Summary of the Invention

[0003] Various aspects of the present application provide a distributed streaming media playback system, method, device, and storage medium for achieving seamless playback.

[0004] The embodiment of the present application provides a distributed streaming media playback system, comprising: a front-end processing module and a back-end rendering module decoupled from the front-end processing module, wherein the back-end rendering module can be implemented based on different rendering frameworks;

[0005] A front-end processing module is used to obtain a streaming media data stream, which includes multiple audio and video data packets in different encapsulation formats and / or different encoding methods; for each audio and video data packet, demultiplexing operations and decoding operations are performed according to its adapted encapsulation format and encoding method to obtain target audio data and its first timestamp, as well as target video data and its second timestamp; if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then according to the third timestamp of the previous audio data, the first timestamp is corrected by timestamp increment to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, then according to the fourth timestamp of the previous video data, the second timestamp is corrected by timestamp increment to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp are provided to the back-end rendering module;

[0006] The back-end rendering module is used to add the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the first queue and the second queue respectively; render the audio data and video data in the first queue and the second queue in sequence, and when rendering to the target audio data and the target video data, synchronously render the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0007] The present application also provides a distributed streaming media playback method, which is applied to a distributed streaming media playback system. The system includes: a front-end processing module and a back-end rendering module decoupled from the front-end processing module. The back-end rendering module can be implemented based on different rendering frameworks. The method includes:

[0008] The front-end processing module obtains a streaming media data stream, which includes multiple audio and video data packets with different encapsulation formats and / or different encoding methods; performs demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, and obtains target audio data and its first timestamp, as well as target video data and its second timestamp; if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then according to the third timestamp of the previous audio data, the first timestamp is corrected by timestamp increment to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, then according to the fourth timestamp of the previous video data, the second timestamp is corrected by timestamp increment to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp are provided to the back-end rendering module;

[0009] The back-end rendering module adds the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the first queue and the second queue respectively; renders the audio data and video data in the first queue and the second queue in sequence, and when rendering to the target audio data and the target video data, synchronously renders the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0010] The present application also provides a distributed streaming media playback method, which is applicable to a front-end processing module deployed on a first terminal device. The method includes:

[0011] Acquire a streaming media data stream, where the streaming media data stream includes a plurality of audio and video data packets in different encapsulation formats and / or different encoding methods;

[0012] Perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, to obtain target audio data and its first timestamp, as well as target video data and its second timestamp;

[0013] If the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, performing timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data;

[0014] If the second timestamp and the fourth timestamp of the previous video data are non-continuously incremented, the second timestamp is corrected by timestamp increment according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp are provided to the back-end rendering module on the second terminal device, so that the back-end rendering module adds the target audio data and its third timestamp, and the target video data and its fourth timestamp to the first queue and the second queue respectively to wait for rendering, and when rendering to the target audio data and the target video data, the target audio data and the target video data are synchronously rendered according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0015] The present application also provides a distributed streaming media playback method, applicable to a backend rendering device on a second terminal device, the method comprising:

[0016] Receive target audio data and its corresponding first timestamp, as well as target video data and its second timestamp, provided by a front-end processing module deployed on a first terminal device; the front-end processing module obtains a streaming media data stream, which includes multiple audio and video data packets in different encapsulation formats and / or different encoding methods; perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, and obtain target audio data and its first timestamp, as well as target video data and its second timestamp; if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then perform timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data, and obtain the third timestamp of the target audio data ; If the second timestamp and the fourth timestamp of the previous video data are non-continuously incremented, the second timestamp is corrected based on the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp are added to the first queue and the second queue respectively; the audio data and video data in the first queue and the second queue are rendered in sequence, and when the target audio data and target video data are rendered, the target audio data and target video data are synchronously rendered according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0017] An embodiment of the present application also provides a terminal device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the distributed streaming media playback method provided in the embodiment of the present application.

[0018] The embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the steps of the distributed streaming media playback method provided in the embodiment of the present application.

[0019] In an embodiment of the present application, the front-end processing module and the back-end rendering module are decoupled, demultiplexing and decoding operations are performed in the front-end processing module, and the timestamps corresponding to the audio data and video data are incrementally corrected, providing a basis for subsequent synchronous rendering. The back-end rendering module adds the audio data and its timestamp after timestamp calibration, as well as the video data and its timestamp, to the first queue and the second queue respectively to wait for rendering. It is only necessary to synchronize the timestamps of the audio data and video data in the first queue and the second queue on demand, which improves rendering efficiency, reduces the situation of sound and picture asynchrony and picture interruption, realizes seamless playback, and enhances user experience.

[0020] Furthermore, since the front-end processing module and the back-end rendering module are decoupled, the back-end rendering module becomes a pluggable structure, which is compatible with different hardware platforms, achieving a high degree of reuse of the front-end processing module and saving development resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A schematic structural diagram of a distributed streaming media playback system provided in an exemplary embodiment of the present application;

[0023] Figure 2 A flowchart of a distributed streaming media playback method provided by an exemplary embodiment of the present application;

[0024] Figure 3 A flowchart of another distributed streaming media playback method provided by an exemplary embodiment of the present application;

[0025] Figure 4 A flowchart of another distributed streaming media playback method provided by an exemplary embodiment of the present application;

[0026] Figure 5 A schematic structural diagram of a terminal device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] In view of the problem that the sound and picture may be out of sync and the picture may be interrupted during the playback process of the above-mentioned player, which affects the user experience, in an embodiment of the present application, the front-end processing module and the back-end rendering module are decoupled, and demultiplexing and decoding operations are performed in the front-end processing module, and the timestamps corresponding to the audio data and video data are incrementally corrected to provide a basis for subsequent synchronous rendering. The back-end rendering module adds the audio data and its timestamp after timestamp calibration, as well as the video data and its timestamp to the first queue and the second queue respectively to wait for rendering. It is only necessary to synchronize the timestamps of the audio data and video data in the first queue and the second queue on demand, which improves the rendering efficiency, reduces the situation of sound and picture out of sync and picture interruption, realizes seamless playback, and enhances the user experience.

[0030] Furthermore, since the front-end processing module and the back-end rendering module are decoupled, the back-end rendering module becomes a pluggable structure, which is compatible with different hardware platforms, achieving a high degree of reuse of the front-end processing module and saving development resources.

[0031] A solution provided by an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0032] Figure 1 This is a structural diagram of a distributed streaming media playback system provided by an exemplary embodiment of the present application. Figure 1 As shown, the system includes: a front-end processing module 101 and a back-end rendering module 102 decoupled from the front-end processing module 101. The back-end rendering module 102 can be implemented based on different rendering frameworks.

[0033] In this embodiment, due to the decoupling between the front-end processing module 101 and the back-end rendering module 102, the back-end rendering module 102 is implemented as a pluggable structure and can be implemented based on different rendering frameworks, so that the front-end processing module 101 becomes a highly reusable module to save development resources. Among them, the specific implementation of the back-end rendering module 102 is related to the hardware, and the back-end rendering module 102 can be independently designed for different hardware platforms. For example, the rendering engine of the Android and IOS platforms is based on the Open Graphics Library for Embedded Systems (OpenGLES) for embedded devices. For another example, the rendering engine of the mobile operating system (Windows Phone 8, WP8) and the application mall (Windows Store) is based on the DirectX11 API. The DirectX11 API is a multimedia programming interface and belongs to the application program interface (API).

[0034] In this embodiment, streaming media refers to the technology of transmitting audio and video content in the form of a stream over a network (such as the Internet or a wireless mobile network). The video data and audio data are encoded separately, and then the encoded audio data and video data are encapsulated to obtain encapsulated audio and video data packets. Multiple audio and video data packets are continuously sent to a receiver at certain time intervals to form a streaming data stream. The receiver obtains the streaming data stream, that is, continuously receives audio and video data packets, and subsequently decapsulates, decodes, and renders and plays the continuously arriving audio and video data packets. In this embodiment, the front-end processing module and the back-end rendering module are combined to implement the actions performed by the receiver.

[0035] Specifically, the front-end processing module obtains a streaming media data stream, which includes multiple audio and video data packets, wherein the multiple audio and video data packets may contain a main video and an auxiliary video, wherein the main video may be a live broadcast content, a TV series content, or a variety show content, and the auxiliary video may be an advertisement content; or the multiple audio and video data packets may contain different audio and video content, for example, a video clip includes audio and video clips from different video sources. In either case, the multiple audio and video data packets may include a portion of audio and video data packets A1 with the same encapsulation format and encoding method, and the multiple audio and video data packets may also include: an audio and video data packet A2 with the same encapsulation format as the audio and video data packet A1 but a different encoding method, or an audio and video data packet A3 with a different encapsulation format than the audio and video data packet A1 but the same encoding method, or the multiple audio and video data packets may include audio and video data packet A1, audio and video data packet A2, and audio and video data packet A3 at the same time; wherein, different encoding methods may refer to different audio encoding methods, different video encoding methods, or different audio encoding methods and video encoding methods at the same time. Among them, the encapsulation formats used for audio and video data packets may include but are not limited to: Moving Picture Experts Group 4 (MP4), Movie digital video technology (MOV), Audio Video Interleaved (AVI), Media Encapsulation Format (MKV) and Streaming Media Format (FLV) etc.; the audio encoding method may include but is not limited to: Moving Picture Experts Group Audio Layer III (MP3), Advanced Audio Coding (AAC) and lossy sound coding format (OPUS) etc.; the video encoding format includes but is not limited to: High Efficiency Video Coding (H.265), Advanced Video Coding (H.264) and Video Compression Format (VP8) etc.

[0036] In this embodiment, the front-end processing module performs demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, obtaining target audio data and its first timestamp, as well as target video data and its second timestamp. The demultiplexing operation mainly separates the compressed audio data and video data from the audio and video data packet, and the decoding operation refers to the process of decompressing the compressed audio data and video data. The target audio data can be pulse-code modulation (PCM) data, and the target video data can be YUV data. YUV is a color coding mode, where Y represents luminance, i.e., grayscale value, and UV represents chrominance and chroma, respectively, and is used to describe image color and saturation and is used to specify the color of a pixel. For example, the first timestamp can be a presentation time stamp (PTS), and the second timestamp can also be a presentation time stamp (PTS).

[0037] In this embodiment, since the timestamps decoded from different audio and video data packets may not be continuous, for example, the timestamp of the target audio data decoded from the current audio and video data packet is not sequentially increasing with the timestamp of the audio data decoded previously, rendering the target audio data according to the first timestamp that is not sequentially increasing may cause the sound and picture to be out of sync and the picture to be discontinuous, making it impossible to achieve seamless playback, affecting the user's viewing experience. Based on this, in the case of discontinuous timestamps, the timestamps are corrected by timestamp increment, and synchronous rendering and playback are performed based on the timestamps after increment correction, so that the sound and picture are synchronized and uninterrupted, achieving seamless connection.

[0038] Specifically, each audio data item has a corresponding playback timestamp. For ease of distinction and description, the playback timestamp of the audio data item is referred to as the third timestamp. After demultiplexing and decoding the first audio and video data packet in the streaming media data stream, the audio data item and its first timestamp are obtained. The first timestamp of the audio data item can be directly used as its third timestamp.

[0039] For non-first audio and video data packets in the streaming media data stream, the front-end processing module determines whether the first timestamp and the third timestamp of the previous audio data are sequentially increasing, where the previous audio data refers to the audio data obtained by performing demultiplexing and decoding operations on the previous audio and video data packets. For example, if the difference between the first timestamp and the previous audio data is less than a set first time threshold, it is considered that the first timestamp and the previous audio data are non-sequentially increasing; if the difference between the first timestamp and the previous audio data is greater than or equal to the set first time threshold, it is considered that the first timestamp and the previous audio data are sequentially increasing. For example, the first time threshold can be 0, 0.1ms, or 0.5us.

[0040] For example, when different audio and video data packets in a streaming media data stream are switched, the demultiplexing module and the decoding module are restarted. After performing demultiplexing and decoding operations on the switched audio and video data packets, the first timestamp of the obtained audio data may start from 0, and the above-mentioned first time threshold is implemented as 0. At this time, the difference between the first timestamp and the third timestamp of the previous audio data is a negative value, and the negative value is less than the first time threshold 0, then it is considered that the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing. For another example, if the sampling rate of the audio data is 44100Hz, the timestamp of the first frame of audio data is 0, the timestamp of the second frame of audio data is 0.0000227s (1 / 44100), and the timestamp of the third frame of audio data is 0.0000454s (2 / 44100), wherein the first frame of audio data, the second frame of audio data, and the third frame of audio data are sequentially increasing.

[0041] Among them, if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then the first timestamp is corrected for the timestamp increment according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data, so that the third timestamp and the third timestamp of the previous audio data are sequentially increasing. Optionally, if the first timestamp and the third timestamp of the previous audio data are sequentially increasing, then the first timestamp is directly used as the third timestamp. Optionally, the first time interval between the audio data can be determined based on the third timestamp of the previous audio data, for example, taking any two adjacent frames of audio data before, and taking the difference between the third timestamps of the two frames of audio data as the first time interval, or taking the difference between the third timestamps of the two frames of audio data before the target audio data frame as the first time interval. The sum of the third timestamp of the previous audio data and the first time interval is used as the third timestamp of the target audio data.

[0042] In this embodiment, each video data item has a corresponding playback timestamp. For ease of distinction and description, the playback timestamp of the video data item is referred to as the fourth timestamp. After demultiplexing and decoding the first audio and video data packet in the streaming media data stream, the video data item and its second timestamp are obtained. The second timestamp of the video data item can be directly used as the fourth timestamp.

[0043] For non-first audio and video data packets in the streaming media data stream, the front-end processing module determines whether the second timestamp and the fourth timestamp of the previous video data are sequentially increasing, where the previous video data refers to the audio data obtained by performing demultiplexing and decoding operations on the previous audio and video data packets. For example, if the difference between the second timestamp and the previous video data is less than a set time threshold, it is considered that the second timestamp and the previous video data are non-sequentially increasing; if the difference between the second timestamp and the previous video data is less than or equal to a set second time threshold, it is considered that the second timestamp and the previous video data are sequentially increasing. For example, the second time threshold can be 0, 0.04, or 0.001, etc.

[0044] For example, when different audio and video data packets in a streaming media data stream are switched, the demultiplexing module and the decoding module are restarted. After performing demultiplexing and decoding operations on the switched audio and video data packets, the second timestamp of the obtained video data may start from 0, and the above-mentioned second time threshold is implemented as 0. At this time, the difference between the second timestamp and the fourth timestamp of the previous audio data is a negative value, and the negative value is less than the second time threshold of 0, then it is considered that the second timestamp and the fourth timestamp of the previous video data are non-sequentially increasing. For another example, if the frame rate of the video data is 25 (frames / second fps), the timestamp of the first frame of video data is 0, the timestamp of the second frame of video data is 0.04s (1 / 25), and the timestamp of the third frame of video data is 0.08s (2 / 25), wherein the first frame of video data, the second frame of video data, and the third frame of video data are sequentially increasing.

[0045] If the second timestamp and the fourth timestamp of the previous audio data are not sequentially increasing, then the second timestamp is corrected for timestamp increment based on the fourth timestamp of the previous video data to obtain the fourth timestamp of the target audio data, so that the fourth timestamp and the fourth timestamp of the previous video data are sequentially increasing. Alternatively, if the second timestamp and the fourth timestamp of the previous video data are sequentially increasing, then the second timestamp is directly used as the fourth timestamp.

[0046] Alternatively, the second time interval between video data can be determined based on the fourth timestamp of the previous video data. For example, the first time interval can be determined by taking the difference between the third timestamps of any two adjacent frames of audio data, or by taking the difference between the third timestamps of the two frames of audio data preceding the target audio data frame. Alternatively, the video frame rate can be obtained, and the reciprocal of the frame rate can be directly used as the second time interval. The sum of the fourth timestamp of the previous video data and the second time interval is used as the fourth timestamp of the target video data.

[0047] In this embodiment, the back-end rendering module maintains a first queue for storing audio data and its third timestamp, and a second queue for storing video data and its fourth timestamp. When the front-end processing module and the back-end rendering module are separated and decoupled, the audio data and video data are cached based on the first queue and the second queue, and the audio and video are rendered and played synchronously, which can achieve seamless playback of audio and video images.

[0048] Specifically, the front-end processing module can provide the target audio data and its third timestamp, and the target video data and its fourth timestamp to the back-end rendering module, and the back-end rendering module adds the target audio data and its third timestamp to the first queue, and adds the target video data and its fourth timestamp to the second queue. For example, the corresponding data can be added to the first queue and the second queue in the order of the timestamps.

[0049] The back-end rendering module can render the audio data and video data in the first queue and the second queue in sequence. There is a corresponding relationship between the third timestamp of the target audio data and the fourth timestamp of the target video data, and this corresponding relationship refers to a relationship that synchronizes the sound with the picture. When rendering to the target audio data and the target video data, the target audio data and the target video data are synchronously rendered according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0050] In an embodiment of the present application, the front-end processing module and the back-end rendering module are decoupled, demultiplexing and decoding operations are performed in the front-end processing module, and the timestamps corresponding to the audio data and video data are incrementally corrected, providing a basis for subsequent synchronous rendering. The back-end rendering module adds the audio data and its timestamp after timestamp calibration, as well as the video data and its timestamp, to the first queue and the second queue respectively to wait for rendering. It is only necessary to synchronize the timestamps of the audio data and video data in the first queue and the second queue on demand, which improves rendering efficiency, reduces the situation of sound and picture asynchrony and picture interruption, realizes seamless playback, and enhances user experience.

[0051] Furthermore, since the front-end processing module and the back-end rendering module are decoupled, the back-end rendering module becomes a pluggable structure, which is compatible with different hardware platforms, achieving a high degree of reuse of the front-end processing module and saving development resources.

[0052] In an optional embodiment, demultiplexing and decoding operations are performed on each audio and video data packet according to its adapted encapsulation format and encoding method, respectively, to obtain original audio data and its first timestamp, as well as original video data and its second timestamp; specifically, a demultiplexing operation can be performed on the audio and video data packet according to its adapted encapsulation method to obtain an audio encoding data packet and a video encoding data packet; a decoding operation is performed on the audio encoding data packet using an adapted audio encoding method to obtain the original audio data and its first timestamp; a decoding operation is performed on the video encoding data packet using an adapted video encoding method to obtain the original video data and its second timestamp.

[0053] For the original video data, the original video data can be directly used as the target video data, or the original video data can be subjected to operations such as resolution conversion to obtain the target video data, which is not limited.

[0054] For the original audio data, the playback parameters of the original audio data may not meet the playback requirements. Therefore, it is necessary to resample the decoded original audio data according to the preset audio playback information so that the resampled audio parameters meet the preset audio playback information. Resampling refers to a sampling technology that changes audio parameters. The original audio data can be resampled according to the preset audio playback information to obtain target audio data that is adapted to the preset audio playback information. The preset audio playback information is a preset information for audio playback parameters, and the preset audio playback information includes at least one of the following audio parameters: a preset audio sampling frequency, a preset number of audio channels, or a preset audio sampling bit depth.

[0055] Among them, the audio sampling frequency refers to how many sampling points of audio data are collected in one second. For example, an audio sampling frequency of 44.1kHz (44100Hz) means that 44100 sampling points are collected in one second (the sampling points represent the size of the audio data, also called amplitude). For example, the audio sampling frequency can also be 8kHz, 16kHz, 32kHz or 48kHz; the number of audio channels is the number of sound channels, for example, mono, stereo (two-channel) and four-sound surround (four-channel); the sampling points need to be quantized after sampling, and each sampling point will have a corresponding quantized value. The audio sampling bit depth refers to how many bits are used to represent each sampling point, for example, 16 bits or 32 bits. In a physical sense, the sampling bit depth represents the accuracy or granularity of the vibration amplitude.

[0056] Optionally, the preset audio playback information includes at least one of the following audio parameters: a preset audio sampling frequency, a preset number of audio channels, or a preset audio sampling bit depth. The front-end processing module, when resampling the original audio data according to the preset audio playback information to obtain target audio data adapted to the preset audio playback information, is specifically used to: obtain the original audio playback information of the original audio data, which can be obtained when decoding the audio and video data packets, wherein the original audio playback information includes the following audio parameters: the original audio sampling frequency, the original number of audio channels, and the original audio sampling bit depth; for the audio parameters existing in the preset audio playback information, if the original audio playback information and the preset audio playback information have different target audio parameters, then the original audio data is resampled according to the target audio parameters to obtain target audio data adapted to the preset audio playback information. The number of target audio parameters can be one or more.

[0057] For example, the target audio parameter is implemented as an audio sampling frequency. The original sampling frequency in the original audio playback information is 8 khz, and the sampling frequency in the preset audio playback information is 16 khz. The original audio data is resampled with a sampling frequency of 16 khz to obtain target audio data that is adapted to the preset audio playback information. For another example, the target audio parameter is implemented as the number of audio channels and the audio sampling bit depth. The original number of audio channels and the original audio sampling bit depth in the original audio playback information are single channel and 16 bits respectively, and the number of audio channels and the audio bit depth in the preset audio playback information are dual channels and 32 bits respectively. The original audio data is resampled with the number of audio channels being dual channels and the audio sampling bit depth being 32 bits to obtain target audio data that is adapted to the preset audio playback information.

[0058] In an optional embodiment, the back-end rendering module can also implement human-computer interaction operations, allowing users to modify preset audio playback information. In response to the modification operation on any preset audio playback information, the back-end rendering module can obtain the modified target audio parameter value and provide the target audio parameter value to the front-end processing module. The front-end processing module is further configured to: determine, based on the parameter type of the target audio parameter value, an audio parameter value to be replaced that is consistent with the parameter type; and replace the audio parameter value to be replaced with the target audio parameter value. The parameter types of the audio playback parameters are: sampling frequency type, channel type, and sampling bit depth type.

[0059] In an optional embodiment, the back-end rendering module is further used to obtain the current display mode information of the back-end rendering module before synchronously rendering the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data. The display mode includes but is not limited to: adapting to the screen (fit) mode or cropping (clip) mode, etc. For example, when the aspect ratio of the target video data is different from the aspect ratio of the screen, in the fit mode, the size information of the target video data is automatically adjusted, and appropriate reduction or enlargement is performed to maintain the integrity of the display without stretching and deformation. For another example, when the aspect ratio of the target video data is different from the aspect ratio of the screen, in the cropping mode, the width or height of the target video data is appropriately cropped so that the target video data fills the screen. Specifically, according to the current display mode information, the target video data is resized to obtain the target video data that is adapted to the display mode information, so as to achieve subsequent adaptive rendering of the target video data, so that the display of the target video data and the previous video data is smoother and less abrupt, thereby improving the user experience.

[0060] In this embodiment, the deployment method of the front-end processing module and the back-end rendering module is not limited. In an optional embodiment, the front-end processing module and the back-end rendering module are deployed on the same terminal device; in another optional embodiment, the front-end processing module and the back-end rendering module are respectively deployed on different first terminal devices and second terminal devices; the first terminal device and the second terminal device are connected by a data cable USB or High Definition Multimedia Interface (HDMI), or the first terminal device and the second terminal device are wirelessly connected by a millimeter wave high-definition multimedia interface HDMI to realize data transmission between the first terminal device and the second terminal device; the front-end processing module is specifically used to: provide the target audio data and its third timestamp, and the target video data and its fourth timestamp to the back-end rendering module through a wired connection or wireless connection between the first terminal device and the second terminal device. Among them, the millimeter wave HDMI is a high-frequency millimeter wave wireless screen projector, which adopts 60GHz independent wireless high-frequency transmission, which can effectively avoid interference from many 2.4G / 5GHz low-frequency electronic devices, support 4k shocking high-definition transmission, and the picture has no delay or freeze.

[0061] In addition to providing a system embodiment, the embodiment of the present application also provides a distributed streaming media playback method. The process of the distributed streaming media playback method provided by the embodiment of the present application is described below.

[0062] Figure 2A flow chart of a distributed streaming media playback method provided by an exemplary embodiment of the present application is provided. The method is applied to a distributed streaming media playback system, which includes: a front-end processing module and a back-end rendering module decoupled from the front-end processing module. The back-end rendering module can be implemented based on different rendering frameworks; Figure 2 As shown, the method includes:

[0063] 201. A front-end processing module obtains a streaming media data stream, where the streaming media data stream includes multiple audio and video data packets in different encapsulation formats and / or different encoding methods;

[0064] 202. Perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding mode, to obtain target audio data and its first timestamp, and target video data and its second timestamp;

[0065] 203. If the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, perform timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data;

[0066] 204. If the second timestamp and the fourth timestamp of the previous video data are non-continuously incremented, perform timestamp increment correction on the second timestamp according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp;

[0067] 205. Provide the target audio data and its third timestamp, and the target video data and its fourth timestamp to a backend rendering module;

[0068] 206. The back-end rendering module adds the target audio data and its third timestamp, and the target video data and its fourth timestamp to the first queue and the second queue respectively;

[0069] 207. Render the audio data and video data in the first queue and the second queue in sequence. When rendering to the target audio data and the target video data, synchronously render the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0070] In an optional embodiment, a demultiplexing operation and a decoding operation are performed on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain target audio data and its first timestamp, as well as target video data and its second timestamp, including: a demultiplexing operation and a decoding operation are performed on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain original audio data and its first timestamp, as well as original video data and its second timestamp; the original audio data is resampled according to preset audio playback information to obtain target audio data adapted to the preset audio playback information; the preset audio playback information includes at least one of the following audio parameters: a preset audio sampling frequency, a preset number of audio channels or a preset audio sampling bit depth; and the original video data is used as the target video data.

[0071] In an optional embodiment, performing a timestamp increment correction on the first timestamp based on the third timestamp of the previous audio data to obtain the third timestamp of the target audio data includes: determining a first time interval between the audio data based on the third timestamp of the previous audio data; and using the sum of the third timestamp of the previous audio data and the first time interval as the third timestamp of the target audio data. Performing a timestamp increment correction on the second timestamp based on the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data includes: determining a second time interval between the video data based on the fourth timestamp of the previous video data; and using the sum of the fourth timestamp of the previous video data and the second time interval as the fourth timestamp of the target video data.

[0072] In an optional embodiment, before synchronously rendering the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data, the back-end rendering module is also used to: obtain the current display mode information of the back-end rendering module; and resize the target video data according to the current display mode information to obtain target video data that is adapted to the display mode information.

[0073] Figure 3 A flow chart of another distributed streaming media playback method provided by an exemplary embodiment of the present application is provided. The method is applicable to a front-end processing module deployed on a first terminal device, such as Figure 3 As shown, the method includes:

[0074] 301. Obtain a streaming media data stream, where the streaming media data stream includes multiple audio and video data packets in different encapsulation formats and / or different encoding methods;

[0075] 302. Perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding mode, to obtain target audio data and its first timestamp, and target video data and its second timestamp;

[0076] 303. If the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, perform timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data;

[0077] 304. If the second timestamp and the fourth timestamp of the previous video data are non-continuously incremented, perform timestamp increment correction on the second timestamp based on the fourth timestamp of the previous video data to obtain a fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp.

[0078] 305. Provide the target audio data and its third timestamp, as well as the target video data and its fourth timestamp, to the back-end rendering module on the second terminal device, so that the back-end rendering module adds the target audio data and its third timestamp, as well as the target video data and its fourth timestamp, to the first queue and the second queue respectively to wait for rendering, and when rendering to the target audio data and the target video data, synchronously render the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0079] In an optional embodiment, a demultiplexing operation and a decoding operation are performed on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain target audio data and its first timestamp, as well as target video data and its second timestamp, including: a demultiplexing operation and a decoding operation are performed on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain original audio data and its first timestamp, as well as original video data and its second timestamp; the original audio data is resampled according to preset audio playback information to obtain target audio data adapted to the preset audio playback information; the preset audio playback information includes at least one of the following audio parameters: a preset audio sampling frequency, a preset number of audio channels or a preset audio sampling bit depth; and the original video data is used as the target video data.

[0080] In an optional embodiment, performing a timestamp increment correction on the first timestamp based on the third timestamp of the previous audio data to obtain the third timestamp of the target audio data includes: determining a first time interval between the audio data based on the third timestamp of the previous audio data; and using the sum of the third timestamp of the previous audio data and the first time interval as the third timestamp of the target audio data. Performing a timestamp increment correction on the second timestamp based on the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data includes: determining a second time interval between the video data based on the fourth timestamp of the previous video data; and using the sum of the fourth timestamp of the previous video data and the second time interval as the fourth timestamp of the target video data.

[0081] In an optional embodiment, the first terminal device and the second terminal device are connected by a wired data cable USB or a high-definition multimedia interface HDMI, or the first terminal device and the second terminal device are connected wirelessly by a millimeter wave HDMI to realize data transmission between the first terminal device and the second terminal device; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp are provided to the back-end rendering module on the second terminal device, including: the front-end processing module provides the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the back-end rendering module on the second terminal device through the wired connection or wireless connection between the first terminal device and the second terminal device.

[0082] Figure 4 A flow chart of another distributed streaming media playback method provided by an exemplary embodiment of the present application is provided. The method is applicable to a back-end rendering device on a second terminal device, such as Figure 4 As shown, the method includes:

[0083] 401. Receive target audio data and its third timestamp, and target video data and its fourth timestamp, provided by a front-end processing module deployed on a first terminal device.

[0084] 402. Add the target audio data and its third timestamp, and the target video data and its fourth timestamp to the first queue and the second queue respectively;

[0085] 403. Render the audio data and video data in the first queue and the second queue in sequence. When the target audio data and the target video data are rendered, the target audio data and the target video data are synchronously rendered according to the third timestamp of the target audio data and the fourth timestamp of the target video data, so as to achieve synchronous playback of audio and video. The front-end processing module generates the target audio data and its third timestamp, and the target video data and its fourth timestamp as follows: obtain a streaming media data stream, the streaming media data stream includes a plurality of audio and video data packets with different encapsulation formats and / or different encoding methods; for each audio and video data packet, according to its appropriate The demultiplexing operation and the decoding operation are performed respectively according to the encapsulation format and the encoding method to obtain the target audio data and its first timestamp, as well as the target video data and its second timestamp; if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, the first timestamp is corrected by timestamp increment according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, the second timestamp is corrected by timestamp increment according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp.

[0086] In an optional embodiment, before synchronously rendering the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data, it also includes: obtaining the current display mode information of the back-end rendering module; adjusting the size of the target video data according to the current display mode information to obtain the target video data adapted to the display mode information.

[0087] In an optional embodiment, the first terminal device and the second terminal device are connected by a wired data cable USB or a high-definition multimedia interface HDMI, or the first terminal device and the second terminal device are connected wirelessly by a millimeter wave HDMI to realize data transmission between the first terminal device and the second terminal device; receiving the target audio data and its third timestamp, as well as the target video data and its fourth timestamp provided by the front-end processing module deployed on the first terminal device, including: receiving the target audio data and its third timestamp, as well as the target video data and its fourth timestamp provided by the front-end processing module on the first terminal device through a wired connection or a wireless connection between the first terminal device and the second terminal device.

[0088] Regarding the embodiments of this application Figure 2-4 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0089] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 201 to 203 can be a device; for another example, the execution entity of steps 201 and 202 can be a device, and the execution entity of step 203 can be device B; and so on.

[0090] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 201, 202, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0091] Figure 5 A schematic diagram of the structure of a terminal device provided by an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the device is applied to a distributed streaming media playback system, which includes: a front-end processing module and a back-end rendering module decoupled from the front-end processing module, and the back-end rendering module can be implemented based on different rendering frameworks; the device includes: a memory 54 and a processor 55.

[0092] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the terminal device. Examples of such data include instructions for any application or method used to operate on the terminal device.

[0093] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54, so as to: obtain a streaming media data stream through the front-end processing module, the streaming media data stream including a plurality of audio and video data packets in different encapsulation formats and / or different encoding methods; perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, and obtain target audio data and its first timestamp, as well as target video data and its second timestamp; if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then perform timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, then perform timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, then perform timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data The fourth timestamp of a video data is obtained by performing timestamp increment correction on the second timestamp to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp are provided to the back-end rendering module; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp are added to the first queue and the second queue respectively through the back-end rendering module; the audio data and the video data in the first queue and the second queue are rendered in sequence, and when the target audio data and the target video data are rendered, the target audio data and the target video data are synchronously rendered according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0094] In an optional embodiment, when the processor 55 performs demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain target audio data and its first timestamp, as well as target video data and its second timestamp, it is specifically used to: perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain original audio data and its first timestamp, as well as original video data and its second timestamp; resample the original audio data according to preset audio playback information to obtain target audio data adapted to the preset audio playback information; the preset audio playback information includes at least one of the following audio parameters: a preset audio sampling frequency, a preset number of audio channels or a preset audio sampling bit depth; and use the original video data as the target video data.

[0095] In an optional embodiment, when the processor 55 performs a timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data, the processor is specifically used to: determine the first time interval between the audio data according to the third timestamp of the previous audio data; and use the sum of the third timestamp of the previous audio data and the first time interval as the third timestamp of the target audio data; when the processor 55 performs a timestamp increment correction on the second timestamp according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, the processor is specifically used to: determine the second time interval between the video data according to the fourth timestamp of the previous video data; and use the sum of the fourth timestamp of the previous video data and the second time interval as the fourth timestamp of the target video data.

[0096] In an optional embodiment, before the back-end rendering module synchronously renders the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data, the processor 55 is also used to: obtain the current display mode information of the back-end rendering module; and resize the target video data according to the current display mode information to obtain target video data that is adapted to the display mode information.

[0097] Regarding the embodiments of this application Figure 5 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0098] Further, if Figure 5 As shown, the terminal device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the terminal equipment only includes Figure 5 In addition, Figure 5The components in the dotted box are optional components, not mandatory components, and may depend on the product form of the terminal device. The terminal device of this embodiment can be implemented as a desktop computer, a laptop computer, a smart phone or an IOT device.

[0099] The embodiment of the present application also provides a terminal device, on which a front-end processing module is deployed. The implementation structure of the terminal device is similar to Figure 5 The implementation structure of the terminal device shown is the same or similar, please refer to Figure 5 The terminal device provided in this embodiment is implemented as shown in the structure of the terminal device. Figure 5 The difference between the terminal devices in the illustrated embodiments lies primarily in the different functions implemented by the processor executing the computer program stored in the memory. For the terminal device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to:

[0100] Acquire a streaming media data stream, where the streaming media data stream includes a plurality of audio and video data packets in different encapsulation formats and / or different encoding methods;

[0101] Perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, to obtain target audio data and its first timestamp, as well as target video data and its second timestamp;

[0102] If the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, performing timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data;

[0103] If the second timestamp and the fourth timestamp of the previous video data are non-continuously incremented, then according to the fourth timestamp of the previous video data, the second timestamp is corrected by timestamp increment to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp;

[0104] The target audio data and its third timestamp, as well as the target video data and its fourth timestamp are provided to the back-end rendering module on the second terminal device, so that the back-end rendering module adds the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the first queue and the second queue respectively to wait for rendering, and when rendering to the target audio data and the target video data, the target audio data and the target video data are synchronously rendered according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

[0105] In an optional embodiment, when the processor performs demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain target audio data and its first timestamp, as well as target video data and its second timestamp, it is specifically used to: perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method to obtain original audio data and its first timestamp, as well as original video data and its second timestamp; resample the original audio data according to preset audio playback information to obtain target audio data adapted to the preset audio playback information; the preset audio playback information includes at least one of the following audio parameters: a preset audio sampling frequency, a preset number of audio channels or a preset audio sampling bit depth; and use the original video data as the target video data.

[0106] In an optional embodiment, when the processor performs timestamp increment correction on the first timestamp based on the third timestamp of the previous audio data to obtain the third timestamp of the target audio data, it is specifically used to: determine the first time interval between the audio data based on the third timestamp of the previous audio data; and use the sum of the third timestamp of the previous audio data and the first time interval as the third timestamp of the target audio data.

[0107] Correspondingly, when the processor performs timestamp increment correction on the second timestamp based on the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, it is specifically used to: determine the second time interval between the video data based on the fourth timestamp of the previous video data; and take the sum of the fourth timestamp of the previous video data and the second time interval as the fourth timestamp of the target video data.

[0108] In an optional embodiment, the first terminal device and the second terminal device are connected by a wired data cable USB or a high-definition multimedia interface HDMI, or the first terminal device and the second terminal device are connected wirelessly by a millimeter wave HDMI to realize data transmission between the first terminal device and the second terminal device; when the processor provides the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the back-end rendering module on the second terminal device, it is specifically used to: the front-end processing module provides the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the back-end rendering module on the second terminal device through the wired connection or wireless connection between the first terminal device and the second terminal device.

[0109] The embodiment of the present application also provides a terminal device, on which a back-end rendering module is deployed. The implementation structure of the terminal device is similar to Figure 5 The implementation structure of the terminal device shown is the same or similar, please refer to Figure 5 The terminal device provided in this embodiment is implemented as shown in the structure of the terminal device. Figure 5 The difference between the terminal devices in the illustrated embodiments mainly lies in: the functions implemented by the processor executing the computer program stored in the memory are different. For the terminal device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: receive the target audio data and its third timestamp, as well as the target video data and its fourth timestamp, provided by the front-end processing module deployed on the first terminal device; add the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the first queue and the second queue respectively; render the audio data and video data in the first queue and the second queue in sequence, and when rendering to the target audio data and target video data, synchronously render the target audio data and target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video; wherein, the steps performed by the front-end processing module include: obtaining streaming media Data stream, streaming media data stream includes multiple audio and video data packets with different encapsulation formats and / or different encoding methods; for each audio and video data packet, demultiplexing operations and decoding operations are performed respectively according to its adapted encapsulation format and encoding method to obtain target audio data and its first timestamp, as well as target video data and its second timestamp; if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then according to the third timestamp of the previous audio data, the first timestamp is timestamp incremented and corrected to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, then according to the fourth timestamp of the previous video data, the second timestamp is timestamp incremented and corrected to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp.

[0110] In an optional embodiment, before synchronously rendering the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data, the processor is also used to: obtain the current display mode information of the back-end rendering module; and resize the target video data according to the current display mode information to obtain target video data that is adapted to the display mode information.

[0111] In an optional embodiment, the first terminal device and the second terminal device are connected by a wired data cable USB or a high-definition multimedia interface HDMI, or the first terminal device and the second terminal device are connected wirelessly by a millimeter wave HDMI to achieve data transmission between the first terminal device and the second terminal device; when the processor receives the target audio data and its third timestamp, as well as the target video data and its fourth timestamp provided by the front-end processing module deployed on the first terminal device, the processor is specifically used to: receive the target audio data and its third timestamp, as well as the target video data and its fourth timestamp provided by the front-end processing module on the first terminal device through a wired connection or a wireless connection between the first terminal device and the second terminal device.

[0112] The detailed implementation and beneficial effects of the terminal device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0113] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the terminal device in the method embodiments shown in 2-4 above.

[0114] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0115] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0116] The above-mentioned display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0117] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0118] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0119] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) that contain computer-usable program code.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0124] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0125] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0127] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A distributed streaming media playback system, characterized in that: include: A front-end processing module and a back-end rendering module decoupled from the front-end processing module, wherein the back-end rendering module can be implemented based on different rendering frameworks; The front-end processing module is configured to obtain a streaming media data stream, the streaming media data stream including a plurality of audio and video data packets in different encapsulation formats and / or different encoding methods; and perform demultiplexing and decoding operations on each audio and video data packet according to the adapted encapsulation format and encoding method to obtain target audio data and its first timestamp, and target video data and its second timestamp; For a non-first audio and video data packet in the streaming media data stream, determining whether a first timestamp and a third timestamp of previous audio data are sequentially increasing, including: if a difference between the first timestamp and the previous audio data is less than a set first time threshold, then determining that the first timestamp and the previous audio data are non-sequentially increasing; If the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then the first timestamp is corrected by timestamp increment according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, then the second timestamp is corrected by timestamp increment according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp, are provided to the back-end rendering module; The back-end rendering module is used to add the target audio data and its third timestamp, as well as the target video data and its fourth timestamp to the first queue and the second queue respectively; render the audio data and video data in the first queue and the second queue in sequence, and when rendering to the target audio data and the target video data, synchronously render the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

2. The system according to claim 1, wherein: Front-end processing module, specifically used for: Performing demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, respectively, to obtain original audio data and its first timestamp, as well as original video data and its second timestamp; resampling the original audio data according to the preset audio playback information to obtain target audio data adapted to the preset audio playback information; The preset audio playback information includes at least one of the following audio parameters: a preset audio sampling frequency, a preset number of audio channels, or a preset audio sampling bit depth; The original video data is used as target video data.

3. The system according to claim 1, wherein: The front-end processing module is specifically used for: Determining a first time interval between audio data according to the third time stamp of the previous audio data; taking the sum of the third time stamp of the previous audio data and the first time interval as the third time stamp of the target audio data; determining a second time interval between video data according to a fourth time stamp of the previous video data; The sum of the fourth timestamp of the previous video data and the second time interval is used as the fourth timestamp of the target video data.

4. The system according to claim 1, wherein: Before synchronously rendering the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data, the back-end rendering module is further configured to: Get the current display mode information of the backend rendering module; The target video data is resized according to the current display mode information to obtain target video data adapted to the display mode information.

5. The system according to claim 1, wherein: The front-end processing module and the back-end rendering module are respectively deployed on different first terminal devices and second terminal devices; The first terminal device and the second terminal device are connected by a wired connection via a USB data cable or a high-definition multimedia interface HDMI, or the first terminal device and the second terminal device are connected wirelessly via a millimeter wave HDMI to achieve data transmission between the first terminal device and the second terminal device; The front-end processing module is specifically used to provide the target audio data and its third timestamp, and the target video data and its fourth timestamp to the back-end rendering module through a wired connection or a wireless connection between the first terminal device and the second terminal device.

6. A distributed streaming media playback method, characterized in that: The invention is applied to a distributed streaming media playback system, the system comprising: a front-end processing module and a back-end rendering module decoupled from the front-end processing module, the back-end rendering module being implementable based on different rendering frameworks; the method comprising: The front-end processing module obtains a streaming media data stream, wherein the streaming media data stream includes a plurality of audio and video data packets in different encapsulation formats and / or different encoding methods; performs demultiplexing and decoding operations on each audio and video data packet according to the adapted encapsulation format and encoding method, and obtains target audio data and its first timestamp, and target video data and its second timestamp; For a non-first audio and video data packet in the streaming media data stream, determining whether a first timestamp and a third timestamp of previous audio data are sequentially increasing, including: if a difference between the first timestamp and the previous audio data is less than a set first time threshold, then determining that the first timestamp and the previous audio data are non-sequentially increasing; If the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, then the first timestamp is corrected by timestamp increment according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, then the second timestamp is corrected by timestamp increment according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; the target audio data and its third timestamp, as well as the target video data and its fourth timestamp, are provided to the back-end rendering module; The back-end rendering module adds the target audio data and its third timestamp, and the target video data and its fourth timestamp to the first queue and the second queue respectively; renders the audio data and video data in the first queue and the second queue in sequence, and when rendering to the target audio data and the target video data, synchronously renders the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

7. A distributed streaming media playback method, characterized in that: Adapted to a front-end processing module deployed on a first terminal device, the method includes: Acquire a streaming media data stream, wherein the streaming media data stream includes a plurality of audio and video data packets in different encapsulation formats and / or different encoding methods; Perform demultiplexing and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, to obtain target audio data and its first timestamp, as well as target video data and its second timestamp; For a non-first audio and video data packet in the streaming media data stream, determining whether a first timestamp and a third timestamp of previous audio data are sequentially increasing, including: if a difference between the first timestamp and the previous audio data is less than a set first time threshold, then determining that the first timestamp and the previous audio data are non-sequentially increasing; If the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, performing timestamp increment correction on the first timestamp according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data; If the second timestamp and the fourth timestamp of the previous video data are non-continuously incremented, performing timestamp increment correction on the second timestamp according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp; The target audio data and its third timestamp, as well as the target video data and its fourth timestamp are provided to a back-end rendering module on the second terminal device, so that the back-end rendering module adds the target audio data and its third timestamp, as well as the target video data and its fourth timestamp, to the first queue and the second queue respectively to wait for rendering, and when rendering to the target audio data and the target video data, the target audio data and the target video data are synchronously rendered according to the third timestamp of the target audio data and the fourth timestamp of the target video data to achieve synchronous playback of audio and video.

8. A distributed streaming media playback method, characterized in that: Applicable to a backend rendering device on a second terminal device, the method includes: Receiving target audio data and its third timestamp, and target video data and its fourth timestamp provided by a front-end processing module deployed on the first terminal device; Adding the target audio data and its third timestamp, and the target video data and its fourth timestamp to the first queue and the second queue respectively; Rendering the audio data and video data in the first queue and the second queue in sequence, and when rendering to the target audio data and the target video data, synchronously rendering the target audio data and the target video data according to the third timestamp of the target audio data and the fourth timestamp of the target video data, so as to achieve synchronous playback of audio and video; The steps performed by the front-end processing module include: obtaining a streaming media data stream, wherein the streaming media data stream includes a plurality of audio and video data packets in different encapsulation formats and / or different encoding methods; performing demultiplexing operations and decoding operations on each audio and video data packet according to its adapted encapsulation format and encoding method, and obtaining target audio data and its first timestamp, as well as target video data and its second timestamp; for non-first audio and video data packets in the streaming media data stream, judging whether the first timestamp and the third timestamp of the previous audio data are sequentially increasing, including: if the difference between the first timestamp and the third timestamp of the previous audio data is less than the set third timestamp, the first timestamp is judged to be the same as the first timestamp of the previous audio data; a time threshold, it is considered that the first timestamp and the previous audio data are non-sequentially increasing; if the first timestamp and the third timestamp of the previous audio data are non-sequentially increasing, the first timestamp is timestamp incremented according to the third timestamp of the previous audio data to obtain the third timestamp of the target audio data; if the second timestamp and the fourth timestamp of the previous video data are non-continuously increasing, the second timestamp is timestamp incremented according to the fourth timestamp of the previous video data to obtain the fourth timestamp of the target video data, and the third timestamp corresponds to the fourth timestamp.

9. A terminal device, characterized in that: include: memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 6 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 6 to 8.

Images