Media data transmission method and apparatus, and electronic device, storage medium and program product
By sending traffic message collections and response messages, predicting the client cache status and adjusting the data transmission plan, the problem that the data sending side is difficult to perceive the player cache, and improving the audio and video data transmission efficiency and playback quality.
Patent Information
- Application Number
- PCT/CN2023/134541
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2023-11-28
- Publication Date
- 2024-12-05
AI Technical Summary
In the transmission of audio and video data, the data sending end is difficult to perceive the client's player cache in a timely manner, resulting in difficulty in adjusting the data sending strategy, affecting the flow control effect and audio and video playback quality.
By sending a collection of traffic messages to the data receiver, and predicting the cache status according to the response messages and playback order of the receiver, calling a preset data transmission scheme to adjust the data transmission strategy to ensure the efficiency and playback quality of data transmission.
It effectively improves data transmission efficiency and audio and video playback quality, adapts to different cache states, and improves the user experience of audio and video live broadcast scenarios.
Smart Images

Figure CN2023134541_05122024_PF_FP_ABST
Abstract
Description
Media data transmission method, device, electronic device, storage medium and program product
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 31, 2023, with application number 202310632510.X and application name “Audio and video data transmission method and related device”. Technical Field
[0002] The present application relates to the field of communication technology and provides a media data transmission method, device, electronic device, storage medium and program product.
[0003] Background of the Invention
[0004] With the continuous development of audio and video technology, various forms of audio and video services, such as live streaming, audio and video on demand, and audio and video calls, have brought users a diverse user experience. However, audio and video services mainly rely on network transmission services to achieve their goals. Therefore, the transmission process of audio and video data is crucial.
[0005] In related technologies, when the data sender transmits audio and video data to the client, it adjusts the sending strategy of the audio and video data according to the real-time network status (such as packet loss rate, latency, etc.) through protocol improvement, congestion control, etc., to ensure that the client receives complete audio and video data and guarantees the quality of audio and video playback.
[0006] However, the audio and video playback quality of the client is not entirely dependent on network transmission performance; it may also be related to the client's player cache status. If the data sender adjusts its data transmission strategy based solely on network transmission performance, this will affect data transmission efficiency and audio playback quality. Furthermore, due to the highly dynamic nature of player cache, the data sender cannot promptly detect the client's player cache status, making it difficult for the data sender to adjust its data transmission strategy based on the player cache status, thus affecting traffic control effectiveness.
[0007] Summary of the Invention
[0008] The embodiments of the present application provide a media data transmission method, apparatus, electronic device, storage medium, and program product to improve data transmission efficiency and audio and video playback quality.
[0009] In one aspect, an embodiment of the present application provides a method for transmitting media data, including:
[0010] Acquire multiple first media frames, send flow message sets of the multiple first media frames to a data receiving end, and receive at least one response message corresponding to the multiple first media frames returned by the data receiving end;
[0011] performing a first determination step of whether the data receiving end can receive media frame data of at least one first media frame based on the at least one response message;
[0012] When the first determination result of the first determining step is positive, performing a second determination step of determining whether the data receiving end can play the at least one first media frame for each of the at least one first media frame according to a preset playback order of the plurality of first media frames;
[0013] Based on the second determination result of the second determination step, predicting a storage state of a storage unit, such as a cache unit, at the data receiving end; and
[0014] A corresponding preset data transmission scheme is called according to the storage state to send a second media frame to the data receiving end using the data transmission scheme, where the second media frame is different from the multiple first media frames, or the second media frame includes at least one of the first media frames.
[0015] On the other hand, an embodiment of the present application provides a media data transmission device, including:
[0016] a traffic transceiver unit, configured to obtain a plurality of first media frames, send a set of traffic messages of each of the plurality of first media frames to a data receiving end, and receive at least one response message corresponding to the plurality of first media frames returned by the data receiving end;
[0017] a first determining unit, configured to perform a first determining step of whether the data receiving end can receive media frame data of at least one first media frame based on the at least one response message;
[0018] a second determining unit configured to, when a first determination result of the first determining step is positive, perform, for each of at least one of the first media frames, a second determining step of determining whether the data receiving end can play at least one of the first media frames according to a preset playback order of the plurality of first media frames;
[0019] a cache prediction unit, configured to predict a storage state of a storage unit, such as a cache unit, at the data receiving end based on a second determination result of the second determination step; and
[0020] A transmission scheme calling unit is used to call the corresponding preset data transmission scheme according to the storage state, so as to adopt the data transmission scheme to send a second media frame to the data receiving end, where the second media frame is different from the multiple first media frames, or the second media frame includes at least one of the first media frames.
[0021] On the other hand, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above method.
[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the above method.
[0023] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of the above method.
[0024] BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0026] FIG1 is a schematic diagram of a first application scenario provided in an embodiment of the present application;
[0027] FIG2 is a schematic diagram of a second application scenario provided in an embodiment of the present application;
[0028] FIG3 is a flow chart of a method for transmitting media data provided in an embodiment of the present application;
[0029] FIG4 is a schematic diagram of a flow message provided in an embodiment of the present application;
[0030] FIG5 is a schematic diagram of a response message provided in an embodiment of the present application;
[0031] FIG6A is a logic diagram of detecting a complete frame provided in an embodiment of the present application;
[0032] FIG6B is a schematic diagram of another logic for detecting a complete frame provided in an embodiment of the present application;
[0033] FIG7 is a logic diagram of a status information table update process provided in an embodiment of the present application;
[0034] FIG8 is a schematic diagram of a playback sequence provided in an embodiment of the present application;
[0035] FIG9 is a logic diagram of a cache duration determination process provided in an embodiment of the present application;
[0036] FIG10 is a schematic diagram of data transmission schemes for two storage states provided in an embodiment of the present application;
[0037] FIG11 is a schematic diagram of another application scenario provided in an embodiment of the present application;
[0038] FIG12 is a schematic structural diagram of a media data transmission device provided in an embodiment of the present application;
[0039] FIG13 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
[0040] Implementation Method
[0041] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.
[0042] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein.
[0043] The collection and processing of relevant data in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.
[0044] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology encompasses network, information, integration, management platform, and application technologies, all based on the cloud computing business model. It can be used to create a resource pool that is flexible and convenient, available on demand.
[0045] Backend services for technical network systems, such as those for video sites, image sites, and numerous portals, require significant computing and storage resources. With the rapid development and application of the internet, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Data of varying levels will be processed separately, requiring robust system support across all industries, a reality only possible through cloud computing.
[0046] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis.
[0047] As a provider of cloud computing infrastructure, a cloud computing resource pool (referred to as a cloud platform, often referred to as Infrastructure as a Service (IaaS)) is established. Within this pool, various virtual resources are deployed for external clients to choose from. This resource pool primarily includes computing devices (virtualized machines, including operating systems), storage devices, and network equipment.
[0048] Based on logical functional divisions, the Platform as a Service (PaaS) layer can be deployed on top of the IaaS layer, and the Software as a Service (SaaS) layer can be deployed on top of the PaaS layer. SaaS can also be deployed directly on top of IaaS. PaaS is a platform for software execution, such as databases and web containers. SaaS is a variety of business software, such as web portals and text messaging apps. Generally speaking, SaaS and PaaS are layers above IaaS.
[0049] With the continuous development of audio and video technology, various forms of audio and video services, such as live streaming, audio and video on demand, and audio and video calls, have brought users a diverse user experience. However, audio and video services mainly rely on network transmission services to achieve their goals. Therefore, the transmission process of audio and video data is crucial.
[0050] In related technologies, when the data sending end transmits audio and video data to the data receiving end, it adjusts the sending strategy of the audio and video data according to the real-time network status (such as packet loss rate, latency, etc.) through protocol improvement, congestion control, etc., to ensure that the data receiving end receives complete audio and video data and guarantees the playback quality of audio and video.
[0051] However, the audio and video playback quality of the client at the data receiving end doesn't solely depend on network transmission performance; it may also be related to the client's player cache status. If the data sending end adjusts its data transmission strategy based solely on network transmission performance, this strategy may not be effective and may seriously affect data transmission efficiency and audio playback quality. Furthermore, due to the highly dynamic nature of player caches, the data sending end cannot promptly detect the client's player cache status, making it difficult for the data sending end to adjust its data transmission strategy based on the player cache status, thus affecting traffic control effectiveness.
[0052] For example, if audio or video data is lost during transmission, and the amount of cacheable data in the client's player cache is small, even if the message is retransmitted based on the network transmission performance and the lost message is retransmitted to the client at the data receiving end, playback will still be interrupted because the client has no cached data.
[0053] In an embodiment of the present application, a data sending end sends a traffic message set of each first media frame to be transmitted to a data receiving end, and receives at least one response message corresponding to multiple first media frames returned by the data receiving end; based on the at least one response message, a first determination step is performed to determine whether the data receiving end can receive the media frame data of at least one first media frame; when the first determination result of the first determination step is positive, a second determination step is performed for each of the at least one first media frame, according to a preset playback order of the multiple first media frames, to determine whether the data receiving end can play at least one first media frame; based on the second determination result of the second determination step, a storage status of a storage unit, such as a cache unit, of the data receiving end is predicted; and, according to the storage status, a corresponding preset data transfer scheme is called to use the data transfer scheme to send a second media frame to the data receiving end, where the second media frame is different from the multiple first media frames, or the second media frame includes at least one of the first media frames.
[0054] In this way, the data sender determines, through the response message, the first media frame that the client has fully received in each first media frame, and predicts the playable frames that the client can fully play, thereby giving the data sender the ability to predict the cache information of the player in the client. Based on the predicted cache information of the player, the data sending strategy can be adjusted in a timely and effective manner, thereby improving data transmission efficiency and enhancing audio and video playback effects. In addition, based on the use of a data transmission scheme preset for storage status (such as sufficient available cache or insufficient available cache) for subsequent data transmission, it can better adapt to the client's service experience needs in dubbing and video live broadcast scenarios, thereby improving the audio and video service quality of the service provider and, in turn, the user experience.
[0055] Refer to FIG. 1 , which is a schematic diagram of a possible application scenario provided in an embodiment of the present application. The scenario includes: a data sending end 110 and a data receiving end 120 .
[0056] The data sending end 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these.
[0057] The data receiving terminal 120 can be a terminal with audio and video playback requirements, and the terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited thereto. The data receiving terminal 120 is installed with a client for playing media frames (including audio frames and video frames).
[0058] The data sending end 110 and the data receiving end 120 may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.
[0059] In an embodiment of the present application, the data sending end 110 sends a traffic message set of each first media frame to be transmitted to the data receiving end 120, and receives each response message returned by the data receiving end 120. Then, when it is determined based on each response message that the data receiving end 120 has received at least one first media frame, based on the playback order of multiple first media frames, it is detected whether the client in the data receiving end 120 can play at least one first media frame; then, the storage status of the cache in the client's player is predicted, and the data transmission scheme preset for the storage status is adopted to send the second media frame to the data receiving end 120.
[0060] In the embodiments of the present application, the media data transmission method can be applied to various audio and video data transmission scenarios, such as live broadcast scenarios, instant messaging scenarios, online conference scenarios, etc., but is not limited thereto.
[0061] Referring to Figure 2, taking a live broadcast scenario as an example, the live broadcast scenario includes a host client 201, a cloud server 202, and a viewer client 203. Host client 201 obtains audio and video data through a device with audio and video recording capabilities, encodes the data, and then uploads the encoded data to cloud server 202. After receiving the encoded audio and video data, cloud server 202 decodes the data and generates first audio and video frames at one or more bit rates (i.e., resolutions) through one or more encoding steps.
[0062] When the cloud server 202 receives a viewing request from a viewer client 203, it transmits a set of traffic messages carrying each first audio and video frame with a specified bit rate to the viewer client 203, and then determines the second audio and video frame completely received by the viewer client 203 based on each response message returned by the viewer client 203, and predicts the playable second audio and video frame that supports the complete playback of the viewer client 203. Then, based on the second audio and video frame completely received by the viewer client 203 and the playable second audio and video frame that supports the complete playback of the viewer client 203, the storage status of the viewer client 203 is predicted.
[0063] Finally, the cloud server 202 uses a data transmission scheme preset for the storage state to send the third audio and video frame to the viewer client 203 .
[0064] FIG3 is a flow chart of a method for transmitting media data according to an embodiment of the present application. The flow chart is applied to an electronic device serving as a data transmitter. For example, the data transmitter is the server 110 shown in FIG1 , and the data receiver may be a viewing client (also referred to as a client). The specific flow chart is as follows:
[0065] S301. Acquire multiple first media frames to be transmitted, send a flow message set of each of the multiple first media frames to a data receiving end, and receive at least one response message corresponding to the multiple first media frames returned by the data receiving end.
[0066] In the embodiments of the present application, a media frame refers to an audio or video frame. An audio or video frame may refer to an audio frame, a video frame, or a combination thereof, without limitation. Each first media frame corresponds to a traffic message set, which includes at least one traffic message, and a first media frame is transmitted via the at least one traffic message.
[0067] In a live broadcast scenario, the first media frames to be transmitted by the server can be those transmitted to the server by the host client. Referring to Figure 2 , when the server receives a viewing request from a viewer client, it sends a traffic message set for each first audio and video frame to the viewer client and receives response messages from the viewer client. It should be noted that the data transmission process between the host client and the server is not limited in this embodiment and will not be further described here.
[0068] After obtaining each first media frame to be transmitted, the server may further identify the play time or frame type (frame_type) of each first media frame.
[0069] Taking the media frames as audio and video frames as an example, the playback time of each audio and video frame can be represented by presentation time stamps (pts). Pts is a relative timestamp, usually in milliseconds (ms). Pts is used to represent the playback time of the audio and video frame in the client relative to the playback time of the first audio and video frame in the client. For example, if the pts of the first audio and video frame is 0, then when the pts of an audio and video frame is 1, it indicates that the audio and video frame is played 1ms after the first audio and video frame is played.
[0070] The pts of each audio or video frame can be obtained based on the audio or video protocol used. Audio or video protocols include, but are not limited to, the Flash Video (FLV) protocol. For example, in each FLV file, the "Timestamps" field in the tag header of each audio or video frame contains the pts of that frame.
[0071] The frame_type of each audio and video frame is used to characterize the frame type corresponding to the audio and video frame. In the embodiment of the present application, the frame type includes but is not limited to audio frames and video frames. For video frames, the video frame type can be further subdivided into key frames and non-key frames, where key frames refer to video I frames, and non-key frames include video P frames and video B frames. Video I frames do not depend on the information of other frames, while video P frames depend on the previous I frames or P frames, and video B frames depend on adjacent I frames or P frames.
[0072] The frame_type of each audio and video frame can be obtained based on the adopted audio and video protocol. Still taking the FLV protocol as an example, in the FLV file of each audio and video frame, the frame type of the audio and video frame is recorded in the Type field in the Tag header. For example, when the value of the Type field is 0x08, it means that the audio and video frame is an audio frame, and when the value of the Type field is 0x09, it means that the audio and video frame is a video frame. For video frames, when the value of the Frame type field in the tag data (Tag Data) is 2, it means that the video frame is a non-key frame, and when the value of the Frame type field is 1, it means that the video frame is a key frame.
[0073] In an embodiment of the present application, as shown in Table 1, the server can also maintain a status information table for each audio and video frame, and the status information table includes: one or more of: frame type (frame_type), playback time (pts), message identifier (Pkt_num), complete reception (Rcv_bool), and complete playback (Play_bool).
[0074] Table 1 Status information table
[0075] In Table 1, the Pkt_num field of each audio / video frame contains the message identifier Pkt_num of each traffic message in the traffic message set of the audio / video frame. In the embodiment of the present application, the server can transmit an audio / video frame via one or more traffic messages. The Pkt_num field in Table 1 is used to record which traffic messages are used to send an audio / video frame. In the Pkt_num field, the initial value of each Pkt_num can be empty, and the content of the Pkt_num field can be updated after some or all traffic messages in the traffic message set of the audio / video frame are sent.
[0076] The Rcv_bool field of each audio / video frame is used to indicate whether the audio / video frame can be completely received by the data receiving end. In the embodiment of the present application, the so-called "complete reception" means that the data receiving end has received the complete data of an audio / video frame. The initial value of the Rcv_bool field can be a set value (such as 0) indicating that the audio / video frame cannot be completely received. The content of the Rcv_bool field is updated as shown in S302.
[0077] The Play_bool field of each audio or video frame is used to indicate whether the audio or video frame can be played completely by the client. In the embodiment of the present application, the so-called "complete playback" means that the client can play the complete data of an audio or video frame. The initial value of the Play_bool field can be a set value (such as 0) indicating that the audio or video frame cannot be played completely.
[0078] After the server obtains each first audio and video frame to be transmitted, it can also detect whether there is a predicted storage status cached in the player (i.e., historical cache storage status) before sending the traffic message set of each first audio and video frame to the client. If there is no historical cache storage status, or the existing historical cache storage status does not meet the cache occupancy sufficient condition and the cache occupancy insufficient condition, the traffic message of each first audio and video frame will be sent to the network according to the transmission parameters calculated by the congestion control method.
[0079] In the congestion control mode, on the one hand, the server can use optimized protocols for data transmission, including but not limited to Quick User Datagram Protocol (UDP) Internet Connection (QUIC), multipath transmission protocol, etc. Multipath transmission protocol includes but is not limited to Multipath TCP (MPTCP) based on Transmission Control Protocol (TCP) and Multipath QUIC (MPQUIC) based on QUIC.
[0080] Among them, the QUIC protocol can largely solve the problems faced by the existing widely deployed TCP protocol. While improving traffic transmission performance, the QUIC protocol adopts the multiplexing of Hypertext Transfer Protocol 2.0 (HTTP2), the security of HTTPS and the reliability of the TCP protocol. Multi-path transmission (such as MPTCP and MPQUIC) solves the problem of low transmission performance caused by limited transmission resources faced by the existing TCP protocol and QUIC protocol on a single path. By adding a new transmission path, it increases the problem of audio and video traffic failing to reach the terminal in time during traffic transmission and packet loss repair, thereby greatly improving the efficiency of traffic transmission.
[0081] On the other hand, servers can sense the level of congestion in intermediate networks and the capacity of their transmission resources, and use this information to adjust data transmission strategies, such as traffic delivery and packet retransmission. For example, servers can use the TCP congestion control algorithm (Reno) to determine whether intermediate networks are congested based on packet loss; the Timely algorithm to determine whether intermediate networks are congested based on traffic transmission latency; and the bottleneck bandwidth and round-trip time (RTT)-based congestion control algorithm (BBR) to calculate the available transmission resources (i.e., available network bandwidth) in the intermediate networks based on network modeling. These algorithms determine the level of congestion in the intermediate networks based on various signals and then adjust the sender's send window or send rate to maximize the use of intermediate network resources, thereby improving network transmission efficiency and, in the hope of improving the user experience on the end user.
[0082] If the existing historical cache storage status meets the sufficient cache occupancy condition or the insufficient cache occupancy condition, target transmission parameters are calculated according to the data transmission scheme in S305, such as the congestion control method, and the traffic message of the second media frame is sent to the data receiving end using the target transmission parameters. See S305 for details.
[0083] As a possible implementation method, after the server sends the traffic message set of each first audio and video frame to the client, for each first audio and video frame, the message identifier Pkt_num of each traffic message in the traffic message set of the audio and video frame is filled in the Pkt_num column in Table 1.
[0084] For example, the media frames are video frames. Referring to FIG. 4 , it is assumed that the first video frames to be transmitted include: frame 3, frame 4, etc. In Table 1, the frame type of frame 3 is a video frame, specifically a video P frame, Pts=30, i.e., the video frame is played 30ms after the viewer client 402 plays the first video frame, and Pkt_num includes: 50, 51, ..., 60, i.e., the video frame is transmitted via traffic packets with message identifiers 50, 51, ..., 60. In Table 1, frame 4 is a video B frame, Pts=31, i.e., the video frame is played 31ms after the viewer client 402 plays the first video frame, and Pkt_num includes: 61, 63, 64, ..., 72, i.e., the video frame is transmitted via traffic packets with message identifiers 61, 63, 64, ..., 72.
[0085] It should be noted that, in the embodiment of the present application, in the Pkt_num column of Table 1, the first bit of each Pkt_num is used to characterize whether the traffic message is a retransmitted message. For example, if it is a retransmitted message, the value of the first bit is the binary number 1, otherwise, the value of the first bit is the binary number 0. When the value of the first bit of Pkt_num is the binary number 1, the remaining m bits of Pkt_num respectively represent the message identifier of the original traffic message and the message identifier of the retransmitted traffic message. Among them, the message identifier of the retransmitted traffic message can be represented by the relative number of the traffic message in each traffic message corresponding to the audio and video frame, or it can be directly represented by the message identifier of the traffic message, and there is no limitation on this.
[0086] For example, assume that in Table 1, the Pkt_num column of frame 2 contains the following Pkt_num: {0, 40}, {0, 41}, ..., {0, 49}, {1, 1, 62}, where {0, 40} represents a traffic message with a message identifier of 40, {0, 41} represents a traffic message with a message identifier of 41, {0, 49} represents a traffic message with a message identifier of 49, and {1, 1, 62} respectively represent a retransmitted message, an original traffic message with a message identifier of 1, and a retransmitted traffic message with a message identifier of 62, indicating that the traffic message with a message identifier of 62 is a retransmitted message of the first traffic message (i.e., the traffic message with a message identifier of 40), and the traffic message with a message identifier of 62 is used to repair the loss of the first traffic message in frame 2.
[0087] In an embodiment of the present application, after receiving a traffic message in the traffic message set of each first media frame sent by the server, the data receiving end returns at least one response message to the server. The response message may carry the message identifiers of one or more received traffic messages. The response message may indicate, through the message identifier, whether the data receiving end has received the corresponding traffic message or has not received the corresponding traffic message.
[0088] The data receiving end may return at least one response message in various ways, including:
[0089] Each received traffic message corresponds to a response message, and the response message carries the message identifier of the traffic message; or,
[0090] One first media frame corresponds to one response message, and the response message carries a message identifier of a traffic message received in the traffic message set of the first media frame, and the message identifier may be one or more;
[0091] Multiple first media frames correspond to one response message, and the response message carries the message identifier of the traffic message received in the traffic message set of each first media frame. The message identifier may be one or more.
[0092] In the following, an example is given in which a response message is used to indicate that a data receiving end has received a corresponding traffic message.
[0093] For example, referring to Figure 5, the response message is an acknowledgment (ACK) message, which is used to indicate that the viewer client 502 has received the corresponding traffic message. Each ACK message carries a message identifier of a traffic message. After the viewer client 502 receives the traffic messages with message identifiers 50, 51, 52, ..., 74, etc. sent by the cloud server 501, it returns a response message carrying message identifiers such as 50, 52, ..., 74, etc. to the cloud server 501, so as to indicate to the cloud server 501 that the data receiving end of the viewer client 502 has received the traffic messages with message identifiers 50, 51, 52, ..., 74, etc.
[0094] S302: Based on at least one response message, perform a first determination step of whether the data receiving end can receive media frame data of at least one first media frame.
[0095] The “first determination step” specifically refers to determining whether the data receiving end can completely receive the media frame data of at least one first media frame, that is, whether the data receiving end can receive all the media frame data of the at least one first media frame.
[0096] When executing the "first determination step," one possible approach is to perform the first determination step frame by frame based on the message identifier carried in each response message, and obtain a corresponding determination result for each first media frame, i.e., "received / not received" or "yes / no." Another possible approach is to determine, based on the message identifier carried in each response message, whether all messages in the traffic message set of each first media frame have been received.
[0097] After the server receives at least one response message, as a possible implementation method, for each response message in each response message, based on the message identifier carried by each response message, it detects whether the client has received each first audio and video frame, and can use at least one first audio and video frame received by the client as the second audio and video frame.
[0098] In the embodiment of the present application, the so-called "receiving" refers to receiving a complete frame of message data, wherein the second audio and video frame is also called a "complete frame", that is, what is received is a complete frame of message data.
[0099] When the server detects whether the data receiving end has received each first media frame based on each received response message, it may use, but is not limited to, the following methods:
[0100] Obtain a message identifier set of each first media frame recorded locally, where the message identifier set includes a message identifier of each flow message in a flow message set of the first media frame, where each message identifier is numbered according to the order in which each flow message is sent;
[0101] Based on each message identifier included in each message identifier set and each message identifier carried by at least one response message, it is determined that the data receiving end has received media frame data of at least one first media frame.
[0102] Taking the first media frame as audio / video frame x as an example, in the embodiment of the present application, the message identifiers of each message in the message identifier set of audio / video frame x are recorded in the Pkt_num column in Table 1 above. After the server obtains each response message, based on the message identifiers carried in each response message, it determines that the data receiving end has received each traffic message of audio / video frame x, and determines that audio / video frame x has been received, and treats audio / video frame x as a complete frame.
[0103] For example, referring to Figure 6A, frame 3 is sent via traffic messages with message identifiers 50, 51, ..., 60. In each response message (such as ACK message) received by the server, based on the message identifiers 50, 52, ..., 60 carried in each ACK message, it is determined that there is an ACK message for the traffic message with message identifiers 50, 52, ..., 60 that the data receiving end has received, but the server has not received the ACK message carrying the message identifier 51, that is, it is determined that the data receiving end has not received the ACK message for the traffic message with message identifier 51. At this time, the server determines that the client data receiving end has not received frame 3, and frame 3 is not a complete frame.
[0104] For another example, referring to FIG6B , frame 4 is sent via traffic messages with message identifiers 63, 64, ..., 72. If the response messages received by the server carry identifiers 63, 64, ..., 72, respectively, then it is determined that the data receiving end has received the traffic messages with message identifiers 63, 64, ..., 72, and then it is determined that the data receiving end has received frame 4 and regards frame 4 as a complete frame.
[0105] It should be noted that in an embodiment of the present application, if the server determines that all traffic messages of a certain audio and video frame have been received by the data receiving end, the Rcv_bool value of the frame in the status information table shown in Table 1 above can be set to 1, that is, if it is determined that the audio and video frame x is a complete frame, the value of the Rcv_bool field in Table 1 is set to 1.
[0106] Step S303: When the determination result of the first determination step is positive, a second determination step is performed for each of the at least one first media frame according to a preset playback order of the plurality of first media frames to determine whether the data receiving end can play the at least one first media frame.
[0107] In which, according to the preset playback order, the second determination step is performed frame by frame for each of the at least one first media frame, that is, a corresponding second determination result is obtained from each first media frame, thereby obtaining at least one second determination result. When the second determination result is expressed as "yes / no", at least one "yes / no" is obtained. "The determination result is positive" indicates that the data receiving end is able to (e.g., completely) receive (e.g., all) media frame data of at least one first media frame. For example, referring to the status information table of frame 3 shown in Figure 7, when the server determines that frame 3 is a complete frame, the value of the Rcv_bool field (i.e., complete reception) in Table 1 is set from 0 to 1.
[0108] In the embodiment of the present application, a client for playing media is installed on the data receiving end, and "whether the data receiving end can play" specifically refers to whether the client can play. For example, when a video application client (such as Tencent Video) is running on the data receiving end (such as a mobile phone), the second determination step determines whether the video application client can play at least one first media frame.
[0109] Furthermore, if the server determines that frame 3 is a playable frame that supports complete playback by the client, the value of the Play_bool field (ie, complete playback) in Table 1 is set from 0 to 1.
[0110] In an embodiment of the present application, when the server detects that a certain audio or video frame has been completely received by the data receiving end, it further detects whether the audio or video frame can be played by the client.
[0111] In one example, whether the media frame can be played by the client means that all data in the media frame can be played by the client, that is, can be played completely by the client.
[0112] In another embodiment, whether the media frame can be played by the client refers to whether some of the data in the media frame can be played. Specifically, if the percentage of the data in the first media frame that can be played by the client is greater than a preset threshold, the media frame is determined to be playable by the client.
[0113] Specifically, the server performs the following operations on each first media frame of at least one first media frame:
[0114] When it is determined based on the preset playback order that there is at least one other media frame preceding the first media frame among the multiple first media frames, based on the first determination result corresponding to each of the at least one other media frames, it is detected whether the data receiving end can play the first media frame.
[0115] Whether the data receiving end can play the video may specifically refer to whether the client in the data receiving end can play the video.
[0116] According to the above embodiment, it is detected that a certain audio and video frame has been completely received by the data receiving end, which can represent the current network status information, that is, the transmission is reliable; on this basis, it is further detected whether the client can play the audio and video frame, which is used to determine the information of the client cached audio and video frames. Therefore, from the overall point of view, a traffic transmission and control method is realized in which the information of the client cached audio and video frames and the network status information are effectively "linked". This linkage adjustment of audio and video applications and network status information is conducive to improving the user experience of existing Internet services based on audio and video live broadcasts and enhancing the transmission capacity of content distribution networks.
[0117] The preset playback order of the multiple first media frames may be obtained from the data sending end when the server receives the multiple first media frames.
[0118] The audio and video frame x has been completely received by the data receiving end. Taking the audio and video frame x as an example, the process of how to detect whether a certain audio and video frame can be completely played by the client is described.
[0119] As a possible situation, when it is determined based on the playback order of each audio and video frame that there are no other media frames before the audio and video frame x in each audio and video frame, that is, when the audio and video frame x is the first frame, the audio and video frame x can be directly regarded as a playable frame that supports complete playback by the client.
[0120] As another possible situation, when it is determined based on the playback order of each audio and video frame that there is at least one other media frame before the audio and video frame x in each audio and video frame, based on the first determination result corresponding to each of the at least one other media frames, it is detected whether the audio and video frame x is a playable frame that supports complete playback by the client.
[0121] Specifically, detecting whether the data receiving end can play the first media frame based on the first determination result corresponding to each of the at least one other media frames includes:
[0122] If it is determined that the data receiving end receives each media frame of the at least one other media frame based on the first determination result corresponding to each of the at least one other media frames, determining that the client is capable of playing the first media frame;
[0123] If, based on the first determination result corresponding to each of the at least one other media frames, it is determined that the at least one other media frame includes a media frame that the data receiving end has not received, it is determined that the client cannot play the first media frame.
[0124] For example, referring to FIG8 , the playback order of each video frame is: frame 1, frame 2, frame 3, frame 4, frame 5, etc., among which frame 1, frame 2, frame 4, and frame 5 are complete frames, that is, the corresponding Rec-bool field value is 1; and frame 3 is not a complete frame, that is, the corresponding Rec-bool field value is 0. For frame 1, there are no other media frames before frame 1 in each audio and video frame, so frame 1 is directly regarded as a playable frame that supports complete playback by the client, that is, the corresponding Play-bool field value is 1. For frame 2, there is frame 1 before frame 2 in each audio and video frame, so based on the first determination result of frame 1, it can be detected whether frame 2 is a playable frame that supports complete playback by the client. Similarly, for frame 4, based on the first determination results of frames 1, 2, and 3, it is detected whether frame 4 is a playable frame that supports complete playback by the client. For frame 5, based on the first determination results of each of frames 1, 2, 3 and 4, it is detected whether frames 1, 2, 3 and 4 are playable frames that support complete playback by the client.
[0125] As a possible implementation, when detecting whether a client can play audio / video frame x based on the first determination result corresponding to each of the at least one other media frames, if each media frame in the at least one other media frame is determined to be a playable frame based on the first determination result corresponding to each of the at least one other media frames, then the client is determined to be able to play audio / video frame x, or in other words, audio / video frame x is a playable frame that supports complete playback by the client. As a specific implementation, when each of the at least one other media frames is determined to be a complete frame, each media frame in the at least one other media frame is determined to be a playable frame.
[0126] As a possible implementation method, if, based on the first determination result corresponding to at least one other media frame, it is determined that there is a media frame in at least one other media frame that has not been received by the data receiving end, then it is determined that the client cannot play the audio and video frame x, or in other words, it is determined that the audio and video frame x is not a playable frame that supports complete playback by the client.
[0127] In another embodiment of the present application, whether the data receiving end can play the first media frame can also be detected based on the second determination result corresponding to at least one other media frame. The second determination result corresponding to at least one other media frame can be determined by the Play_bool field of the at least one other media frame. For example, when the value of Play_bool is 1, it indicates that the corresponding frame is a playable frame that supports complete playback by the client. When the value of Play_bool is 0, it indicates that the corresponding frame is not a playable frame that supports complete playback by the client.
[0128] Still referring to FIG8 , taking frames 2 and 5 as examples for explanation, the server determines, based on the second determination result of frame 1 (the value of Play_bool of frame 1 is 1), that frame 2 is a playable frame that supports complete playback by the client, that is, the value of the corresponding Play-bool field is 1. The server determines, based on the second determination results of frames 1, 2, 3, and 4, that the values of Play_bool of frames 1 and 2 are both 1, and the values of Play_bool of frames 3 and 4 are both 0, that is, frames 1 and 2 are both playable frames that support complete playback by the client, but frames 3 and 4 are not playable frames. Therefore, frame 5 is not a playable frame that supports complete playback by the client, that is, the values of the Play-bool fields corresponding to frames 3, 4, and 5 are all 1.
[0129] The following describes whether audio frames and video frames support complete playback on the client.
[0130] If audio / video frame x is an audio frame, then the server determines whether there are other media frames preceding audio frame x in each audio frame based on the playback order of each audio frame. If not (i.e., audio frame x is the first frame), then audio frame x is a playable frame that supports complete playback by the client. If so (i.e., audio frame x is not the first frame), then based on the second determination result corresponding to at least one other media frame, the server further detects whether audio frame x is a playable frame that supports complete playback by the client. In the further detection process, if the Play_bool corresponding to at least one other media frame is all 1, then it is determined that audio frame x is a playable frame that supports complete playback by the client. If the Play_bool corresponding to at least one other media frame is not all 1, then it is determined that audio frame x is not a playable frame that supports complete playback by the client.
[0131] If the audio / video frame x is an I-frame, the audio / video frame x is determined to be a playable frame that supports complete playback by the client when the audio / video frame x meets any of the following conditions:
[0132] Condition 1: There is no I-frame, P-frame, or B-frame preceding the video I-frame. In other words, there is no other media frame preceding the video I-frame in each audio and video frame.
[0133] Condition 2: The Play_bool values of all video frames preceding the video I frame are all 1. In other words, the Play_bool values of at least one other media frame preceding the video I frame are all 1.
[0134] If the audio / video frame x is a video P frame, when the Play_bool values of all I frames, P frames, and B frames preceding the video P frame are all 1, the audio / video frame x is determined to be a playable frame that supports complete playback by the client.
[0135] If the audio / video frame x is a video B frame, when the Play_bool values of all I frames, P frames, and B frames preceding the video P frame are all 1, the audio / video frame x is determined to be a playable frame that supports complete playback by the client.
[0136] Furthermore, when it is determined that the audio / video frame x is a playable frame that supports complete playback by the client, the Play_bool of the audio / video frame x is set to 1; otherwise, the Play_bool of the audio / video frame x is maintained to 0.
[0137] In a possible implementation, the playback order of the multiple first media frames may be determined in the following manner:
[0138] Obtain a message identifier set of each first media frame recorded locally, where the message identifier set includes a message identifier of each flow message in a flow message set of the first media frame, where each message identifier is numbered according to the order in which each flow message is sent;
[0139] A playback order of the plurality of first media frames is determined based on each message identifier included in each message identifier set.
[0140] Exemplarily, in the playback order, if the minimum packet number of one audio / video frame is less than the minimum packet number of another audio / video frame, then one audio / video frame is located before the other audio / video frame. That is, referring to Table 1 above, the basis for determining the "front and back" order of audio / video frames can be the minimum packet identifier Pkt_num corresponding to the audio / video frames. Specifically, if the minimum Pkt_num of one frame (frame a) is less than the minimum Pkt_num of another frame (frame b), it means that frame a is before frame b. For example, the Pkt_num of frame a includes: 50, 51, 52, ..., 60, and the Pkt_num of frame b includes: 61, 62, 63, ..., 66. The minimum Pkt_num of frame a is 50, which is less than the minimum Pkt_num of the other frame (frame b) is 61. Therefore, it means that frame a is before frame b.
[0141] S304: Based on the second determination result of the second determination step, predict the storage status of a storage unit, such as a cache unit, at the data receiving end.
[0142] In the embodiments of the present application, a data receiving end uses a storage unit when playing media frames. For example, when a client running on the data receiving end plays a media frame, the operating system of the data receiving end allocates a cache unit to it. The client uses the cache unit to cache the media frames to be played, facilitating fast access and smooth playback. Therefore, the predicted storage state refers to the storage state of the cache allocated to the client running on the data receiving end.
[0143] Specifically, the server predicts the storage state of the storage unit of the data receiving end based on the second determination result of the second determination step, including:
[0144] Based on the second determination result and in combination with the media frame currently played by the data receiving end, obtaining the cache usage data;
[0145] Based on the usage data, the storage status is determined.
[0146] The storage status may include an insufficient cache occupancy status (indicating that the current client player has cached fewer audio and video frames) or a sufficient cache occupancy status (indicating that the current client player has cached more audio and video frames).
[0147] After obtaining the client's storage status based on the second determination result, the server can use a preset data transmission scheme for the insufficient or sufficient cache state to send a second media frame to the client, thereby dynamically adjusting subsequent data transmission, improving data transmission efficiency, and enhancing audio and video playback quality. The second media frame can be a new media frame to be transmitted or a retransmitted first media frame. The new media frame to be transmitted refers to a media frame that needs to be transmitted after the first media frame is sent.
[0148] In a possible implementation, the cache usage data includes: the cache quantity and cache duration of media frames in the cache.
[0149] The cache duration and number of audio and video frames cached in the client can be predicted by, but not limited to, the following methods:
[0150] When it is determined based on the second determination result that at least one playable frame exists in at least one of the first media frames, determining, based on the respective play times of the at least one playable frame, a target frame whose play time satisfies a set reference condition from the at least one playable frame;
[0151] Obtaining a buffering duration based on the respective playback times of the media frame currently played by the data receiving end and the target frame;
[0152] And, the number of the at least one playable frame is used as the buffer number.
[0153] The target frame meeting the set reference condition may be a playable frame with the maximum pts among at least one playable frame, ie, play_bool=1, and an audio or video frame with the largest pts value, but is not limited thereto.
[0154] The cache duration of the audio and video frames cached in the client may be determined based on the difference between the playback time of the audio and video frame currently played by the client and the playback time of the target frame, but is not limited thereto.
[0155] Exemplarily, the server determines the audio and video frame currently played by the client (denoted as frame_cur), and determines the pts of the audio and video frame currently played by the client (denoted as pts_cur). Then, based on the pts of each of the at least one playable frame, the playable frame with the maximum pts among the at least one playable frame is used as the target frame. Based on the pts of the target frame (denoted as pts_max) and combined with pts_cur, the cache duration of the audio and video frame cached in the client (denoted as Frame_time_len) can be calculated using formula (1): Frame_time_len = pts_max - pts_cur Formula (1)
[0156] It should be noted that, in the embodiment of the present application, the audio and video frames currently played by the client can be fed back to the server by the client in real time, but is not limited to this.
[0157] For example, referring to Figure 9, the playable frames include frame 1 and frame 2, and the pts of frame 1 and frame 2 are 50 and 51 respectively. The audio and video frame currently played by the client is frame 1. The server determines the audio and video frame currently played by the client, and determines that the pts value of the audio and video frame currently played by the client is 50 (that is, pts_cur=50). Then, based on the pts of frame 1 and frame 2, frame 2 with the largest pts among frame 1 and frame 2 is used as the target frame. Based on the pts of the target frame (that is, pts_max=51) and combined with pts_cur, the cache duration of the audio and video frames cached in the client is Frame_time_len=51-50=1ms.
[0158] In the embodiment of the present application, the number of buffered audio and video frames in the client can be predicted by, but not limited to, the following methods:
[0159] When it is determined that there is at least one playable frame in at least one complete frame, the server obtains the cached number of audio and video frames cached in the client (denoted as frame_buffer_amount) based on the number of at least one playable frame and the audio and video frames currently played by the client.
[0160] Specifically, as a possible implementation, the server may use the number of playable frames in at least one playable frame whose corresponding pts is not less than pts_cur as the number of audio and video frames cached in the client. For example, referring to FIG9 , pts_cur = 50, the playable frames include frame 1 and frame 2, the pts of frames 1 and 2 are 50 and 51, respectively, the playable frames whose corresponding pts is not less than pts_cur include frame 1 and frame 2, and the number of audio and video frames cached in the client is 2.
[0161] As another possible implementation method, in order to reduce memory usage and improve the efficiency of audio and video frame detection, if the server detects that a certain video frame has been played in the client player, that is, the pts of the audio and video frame is less than pts_cur, then the relevant information of the audio and video frame is deleted from the status information table. Accordingly, when predicting the cached number of audio and video frames cached in the client, the number of playable frames contained in the status information table can be used as the cached number of audio and video frames cached in the client.
[0162] After the number and duration of the audio and video frames cached in the client are predicted, in one possible implementation, based on the number and duration of the audio and video frames cached in the client, there are two possible storage states:
[0163] A first possible state: if the cache duration is less than a preset first duration threshold and the cache quantity is less than a preset first quantity threshold, it is determined that the storage state is insufficient cache occupancy.
[0164] The preset cache occupancy insufficient condition is met when at least one of the following conditions is met:
[0165] Condition a: Cache duration (frame_time_len) < cache duration threshold (frame_time_len_thres);
[0166] Condition b: Buffer quantity (frame_buffer_amount) < buffer quantity threshold (frame_buffer_amount_thres).
[0167] Among them, frame_time_len_thres and frame_buffer_amount_thres can be configured by the administrator and reflected in the configuration file. For example, by default, frame_time_len_thres=100ms and frame_buffer_amount_thres=5.
[0168] A second possible state: if the cache duration is greater than a preset second duration threshold and the cache quantity is greater than a preset second quantity threshold, it is determined that the storage state is that the cache is sufficiently occupied.
[0169] The preset cache occupancy sufficient condition is met when at least one of the following conditions is met:
[0170] Condition c: frame_time_len>N*frame_time_len_thres;
[0171] Condition d: frame_buffer_amount>N*frame_buffer_amount_thres.
[0172] The value of N is a positive integer, which can be configured by an administrator and reflected in a configuration file. For example, by default, N=5.
[0173] S305: Call a corresponding preset data transmission scheme according to the storage state to send a second media frame to the data receiving end using the data transmission scheme, where the second media frame is different from the multiple first media frames, or the second media frame includes at least one first media frame.
[0174] In one possible implementation, the server provides a queryable database, which includes a data transmission scheme lookup table (LUT), which stores multiple storage states and multiple preset data transmission schemes and the mapping relationship between the two. The corresponding data transmission scheme can be queried based on the previously predicted cache storage state.
[0175] In another possible implementation, using a data transmission scheme preset for the storage state to send the second media frame to the data receiving end includes:
[0176] Determining initial transmission parameters based on network transmission performance between the data sending end and the data receiving end;
[0177] Based on the data transmission plan, the initial transmission parameters are adjusted to obtain the target transmission parameters;
[0178] Based on the target transmission parameter, the second media frame is sent to the data receiving end.
[0179] The transmission parameters include, but are not limited to, one or more of the sending rate and the sending window. The sending rate in the initial transmission parameters is denoted as pacing_cpt, and the sending window in the initial transmission parameters is denoted as cwnd_cpt. Pacing_cpt and cwnd_cpt can be calculated using a congestion control algorithm. The specific congestion control algorithm is not limited and will not be described here. The sending rate in the target transmission parameters is denoted as pacing_final, and the sending window in the target transmission parameters is denoted as cwnd_final.
[0180] In the embodiment of the present application, different data transmission schemes can be used for data transmission in the insufficient cache occupancy state and the sufficient cache occupancy state. The following describes data transmission in the insufficient cache occupancy state and the sufficient cache occupancy state respectively.
[0181] 10 , if the storage status indicates that the client is in a state of insufficient cache occupancy, the initial transmission parameters are adjusted according to a data transmission scheme for increasing the amount of transmitted data to obtain target transmission parameters.
[0182] In the embodiment of the present application, the data transmission scheme for increasing the amount of transmitted data includes: increasing the flow sending window, or retransmitting packets when they are lost. This is also called an aggressive packet sending strategy, which aims to cache more audio and video frames in the client player cache within the same time.
[0183] For example, Pacing_final can be calculated by formula (2), and Cwnd_final can be calculated by formula (3): Cwnd_final = cwnd_cpt * cwnd_gain_weight_1 Formula (2) Pacing_final = pacing_cpt * pacing_gain_weight_1 Formula (3)
[0184] Among them, cwnd_gain_weight_1 is the first weight corresponding to cwnd_cpt, pacing_gain_weight_1 is the first weight corresponding to pacing_cpt, cwnd_gain_weight_1 and pacing_gain_weight_1 can be configured by the administrator and can be reflected in the configuration manual. For example, by default, pacing_gain_weight_1=cwnd_gain_weight_1=1.5.
[0185] When the storage status indicates that the client is experiencing insufficient cache occupancy, the server can also perform packet loss repair operations in an aggressive manner. Specifically, the server can use redundant retransmission to address packet loss, i.e., retransmit M copies of the same lost packet data, where these M copies of lost packet data can be sent at the same time or at intervals of SRTT / 2. SRTT represents smoothed RTT, which can be measured by the server. Packet loss detection can be implemented using, but is not limited to, algorithms such as Forward Acknoledgements (FACK) and time-based packet loss detection (RACK).
[0186] If the storage status indicates that the client is in a state where the cache is fully occupied, the initial transmission parameters are adjusted according to a data transmission scheme for reducing the amount of transmitted data to obtain target transmission parameters.
[0187] In the embodiments of the present application, the data transmission scheme for reducing the amount of transmitted data includes: reducing the traffic transmission window, reducing the transmission rate of traffic transmission, or reducing the size of the traffic packet set. This is also called a mitigation packet transmission strategy, which is intended to minimize the loss of transmitted traffic packets.
[0188] For example, Pacing_final can be calculated by formula (4), and Cwnd_final can be calculated by formula (5): Cwnd_final = cwnd_cpt * cwnd_gain_weight_2 Formula (4) Pacing_final = pacing_cpt * pacing_gain_weight_2 Formula (5)
[0189] Among them, cwnd_gain_weight_2 is the second weight corresponding to cwnd_cpt, and pacing_gain_weight_2 is the second weight corresponding to pacing_cpt. cwnd_gain_weight_2 and pacing_gain_weight_2 can be configured by the administrator and can be reflected in the configuration manual. For example, by default, pacing_gain_weight_2=cwnd_gain_weight_2=0.75.
[0190] It should be noted that in an embodiment of the present application, when the client caches fewer audio and video frames, the data sending end can adopt a "suitably aggressive" data transmission method (such as increasing the traffic sending window) to achieve caching more audio and video frames in the client player cache within the same time; or, the data sending end adopts a "more aggressive" data transmission method (such as a packet loss retransmission strategy) to address packet loss, to ensure that the lost traffic packets are repaired in the shortest possible time. Therefore, the Cwnd_final obtained based on cwnd_gain_weight_2 should be less than the Cwnd_final obtained based on cwnd_gain_weight_1, and the Pacing_final obtained based on pacing_gain_weight_2 should be less than the Pacing_final obtained based on pacing_gain_weight_1.
[0191] If the storage status indicates that the client's cache is fully occupied, the server can also perform tasks such as bandwidth probing or minimum latency probing. For example, in the BBR algorithm, the server will perform one or more of the following operations: bandwidth probing (ProbeBW) or latency probing (ProbeRTT) to obtain at least one of the latest bandwidth and latency information. Based on this latest bandwidth and latency information, the values of the various parameters involved in the packet mitigation strategy can be determined.
[0192] Based on the above embodiments, compared with the method in the related art that only adjusts the sending and retransmission strategy based on the network status (such as packet loss rate, available bandwidth, etc.), the present application enables the data sending end to predict the ability of the user terminal player to cache audio and video data, and adaptively adjusts the traffic transmission and control strategy of the sending end based on the number and duration of audio and video frames cached by the player. This can solve the problem of client lag to a large extent and improve the service experience such as lag of user terminals in audio and video live broadcast scenarios.
[0193] Below, this application is described with reference to a specific embodiment.
[0194] Taking an online conference scenario as an example, as shown in Figure 11, this application scenario includes client A for object A, client B for object B, client B for object C, and a server providing voice call services. Object A is the conference speaker. For example, clients A, B, and C are all mobile clients.
[0195] Client A obtains each audio frame through a device with audio recording function, encodes each audio frame, and uploads the encoded audio frame to the server. After receiving each encoded audio frame, the server obtains each audio frame by decoding.
[0196] After receiving a conference join request from client B of object B, the server transmits each traffic message in the traffic message set of each audio and video frame to client B, and receives each response message returned by client B. Then, when it is determined based on each response message that client B has received at least one complete frame of each audio and video frame, the server detects a playable frame that supports complete playback by client B from the at least one complete frame based on the playback order of the audio and video frames.
[0197] Furthermore, based on the detection result of the playable frame, the storage status of the cache in the player of client B is determined.
[0198] As the first case, the storage status of client B is insufficient cache occupancy, then the third audio and video frame is sent to client B according to the data transmission scheme that increases the amount of transmitted data (if the above-mentioned aggressive packet sending and redundant retransmission are performed).
[0199] As the second case, the storage status of client B is a state where the cache is fully occupied. Then, according to the data transmission scheme that reduces the amount of transmitted data (if the above-mentioned packet sending and network detection are relaxed), a new third audio and video frame to be transmitted is sent to client B.
[0200] As a third case, the storage status of client B is neither in a state of sufficient cache occupancy nor in a state of insufficient cache occupancy. In this case, the previous data transmission plan can be left unchanged, or the third audio and video frame can be sent to client B according to the data transmission plan based on the conventional congestion control algorithm.
[0201] Similarly, after receiving a conference participation request from client C of object C, the server transmits each traffic message in the traffic message set of each audio and video frame to client C, and receives each response message returned by client C. Then, based on each response message returned by client C, the server predicts the audio and video frames that client C has completely received, as well as the audio and video frames that client C supports complete playback. Then, based on the predicted audio and video frames that client C has completely received and the audio and video frames that client C supports complete playback, the server obtains the storage status of the cache in the client's player, and then uses a data transmission scheme preset for the storage status to send a third audio and video frame to client C.
[0202] Based on the same inventive concept, an embodiment of the present application provides a media data transmission device. As shown in FIG12 , it is a schematic structural diagram of a media data transmission device 1200 , which may include:
[0203] The traffic transceiver unit 1201 is configured to obtain a plurality of first media frames, send a set of traffic messages of each of the plurality of first media frames to a data receiving end, and receive at least one response message corresponding to the plurality of first media frames returned by the data receiving end;
[0204] A first determining unit 1202 is configured to perform a first determining step of whether the data receiving end can receive media frame data of at least one first media frame based on the at least one response message;
[0205] a second determining unit 1203 configured to, when the first determination result of the first determining step is positive, perform, for each of the at least one first media frame, a second determining step of determining whether the data receiving end can play the at least one first media frame according to a preset playback order of the plurality of first media frames;
[0206] a cache prediction unit 1204, configured to predict a storage state of a storage unit, such as a cache unit, at the data receiving end based on a second determination result of the second determination step; and
[0207] The transmission scheme calling unit 1205 is used to call the corresponding preset data transmission scheme according to the storage status, so as to adopt the data transmission scheme to send a second media frame to the data receiving end, where the second media frame is different from the multiple first media frames, or the second media frame includes at least one of the first media frames.
[0208] As a possible implementation method, the second determination unit 1203 is specifically used to perform the following operations for each first media frame of at least one of the first media frames: when it is determined based on the playback order that there is at least one other media frame before the first media frame among the multiple first media frames, based on the first determination results corresponding to each of the at least one other media frames, detect whether the data receiving end can play the first media frame.
[0209] As a possible implementation method, the second determination unit 1203 is specifically used to, if based on the first determination result corresponding to each of the at least one other media frames, determine that the data receiving end has received each media frame in the at least one other media frame, determine that the client can play the first media frame; if based on the first determination result corresponding to each of the at least one other media frames, determine that there is a media frame in the at least one other media frame that the data receiving end has not received, determine that the client cannot play the first media frame.
[0210] As a possible implementation manner, the cache prediction unit 1204 is specifically configured to obtain usage data of the cache based on the second determination result and in combination with the media frame currently played by the data receiving end; and determine the storage status based on the usage data.
[0211] As a possible implementation, the cache usage data includes: the cache quantity and cache duration of media frames in the cache;
[0212] The cache prediction unit 1204 is specifically used to, when it is determined based on the second determination result that there is at least one playable frame in at least one of the first media frames, determine, from the at least one playable frame, a target frame whose playback time meets the set reference condition based on the respective playback times of the at least one playable frame; obtain the cache duration based on the respective playback times of the media frame currently played by the data receiving end and the target frame; and use the number of the at least one playable frame as the cache number.
[0213] As a possible implementation, the cache prediction unit 1204 is specifically configured to:
[0214] If the cache duration is less than a preset first duration threshold and the cache quantity is less than a preset first quantity threshold, determining that the storage state is insufficient cache occupancy;
[0215] If the cache duration is greater than a preset second duration threshold and the cache quantity is greater than a preset second quantity threshold, it is determined that the storage status is that the cache is sufficiently occupied.
[0216] As a possible implementation manner, the transmission scheme calling unit 1205 is specifically configured to:
[0217] determining initial transmission parameters based on network transmission performance between the data sending end and the data receiving end;
[0218] Based on the data transmission scheme, adjusting the initial transmission parameters to obtain target transmission parameters;
[0219] Based on the target transmission parameter, the second media frame is sent to the data receiving end.
[0220] As a possible implementation manner, the transmission scheme calling unit 1205 is specifically configured to:
[0221] If the storage status indicates that the cache is insufficiently occupied, adjusting the initial transmission parameters according to a data transmission scheme that increases the amount of transmitted data to obtain the target transmission parameters;
[0222] If the storage status indicates that the cache is sufficiently occupied, the initial transmission parameters are adjusted according to a data transmission scheme for reducing the amount of transmitted data to obtain the target transmission parameters.
[0223] As a possible implementation, the second determining unit 1203 is further configured to: obtain a locally recorded message identifier set for each first media frame, where the message identifier set includes a message identifier of each flow message in a flow message set of the first media frame, where each message identifier is numbered according to the order in which the flow messages are sent;
[0224] The play order of the multiple first media frames is determined based on each message identifier included in each message identifier set.
[0225] As a possible implementation manner, the first determining unit 1202 is specifically configured to:
[0226] Obtain a message identifier set of each first media frame recorded locally, wherein the message identifier set includes a message identifier of each flow message in a flow message set of the first media frame, and each message identifier is numbered according to the sending order of each flow message;
[0227] Based on each message identifier included in each message identifier set and the message identifier carried by each response message, it is determined that the data receiving end has received at least one first media frame.
[0228] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.
[0229] Regarding the apparatus in the above embodiment, the specific manner in which each unit executes the request has been described in detail in the embodiment of the method, and will not be elaborated on here.
[0230] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."
[0231] Based on the same inventive concept, an embodiment of the present application further provides an electronic device. In one embodiment, the electronic device can be a data transmitter or a terminal device. Referring to FIG. 13 , which is a schematic diagram of the structure of a possible electronic device provided in an embodiment of the present application, electronic device 1300 includes a processor 1310 and a memory 1320.
[0232] The memory 1320 stores a computer program that can be executed by the processor 1310 . The processor 1310 can execute the steps of the above-mentioned media data transmission method by executing the instructions stored in the memory 1320 .
[0233] Memory 1320 can be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1320 can also be a combination of the above memories.
[0234] The processor 1310 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1310 is configured to implement the above-mentioned media data transmission method when executing the computer program stored in the memory 1320 .
[0235] In some embodiments, the processor 1310 and the memory 1320 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.
[0236] The specific connection medium between the processor 1310 and the memory 1320 is not limited in the embodiments of the present application. In the embodiments of the present application, the bus connection between the processor 1310 and the memory 1320 is used as an example. The bus is depicted as a bold line in FIG13 . The connection methods between other components are merely illustrative and are not intended to be limiting. Buses can be divided into address buses, data buses, control buses, etc. For ease of description, FIG13 depicts only a single bold line, but this does not imply that there is only one bus or only one type of bus.
[0237] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is executed on an electronic device, the computer program is used to cause the electronic device to perform the steps of the above-mentioned media data transmission method. In some possible implementations, various aspects of the media data transmission method provided in this application can also be implemented in the form of a program product, which includes a computer program. When the program product is executed on an electronic device, the computer program is used to cause the electronic device to perform the steps of the above-mentioned media data transmission method. For example, the electronic device can perform the steps shown in Figure 3.
[0238] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0239] The program product of the embodiments of the present application may be a CD-ROM and include a computer program, and may be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a computer program that can be used by or in conjunction with a command execution system, apparatus, or device.
[0240] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a computer program for use by or in conjunction with a command execution system, apparatus, or device.
[0241] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.
[0242] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.
Claims
1. A media data transmission method, performed by an electronic device, comprising: Acquire multiple first media frames, send flow message sets of the multiple first media frames to a data receiving end, and receive at least one response message corresponding to the multiple first media frames returned by the data receiving end; Based on the at least one response message, performing a first determination step of whether the data receiving end can receive at least one media frame data of the first media frame; When the first determination result of the first determination step is positive, for each of at least one of the first media frames, according to a preset playback order of the plurality of first media frames, a second determination step is performed on whether the data receiving end can play at least one of the first media frames; Based on the second determination result of the second determination step, predicting a storage state of a storage unit, such as a cache unit, of the data receiving end; and, A corresponding preset data transmission scheme is called according to the storage state to send a second media frame to the data receiving end by using the data transmission scheme, where the second media frame is different from the multiple first media frames, or the second media frame includes at least one of the first media frames.
2. The method of claim 1, wherein: When the determination result of the first determining step is positive, for each of at least one of the first media frames, according to the preset playback order of the plurality of first media frames, performing a second determining step of whether the data receiving end can play at least one of the first media frames comprises: For each first media frame of at least one of the first media frames, perform the following operations: When it is determined based on the playback order that there is at least one other media frame before the first media frame among the multiple first media frames, it is detected whether the data receiving end can play the first media frame based on the first determination result corresponding to each of the at least one other media frame.
3. The method of claim 2, wherein: The detecting, based on the first determination result corresponding to each of the at least one other media frames, whether the data receiving end can play the first media frame includes: If it is determined that the data receiving end receives each media frame in the at least one other media frame based on the first determination result corresponding to each of the at least one other media frames, it is determined that the client can play the first media frame; If it is determined based on the first determination result corresponding to each of the at least one other media frames that there is a media frame that has not been received by the data receiving end in the at least one other media frame, it is determined that the client cannot play the first media frame.
4. The method of claim 1, 2 or 3, wherein: The predicting the storage state of the storage unit of the data receiving end based on the second determination result of the second determination step includes: Based on the second determination result and in combination with the media frame currently played by the data receiving end, obtaining the usage data of the cache; Based on the usage data, the storage status is determined.
5. The method of claim 4, wherein: The cache usage data includes: the cache quantity and cache duration of media frames in the cache; Based on the second determination result and in combination with the media frame currently played by the data receiving end, the usage data of the cache is obtained, including: When it is determined based on the second determination result that at least one of the first media frames has at least one playable frame, determining a target frame whose play time meets a set reference condition from the at least one playable frame based on the play time of each of the at least one playable frame; Obtaining the cache duration based on the respective playback times of the media frame currently played by the data receiving end and the target frame; The number of the at least one playable frame is used as the buffer number.
6. The method of claim 5, wherein: The determining the storage status based on the usage data includes: If the cache duration is less than a preset first duration threshold and the cache quantity is less than a preset first quantity threshold, determining that the storage state is insufficient cache occupancy; If the cache duration is greater than a preset second duration threshold and the cache quantity is greater than a preset second quantity threshold, it is determined that the storage state is that the cache is sufficiently occupied.
7. The method of claim 1, 2 or 3, wherein: The adopting the data transmission scheme to send the second media frame to the data receiving end includes: Determining initial transmission parameters based on network transmission performance between the data sending end and the data receiving end; Based on the data transmission scheme, adjusting the initial transmission parameters to obtain target transmission parameters; Based on the target transmission parameter, the second media frame is sent to the data receiving end.
8. The method of claim 7, wherein: The adjusting the initial transmission parameters based on the data transmission scheme to obtain target transmission parameters includes: If the storage state indicates that the cache is insufficiently occupied, adjusting the initial transmission parameters according to a data transmission scheme that increases the amount of transmitted data to obtain the target transmission parameters; If the storage status indicates that the cache is sufficiently occupied, the initial transmission parameters are adjusted according to a data transmission scheme for reducing the amount of transmitted data to obtain the target transmission parameters.
9. The method of claim 1, 2 or 3, wherein: The playback order of the multiple first media frames is determined in the following manner: Obtain a message identification set of each first media frame recorded locally, wherein the message identification set includes the message identification of each flow message in the flow message set of the first media frame, and each message identification is obtained by numbering according to the sending order of each flow message; The play order of the multiple first media frames is determined based on each message identifier included in each message identifier set.
10. The method of claim 1, 2 or 3, wherein: The first step of determining whether the data receiving end can receive at least one media frame data of the first media frame based on the at least one response message includes: Obtain a message identification set of each of the first media frames recorded locally, wherein the message identification set includes the message identification of each flow message in the flow message set of the first media frame, and each message identification is obtained by numbering according to the sending order of each flow message; Based on each message identifier included in each message identifier set and a message identifier carried by each of the at least one response message, it is determined that the data receiving end has received media frame data of at least one of the first media frames.
11. A media data transmission device, comprising: A traffic transceiver unit, configured to obtain a plurality of first media frames, send a set of traffic messages of the plurality of first media frames to a data receiving end, and receive at least one response message corresponding to the plurality of first media frames returned by the data receiving end; A first determining unit, configured to perform a first determining step of whether the data receiving end can receive media frame data of at least one of the first media frames based on the at least one response message; a second determining unit, configured to, when a first determination result of the first determining step is positive, perform, for each of at least one of the first media frames, a second determining step of whether the data receiving end can play at least one of the first media frames according to a preset playback order of the plurality of first media frames; a cache prediction unit, configured to predict a storage state of a storage unit, such as a cache unit, of the data receiving end based on a second determination result of the second determination step; and, A transmission scheme calling unit is used to call the corresponding preset data transmission scheme according to the storage state, so as to adopt the data transmission scheme to send a second media frame to the data receiving end, where the second media frame is different from the multiple first media frames, or the second media frame contains at least one of the first media frames.
12. The device of claim 11, wherein: The second determination unit is used to perform the following operations for each first media frame of at least one of the first media frames: when it is determined based on the playback order that there is at least one other media frame before the first media frame among the multiple first media frames, based on the first determination results corresponding to each of the at least one other media frames, detect whether the data receiving end can play the first media frame.
13. The device of claim 12, wherein: The second determination unit is configured to determine that the client can play the first media frame if it is determined that the data receiving end receives each media frame in the at least one other media frame based on the first determination result corresponding to each of the at least one other media frames; If it is determined based on the first determination result corresponding to each of the at least one other media frames that there is a media frame that has not been received by the data receiving end in the at least one other media frame, it is determined that the client cannot play the first media frame.
14. The device of claim 11, 12 or 13, wherein: The cache prediction unit is used to obtain usage data of the cache based on the second determination result and in combination with the media frame currently played by the data receiving end; and determine the storage state based on the usage data.
15. The device of claim 11, 12 or 13, wherein: The transmission scheme calling unit is used to determine initial transmission parameters based on network transmission performance between the data sending end and the data receiving end; Based on the data transmission scheme, the initial transmission parameters are adjusted to obtain target transmission parameters; based on the target transmission parameters, the second media frame is sent to the data receiving end.
16. The device of claim 11, 12 or 13, wherein: The second determination unit is further used to obtain a message identification set of each first media frame recorded locally, wherein the message identification set includes a message identification of each flow message in a flow message set of the first media frame, and each message identification is obtained by numbering according to the sending order of each flow message; The play order of the multiple first media frames is determined based on each message identifier included in each message identifier set.
17. The device of claim 11, 12 or 13, wherein: The first determination unit is used to obtain a message identification set of each first media frame recorded locally, wherein the message identification set includes the message identification of each flow message in the flow message set of the first media frame, and each message identification is obtained by numbering according to the sending order of each flow message; Based on each message identifier included in each message identifier set and a message identifier carried by each of the at least one response message, it is determined that the data receiving end has received media frame data of at least one of the first media frames.
18. An electronic device comprising a processor and a memory, wherein: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of any one of the methods of claims 1 to 10.
19. A computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on an electronic device, the computer program is used to cause the electronic device to execute the steps of any one of the methods of claims 1 to 10.
20. A computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of any method described in claims 1 to 10.
Citation Information
Patent Citations
Video code rate selecting method driven by user experience under HSPA system
CN103338393A
Video coding stream control method facing network live broadcast
CN103475934A
Data buffering method and network equipment
CN108881931A
Streaming media real-time playing method and a device
CN109168083A
Practical network bandwidth allocation method for fairness of video experience quality
CN113038187A
Cited By
Multimedia data processing method, device, and computer-readable storage medium
WO2026199440A1
