Method, device, computer device and readable storage medium for transmitting media stream data

Abstract

The application relates to a media stream data transmission method and device, computer equipment and a readable storage medium. The method comprises the following steps: obtaining media stream data to be sent, and writing the media stream data into a buffer queue; obtaining network feedback information of a current transmission channel; respectively smoothing a packet loss factor and a transmission code rate in the network feedback information to obtain a smoothed packet loss factor and a smoothed transmission code rate; determining a first transmission interval according to the smoothed packet loss factor, and determining a second transmission interval according to the number of bytes to be sent in the buffer queue and the smoothed transmission code rate; determining a target transmission interval according to the smoothed packet loss factor, the first transmission interval and the second transmission interval; and determining to send the media stream data in the buffer queue to a corresponding receiving end according to the target transmission interval. In this way, the communication problems such as congestion and sending adjustment lag are effectively solved, the network queuing delay is reduced, the link congestion probability is reduced, and the real-time communication fluency and stability are improved.

Description

Technical Field

[0001] This application relates to the field of real-time communication technology, and in particular to a method, apparatus, computer device, and readable storage medium for transmitting media stream data. Background Technology

[0002] In the field of real-time communication, after receiving media stream data from the encoder, the sending end can transmit the media stream data and other media data to the remote end through the network protocol stack. During the transmission of media stream data, data bursts from upstream may be written to the underlying network sending interface in a concentrated manner, causing data congestion during transmission. This congestion may further lead to network queuing, increased instantaneous jitter, increased packet loss rate, and buffer instability, thus affecting the communication quality of real-time communication. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, computer device, and readable storage medium for transmitting media stream data to address the aforementioned technical problems. This method can make the communication process more stable and controllable while meeting the requirements of real-time communication, thereby enabling smoother real-time communication.

[0004] In a first aspect, this application provides a method for transmitting media stream data, the method comprising:

[0005] Acquire the media stream data to be sent and write the media stream data to the buffer queue;

[0006] Obtain network feedback information for the current transmission channel; the network feedback information includes at least the packet loss factor and transmission rate.

[0007] The packet loss factor and transmission rate in the network feedback information are smoothed to obtain the smoothed packet loss factor and smoothed transmission rate.

[0008] The first transmission interval is determined based on the smooth packet loss factor, and the second transmission interval is determined based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate.

[0009] The target transmission interval is determined based on the smooth packet loss factor, the first transmission interval, and the second transmission interval.

[0010] Based on the target transmission interval, the media stream data in the buffer queue is sent to the corresponding receiving end.

[0011] In one embodiment, the sending end includes a transport layer, and the step of obtaining network feedback information of the current transmission channel includes:

[0012] Obtain the reception statistics of the transport layer; wherein, the reception statistics include at least the transmission sequence number and arrival timestamp of the data packets received by the transport layer;

[0013] Calculate the reception time interval between data packets and adjacent data packets based on the arrival timestamp;

[0014] According to the transmission sequence number, data packets with a reception time interval less than or equal to a preset grouping threshold are divided into the same data packet group, and the transmission code rate of the data packet group is calculated.

[0015] The packet loss factor is obtained by calculating the difference between the data packets sent and received by the transport layer.

[0016] In one embodiment, the step of smoothing the packet loss factor and transmission rate in the network feedback information to obtain the smoothed packet loss factor and smoothed transmission rate includes:

[0017] The packet loss factor is subjected to exponential weighted moving smoothing using the first time constant to filter out packet loss noise data in the packet loss factor and obtain a smoothed packet loss factor.

[0018] The transmission bitrate is smoothed by an exponentially weighted moving average using a second time constant to filter out bitrate fluctuation noise data and obtain a smooth transmission bitrate; wherein, the transmission bitrate is the instantaneous bitrate, and the second time constant is greater than the first time constant.

[0019] In one embodiment, the steps of determining a first transmission interval based on a smooth packet loss factor and determining a second transmission interval based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate include:

[0020] If the smoothing packet loss factor is less than or equal to the first packet loss threshold, the current transmission interval of the current transmission channel is multiplied by the first multiplier to obtain the first transmission interval; wherein the first multiplier is less than 1.

[0021] Alternatively, if the smoothing packet loss factor is greater than or equal to the second packet loss threshold, the current transmission interval of the current transmission channel is multiplied by the second multiplier to obtain the first transmission interval; wherein the first multiplier is greater than 1.

[0022] Alternatively, if the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, the current transmission interval of the current transmission channel is linearly interpolated according to the first multiplier and the second multiplier to obtain the first transmission interval; wherein the first packet loss threshold is lower than the second packet loss threshold.

[0023] Divide the number of bytes to be sent by the smoothed transmission rate to determine the duration of continuous transmission required for the number of bytes to be sent, and calculate the second transmission interval for each data packet based on the duration of continuous transmission.

[0024] In one embodiment, the step of determining the target transmission interval based on the smoothing packet loss factor, the first transmission interval, and the second transmission interval includes:

[0025] If the smoothing packet loss factor is less than or equal to the first packet loss threshold, the smaller value between the first transmission interval and the second transmission interval shall be taken as the target transmission interval.

[0026] Alternatively, if the smoothing packet loss factor is greater than or equal to the second packet loss threshold, the larger of the first transmission interval and the second transmission interval shall be taken as the target transmission interval.

[0027] Alternatively, if the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, calculate the normalized ratio of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold; wherein, the normalized ratio is used to represent the relative position of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold.

[0028] Based on the normalization ratio, a linear interpolation is performed between the smaller and larger values ​​in the first and second transmission intervals to obtain the target transmission interval.

[0029] In one embodiment, the step of sending the media stream data in the buffer queue to the corresponding transmission interface at the target transmission time includes:

[0030] Calculate the target transmission duration of the media stream data in the buffer queue based on the target transmission interval;

[0031] The target transmission duration is transmitted along with the media stream data to the transmission layer at the sending end, so that the media stream data is split into multiple corresponding transmission data packets;

[0032] Based on the target transmission duration and the number of data packets, the data packet distribution transmission parameters are calculated, and the system sleeps during the intervals between data packet transmissions according to the data packet distribution transmission parameters to avoid all data packets of the same media stream being sent at once until multiple data packets are sent to the corresponding transmission interface.

[0033] In one embodiment, the media stream data includes audio data frames and video data frames, the buffer queue includes an audio buffer queue and a video buffer queue, and the method further includes:

[0034] Write audio data frames to the audio buffer queue and video data frames to the video buffer queue.

[0035] Audio data frames in the audio buffer queue are sent first, and are not limited by the target transmission interval;

[0036] If the video buffer queue includes multiple non-empty video buffer queues, the transmission channel containing the video data frame with the earliest timestamp in the video buffer queue shall be selected for priority transmission.

[0037] Secondly, this application also provides a media stream data transmission apparatus, the apparatus comprising:

[0038] The first acquisition module is used to acquire the media stream data to be sent and write the media stream data into the buffer queue;

[0039] The second acquisition module is used to acquire network feedback information of the current transmission channel; wherein, the network feedback information includes at least the packet loss factor and the transmission rate;

[0040] The smoothing module is used to smooth the packet loss factor and transmission rate in the network feedback information respectively, so as to obtain the smoothed packet loss factor and smoothed transmission rate.

[0041] The first determining module is used to determine the first transmission interval based on the smooth packet loss factor, and to determine the second transmission interval based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate.

[0042] The second determining module is used to determine the target transmission interval based on the smooth packet loss factor, the first transmission interval, and the second transmission interval.

[0043] The transmission module is used to determine the media stream data in the buffer queue to be sent to the corresponding receiving end based on the target transmission interval.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the media stream data transmission method of the first aspect.

[0045] Fourthly, this application also provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the media stream data transmission method of the first aspect.

[0046] The aforementioned media stream data transmission method, apparatus, computer equipment, and readable storage medium acquire the media stream data to be sent and write it into a buffer queue; acquire network feedback information of the current transmission channel; wherein the network feedback information includes at least a packet loss factor and a transmission rate; smooth the packet loss factor and transmission rate in the network feedback information to obtain a smoothed packet loss factor and a smoothed transmission rate; determine a first transmission interval based on the smoothed packet loss factor, and determine a second transmission interval based on the number of bytes to be sent in the buffer queue and the smoothed transmission rate; determine a target transmission interval based on the smoothed packet loss factor, the first transmission interval, and the second transmission interval; and determine the media stream data in the buffer queue to be sent to the corresponding receiving end based on the target transmission interval. In this way, by acquiring media stream data and writing it into a buffer queue, combining it with network feedback information for smoothing, dynamically calculating the first and second transmission intervals, and comprehensively determining the target transmission interval based on the smoothed packet loss factor to control the transmission rhythm, this effectively solves the problems of excessively high instantaneous traffic peaks, congestion caused by concentrated transmission of key frames, and delayed transmission adjustments in existing technologies. It has the advantages of reducing network queuing delay, reducing the probability of link congestion, and improving the smoothness and stability of real-time communication. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the system architecture of the media stream data transmission method in this embodiment;

[0049] Figure 2 This is a schematic diagram of the first process of the media stream data transmission method in this embodiment;

[0050] Figure 3 This is a schematic diagram of the second process of the media stream data transmission method in this embodiment;

[0051] Figure 4 This is a schematic diagram of the third process of the media stream data transmission method in this embodiment;

[0052] Figure 5 This is a schematic diagram of the structure of a media stream data transmission device in one embodiment;

[0053] Figure 6 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0056] The media stream data transmission method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 can be a media stream data transmission device. It can acquire media stream data to be sent and write the media stream data into a buffer queue; acquire network feedback information of the current transmission channel; wherein the network feedback information includes at least a packet loss factor and a transmission rate; smooth the packet loss factor and transmission rate in the network feedback information to obtain a smoothed packet loss factor and a smoothed transmission rate; determine a first transmission interval based on the smoothed packet loss factor, and determine a second transmission interval based on the number of bytes to be sent in the buffer queue and the smoothed transmission rate; determine a target transmission interval based on the smoothed packet loss factor, the first transmission interval, and the second transmission interval; and determine the media stream data in the buffer queue to be sent to the corresponding receiving end based on the target transmission interval.

[0057] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0058] In real-time audio and video communication systems, the transmitting end continuously receives data frames output by the encoder and transmits the media data to the remote end via the network protocol stack. Due to factors such as large fluctuations in encoder output frame size, short-term fluctuations in network feedback, asynchronous accumulation of multi-channel buffers, fixed granularity of sending thread scheduling, and controlling only the average bitrate without controlling the sending rhythm within a short time window, the transmitting end is prone to sending multiple media packets in a short period of time, resulting in excessively high instantaneous traffic peaks. This sudden sending behavior increases network queuing latency, instantaneous jitter, packet loss rate, and receiver buffer instability, and in turn, it perturbs the bandwidth estimation results, ultimately affecting the overall real-time communication quality.

[0059] For example, in video conferencing applications, when multiple participants simultaneously open video channels and one person switches camera views, the encoder generates keyframes, which are significantly larger than regular frames. Simultaneously, network conditions may deteriorate briefly due to wireless signal interference, causing a rapid drop in the transmission bitrate. At this point, the sending end may write keyframes and adjacent regular frames together into the network interface, causing momentary link congestion. Specifically, this manifests as a buildup of keyframes in the transmission queue, a rapidly increasing packet loss factor in network feedback, but the transmission interval is not adjusted in time, resulting in stuttering and screen tearing in the receiving end, intermittent audio, and impaired timing consistency.

[0060] If these problems are not addressed, network congestion will worsen, packet loss rates will continue to rise, and the bandwidth estimation mechanism will output inaccurate target transmission rates due to feedback distortion. This will lead to further instability in the transmission rhythm, creating a vicious cycle that ultimately degrades the quality of real-time audio and video communication, causes receiver buffer management to fail, and reduces the overall system reliability.

[0061] Based on this, this embodiment proposes a method for transmitting media stream data, which dynamically adjusts the data transmission rhythm by comprehensively considering network feedback information and buffer queue status, in order to achieve smooth and stable data transmission. Figure 2 As shown, this method is exemplarily applied to Figure 1 The terminal 102 shown may include the following steps 21 to 26, wherein:

[0062] Step 21: Obtain the media stream data to be sent and write the media stream data into the buffer queue.

[0063] Media streaming data refers to continuous data transmitted over a network, such as real-time audio and video data. Media streaming data is typically generated by an encoder and needs to be sent to the receiving end in a continuous and stable manner to ensure user experience.

[0064] Here, the buffer queue can be a data structure used to temporarily store media stream data to be sent. During data transmission, media stream data is first written to the buffer queue, and then retrieved from the queue and sent according to the transmission strategy. The buffer queue helps smooth the data flow and address the mismatch between the data generation rate and the transmission rate.

[0065] As one implementation method, media stream data can be generated in real time by a local encoder. For example, in video conferencing applications, the raw video frames captured by the camera are encoded to form video stream data. The video stream data can be directly written into a first-in-first-out (FIFO) buffer queue, waiting to be sent.

[0066] As another implementation, media stream data can also be audio and video files pre-stored in local memory, read from memory and sequentially placed into a buffer queue during transmission. The purpose of the buffer queue is to provide a temporary storage area to cope with the instantaneous difference between the media stream data generation rate and the network transmission rate, avoiding data loss or transmission interruption.

[0067] Step 22: Obtain network feedback information for the current transmission channel.

[0068] The network feedback information includes at least the packet loss factor and the transmission rate.

[0069] Here, network feedback information refers to data about the network status obtained from the transmission channel. Network feedback information can be used to dynamically adjust data transmission strategies to adapt to the constantly changing network environment.

[0070] Specifically, the packet loss factor can be an indicator of the loss of data packets during transmission. A high packet loss factor usually indicates network congestion or poor link quality. The packet loss factor can be used to assess the reliability of the current network.

[0071] In addition, transmission rate can refer to the actual rate at which data is transmitted in the transmission channel. Transmission rate can be used to reflect the amount of data that the current network can carry and is an important parameter for evaluating network bandwidth.

[0072] For example, network feedback information can be obtained in various ways. For instance, the sender can periodically send probe packets to the receiver and use the acknowledgment information returned by the receiver (such as confirmation of receipt or non-receipt) to count packet loss and calculate the packet loss factor. Simultaneously, the sender can also calculate the transmission rate by counting the number of bytes successfully transmitted within a certain time window.

[0073] As another implementation method, network feedback information can also be obtained through reports actively sent by the receiving end. These reports include information such as packet loss rate and received bit rate as statistically observed by the receiving end. This raw network feedback information reflects the instantaneous state of the current network.

[0074] Step 23: Smooth the packet loss factor and transmission rate in the network feedback information to obtain the smoothed packet loss factor and smoothed transmission rate.

[0075] Among them, the smoothed packet loss factor is a packet loss factor after smoothing, which can more accurately reflect the average packet loss trend over a period of time and reduce misjudgments caused by short-term packet loss noise.

[0076] In addition, the smoothed transmission rate is the transmission rate after smoothing, which can more stably reflect the average transmission capacity over a period of time and avoid drastic adjustments in the transmission rate caused by instantaneous bit rate fluctuations.

[0077] It should be noted that smoothing is a process of filtering or averaging the original data, aiming to eliminate instantaneous fluctuations or noise in the data, thereby obtaining more stable and representative trend information. Through smoothing, transmission decisions that overreact based on instantaneous abnormal data can be avoided.

[0078] Understandably, the purpose of smoothing is to filter out noise caused by instantaneous network fluctuations and obtain a more stable network state assessment.

[0079] As an example, a simple moving average algorithm can be used to process the packet loss factor, that is, taking the average of the most recent N packet loss factors as the smoothed packet loss factor. Similarly, the transmission bitrate can also be smoothed using a similar moving average algorithm to obtain a smoothed transmission bitrate. In this way, frequent and drastic adjustments to the transmission strategy caused by instantaneous network jitter can be avoided.

[0080] Step 24: Determine the first transmission interval based on the smooth packet loss factor, and determine the second transmission interval based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate.

[0081] The determination of the first transmission interval can be based on a preset mapping relationship.

[0082] For example, a lookup table can be defined that directly corresponds to a preset first transmission interval value based on different smooth packet loss factor ranges.

[0083] It should be noted that when the smoothing packet loss factor is low, the first transmission interval can be small, allowing for faster transmission; when the smoothing packet loss factor is high, the first transmission interval can be large, in order to slow down the transmission rate.

[0084] Here, the determination of the second transmission interval focuses on the efficiency of clearing the buffer queue.

[0085] Specifically, dividing the number of bytes currently to be sent in the buffer queue by the smoothed transmission rate yields the duration of continuous transmission required to send all this data. Then, based on this duration and the expected size of each data packet, a second transmission interval for transmitting each data packet can be calculated.

[0086] Step 25: Determine the target transmission interval based on the smooth packet loss factor, the first transmission interval, and the second transmission interval.

[0087] The target transmission interval is the final data transmission interval determined by comprehensively considering the smooth packet loss factor, the first transmission interval, and the second transmission interval. The target transmission interval can be used to balance the requirements of network congestion control and real-time data transmission.

[0088] The first transmission interval is a transmission time interval determined based on the smoothing packet loss factor. The first transmission interval is mainly used to respond to network packet loss. When the packet loss rate is high, this interval may be adjusted to slow down the transmission rate.

[0089] The second transmission interval is a transmission time interval determined based on the number of bytes to be sent in the buffer queue and the smooth transmission bit rate. The second transmission interval is mainly used to ensure that the data in the buffer queue can be sent in a timely manner, avoiding queue overflow or data accumulation.

[0090] The target transmission interval is the final data transmission interval determined by comprehensively considering the smooth packet loss factor, the first transmission interval, and the second transmission interval. The target transmission interval can be used to balance the needs of network congestion control and real-time data transmission.

[0091] It should be noted that the determination of the target transmission interval needs to take into account both the network congestion situation (reflected by the smoothing packet loss factor and the first transmission interval) and the pressure on the buffer queue (reflected by the second transmission interval).

[0092] As an example, the smaller of the first and second transmission intervals can be taken as the target transmission interval to clear the buffer queue as quickly as possible while ensuring that the network is not congested.

[0093] As another implementation, when the smoothing packet loss factor is low, the second transmission interval can be given priority to improve transmission efficiency; while when the smoothing packet loss factor is high, the first transmission interval can be given priority to reduce network burden.

[0094] Step 26: Determine the media stream data in the buffer queue to be sent to the corresponding receiving end based on the target transmission interval.

[0095] The receiving end is the device or application that receives the media stream data. The sending end transmits the media stream data to the corresponding receiving end via the network to complete real-time communication or media playback.

[0096] Once the target transmission interval is determined, the sending end can retrieve media stream data from the buffer queue according to that interval, encapsulate it into data packets, and send them out. For example, the sending end can set a timer to send a data packet each time the timer reaches the target transmission interval. This sending mechanism based on dynamically adjusted target transmission intervals allows the data transmission rhythm to flexibly adapt to changes in network conditions and the buffer queue, thereby achieving smooth data stream transmission.

[0097] It's important to note that media streaming data typically consists of continuous audio and video information, which is then divided into discrete data units for transmission. Audio data frames carry sound information and usually have extremely high real-time requirements; even slight delays or jitter can lead to poor listening experience. Video data frames carry image information and, while also requiring some real-time performance, generally have a slightly higher tolerance for latency than audio, and their data volume is much larger.

[0098] In one embodiment, the media stream data includes audio data frames and video data frames, the buffer queue includes an audio buffer queue and a video buffer queue, and the method further includes:

[0099] (1) Write the audio data frame to the audio buffer queue and write the video data frame to the video buffer queue.

[0100] (2) Prioritize sending audio data frames in the audio buffer queue, and are not limited by the target transmission interval.

[0101] (3) If the video buffer queue includes multiple non-empty video buffer queues, the transmission channel containing the video data frame with the earliest timestamp in the video buffer queue shall be selected for priority transmission.

[0102] Here, the buffer queue is a memory area used to temporarily store data to be sent.

[0103] Setting up independent audio and video buffer queues enables independent management and scheduling of different types of media data, avoiding data interference between different media types, and allowing different transmission strategies to be adopted according to their respective characteristics (such as real-time requirements and data size).

[0104] For example, a dedicated, higher-priority buffer area can be allocated for audio data frames, or a larger buffer space can be provided for video data frames to cope with network fluctuations.

[0105] When media stream data enters the transmission system, it can be categorized and processed according to its type. For example, when the encoder generates audio data frames, they are sent to the audio buffer queue; when it generates video data frames, they are sent to the video buffer queue. This ensures that data of different media types can be correctly identified and processed.

[0106] When transmission resources are limited, audio data frames will receive a higher scheduling priority. Unrestricted by the target transmission interval, even if the target transmission interval calculated based on network feedback indicates poor network conditions requiring slower transmission, audio data frames can still be sent at a faster rate or with smaller intervals to ensure low-latency transmission. This can be achieved by setting a separate transmission timer for audio data frames or assigning them the highest priority in the scheduler.

[0107] The earliest timestamped data frame typically means it's the "oldest" data. If it's not sent promptly, it can cause stuttering or desynchronization at the receiving end. When multiple video streams exist (e.g., multi-channel video conferencing or multi-view live streaming), each stream may have its own independent video buffer queue. To ensure smooth and synchronized video playback, the earliest timestamped video frame can be prioritized for transmission, thus maintaining the overall playback quality of the multiple video streams.

[0108] In existing technologies, the transmitting end often struggles to effectively handle instantaneous network fluctuations and buffer queue pressure, easily leading to burst transmissions and consequently causing network congestion, increased jitter, and higher packet loss rates. For example, when the encoder suddenly outputs a large keyframe, existing solutions may immediately transmit it at a high burst rate, causing instantaneous network link congestion. In the aforementioned media stream data transmission method, this embodiment effectively filters out instantaneous noise by introducing smoothing processing of network feedback information (such as packet loss factor and transmission bit rate), making the assessment of network status more stable and accurate. This contrasts with the existing technology's approach of directly using instantaneous feedback information for decision-making, avoiding drastic fluctuations in the transmission rhythm caused by short-term network volatility.

[0109] Specifically, this embodiment comprehensively considers the network congestion control requirements based on a smooth packet loss factor (reflected by the first transmission interval) and the real-time requirements based on the amount of data to be sent in the buffer queue (reflected by the second transmission interval), making the determination of the target transmission interval more comprehensive and precise. Existing technologies typically only focus on average bit rate control or fixed time slice transmission, lacking this comprehensive adjustment capability that combines packet loss feedback and frame buffer status. By dynamically determining the target transmission interval, this embodiment can achieve fine-grained scheduling of the media stream data transmission rhythm. When network conditions deteriorate, the transmission rhythm can automatically slow down, effectively avoiding the problem of continuing to use aggressive transmission intervals in existing technologies. When network conditions recover, the transmission rhythm can also smoothly recover, avoiding new jitter caused by overly abrupt adjustments. This adaptive transmission shaping capability significantly reduces short-term transmission peaks, reduces network queuing latency and jitter, thereby improving the overall quality and stability of real-time communication.

[0110] In some of the above embodiments, this application proposes a method for transmitting media stream data, which relies on obtaining network feedback information of the current transmission channel, particularly the packet loss factor and transmission bit rate, to dynamically adjust the transmission strategy. However, in real-world network environments, how to accurately and reliably extract this crucial network feedback information from the raw network data, especially when network conditions are complex and variable, is a technical problem that needs to be solved. Simply obtaining network feedback information may lead to inaccurate data, thereby affecting the calculation of subsequent transmission intervals and the efficiency and stability of media stream transmission.

[0111] Based on this, an exemplary embodiment is proposed, such as... Figure 3 As shown, the steps of obtaining network feedback information of the current transmission channel and smoothing the network feedback information may include steps 221 to 224 and steps 231 to 232, wherein:

[0112] Step 221: Obtain the reception statistics of the transport layer.

[0113] The received statistics include at least the transmission sequence number and arrival timestamp of the data packets received by the transport layer.

[0114] The transport layer is a crucial layer in the network protocol stack, responsible for end-to-end data transmission. It provides reliable or unreliable data transmission services to upper-layer applications and handles functions such as data segmentation, reassembly, flow control, and congestion control. The transport layer can be a standard protocol, such as Transmission Control Protocol (TCP) or User Datagram Protocol (UDP), or a custom transport protocol built upon these standard protocols to meet the specific latency, reliability, or bandwidth requirements of particular applications (such as real-time media streaming).

[0115] Receive statistics refer to various data collected by the transport layer during data reception regarding the transmission status of data packets, which are crucial for assessing network conditions. In addition to sequence numbers and arrival timestamps, receive statistics may also include round-trip time (RTT), jitter, number of out-of-order packets, and number of duplicate packets. These data collectively depict the quality of data packet transmission within the network.

[0116] It's important to note that the transmission sequence number is a unique identifier for each data packet, used at the receiving end for packet sorting, detection of loss, and duplicates. The arrival timestamp records the precise time the data packet was received by the receiving end's transport layer, typically based on the system clock or a high-precision timer. The transmission sequence number allows us to determine the order and integrity of data packets, while the arrival timestamp allows us to analyze transmission delays and arrival intervals.

[0117] Step 222: Calculate the reception time interval between the data packet and the adjacent data packet based on the arrival timestamp.

[0118] The receiving time interval refers to the time difference between two consecutively received data packets.

[0119] These intervals can be calculated by comparing the arrival timestamps of the data packets. For example, if data packet A arrives at time T1 and data packet B (the next data packet following A) arrives at time T2, the reception interval is T2-T1. This interval may not be used to reflect the transmission rate and jitter of the data packets in the network.

[0120] Step 223: According to the transmission sequence number, data packets with a reception time interval less than or equal to a preset grouping threshold are divided into the same data packet group, and the transmission code rate of the data packet group is calculated.

[0121] Furthermore, data packets with similar transmission characteristics or arriving consecutively within a short period can be grouped together. The logical continuity of data packets within a group is ensured by using transmission sequence numbers, and data packets transmitted "closely" in the network can be identified by comparing the reception time interval with a preset group threshold. For example, if the reception time interval of consecutively arriving data packets is less than a certain threshold (e.g., 5 milliseconds), they are considered to belong to the same burst transmission.

[0122] As an example, the transmission rate can be calculated by dividing the total number of bytes in the data packets within the packet by the duration of the packet (i.e., the time difference between the arrival timestamp of the first data packet and the arrival timestamp of the last data packet).

[0123] As another calculation method, the transmission bit rate of the corresponding data packet can be obtained by multiplying the number of data packets in the group by the average data packet size and then dividing by the group duration.

[0124] Step 224: Calculate the difference between the data packets sent and received by the transport layer to obtain the packet loss factor.

[0125] Here, the packet loss factor is an indicator that measures the network packet loss rate.

[0126] As an example, the packet loss factor can be calculated by comparing the difference between the number of data packets actually sent by the transport layer at the sending end and the number of data packets actually received at the receiving end. For instance, if the sending end sends N data packets, but the receiving end only receives M data packets (M < N), the packet loss factor can be expressed as (NM) / N.

[0127] As another example, the number of missing sequence numbers can be counted based on the continuity detection of transmission sequence numbers, and the missing sequence numbers can be used as the packet loss number to calculate the corresponding packet loss factor.

[0128] Step 231: The packet loss factor is subjected to exponential weighted moving smoothing using the first time constant to filter out packet loss noise data in the packet loss factor and obtain a smoothed packet loss factor.

[0129] Among them, the smoothing loss factor can be a data processing method that performs exponential weighted moving smoothing on the loss factor. The smoothing loss factor gives higher weight to recent data, which can effectively suppress noise while responding to data changes.

[0130] As an example, the iterative formula for smoothing can be as follows:

[0131] Smoothing value = α * current value + (1 - α) * previous smoothing value.

[0132] Here, α is the smoothing coefficient, which has a value between 0 and 1. The smoothing coefficient can be used to reflect the importance attached to the current data.

[0133] As another example, a sliding window can be used, with exponentially decaying weights assigned to the data points within the window, so that data points closer to the current time have greater weights.

[0134] Here, the first time constant is a parameter used to control the response speed of the packet loss factor smoothing process.

[0135] It should be noted that a smaller time constant makes the smoothing results more sensitive to recent changes in the packet loss factor and can reflect instantaneous changes in network conditions more quickly; while a larger time constant makes the smoothing results more stable and less sensitive to instantaneous fluctuations.

[0136] It can be further explained that the time constant can be set according to the requirements of the response speed to changes in the packet loss factor in the actual application scenario. For example, it can be reflected by adjusting the smoothing coefficient in the exponentially weighted moving average algorithm.

[0137] Understandably, smoothing the packet loss factor can filter out packet loss noise data, thereby eliminating atypical packet loss events caused by accidental factors or brief interference in the network, avoiding misjudgments of network conditions by these transient noises, and ensuring that the smoothed packet loss factor can more accurately reflect the actual network congestion or link quality.

[0138] Step 232: The transmission bit rate is smoothed by exponential weighted moving average using a second time constant to filter out bit rate fluctuation noise data in the transmission bit rate and obtain a smooth transmission bit rate.

[0139] Among them, the transmission code rate is the instantaneous code rate, and the second time constant is greater than the first time constant.

[0140] Here, the second time constant is a parameter used to control the response speed of the transmission bit rate smoothing process.

[0141] Similar to the first time constant, this time constant determines the degree to which the smooth transmission bit rate responds to instantaneous bit rate fluctuations.

[0142] Specifically, this time constant can also be set by adjusting the smoothing coefficient in the exponentially weighted moving average algorithm.

[0143] Understandably, smoothing the transmission bitrate can filter out bitrate fluctuation noise data, thereby eliminating short-term, non-continuous fluctuations in the instantaneous transmission bitrate caused by factors such as data packet sending intervals and instantaneous changes in network load, ensuring that the smooth transmission bitrate can reflect the long-term stable carrying capacity of the network channel.

[0144] It should be noted that the transmission rate, calculated by statistically analyzing the ratio of the amount of data successfully transmitted within a very short time window to the length of that time window, reflects the data transmission capacity of the network channel at that moment. For example, the transmission rate can be obtained by counting the number of bytes sent at fixed time intervals (such as 100 milliseconds) and then dividing by that time interval; this transmission rate is the instantaneous rate.

[0145] It can be further explained that the second time constant is greater than the first time constant. That is, smoothing the transmission bit rate has a stronger noise suppression capability and a slower response speed than smoothing the packet loss factor. This makes the smoothing of the transmission bit rate change more stable and better reflects the long-term trend of network bandwidth, while smoothing the packet loss factor can respond to the initial signs of network congestion relatively faster.

[0146] In this embodiment, the packet loss factor and transmission bit rate in the network feedback information are processed using exponentially weighted moving average smoothing, and different time constants are set according to their characteristics, thereby achieving accurate perception of network conditions. Specifically, for the packet loss factor, since it is often a direct and sensitive indicator of network congestion, exponentially weighted moving average smoothing is performed using a first time constant. This allows the smoothed packet loss factor to respond relatively quickly to packet loss in the network, reflecting the deterioration trend of network quality in a timely manner, while simultaneously filtering out occasional, non-persistent packet loss noise, avoiding overreaction of the system to short-lived, unrepresentative packet loss events. For the transmission bit rate, especially the instantaneous bit rate, its fluctuation is usually large. If the response to instantaneous fluctuations is too sensitive, it may lead to frequent adjustments of the transmission strategy, which may reduce transmission stability. Therefore, exponentially weighted moving average smoothing is performed on the transmission bit rate using a second time constant, and the second time constant is greater than the first time constant, resulting in a stronger smoothing effect and a slower response speed for the transmission bit rate smoothing. This allows the smoothed transmission bitrate to effectively filter out drastic fluctuations in instantaneous bitrate noise, providing a more stable and reliable bandwidth estimate, thus better reflecting the continuous carrying capacity of the current transmission channel. Through this differentiated smoothing strategy, this application can obtain smooth network feedback information that both reflects network congestion in a timely manner and provides a stable bandwidth estimate. This makes the transmission strategy for media stream data more intelligent and adaptive, effectively reducing stuttering and latency during media stream transmission and improving the user viewing experience.

[0147] In an exemplary embodiment, after smoothing the packet loss factor and transmission rate in the network feedback information, a first transmission interval and a second transmission interval can be calculated based on the packet loss factor and / or transmission rate.

[0148] The smoothed packet loss factor is a packet loss rate metric that has been smoothed to reflect the current network packet loss situation in the transmission channel. Its function is to filter out instantaneous or occasional packet loss fluctuations, providing a more stable and representative network packet loss trend.

[0149] Specifically, the step of determining the first transmission interval based on the smooth packet loss factor can be to compare the smooth packet loss factor with the packet loss prediction, perform corresponding calculations under different packet loss conditions, and determine the first transmission interval accordingly. The calculation methods for the first transmission interval include the following different calculation strategies:

[0150] (1) When the smooth packet loss factor is less than or equal to the first packet loss threshold, the current transmission interval of the current transmission channel is multiplied by the first multiplier to obtain the first transmission interval.

[0151] The first multiplier is less than 1.

[0152] Here, the first packet loss threshold is a preset, low upper limit for packet loss rate, used to define a packet loss level that is good or acceptable for network conditions.

[0153] When the smoothing packet loss factor is below or equal to this threshold, the network condition is generally considered good, and it is advisable to improve transmission efficiency. This threshold can be set according to the packet loss tolerance requirements of the actual application scenario; for example, it can be set to 0.01 or lower.

[0154] The first multiplier is a factor less than 1, used to reduce the current transmission interval of the current transmission channel when network conditions are good, thereby increasing the data transmission rate. For example, it can be set to 0.9, 0.8, etc., and the specific value will affect the aggressiveness of the transmission rate adjustment.

[0155] (2) When the smooth packet loss factor is greater than or equal to the second packet loss threshold, the current transmission interval of the current transmission channel is multiplied by the second multiplier to obtain the first transmission interval.

[0156] The first multiplier is greater than 1.

[0157] Here, the second packet loss threshold is a preset, relatively high lower limit for packet loss rate, used to define the packet loss level when the network condition is poor or there is significant congestion.

[0158] When the smoothing packet loss factor is higher than or equal to this threshold, the network condition is generally considered poor, and a more conservative transmission strategy is needed to avoid further deterioration. This threshold is usually higher than the first packet loss threshold; for example, it can be set to 0.05 or higher.

[0159] The second multiplier is a factor greater than 1, used to increase the current transmission interval of the current transmission channel when network conditions are poor, thereby reducing the data transmission rate and alleviating network congestion. For example, it can be set to 1.1, 1.2, etc., and the specific value chosen will affect the degree of conservatism in adjusting the transmission rate.

[0160] (3) When the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, the current transmission interval of the current transmission channel is linearly interpolated according to the first multiplier and the second multiplier to obtain the first transmission interval.

[0161] The first packet loss threshold is lower than the second packet loss threshold.

[0162] The current transmission interval of a current transmission channel refers to the time interval between the transmission of two adjacent data packets on the currently used transmission channel. The current transmission interval is dynamic and can reflect the transmission rate in the previous cycle or at the current moment.

[0163] Here, the current transmission interval can be calculated by recording the time of the last data packet transmission and the time of the current data packet transmission, or by using the reciprocal of the current transmission rate.

[0164] When the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, an adjustment factor between the first and second multipliers can be calculated by linear interpolation based on the relative position of the smoothing packet loss factor with respect to the two thresholds, thereby determining the first transmission interval.

[0165] For example, the specific calculation formula can be as follows:

[0166] Y = Y1+ (X-X1) * (Y2-Y1) / (X2-X1).

[0167] Where X is the smoothing packet loss factor, X1 and X2 are the first packet loss threshold and the second packet loss threshold, respectively, and Y1 and Y2 are the first multiplier and the second multiplier, respectively.

[0168] Furthermore, the method for determining the second transmission interval based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate may include:

[0169] (4) The number of bytes to be sent is divided by the smooth transmission rate to determine the duration of continuous transmission required for the number of bytes to be sent, and the second transmission interval for each data packet is calculated based on the duration of continuous transmission.

[0170] The number of bytes to be sent refers to the total number of bytes of media stream data currently waiting to be sent in the buffer queue. The number of bytes to be sent can be used to reflect the backlog of the buffer queue and is an important indicator for measuring the data backlog at the sending end.

[0171] The number of bytes to be sent can be obtained by accumulating the number of bytes in all data packets to be sent in the buffer queue.

[0172] Here, the smoothed transmission rate is the data transmission rate after smoothing, used to reflect the effective transmission capacity of the current transmission channel. Compared with the instantaneous transmission rate, the smoothed transmission rate can better reflect the long-term transmission capacity of the network and reduce the impact of instantaneous fluctuations on transmission decisions.

[0173] Furthermore, the sustained transmission duration can refer to the time required to continuously transmit all bytes of data in the buffer queue at the current smooth transmission rate. Specifically, the sustained transmission duration can be calculated by dividing the number of bytes to be transmitted by the smooth transmission rate.

[0174] For example, if there is 1MB of data in the buffer queue and the smooth transfer bitrate is 1Mbps, the continuous transfer time is 8 seconds.

[0175] It should be noted that the second transmission interval is calculated based on the number of bytes to be sent in the buffer queue and the smooth transmission rate. It is used to control the transmission pace of data packets to avoid buffer queue overflow or underload. The second transmission interval can reflect the time required to clear the current buffer queue and is allocated to the transmission interval of each data packet.

[0176] For example, if the continuous transmission duration is T and N data packets need to be sent, the second transmission interval can be simply calculated as T / N.

[0177] Furthermore, after determining the first transmission interval and the second transmission interval, the target transmission interval can be determined based on the smoothing packet loss factor, the first transmission interval, and the second transmission interval. Here, depending on the value of the smoothing packet loss factor, different strategies can be used to calculate the target transmission interval. Specifically, the target transmission interval can be calculated using the following different strategies:

[0178] (1) When the smooth packet loss factor is less than or equal to the first packet loss threshold, the smaller value between the first transmission interval and the second transmission interval shall be taken as the target transmission interval.

[0179] (2) When the smooth packet loss factor is greater than or equal to the second packet loss threshold, the larger value between the first transmission interval and the second transmission interval shall be taken as the target transmission interval.

[0180] (3) When the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, calculate the normalization ratio of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold, and perform linear interpolation between the smaller and larger values ​​in the first transmission interval and the second transmission interval according to the normalization ratio to obtain the target transmission interval.

[0181] The normalized ratio is used to represent the relative position of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold.

[0182] It should be noted that, based on the relative position of the smooth packet loss factor between the two thresholds, the value of the new data point can be estimated between known data points, smoothly transitioning the target transmission interval and avoiding sudden changes in transmission rate.

[0183] Specifically, when the smoothing packet loss factor is between two thresholds, the target transmission interval can be calculated using the following specific formula:

[0184] Smaller value * (1 - normalization ratio) + Larger value * normalization ratio.

[0185] In the above technical solution, the final target transmission interval is determined by range judgment of the smoothed packet loss factor and adaptive selection or interpolation combined with the first and second transmission intervals. Specifically, when the smoothed packet loss factor is at a low level (less than or equal to the first packet loss threshold), it indicates that the network condition is good. In this case, transmission efficiency and response speed are prioritized, so the smaller value between the first and second transmission intervals is selected as the target transmission interval to maximize the transmission rate. When the smoothed packet loss factor is at a high level (greater than or equal to the second packet loss threshold), it indicates that the network condition is poor or there is congestion. In this case, network stability and congestion avoidance are prioritized, so the larger value between the first and second transmission intervals is selected as the target transmission interval to reduce the transmission rate and alleviate network pressure. When the smoothed packet loss factor is in an intermediate state between the first and second packet loss thresholds, a more refined strategy is adopted. First, the normalized ratio of the smoothed packet loss factor within this interval is calculated. This ratio reflects the relative position of the current packet loss factor between good and poor network conditions. Subsequently, based on this normalization ratio, linear interpolation is performed between the smaller and larger values ​​in the first and second transmission intervals to obtain a target transmission interval with a smooth transition. This allows the target transmission interval to dynamically balance network congestion control (reflected by the first transmission interval) and sender buffer queue management (reflected by the second transmission interval), ensuring stable and efficient transmission of media stream data under various network environments.

[0186] This embodiment provides a more intelligent and adaptive media stream data transmission interval determination mechanism. It dynamically fuses a first transmission interval based on network quality and a second transmission interval based on buffer queue status according to real-time changes in network packet loss. This avoids overly conservative transmission rates under good network conditions and overly aggressive transmission rates under poor network conditions. Therefore, it effectively prevents further deterioration of network congestion while maximizing the transmission efficiency and smoothness of media stream data, significantly improving the user's viewing experience in different network environments and reducing stuttering and buffering issues.

[0187] In one exemplary embodiment, such as Figure 4 As shown, after determining the target transmission interval, the media stream data in the buffer queue can be sent to the corresponding receiving end according to the target transmission interval. Specifically, this can include steps 261 to 263, wherein:

[0188] Step 261: Calculate the target transmission duration of the media stream data in the buffer queue based on the target transmission interval.

[0189] The target duration is defined as transforming a discrete transmission interval into a continuous transmission time period.

[0190] Here, the target transmission interval usually refers to the time between two transmission operations, while the target transmission duration represents the time required to send all the data in the current buffer queue in one transmission operation.

[0191] Specifically, the duration can be obtained by dividing the total amount of media stream data to be sent in the buffer queue by the currently determined transmission bitrate, or by estimating the number of data packets that can be sent within this interval based on the target transmission interval and the preset size of each data packet, and then calculating the total transmission time of these data packets.

[0192] Step 262: The target transmission duration and media stream data are transmitted to the transmission layer of the sending end to split the media stream data into multiple corresponding transmission data packets.

[0193] Specifically, the transport layer can split the received media stream data stream or data block into multiple fixed-size or variable-size transport data packets according to the maximum transmission unit (MTU) or a preset data packet size, or encapsulate each media data frame into one or more transport data packets according to the media data frame boundaries provided by the application layer.

[0194] Step 263: Calculate the data packet distribution transmission parameters based on the target transmission duration and the number of data packets to be transmitted, and sleep during the intervals between data packet transmissions according to the data packet distribution transmission parameters, so as to avoid all data packets of the same media stream being sent at the same time, until multiple data packets are sent to the corresponding transmission interface.

[0195] Among them, the data packet distribution transmission parameter is used to guide how to send multiple data packets evenly or in a specific pattern within the target transmission duration. Its purpose is to avoid instantaneous traffic peaks caused by concentrated data packet transmission.

[0196] Specifically, the data packet distribution transmission parameters can be obtained by dividing the target transmission duration by the number of data packets to obtain the average transmission interval of each data packet, or by calculating the transmission rate or transmission time of each data packet based on the target transmission duration and the number of data packets, combined with a certain scheduling algorithm.

[0197] Here, "sleep" can be a brief delay introduced between sending two consecutive data packets to achieve the purpose of smoothing data transmission.

[0198] After sending a data packet, the sending end can call the system-provided delay function to wait for the time specified by the data packet distribution transmission parameters before sending the next data packet, or it can trigger the sending of the next data packet at a predetermined time point by setting a timer or using an event-driven mechanism.

[0199] A transmission interface refers to the interface through which the sending end interacts with the network physical layer or data link layer, such as a network interface card (NIC). Data packets are processed layer by layer through the network protocol stack via the socket interface provided by the operating system, and finally the NIC driver sends the data frame to the physical network medium.

[0200] The proposed solution, after determining the target transmission interval, does not simply send all data at once. Instead, it first calculates the target transmission duration of the media stream data. Subsequently, the media stream data is transmitted to the transport layer at the sending end and split into multiple transmission data packets. Crucially, based on the target transmission duration and the number of transmission data packets, data packet distribution parameters are calculated, and the data packets are strictly controlled according to these parameters during the intervals between transmission packet transmissions. This ensures that multiple transmission data packets are smoothly distributed throughout the target transmission duration, rather than being sent all at once. This not only maintains the overall transmission rate within the target transmission interval setting, but more importantly, it effectively avoids the impact of instantaneous traffic spikes on the network, thereby significantly reducing the risk of network congestion and packet loss, and improving the stability and smoothness of media stream transmission.

[0201] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0202] Based on the same inventive concept, this application also provides a media stream data transmission apparatus for implementing the media stream data transmission method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more media stream data transmission apparatus embodiments provided below can be found in the limitations of the media stream data transmission method described above, and will not be repeated here.

[0203] In one exemplary embodiment, such as Figure 5As shown, a media stream data transmission device is provided, comprising: a first acquisition module, a second acquisition module, a smoothing processing module, a first determination module, a second determination module, and a transmission module, wherein:

[0204] The first acquisition module is used to acquire the media stream data to be sent and write the media stream data into the buffer queue;

[0205] The second acquisition module is used to acquire network feedback information of the current transmission channel; wherein, the network feedback information includes at least the packet loss factor and the transmission rate;

[0206] The smoothing module is used to smooth the packet loss factor and transmission rate in the network feedback information respectively, so as to obtain the smoothed packet loss factor and smoothed transmission rate.

[0207] The first determining module is used to determine the first transmission interval based on the smooth packet loss factor, and to determine the second transmission interval based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate.

[0208] The second determining module is used to determine the target transmission interval based on the smooth packet loss factor, the first transmission interval, and the second transmission interval.

[0209] The transmission module is used to determine the media stream data in the buffer queue to be sent to the corresponding receiving end based on the target transmission interval.

[0210] In one embodiment, the second acquisition module includes a third acquisition unit, a receiving interval calculation unit, a data packet grouping unit, and a difference calculation unit, wherein:

[0211] The third acquisition unit is used to acquire the reception statistics of the transport layer; wherein, the reception statistics include at least the transmission sequence number and arrival timestamp of the data packets received by the transport layer;

[0212] The receiving interval calculation unit is used to calculate the receiving time interval between a data packet and an adjacent data packet based on the arrival timestamp.

[0213] The data packet grouping unit is used to divide data packets with a reception time interval less than or equal to a preset grouping threshold into the same data packet according to the transmission sequence number, and to calculate the transmission code rate of the data packet.

[0214] The difference calculation unit is used to calculate the difference between the data packets sent and received by the transport layer to obtain the packet loss factor.

[0215] In one embodiment, the smoothing module includes a first smoothing unit and a second smoothing unit, wherein:

[0216] The first smoothing processing unit is used to perform exponential weighted moving smoothing on the packet loss factor using a first time constant, so as to filter out the packet loss noise data in the packet loss factor and obtain the smoothed packet loss factor.

[0217] The second smoothing unit is used to perform exponentially weighted moving average smoothing on the transmission bit rate using a second time constant to filter out bit rate fluctuation noise data in the transmission bit rate and obtain a smooth transmission bit rate; wherein, the transmission bit rate is the instantaneous bit rate, and the second time constant is greater than the first time constant.

[0218] In one embodiment, the first determining module includes a first transmission interval determining unit and a second transmission interval determining unit, wherein:

[0219] The first transmission interval determination unit is used to multiply the current transmission interval of the current transmission channel by a first multiplier when the smoothing packet loss factor is less than or equal to the first packet loss threshold, to obtain the first transmission interval; wherein the first multiplier is less than 1.

[0220] The first transmission interval determination unit is further configured to multiply the current transmission interval of the current transmission channel by the second multiplier when the smoothing packet loss factor is greater than or equal to the second packet loss threshold, to obtain the first transmission interval; wherein the first multiplier is greater than 1.

[0221] The first transmission interval determination unit is further configured to perform linear interpolation calculation on the current transmission interval of the current transmission channel according to the first multiplier and the second multiplier when the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, to obtain the first transmission interval; wherein the first packet loss threshold is lower than the second packet loss threshold.

[0222] The second transmission interval determination unit is used to divide the number of bytes to be sent by the smooth transmission code rate to determine the continuous transmission duration required for the number of bytes to be sent, and to calculate the second transmission interval for each data packet to be transmitted based on the continuous transmission duration.

[0223] In one embodiment, the first determining module includes a first parallel unit, a second parallel unit, and a third parallel unit, wherein:

[0224] The first parallel unit is used to take the smaller value between the first transmission interval and the second transmission interval as the target transmission interval when the smoothing packet loss factor is less than or equal to the first packet loss threshold.

[0225] Or a second parallel unit, used to take the larger value of the first transmission interval and the second transmission interval as the target transmission interval when the smoothing packet loss factor is greater than or equal to the second packet loss threshold.

[0226] Alternatively, a third parallel unit is used to calculate the normalized ratio of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold when the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, and to perform linear interpolation between the smaller and larger values ​​in the first transmission interval and the second transmission interval according to the normalized ratio to obtain the target transmission interval; wherein, the normalized ratio is used to represent the relative position of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold.

[0227] In one embodiment, the transmission module includes a duration calculation unit, a data packet splitting unit, and a transmission execution unit, wherein:

[0228] The duration calculation unit is used to calculate the target transmission duration of media stream data in the buffer queue based on the target transmission interval.

[0229] The packet splitting unit is used to transmit the target transmission duration and media stream data to the transport layer of the sending end, so as to split the media stream data into multiple corresponding transmission packets;

[0230] The transmission execution unit is used to calculate the data packet distribution transmission parameters based on the target transmission duration and the number of data packets to be transmitted, and to sleep during the intervals between data packet transmissions according to the data packet distribution transmission parameters, so as to avoid all data packets of the same media stream being sent at the same time, until multiple data packets are sent to the corresponding transmission interface.

[0231] In one embodiment, the media stream data transmission device further includes a data classification module, wherein the data classification module includes: a data writing unit, a first transmission unit, and a second transmission unit, wherein:

[0232] The data writing unit is used to write audio data frames to the audio buffer queue and video data frames to the video buffer queue.

[0233] The first transmission unit is used to prioritize sending audio data frames in the audio buffer queue, and is not limited by the target transmission interval;

[0234] The second transmission unit is used to select the transmission channel containing the video data frame with the earliest timestamp in the video buffer queue for priority transmission when the video buffer queue includes multiple non-empty video buffer queues.

[0235] Each module in the aforementioned media stream data transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processor in a computer device, or stored in software within the memory of the computer device, so that the processor can invoke and execute the operations corresponding to each module.

[0236] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores matrix data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a method for transmitting media stream data.

[0237] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0238] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps performed by the media stream data transmission method described above.

[0239] In one embodiment, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps performed by the media stream data transmission method described above.

[0240] In one embodiment, a program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0241] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0242] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0243] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for transmitting media stream data, characterized in that, The method includes: Acquire the media stream data to be sent and write the media stream data into the buffer queue; Obtain network feedback information for the current transmission channel; wherein, the network feedback information includes at least packet loss factor and transmission rate; The packet loss factor and transmission rate in the network feedback information are smoothed to obtain a smoothed packet loss factor and a smoothed transmission rate; wherein, the smoothed packet loss factor is used to reflect the average packet loss trend over a period of time, and the smoothed transmission rate is used to reflect the average transmission capacity over a period of time. The first transmission interval is determined based on the smooth packet loss factor, and the second transmission interval is determined based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate. The target transmission interval is determined based on the smooth packet loss factor, the first transmission interval, and the second transmission interval; Based on the target transmission interval, the media stream data in the buffer queue will be sent to the corresponding receiving end.

2. The method according to claim 1, characterized in that, The method is applied to a transmitting end, which includes a transport layer. The step of obtaining network feedback information of the current transmission channel includes: Obtain the reception statistics of the transport layer; wherein the reception statistics include at least the transmission sequence number and arrival timestamp of the data packets received by the transport layer; Calculate the reception time interval between the data packet and the adjacent data packet based on the arrival timestamp; According to the transmission sequence number, data packets with a reception time interval less than or equal to a preset grouping threshold are divided into the same data packet group, and the transmission code rate of the data packet group is calculated. The packet loss factor is obtained by calculating the difference between the data packets sent and received by the transport layer.

3. The method according to claim 1, characterized in that, The step of smoothing the packet loss factor and transmission rate in the network feedback information to obtain the smoothed packet loss factor and smoothed transmission rate includes: An exponentially weighted moving average smoothing process is applied to the packet loss factor using a first time constant to filter out packet loss noise data in the packet loss factor, thereby obtaining a smoothed packet loss factor. The transmission bitrate is smoothed by an exponentially weighted moving average using a second time constant to filter out bitrate fluctuation noise data in the transmission bitrate and obtain a smooth transmission bitrate; wherein the transmission bitrate is an instantaneous bitrate and the second time constant is greater than the first time constant.

4. The method according to claim 1, characterized in that, The steps of determining the first transmission interval based on the smooth packet loss factor and determining the second transmission interval based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate include: If the smoothing packet loss factor is less than or equal to the first packet loss threshold, the current transmission interval of the current transmission channel is multiplied by the first multiplier to obtain the first transmission interval; wherein the first multiplier is less than 1; or If the smoothing packet loss factor is greater than or equal to the second packet loss threshold, the current transmission interval of the current transmission channel is multiplied by the second multiplier to obtain the first transmission interval; wherein the first multiplier is greater than 1; or When the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, the current transmission interval of the current transmission channel is linearly interpolated according to the first multiplier and the second multiplier to obtain the first transmission interval; wherein, the first packet loss threshold is lower than the second packet loss threshold; Divide the number of bytes to be sent by the smooth transmission bitrate to determine the continuous transmission duration required for the number of bytes to be sent, and calculate the second transmission interval for each data packet based on the continuous transmission duration.

5. The method according to claim 1, characterized in that, The step of determining the target transmission interval based on the smoothing packet loss factor, the first transmission interval, and the second transmission interval includes: If the smoothing packet loss factor is less than or equal to the first packet loss threshold, the smaller value between the first transmission interval and the second transmission interval is taken as the target transmission interval; or If the smoothing packet loss factor is greater than or equal to the second packet loss threshold, the larger of the first transmission interval and the second transmission interval is taken as the target transmission interval; or When the smoothing packet loss factor is between the first packet loss threshold and the second packet loss threshold, the normalized ratio of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold is calculated, and a linear interpolation is performed between the smaller and larger values ​​of the first transmission interval and the second transmission interval based on the normalized ratio to obtain the target transmission interval; wherein, the normalized ratio is used to represent the relative position of the smoothing packet loss factor between the first packet loss threshold and the second packet loss threshold.

6. The method according to claim 1, characterized in that, The step of determining, based on the target transmission interval, to send the media stream data in the buffer queue to the corresponding receiving end includes: Calculate the target transmission duration of the media stream data in the buffer queue based on the target transmission interval; The target transmission continues while the media stream data is transmitted to the transmission layer of the sending end, so as to split the media stream data into multiple corresponding transmission data packets; Based on the target transmission duration and the number of transmission data packets, the data packet distribution transmission parameters are calculated, and the system sleeps during the intervals between data packet transmissions according to the data packet distribution transmission parameters, so as to avoid all transmission data packets of the same media stream being sent at the same time, until the multiple transmission data packets are sent to the corresponding transmission interface.

7. The method according to any one of claims 1 to 6, characterized in that, The media stream data includes audio data frames and video data frames, the buffer queue includes an audio buffer queue and a video buffer queue, and the method further includes: The audio data frame is written to the audio buffer queue, and the video data frame is written to the video buffer queue; Audio data frames in the audio buffer queue are sent first, and are not limited by the target transmission interval; If the video buffer queue includes multiple non-empty video buffer queues, the transmission channel containing the video data frame with the earliest timestamp in the video buffer queue shall be selected for priority transmission.

8. A media stream data transmission device, characterized in that, The device includes: The first acquisition module is used to acquire media stream data to be sent and write the media stream data into a buffer queue; The second acquisition module is used to acquire network feedback information of the current transmission channel; wherein, the network feedback information includes at least packet loss factor and transmission rate; The smoothing module is used to smooth the packet loss factor and transmission rate in the network feedback information respectively to obtain the smoothed packet loss factor and smoothed transmission rate; wherein, the smoothed packet loss factor is used to reflect the average packet loss trend over a period of time, and the smoothed transmission rate is used to reflect the average transmission capacity over a period of time. The first determining module is used to determine a first transmission interval based on the smooth packet loss factor, and to determine a second transmission interval based on the number of bytes to be sent in the buffer queue and the smooth transmission code rate. The second determining module is used to determine the target transmission interval based on the smooth packet loss factor, the first transmission interval, and the second transmission interval; The transmission module is used to determine, based on the target transmission interval, to send the media stream data in the buffer queue to the corresponding receiving end.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images