Method and apparatus for transmitting streaming media data

Abstract

This disclosure provides a method and apparatus for transmitting streaming media data. The method for transmitting streaming media data, performed by a first device, includes: reducing the resolution of newly arriving streaming media data in response to satisfying the predetermined conditions; and transmitting the streaming media data in a transmission buffer to a second device, wherein the predetermined conditions include that the amount of streaming media data buffered in the transmission buffer is greater than or equal to a data amount threshold and that a first time interval has elapsed since the most recent reduction in the resolution of the newly arriving streaming media data, and the data amount threshold is determined according to a predetermined maximum buffering time and is variable according to the resolution of the newly arriving streaming media data.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more specifically to methods and apparatus for transmitting streaming media data. Background Technology

[0002] When users preview live audio and video streams from remote network cameras through applications, they are typically offered resolution options such as High Density (HD) and Standard Density (SD) for selection. HD previews offer higher resolution and image quality, but also come with a higher bitrate, resulting in higher bandwidth usage during transmission. SD previews, on the other hand, have a lower bitrate, making remote transmission easier, but at the cost of lower image resolution.

[0003] When the pan-tilt-zoom (PTZ) of a network camera rotates, the camera rotates as well, causing a surge in the bitrate of the captured audio and video streams. This leads to delays in the remote transmission of the live feed, preventing the real-time footage from being displayed promptly. Similarly, when a stationary network camera captures dynamic and vibrant footage, the bitrate of the captured audio and video also surges. In such cases, if the remote transmission of the audio and video streams encounters weak network conditions with insufficient bandwidth, stuttering and delays often occur. This results in a very poor user experience. However, if only standard definition previews are allowed, prolonged periods of poor audio and video quality, even with a good network connection or when the PTZ is stationary, will also lead to a poor user experience.

[0004] We need technologies that ensure a good user experience and deliver streaming data in real time. Summary of the Invention

[0005] According to one aspect of this disclosure, a method for transmitting streaming media data is provided, performed by a first device, the method comprising: reducing the resolution of newly arriving streaming media data in response to satisfying predetermined conditions; and transmitting streaming media data in a transmission buffer to a second device, wherein the predetermined conditions include that the amount of streaming media data buffered in the transmission buffer is greater than or equal to a data amount threshold and that a first duration has elapsed since the most recent reduction in the resolution of the newly arriving streaming media data, and the data amount threshold is determined according to a predetermined maximum buffering time and is variable according to the resolution of the newly arriving streaming media data.

[0006] In at least one embodiment, the method further includes: in response to reducing the resolution of newly arrived streaming media data for a first duration, reducing a data volume threshold based on a predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data, wherein the first duration is greater than or equal to the predetermined maximum buffer time.

[0007] In at least one embodiment, the method further includes: reducing the resolution of newly arriving streaming media data in response to movement of the first device and where the most recent resolution reduction was not in response to movement of the first device; reducing the resolution of newly arriving streaming media data in response to movement of the first device and where the resolution has been increased since the most recent resolution reduction in response to movement of the first device; or reducing the resolution of newly arriving streaming media data in response to receiving a request to reduce the resolution from a second device.

[0008] In at least one embodiment, the method further includes: increasing the resolution of the newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the first device initially sent streaming media data to the second device; increasing the resolution of the newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the most recent decrease or increase in the resolution of the newly arrived streaming media data; increasing the resolution of the newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the first device stopped moving; or increasing the resolution of the newly arrived streaming media data in response to receiving a request to increase the resolution from the second device, determining that the resolution of the newly arrived streaming media data is lower than the maximum resolution and a second time elapsed since the most recent decrease or increase in the resolution.

[0009] In at least one embodiment, the method further includes: in response to increasing the resolution of newly arrived streaming media data, increasing the data volume threshold based on a predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data.

[0010] In at least one embodiment, the method further includes extending a second duration in response to at least one of the following: the resolution of the newly arrived streaming media data reaches a maximum resolution; and during a third duration since the resolution of the newly arrived streaming media data was increased, the first device moves or decreases the resolution of the newly arrived streaming media data.

[0011] In at least one embodiment, the second duration has an upper limit.

[0012] In at least one embodiment, the size of the transmit buffer is set to be greater than or equal to a data volume threshold.

[0013] In at least one embodiment, the data volume threshold is determined by multiplying a predetermined maximum buffer time by the maximum bitrate of the streaming data corresponding to the resolution of the newly arrived streaming data.

[0014] According to one aspect of this disclosure, an apparatus for transmitting streaming media data is provided, comprising: at least one memory configured to store program instructions; and one or more processors coupled to the at least one memory and configured to execute the program instructions to perform a method according to at least one embodiment of this disclosure.

[0015] According to one aspect of this disclosure, when the amount of buffered streaming media data obtained from shooting is greater than or equal to a data volume threshold, it indicates that the communication network is congested or bandwidth is insufficient, or the bit rate of the streaming media to be sent is too high, causing an excessive amount of data to be buffered and unable to be sent in time. In this case, reducing the resolution of the streaming media reduces the bit rate, thereby reducing the storage space occupied and the amount of data sent, and thus enabling the streaming media obtained from real-time shooting to be sent out as quickly as possible, satisfying the real-time transmission of streaming media and the smoothness of the preview image at the receiving end. Moreover, compared to the prior art which needs to judge bit rate spikes or communication network speed, this solution only needs to determine whether to reduce the resolution based on an appropriate data volume threshold. The data volume threshold changes with the resolution, which can avoid data of any resolution being buffered in the buffer for more than the predetermined maximum buffering time, so as to ensure the real-time preview of streaming media at the receiving end. Attached Figure Description

[0016] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to offer a further understanding of the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 A schematic diagram of an application scenario in which at least one embodiment of the present disclosure is shown.

[0018] Figure 2 A schematic diagram illustrating the flow of an interaction process between a device and a client according to at least one embodiment of the present disclosure is shown.

[0019] Figure 3 A schematic diagram illustrating the flow of the interaction process between the device, the client, and the relay server according to at least one embodiment of the present disclosure is shown.

[0020] Figure 4 A flowchart illustrating a method for transmitting streaming media data performed by a first device according to at least one embodiment of the present disclosure is shown.

[0021] Figure 5A flowchart is shown illustrating a method performed by a second device to adjust the resolution of a streaming media transmitted by a first device, according to at least one embodiment of the present disclosure.

[0022] Figure 6 A flowchart of a device-side resolution reduction mechanism according to at least one embodiment of the present disclosure is shown.

[0023] Figure 7 A flowchart of a device-side resolution upscaling mechanism according to at least one embodiment of the present disclosure is shown.

[0024] Figure 8 A flowchart illustrating a resolution-boosting and resolution-lowering mechanism for a client according to at least one embodiment of the present disclosure is shown.

[0025] Figure 9 A block diagram of a device for transmitting streaming media data according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same elements throughout. It should be understood that the embodiments described in this disclosure are merely illustrative and should not be construed as limiting the scope of this disclosure.

[0027] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as superior to or better than other aspects. Similarly, the term “aspects of this disclosure” does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed.

[0028] We need technologies that ensure a good user experience and deliver streaming data in real time.

[0029] This disclosure provides a method for transmitting streaming media data, executed by a first device. The method includes: reducing the resolution of newly arriving streaming media data in response to meeting predetermined conditions; and transmitting the streaming media data in a transmission buffer to a second device. The predetermined conditions include that the amount of buffered streaming media data in the transmission buffer is greater than or equal to a data amount threshold and a first time interval has elapsed since the most recent reduction in the resolution of the newly arriving streaming media data. The data amount threshold is determined based on a predetermined maximum buffering time and is variable according to the resolution of the newly arriving streaming media data. Thus, when the amount of buffered streaming media data obtained from recording is greater than or equal to the data amount threshold, it indicates that the communication network is congested or bandwidth is insufficient, or the bitrate of the streaming media to be transmitted is too high, causing excessive data to be buffered and unable to be transmitted in a timely manner. In this case, reducing the resolution of the streaming media reduces the bitrate, decreases the storage space occupied, and reduces the amount of data transmitted, thereby enabling the streaming media obtained from real-time recording to be transmitted as quickly as possible, satisfying the real-time transmission requirements of streaming media and the smoothness of the preview image at the receiving end. Moreover, compared to existing technologies that require judging bit rate spikes or communication speeds, this solution only needs to determine whether to reduce the resolution based on an appropriate data volume threshold. The data volume threshold changes with the resolution, which can minimize the buffering of data of any resolution in the buffer for longer than the predetermined maximum buffering time, thus ensuring the real-time preview of streaming media at the receiving end.

[0030] In some embodiments, the method further includes: in response to reducing the resolution of newly arrived streaming media data for a first duration, reducing a data volume threshold based on a predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data, wherein the first duration is greater than or equal to the predetermined maximum buffer time. Thus, reducing the resolution for at least the first duration ensures that the high-resolution streaming media data generated during the predetermined maximum buffer time, which was previously blocked in the buffer, is sent out as much as possible. Reducing the data volume threshold at this point avoids the situation where immediately reducing the data volume threshold after reducing the resolution causes unsent streaming media data to continuously trigger an instruction to reduce the resolution due to exceeding the data volume threshold.

[0031] In some embodiments, the method further includes reducing the resolution of newly arriving streaming media data in response to movement of the first device, provided that the most recent resolution reduction was not in response to movement of the first device. Thus, when the first device moves while capturing streaming media data, the dynamic range of the captured image may be too large, causing the bitrate of the streaming media to spike and potentially leading to excessive data volume being buffered and unable to be sent in time. Reducing the resolution of the streaming media data in this situation lowers the bitrate, reduces the storage space required, and decreases the amount of data sent, thereby satisfying the real-time nature of streaming media transmission and the smoothness of the preview. However, since the first device may move frequently when remotely controlled by the user, frequent movement of the first device leading to frequent resolution reductions would affect the image quality and viewing experience of the streaming media. Therefore, it is necessary to limit the most recent resolution reduction to not being in response to movement of the first device to avoid frequent resolution reductions caused by frequent movement of the first device.

[0032] In some embodiments, the method further includes: reducing the resolution of newly arriving streaming media data in response to movement of the first device, where the resolution has been increased since the most recent reduction in response to movement of the first device. Since the first device moves frequently, if the resolution has already been reduced due to movement of the first device, it is not advisable to continue reducing the resolution if the first device moves again. However, if the resolution has already been increased, it can be reduced again when the first device moves, thereby avoiding frequent resolution reductions caused by frequent movement of the first device.

[0033] In some embodiments, the method further includes: reducing the resolution of newly arrived streaming media data in response to receiving a request to reduce resolution from the second device. When stuttering is detected at the second device, a request to reduce resolution may be sent. In this case, the first device should ideally reduce the resolution immediately to ensure the real-time transmission of the streaming media and the smoothness of the preview image at the receiving end.

[0034] In some embodiments, the method further includes: increasing the resolution of newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the first device initially sent streaming media data to the second device; increasing the resolution of newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the last decrease or increase in the resolution of the newly arrived streaming media data; increasing the resolution of newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the first device stopped moving; or increasing the resolution of newly arrived streaming media data in response to receiving a request to increase the resolution from the second device, determining that the resolution of the newly arrived streaming media data is lower than the maximum resolution and a second time elapsed since the last decrease or increase in the resolution. Thus, periodically increasing the resolution according to the second time elapsed is to restore the clarity of the streaming media and avoid pushing low-resolution streaming media for extended periods, which would result in a poor user viewing experience.

[0035] In some embodiments, the method further includes: in response to increasing the resolution of newly arrived streaming media data, increasing a data volume threshold based on a predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data. Thus, the data volume threshold dynamically increases as the current resolution increases, thereby allowing for a more appropriate determination of whether to reduce the current streaming media resolution based on a larger data volume threshold. If the resolution increases, the amount of data allowed to be buffered after the predetermined maximum buffer time also increases. Increasing the data volume threshold allows for the buffering of larger amounts of high-resolution data without easily triggering a resolution reduction, thereby improving the clarity of the viewed image.

[0036] In some embodiments, the method further includes extending a second duration in response to at least one of the following: the resolution of the newly arrived streaming data reaches its maximum resolution; and during a third duration since the increase in the resolution of the newly arrived streaming data, the first device moves or decreases the resolution of the newly arrived streaming data. Thus, extending the timing period for attempting to increase the resolution when it is not suitable avoids frequent increases in resolution when it is not appropriate.

[0037] Figure 1 A schematic diagram of an application scenario in which at least one embodiment of the present disclosure is shown.

[0038] like Figure 1As shown, the device 101 (e.g., a camera) can be mounted on the pan / tilt head 102. The pan / tilt head 102 typically has two or three rotation axes (such as horizontal axis, pitch axis, roll axis) and other moving devices, allowing the camera 101 to move freely in three-dimensional space, covering a wider field of view. For example, it can achieve 360° horizontal rotation and ±90° vertical pitch, and even translation, etc.

[0039] In local or peer-to-peer (P2P) connection methods, client 102 (e.g., mobile terminal such as a phone, fixed terminal such as a desktop computer) can directly request device 101 to send audio and video streaming media captured by device 101, so that client 102 can preview or play such audio and video streaming media on its display. In this case, client 102 and device 101 can communicate directly through encoding and transmission protocols (such as Real-time Message Transfer Protocol (RTMP) or Real-time Streaming Protocol (RTSP)). Client 102 can pull streams to device 101, and device 101 can push streams to client 102. This application scenario can include audio and video on demand, interactive live streaming, distance education, and intelligent monitoring.

[0040] In the relay connection method, client 102 can communicate with device 101 through relay server 104. Client 102 can send a streaming request to relay server 104, relay server 104 forwards the streaming request to device 101, device 101 pushes the captured audio and video streaming media to relay server 104, and relay server 104 converts the audio and video streaming media to client 102.

[0041] Figure 2 A schematic diagram illustrating the flow of an interaction process between a device and a client according to at least one embodiment of the present disclosure is shown.

[0042] Figure 2 This describes the flow of interaction between the device and the client in local direct connection or end-to-end scenarios.

[0043] The client can determine in advance whether the device supports automatic resolution (Auto) mode (hereinafter referred to as Auto mode) by obtaining the device's capability set. Automatic resolution is the automatic change of the resolution of the audio and video streaming media sent by the device, implemented according to at least one embodiment of this disclosure.

[0044] For example, in at least one embodiment of this disclosure, different resolutions may include high definition (HD) resolution, medium definition (MD) resolution, standard definition (SD) or low definition (e.g., Video Graphics Array (VGA)) resolution, and even lower resolution. Audio and video streaming media with different resolutions typically have different bitrate ranges.

[0045] High-definition resolution typically refers to 720p (1280×720) or 1080p (1920×1080). For 720p, the bitrate is generally above 2Mbps. For example, when using H.264 encoding, the bitrate of 720p video might be between 2Mbps and 5Mbps. For 1080p, the bitrate is usually even higher to meet the demands of higher image quality. For example, on video platforms, the bitrate of 1080p video might range from around 1.5Mbps to 6Mbps, but for some high-quality content or special scenarios, the bitrate might exceed 6Mbps.

[0046] Medium definition (MD) resolution falls between high definition (HD) and standard definition (SD) resolutions (such as 480p or similar), with bitrates ranging from 500Kbps to 1.5Mbps, depending on frame rate, encoding method, and content complexity. For example, on video streaming platforms, standard definition (SD, which may include 480p) bitrates can range from 500Kbps to 1.5Mbps.

[0047] Standard definition (SD) resolution, such as VGA, has a resolution of 640×480 pixels, which is the lower limit for standard resolution (SD) video. The bitrate of VGA video may be below 500Kbps, depending on the frame rate, encoding method, and content complexity. For example, in some network video surveillance systems, the bitrate of VGA resolution video may be set between 300Kbps and 500Kbps.

[0048] Lower resolution videos, such as 360p (640×360) or lower, may have lower bitrates to accommodate worse network environments or storage conditions.

[0049] In step 201, in order for the client to preview the audio and video streaming media captured by the device, the client and the device establish a streaming connection through a local Transmission Control Protocol (TCP) or a remote P2P connection.

[0050] In step 202, according to the protocol design, the client carries an "auto-resolution" type field in the preview request (requesting the device to push the stream, or the push request) to inform the device that the type of the push stream is auto-resolution.

[0051] In step 203, the device marks the stream as an automatic resolution type. The device defaults to main stream streaming. Default main stream streaming means that during video streaming, the system selects the maximum resolution main stream output by the camera or encoder for real-time transmission to meet local storage or high-definition live streaming needs. Subsequently, both parties begin streaming according to the automatic resolution strategy. Specifically, the device can directly switch an existing stream to automatic resolution type without creating a new stream. The device marks the stream as automatic resolution, and subsequently, both parties begin streaming and previewing audio and video media according to the automatic resolution strategy.

[0052] The above is the first method by which both parties determine the automatic resolution type.

[0053] In step 204, if the client already has other types of streaming, the client needs to send a request to the device to change to automatic resolution.

[0054] In step 205, the device identifies this stream as an auto-resolution stream. The device defaults to streaming the main stream. Both parties then begin streaming according to the auto-resolution strategy.

[0055] The above is the second method for both parties to determine the automatic resolution type.

[0056] Here, either the client or the device can independently decide whether to switch resolution according to at least one embodiment of this disclosure. In the case of device-side decision-making, after deciding to switch to a certain resolution, the device automatically pushes the switched streaming media to the client, as described in steps 206 and 207. In the case of client-side decision-making, after deciding to switch to a certain resolution, the client needs to send a request to the device to change the resolution to that resolution. The device then pushes the switched streaming media to the client, as described in steps 208, 209, and 210.

[0057] In step 206, during the streaming of the media, the device can adaptively adjust the resolution of the streaming media to a new resolution by using an internal strategy of automatic resolution design, such as when encountering scenarios of first device movement or communication network congestion, according to the method of at least one embodiment of the present disclosure.

[0058] In step 207, the device can switch the audio and video streaming media to be streamed to the client to a new resolution.

[0059] In step 208, the client can, according to the method of at least one embodiment of the present disclosure, use an internal strategy for automatic resolution design, such as when encountering network congestion at the client, to autonomously decide what resolution to change the current resolution to, and thus actively send a request to the device to change the resolution (e.g., change the resolution to VGA mode). The device then dynamically adjusts the resolution of the streaming media and switches to the resolution requested by the client to stream audio and video media.

[0060] In step 211, after ending the automatic resolution streaming preview of audio and video media, the streaming connection is released, allowing the client to establish a new streaming connection and preview the audio and video media in a fixed bitrate mode. Here, fixed bitrate mode refers to streaming in a fixed bitrate mode (e.g., high definition, medium definition, or low definition) without automatically deciding which bitrate mode to switch to based on at least one embodiment of this disclosure. However, when the client needs to switch to fixed bitrate mode to preview audio and video media, it can directly switch from automatic resolution streaming to the target bitrate preview by removing the automatic resolution identifier and switching to the target fixed bitrate, without needing to establish a new streaming connection.

[0061] Figure 3 A schematic diagram illustrating the flow of the interaction process between the device, the client, and the relay server according to at least one embodiment of the present disclosure is shown.

[0062] Figure 3 This describes the flow of interaction between the device, client, and relay server in a relay scenario.

[0063] The client can determine in advance whether the device supports automatic resolution by obtaining the device's capability set.

[0064] In step 301, the client establishes an auto-resolution streaming connection with the relay server;

[0065] In step 302, the device establishes an auto-resolution streaming connection with the relay server. Subsequent communication is relayed through the relay server to complete remote transmission.

[0066] In step 303, as per the protocol design, the client sends a preview request to the relay server, which carries an "auto-resolution" type field as per the protocol.

[0067] In step 304, the relay server forwards a preview request to the device to inform the device that the type of the push stream is automatic resolution.

[0068] In step 305, the device identifies the push stream as an automatic resolution type. By default, the device pushes the main stream to the relay server for audio and video streaming media.

[0069] In step 306, the relay server forwards the audio and video streaming media of the main stream pushed by the device to the client.

[0070] Here, either the client or the device can independently decide whether to switch resolution according to at least one embodiment of this disclosure. In the case of device-side decision-making, after deciding to switch to a certain resolution, the device automatically pushes the streaming media to the client at that resolution, as described in steps 307, 308, and 309. In the case of client-side decision-making, after deciding to switch to a certain resolution, the client needs to send a request to the device to change the resolution to that resolution. The device then pushes the streaming media to the client at the changed resolution, as described in steps 310-314.

[0071] In step 307, during the streaming of the media, the device can adaptively adjust the resolution of the streaming media to a new resolution by using an internal strategy of automatic resolution design, such as when encountering scenarios of first device movement or communication network congestion, according to the method of at least one embodiment of this disclosure.

[0072] In step 308, the device can switch the audio and video streaming media to be pushed to a new resolution to push the stream to the relay server.

[0073] In step 309, the relay server forwards the new resolution of the pushed audio and video streaming media to the client.

[0074] In step 311, the client may, according to the method of at least one embodiment of the present disclosure, make an autonomous decision on what resolution to change the current resolution to, such as when encountering communication network congestion at the client's location, through an internal strategy of automatic resolution design, thereby actively sending a request to the relay server to change the resolution (e.g., change the resolution to VGA mode).

[0075] In step 312, the relay server forwards the request to change the resolution to the device.

[0076] In step 313, the device dynamically adjusts the resolution of the streaming media and switches to the resolution requested by the client to push audio and video streaming media to the relay server.

[0077] In step 314, the relay server forwards the pushed audio and video streaming media to the client.

[0078] In step 315, the client and the relay server end the automatic resolution preview and release the streaming connection.

[0079] In step 316, the device ends the automatic resolution preview with the relay server and releases the push streaming connection.

[0080] In addition, because the relay server has a push and distribution mechanism, meaning that a device-side push stream can be distributed to multiple clients for simultaneous viewing through the relay, the connection cannot be directly reused for automatic resolution and other preview modes as in local direct connection and P2P preview. Automatic resolution connection must be created every time it is needed.

[0081] The only difference between the direct connection and relay methods mentioned above is the connection method, but the automatic resolution strategy on the device or client side is almost the same.

[0082] The following describes in detail how the device or client, according to at least one embodiment of the present disclosure, autonomously decides whether to reduce or increase the current resolution through an internal strategy of automatic resolution design.

[0083] Figure 4 A flowchart is shown of a method 400 for transmitting streaming media data performed by a first device according to at least one embodiment of the present disclosure.

[0084] Method 400 includes steps 401 and 402. Here, the first device can be the device end described above, and the second device can be the client end described above. The device end can include a camera that captures streaming media and the first device equipped with the camera.

[0085] In step 401, in response to meeting predetermined conditions, the resolution of newly arriving streaming media data is reduced. The predetermined conditions include that the amount of streaming media data buffered in the transmit buffer is greater than or equal to a data volume threshold and that a first duration has elapsed since the most recent reduction in the resolution of the newly arriving streaming media data. The data volume threshold is determined based on a predetermined maximum buffering time and varies according to the resolution of the newly arriving streaming media data.

[0086] In step 402, the streaming media data in the transmit buffer is sent to the second device.

[0087] If the amount of streaming media data buffered in the send buffer is greater than or equal to the data volume threshold, it means that the streaming media to be sent is being buffered in the send buffer and has not yet been sent due to communication network congestion or bitrate spikes. This indicates that packet transmission is congested. At this time, it is necessary to reduce the resolution of the streaming media to lower the bitrate, thereby reducing the storage space occupied and the amount of data sent. This will allow the streaming media captured in real time to be sent out as soon as possible, satisfying the real-time transmission of streaming media and the smoothness of the preview image at the receiving end.

[0088] Compared to existing technologies that require judging bitrate spikes or network speed, this solution only needs to determine whether to reduce the resolution based on an appropriate data volume threshold. The data volume threshold changes with the resolution, which can minimize the buffering of data of any resolution in the buffer for longer than the predetermined maximum buffering time, thus ensuring the real-time preview of streaming media at the receiving end.

[0089] In some embodiments, the size of the send buffer can be set to be greater than or equal to a data volume threshold. Setting the send buffer size to equal the data volume threshold allows for the determination that the amount of buffered streaming media data in the send buffer is greater than or equal to the data volume threshold when the send buffer is full or overflowing. In this case, newly arriving streaming media data can be cached in other storage spaces without data loss or frame skipping. Once network congestion is resolved, this cached data will be sent out quickly. If network issues cause excessive data transmission delays, data loss or frame skipping may be necessary. However, setting the send buffer size to be greater than the data volume threshold ensures that even if the amount of buffered streaming media data in the send buffer is greater than or equal to the data volume threshold, subsequent newly arriving streaming media data can still be buffered within the data volume threshold, preventing data loss or frame skipping.

[0090] The data volume threshold needs to be set reasonably to avoid triggering resolution adjustment due to slight fluctuations in the communication network conditions causing a small number of data packets to not be sent in time; while ensuring a relatively sensitive mechanism for triggering resolution adjustment.

[0091] The amount of data congested due to actual communication network congestion can be calculated using the bandwidth-delay product: BDP = Bandwidth * RTT. Bandwidth is the communication network bandwidth, representing the transmission rate of the network link, usually measured in bits per second (bps, such as Mbps or Gbps). RTT is Round-Trip Time, the total time required for data to travel from the sender to the receiver and back, measured in seconds (s). It includes propagation delay, processing delay, queuing delay, etc., and is a core indicator of network latency. For example, if the RTT is 50 milliseconds (0.05 s), it means the signal takes 0.05 seconds to travel round trip.

[0092] However, due to the complex and variable operating environment of the devices, bandwidth and RTT are unpredictable, and the communication network environment is often unstable, or in scenarios where the bit rate spikes, bandwidth and RTT change too rapidly. Therefore, in at least one embodiment, the size of the data volume threshold is determined based on the predetermined maximum buffer time Δt (in seconds) and the maximum bit rate corresponding to the current resolution, which can approximately estimate the data volume caused by the maximum tolerable congestion delay. Typically, this predetermined maximum buffer time is set and not easily changed, for example, allowing a maximum buffer delay of 2 seconds.

[0093] In some embodiments, the data volume threshold is determined by multiplying a predetermined maximum buffer time by the maximum bitrate of the streaming data corresponding to the resolution of the newly arrived streaming data.

[0094] For example, the formula for the data volume threshold SendBufferSize (hereinafter referred to as SBS, unit: bytes) can be: SBS = (Max)Bitrate / 8 * Δt / 2; where (Max)Bitrate is the maximum bitrate at the resolution corresponding to the streaming media data generated by the current device and to be stored in the send buffer. Dividing Bitrate by 8 converts the bitrate to byte rate (one way to represent the maximum bitrate); since the Linux kernel of the device's operating system doubles the set data volume threshold, the actual size of the send buffer that needs to be set also needs to be divided by 2.

[0095] For example, in HD (high definition) bitrate, the default maximum bitrate is 6144kbps (6Mbps). For MD (medium resolution) bitrate, the maximum bitrate might be 3072kbps (3Mbps). For SD (low definition) bitrate, the maximum bitrate might be 1024kbps (1Mbps).

[0096] For example, if the device's current resolution is HD, the default maximum bitrate is 6144kbps, and the device's predetermined maximum buffering time Δt is 2 seconds, then:

[0097] SBS = 6144kbps / 8 * 2s / 2 = 768KB;

[0098] That is, setting the data size threshold SendBufferSize at HD resolution to 768KB will satisfy the requirement that when the buffer exceeds the amount of data that can be generated in 2 seconds at HD resolution on the device side, the resolution will be lowered as much as possible, thus ensuring the real-time performance and smoothness of streaming media.

[0099] In some embodiments, method 400 further includes: in response to reducing the resolution of newly arrived streaming media data for a first duration, reducing a data volume threshold based on a predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data, wherein the first duration is greater than or equal to the predetermined maximum buffer time.

[0100] Therefore, the resolution must be reduced for at least the first duration to ensure that the high-resolution streaming data generated during the predetermined maximum buffer time that was previously blocked in the buffer is sent out as much as possible. Then, the data volume threshold is reduced to avoid the situation where reducing the data volume threshold immediately after reducing the resolution would cause unsent streaming data to continuously trigger the resolution reduction instruction due to exceeding the data volume threshold. For example, the predetermined maximum buffer time could be 2 seconds, and the first duration could be set to 4 seconds, etc.

[0101] For example, when the resolution is adjusted from HD to MD or SD, the corresponding data volume threshold will be calculated based on the maximum bitrate at MD or SD resolution, and the appropriate data volume threshold will be redefined each time the resolution is adjusted.

[0102] For example, when a device switches to a lower resolution stream (e.g., from HD to MD), the bitrate decreases, and the amount of data generated per unit time decreases. Therefore, theoretically, a larger data size threshold is not needed to tolerate buffer latency. If the original data size threshold (e.g., 768KB) is maintained, the device will tolerate a latency exceeding, for example, 2 seconds, because the original data size threshold can store a longer period of low-resolution streaming media. This leads to unnecessary latency accumulation, affecting real-time performance; wastes memory and system resources because the buffer is too large, which is unnecessary for the current bitrate; and is less responsive to network conditions because the large data size threshold masks actual network congestion. If the data size threshold is not reduced, the device may mistakenly believe that the network conditions are good, thus delaying the timing of bitrate adjustment and resulting in a poor user experience. By dynamically adjusting the data size threshold according to the maximum bitrate corresponding to the current resolution, network fluctuations can be detected more quickly, allowing for a more timely response. A smaller buffer means lower memory usage and less resource consumption. Especially on devices with multiple concurrent streams and limited resources, optimizing resource usage is crucial. Adapting to the network requirements of different bitrate streams is also essential. Different resolutions correspond to different bitrates and network requirements. Reducing the resolution decreases network bandwidth requirements, allowing devices to decrease buffer size and reduce latency tolerance, thus enabling faster network condition detection and more flexible resolution adjustment strategies.

[0103] Assuming the device's maximum bitrate in HD mode is 6144kbps and the maximum buffer time Δt is 2 seconds, then: SBS = 6144 / 8 × 2 / 2 = 768KB.

[0104] When the device detects that the amount of buffered streaming media data is greater than or equal to the data volume threshold, it indicates that the current network bandwidth is insufficient to support the HD stream, so it switches to the MD stream (e.g., 2048kbps). At this time, the device recalculates the SBS: SBS = 2048 / 8 × 2 / 2 = 256KB.

[0105] By reducing the data volume threshold, the device can more sensitively detect subsequent network fluctuations or bitrate changes, thus responding faster.

[0106] Of course, the formulas and values ​​above are merely examples and not limitations.

[0107] In some embodiments, the method further includes reducing the resolution of newly arriving streaming data in response to motion of the first device, provided that the most recent resolution reduction was not in response to motion of the first device.

[0108] When the first device is in motion, the image changes rapidly in the video captured by the camera, causing the bitrate of the captured video to spike. Therefore, once the first device starts moving, regardless of whether the preview video is choppy, it is necessary to consider lowering the resolution of the streaming media. Furthermore, during the movement of the first device, it is possible to configure settings to disallow increasing the resolution of the streaming media.

[0109] Thus, when the first device capturing streaming data moves, the dynamic range of the captured image can cause the bitrate of the streaming media to spike, potentially leading to excessive data volume being buffered and unable to be sent in time. Reducing the resolution of the streaming media data in this situation lowers the bitrate, reducing storage space and the amount of data sent, thereby satisfying the real-time nature of streaming media transmission and the smoothness of the preview. However, when the user remotely controls the first device, it may move frequently. If frequent movement of the first device leads to frequent resolution reductions, it will affect the image quality and viewing experience of the streaming media. Therefore, it is necessary to ensure that the most recent resolution reduction is not in response to the movement of the first device, thus avoiding frequent resolution reductions caused by frequent movement of the first device.

[0110] In some embodiments, the method further includes: reducing the resolution of newly arriving streaming data in response to motion of the first device, having increased the resolution since the most recent reduction in resolution in response to motion of the first device.

[0111] Since the first device moves frequently, if the resolution has already been reduced due to the movement of the first device, it is not advisable to continue reducing the resolution if the first device moves again. However, if the resolution has already been increased, it can be reduced again when the first device moves, thus avoiding frequent resolution reductions caused by the frequent movement of the first device.

[0112] Here, a flag can be designed, which, when set, records that the movement of the first device has controlled the resolution adjustment, lowering the streaming media resolution and failing to restore the higher resolution. That is, the flag is set in response to the determination that the first device movement has lowered the resolution. If the flag is set, the first device movement will not trigger the operation of adjusting the resolution to lower the streaming media resolution. The flag is only reset after the resolution adjustment has increased the streaming media resolution. At this point, if it is determined that the flag has been reset and the first device movement is confirmed, the resolution can be lowered.

[0113] In some embodiments, the method further includes: reducing the resolution of newly arriving streaming media data in response to receiving a request to reduce the resolution from a second device.

[0114] If the second device detects video stuttering, it may send a request to reduce the resolution. In this case, the first device should immediately reduce the resolution to ensure the real-time transmission of the streaming media and the smoothness of the preview on the receiving end.

[0115] In some embodiments, the method further includes: increasing the resolution of the newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the first device initially sent streaming media data to the second device; increasing the resolution of the newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the most recent decrease or increase in the resolution of the newly arrived streaming media data; increasing the resolution of the newly arrived streaming media data in response to the resolution of the newly arrived streaming media data being lower than the maximum resolution and a second time elapsed since the first device stopped moving; or increasing the resolution of the newly arrived streaming media data in response to receiving a request to increase the resolution from the second device, determining that the resolution of the newly arrived streaming media data is lower than the maximum resolution and a second time elapsed since the most recent decrease or increase in the resolution.

[0116] Therefore, increasing the resolution periodically according to the second duration (e.g., t=32s) is to restore the clarity of the streaming and avoid pushing low-resolution streaming media for extended periods, which would result in a poor user experience.

[0117] In some embodiments, the method further includes: in response to increasing the resolution of newly arrived streaming media data, increasing the data volume threshold based on a predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data.

[0118] In this way, the data volume threshold increases dynamically as the current resolution increases. Based on the larger data volume threshold, it is more appropriate to determine whether to reduce the current streaming resolution. If the resolution increases, the amount of data generated by the allowed delay of the maximum buffer time will also increase. Increasing the data volume threshold allows for the buffering of larger data at higher resolutions without easily triggering a reduction in resolution, thus improving the clarity of the viewing image.

[0119] For example, when increasing the resolution (e.g., switching from SD to HD), the maximum bitrate corresponding to the resolution increases. Therefore, according to the formula, the data volume threshold needs to be increased proportionally. This ensures that during brief network fluctuations, a larger data volume threshold can accommodate more high-resolution video data, avoiding misjudgments of network congestion due to exceeding the originally smaller data volume threshold, which would reduce the clarity of the streaming media and affect the user's viewing experience.

[0120] However, this second duration can be extended. If it is not currently suitable to continue increasing the resolution, the second duration can be extended, allowing more time before increasing the resolution. This avoids frequently increasing the resolution when it is not appropriate.

[0121] In some embodiments, the method further includes extending a second duration in response to at least one of the following: the resolution of the newly arrived streaming data reaches its maximum resolution; and during a third duration since the increase in the resolution of the newly arrived streaming data, the first device moves or decreases the resolution of the newly arrived streaming data. Thus, extending the timing period for attempting to increase the resolution when it is not suitable avoids frequent increases in resolution when it is not appropriate.

[0122] For example, if the resolution has already been increased to the maximum resolution, it is no longer appropriate to increase the resolution again in a short period of time. Therefore, the second duration is extended (for example, twice the original second duration, such as twice 32 seconds), so that the resolution is increased after a longer period of time.

[0123] For example, if the resolution is successfully increased, but then decreased again within the third time interval (e.g., 10 seconds), it indicates that the current situation is insufficient to stably recover the increased resolution. Therefore, the second time interval is extended, allowing more time for the resolution to be improved.

[0124] If you attempt to increase the resolution but the first device is moving during the third duration (e.g., 10 seconds) (because it's stipulated that increasing the resolution is not allowed when the first device is moving), you can similarly extend the second duration, allowing more time before increasing the resolution. And so on.

[0125] In some embodiments, the second duration has an upper limit to prevent frequent switching between lowering and raising the resolution, and also to prevent the resolution from failing to recover to high resolution for an extended period after being lowered. For example, the second duration can be set to be extended until the upper limit of 256 seconds is reached. This second duration can be set by the duration of a timer.

[0126] In some embodiments, the degree to which the resolution is increased or decreased can be determined based on the transmit buffer space occupancy rate and the movement speed of the first device. This allows for more precise adjustment of the resolution.

[0127] In some embodiments, reducing the resolution of newly arriving streaming data includes reducing it to a lower resolution level among a plurality of selectable resolution levels from high to low. Alternatively, increasing the resolution of newly arriving streaming data includes increasing it to a higher resolution level among a plurality of selectable resolution levels from high to low. That is, if multiple resolution levels exist, each increase or decrease in resolution can increase or decrease the resolution to a higher or lower resolution level adjacent to the current resolution level. Increasing or decreasing resolution can be done incrementally. If the preview streaming is still not smooth after decreasing the resolution by one level, the resolution can be decreased by one level again. If the above conditions are met after the necessary time, the resolution can be increased by one level again. For example, resolutions from high to low are divided into HD, MD (if supported), and SD (e.g., VGA). When the first device moves or the amount of buffered data exceeds a threshold, the resolution level is reduced by one level. This is done incrementally, i.e., the resolution is reduced from HD to MD, and if the streaming is still not smooth, the resolution is reduced to SD. Similarly, increasing resolution can also be done incrementally.

[0128] Figure 5 A flowchart is shown of a method 500 performed by a second device to adjust the resolution of a streaming media transmitted by a first device, according to at least one embodiment of the present disclosure.

[0129] Method 500 includes steps 501 and 502. Here, the first device can be the device end described above, and the second device can be the client end described above. The device end can include a camera that captures streaming media and the first device equipped with the camera.

[0130] In step 501, in response to determining that the time difference between the generation time of the streaming media data and the reception time of the streaming media data exceeds a threshold, it is determined to reduce the resolution of the streaming media data.

[0131] In step 502, a request to reduce the resolution of the streaming media data is sent to the first device.

[0132] For example, after the client's underlying buffer receives streaming media data sent from the device, the upper layer of the client can process it immediately by comparing the current actual time T (the time the streaming media data was received) with the streaming media data generation timestamp (added by the first device and can be approximated as the time the streaming media data was sent) T. 帧 Calculate the time difference ΔT and determine whether the time difference ΔT exceeds the threshold ΔT. 阈 If it exceeds the threshold ΔT 阈 This determines to reduce the resolution of the streaming media data.

[0133] In some embodiments, the threshold is determined by the data transmission latency of the streaming media, the data transmission latency of the streaming media communication link, and the internal data processing latency of the second device.

[0134] For example, ΔT 阈 The calculation formula is as follows:

[0135] ΔT = T - T 帧 ;

[0136] ΔT 阈 = Δt + RTT* + T 内 ;

[0137] Where Δt is the data transmission delay of the streaming media sent by the aforementioned device; RTT* is the data transmission delay of the streaming media communication link, including the relay delay in the relay server scenario, T 内 This addresses the latency of data processing within the client, including client-side caching latency, decryption and decoding processing, etc.

[0138] In this way, the second device can also determine that the communication network is congested if the time difference between the generation time and the reception time of the streaming media data exceeds a threshold, thereby causing the first device to reduce the resolution of the streaming media data to ensure the real-time performance and smoothness of the streaming media data received by the second device.

[0139] In some embodiments, method 500 may further include: in response to a first duration not exceeding since the most recent request to reduce resolution, not sending a request to reduce resolution. This avoids situations where data reception latency persists for a period even after resolution reduction, leading to frequent requests to the first device to lower the resolution during this time, resulting in excessively low resolution and negatively impacting the user's experience with the clarity of streaming media.

[0140] In some embodiments, method 500 may further include: determining to increase resolution in response to a second time elapsed after a request to decrease resolution; and sending a request to increase resolution to a first device.

[0141] This is so that the second device can also periodically request a higher resolution to prevent the resolution from remaining consistently low and affecting the user's streaming experience.

[0142] The following section describes the overall process of increasing and decreasing resolution on the device and client sides, as well as the communication and interaction process between them, using specific examples.

[0143] Figure 6 A flowchart of a device-side resolution reduction mechanism according to at least one embodiment of the present disclosure is shown.

[0144] The device side needs to be combined Figure 4 A set of rules is described to determine whether to lower (i.e. reduce) the resolution of the transmitted streaming media. Subsequently, the movement of the first device is equated to the rotation of the gimbal carrying the first device.

[0145] Specifically, in step 601, it is determined whether the pull-up timer (e.g., 32s) (i.e. the second duration mentioned above) has timed out.

[0146] If the pull-up timer (e.g., 32s) is determined to have timed out, i.e., 32s have elapsed, then in step 602, an attempt is made to increase the resolution of the streaming media and reset the pull-up timer to 0 to continue timing.

[0147] If the resolution of the streaming media is successfully increased (e.g., the PTZ is not rotating, or the resolution was not at the highest level before the increase, etc.), then in step 603, the flag bit is reset to 0, indicating that the resolution was not reduced due to the rotation of the PTZ.

[0148] In step 604, the stream continues to be pushed to the client at the increased resolution. That is, the client receives the streaming media with the increased resolution at this time, and the user can preview the streaming media with the increased resolution through the client to achieve a better viewing experience.

[0149] In step 605, in response to determining whether a silent duration t (e.g., t=4) seconds (i.e., the first duration mentioned earlier) has elapsed since the most recent resolution reduction, and determining that the buffer exceeds the data volume threshold, if the previous resolution was increased, there is no need to wait for the silent duration. This is because the silent duration is mainly to give the send buffer some time to empty if the resolution was reduced due to the buffer exceeding the data volume threshold. After the silent duration, the amount of data buffered in the send buffer may be gradually reduced, causing the send buffer to be in a state of dissipation. Only in this case can the resolution be reduced again due to the buffer exceeding the data volume threshold (e.g., congestion in the streaming media communication link, resulting in data not being discharged from the send buffer in time, or due to excessive dynamic changes in the captured image (e.g., the subject moving too much or too fast), causing the video image bitrate to spike, resulting in a larger amount of data buffered in the send buffer, etc.). Therefore, this design can avoid the situation where the buffer exceeds the data volume threshold for a prolonged period of time, leading to frequent resolution reductions during this period, resulting in excessively low resolution and affecting the user's experience of viewing streaming media clarity. If the previous resolution was increased, there is no need to consider avoiding frequent reductions in resolution. You can reduce the resolution without waiting for the silent period to expire, i.e., once you determine that the buffer exceeds the data volume threshold.

[0150] If it is determined that the backoff has exceeded the silence duration t (e.g., t=4) seconds and the buffer exceeds the data volume threshold, then in step 606, the resolution is reduced, an attempt is made to lower the resolution, and the timer is reset to 0 to continue timing.

[0151] If the backoff has not exceeded the silence duration t (e.g., t=4) seconds, or the buffer has not exceeded the data volume threshold, then return to step 601 to determine whether the timer has timed out to decide whether to increase the resolution.

[0152] Step 606 can also be triggered by steps 607 and 608. In step 607, it is determined that the pan-tilt unit (PTZ) has started rotating. In step 608, it is determined whether the flag bit is set to 1. If the flag bit is set, it means that the PTZ rotation has controlled the adjustment of the resolution, thus lowering the resolution of the streaming media and has not restored the increased resolution. Therefore, it should not continue to lower the resolution; otherwise, the continuous rotation of the PTZ will cause the resolution to be continuously lowered, affecting the user's experience of viewing the clarity of the streaming media.

[0153] Therefore, if the flag bit is set to 1, the streaming continues. If the flag bit is not set to 1 (i.e., 0), then in step 606, the resolution can be reduced to attempt to lower the resolution, and the pull-high timer can be reset to 0 to continue timing.

[0154] Next, in step 609, it is determined whether the resolution reduction was successful. If the resolution reduction was successfully caused by the gimbal rotation, then in step 610, the flag is set to 1, indicating that the gimbal rotation has controlled and adjusted the resolution, thus reducing the streaming media resolution and not restoring the high resolution. Furthermore, in step 611, a silence duration t=4s is set, indicating that the resolution can only be reduced again after 4 seconds, and streaming can continue.

[0155] If lowering the resolution fails (for example, the current resolution is already at the lower limit, such as the SD level among the three levels of HD, MD and SD), then in step 612, set the silence duration t=4s and continue streaming.

[0156] Figure 7 A flowchart of a device-side resolution upscaling mechanism according to at least one embodiment of the present disclosure is shown.

[0157] Figure 7 Detailed description Figure 6 The steps related to increasing the resolution that occur in the process.

[0158] Specifically, in step 701, the communication link for sending streaming media to the first device is initially established, i.e., the initial streaming begins. In step 702, the resolution is successfully lowered. In step 703, the pan-tilt rotation ends. After steps 701, 702, and 703, in step 704, the pull-up timer is reset to 32 seconds. After the pull-up timer expires, it is determined that the resolution needs to be increased, and in step 705, it is determined whether the resolution can be successfully increased.

[0159] If it is determined that increasing the resolution cannot be successfully achieved (e.g., the gimbal starts rotating, or the current resolution has reached its maximum, such as HD), then in step 706, the timer for increasing the resolution is extended, for example, by setting the backoff timer to 2 * the timer for increasing the resolution. In this case, each time the timer for increasing the resolution is extended, it is multiplied by 2 from the original timer. That is, the initial timer for increasing the resolution is 32 seconds, the first extension is 64 seconds, the second extension is 128 seconds, the third extension is 256 seconds, and so on. The current backoff timer has a maximum duration, such as 256 seconds. After reaching the maximum duration, it will not be extended further. That is, it will wait a maximum of 256 seconds before attempting to increase the resolution, thus preventing a prolonged period of inability to increase the resolution after decreasing it.

[0160] If the resolution can be successfully increased, then in step 707, determine whether the streaming can be stabilized for a stable duration (e.g., 10s) after the resolution is increased (i.e., the third duration mentioned above).

[0161] If the resolution cannot be successfully increased (e.g., the resolution has already been increased to the maximum resolution; the resolution cannot be increased due to gimbal rotation) (No in step 705), or if the resolution decreases within a stable duration (e.g., 10 seconds) after the resolution is increased (i.e., the stream cannot be stably pulled for 10 seconds after the resolution is increased) (No in step 707), then in step 707, the timing of the pull-up timer is extended, for example, by setting the backoff timer = 2 * pull-up timer. As described above, for example, in this case, each time the timing of the pull-up timer is extended, it can be multiplied by 2 based on the original timer.

[0162] If in step 707 it is determined that the streaming can be stably pulled for a stable duration (e.g., 10s) after the resolution is increased, then in step 704, the pull-up timer is reset to 32s so that the resolution can be increased again when the pull-up timer expires. This can increase the resolution of the streaming media as much as possible and improve the user's streaming media viewing experience.

[0163] Note that if multiple resolution levels exist, each time you increase or decrease the resolution, you can raise or lower the resolution to the next higher or lower resolution level adjacent to the current resolution level. That is, increasing or decreasing the resolution can be done step by step.

[0164] Figure 8 A flowchart illustrating a resolution-boosting and resolution-lowering mechanism for a client according to at least one embodiment of the present disclosure is shown.

[0165] In step 801, the client determines whether the pull-high timer has timed out. The pull-high timer can be 32 seconds.

[0166] If the client determines that the resolution timer has expired, in step 802, it attempts to increase the resolution and resets the resolution timer. The client needs to send a resolution increase request to the device. Upon receiving the resolution increase request from the client, in step 803, the client continues to stream to the device. In step 809, the device determines whether the first device carrying the device is not moving and whether the time interval between the most recent resolution increase or decrease operation exceeds the silence duration T1. If so, in step 810, the device attempts to increase the resolution; otherwise, the device does not increase the resolution and continues to stream the un-upgraded media. If the device successfully increases the resolution (e.g., the client receives a successful response from the device to the resolution increase request), in step 811, the device increases the resolution and resets the silence duration T1=32s. In step 812, the device sends the upgraded media to the client and continues to stream. If the device cannot successfully increase the resolution (e.g., the client receives a response from the device indicating that the request to increase the resolution failed, such as because the device's pan-tilt unit has started rotating again, or the device's current resolution has reached its maximum), then in step 812, the device sends the streaming media without increasing the resolution to the client and continues to push the stream.

[0167] In step 804, the client determines whether the data reception time difference (i.e., the time difference between the generation time and reception time of the streaming media data) exceeds the threshold and whether the backoff silence duration t seconds has been reached. If not, the resolution is not lowered, and the process returns to step 801 to determine whether the timer has timed out. If so, in step 805, the client attempts to lower the resolution, restarts the timer, and sends a request to the device to lower the resolution.

[0168] In step 806, the client determines whether lowering the resolution can be successfully achieved. If successful (e.g., the client receives a response from the device to the request to lower the resolution), then in step 807, a silence duration t=4s is set, and the resolution can only be lowered again after 4s. If unsuccessful (e.g., the client does not receive a response from the device to the request to lower the resolution, such as because the current resolution has reached the lower limit), then in step 808, a silence duration t=4s is set, and the resolution can only be lowered again after 4s.

[0169] In step 814, the device receives a request to lower the resolution and immediately attempts to do so. If not, in step 812, the resolution of the streaming media is not lowered, and the streaming media with the resolution not lowered is sent to the client, continuing to stream. If the resolution can be lowered, in step 813, the resolution is lowered, the silence time T1=32s is reset, and in step 812, the streaming media with the lowered resolution is sent to the client, continuing to stream.

[0170] In step 815, the device detects that the gimbal has started to rotate, and in step 814 it will also try to lower the resolution.

[0171] Secondly, when the client needs to reduce the resolution, the device immediately attempts to reduce the resolution to minimize the time difference in data reception. When the client needs to increase the resolution, the device needs to determine whether the time interval between the last resolution increase or decrease exceeds the silent duration. This is to avoid frequently increasing the resolution, which could lead to situations where the buffer exceeds the data volume threshold or the data reception time difference is too long, requiring the resolution to be reduced, and then frequently reducing the resolution again, resulting in excessively fast resolution switching. The device also needs to determine whether the gimbal is rotating. If the gimbal is rotating, it is not suitable to increase the resolution.

[0172] In actual testing, for devices like high-speed PTZ cameras with wide bitrate ranges and high gimbal rotation speeds, bitrate spikes can easily occur due to gimbal rotation or capturing dynamic footage. In scenarios with insufficient communication network bandwidth, the method of transmitting streaming media data according to at least one embodiment of this disclosure can automatically reduce the resolution for streaming media preview, ensuring the real-time performance and smoothness of the streaming media. When network bandwidth is sufficient, the method of transmitting streaming media data according to at least one embodiment of this disclosure can restore the resolution to a higher resolution, ensuring video quality and improving the user viewing experience. Compared to the commonly supported fixed high and low resolution preview methods in the market, this provides a better live streaming preview experience.

[0173] Note that in the above embodiments, if a relay server is present, the data sent and received by the client and the device are relayed through the relay server. The specific process is not described in detail here, and those skilled in the art can construct it based on at least one embodiment of this disclosure.

[0174] Figure 9 A block diagram of a device 900 for transmitting streaming media data according to at least one embodiment of the present disclosure is shown.

[0175] The device 900 includes at least: at least one memory 920 configured to store program instructions; and one or more processors 910 coupled to the at least one memory and configured to execute program instructions to perform various methods of the device in at least one embodiment of this disclosure.

[0176] For example, only one processor 910 is shown, but there can also be multiple processors. Furthermore, processing can be performed by a single processor, or by more than one processor simultaneously, sequentially, or using other methods. Additionally, processor 910 can be mounted on more than one chip.

[0177] The functions of the device 900 can be implemented, for example, by reading the instructions (programs) stored in the memory 930 into hardware such as the processor 910 and the memory 920, so that the processor 910 can perform operations, control the communication performed by the communication device 930, and control the reading and / or writing of data in the memory 920.

[0178] The processor 910 enables the operating system to operate, thereby controlling the device as a whole. The processor 910 may be composed of a central processing unit (CPU) that includes interfaces with peripheral devices, control devices, arithmetic devices, registers, etc. For example, the aforementioned processing units can be implemented by the processor 910.

[0179] Furthermore, the processor 910 reads programs (program code), data, etc., from the memory 920 and performs various processes accordingly. The program can be one that causes the computer to perform at least a portion of the actions described in the above embodiments. For example, the method executed by the device can be implemented using a control program stored in the memory 920 and operated by the processor 910.

[0180] The memory 920 may be a computer-readable recording medium, such as at least one of a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically programmable read-only memory (EEPROM), a random access memory (RAM), or other suitable storage media. The memory 920 may include registers, caches, main memory (main storage device), etc. The memory 920 may store executable programs (program code), software modules, etc., for implementing the methods according to an embodiment of this disclosure.

[0181] In addition, the memory 920 may also include, for example, a computer-readable recording medium consisting of at least one of a flexible disk, a floppy disk, a magneto-optical disk (e.g., a read-only optical disc (CD-ROM, etc.), a digital universal optical disc, a Blu-ray disc), a removable disk, a hard disk, a smart card, a flash memory device (e.g., a card, a stick, a key driver), a magnetic stripe, a database, a server, or other suitable storage media.

[0182] Communication device 930 is hardware (transmitting and receiving device) used for communication between computers via wired and / or wireless networks, and is also referred to as a network device, network controller, network interface card (NIC), communication module, etc. To implement, for example, frequency division duplex (FDD) and / or time division duplex (TDD), communication device 930 may include high-frequency switches, duplexers, filters, frequency synthesizers, etc. For example, the aforementioned transmitting and receiving operations can be implemented by communication device 930.

[0183] The communication device 930 can communicate bidirectionally via one or more antennas, wired or wireless links, as described herein. For example, the communication device 930 can represent a wireless transceiver and can communicate bidirectionally with another wireless transceiver. The communication device 930 may also include a modem to modulate packets and provide the modulated packets to the antenna for transmission, and to demodulate packets received from the antenna. In some cases, the communication device may include a single antenna 950. However, in some cases, the communication device may have more than one antenna 950, which are capable of simultaneously transmitting or receiving multiple wireless transmissions.

[0184] Furthermore, the processor 910, memory 920, and other devices are connected via a bus 940 for communication of information. The bus 940 can consist of a single bus or different buses between devices.

[0185] In addition, the device may include hardware such as microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and field-programmable gate arrays (FPGAs).

[0186] This disclosure may include a non-transitory computer-readable storage medium. Instructions, such as computer instructions, are stored on the non-transitory computer-readable storage medium. When the computer instructions are executed by a processor, the various methods described above can be performed. Non-transitory computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.). For example, the non-transitory computer-readable storage medium can be connected to a computing device such as a computer, and then, when the computing device executes the computer instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0187] This disclosure may also include a computer program product capable of performing the methods, steps, and operations given herein. For example, such a computer program product may be a computer software package, computer code instructions, or a computer-readable tangible medium having computer instructions tangibly stored (and / or encoded) thereon, which can be executed by a processor to perform the operations described herein. The computer program product may include packaging materials.

[0188] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The term “such as / for example” as used herein refers to the phrase “such as / for example but not limited to,” and is used interchangeably with it.

[0189] The flowcharts and method descriptions in this disclosure are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the given order. As those skilled in the art will recognize, the steps in the above embodiments can be performed in any order. Words such as "then," "next," etc., are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to a singular element, such as the use of the articles "a," "one," or "the," is not to be construed as limiting that element to the singular.

[0190] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit at least one embodiment of the present disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for sending streaming media data, performed by a first device, the method comprising: In response to meeting predetermined conditions, the resolution of newly arriving streaming media data is reduced; as well as Send the streaming media data in the transmit buffer to the second device. The predetermined conditions include that the amount of streaming media data buffered in the transmission buffer is greater than or equal to a data volume threshold and that a first time interval has elapsed since the resolution of the most recently received streaming media data was reduced. The data volume threshold is determined based on a predetermined maximum buffer time and is variable depending on the resolution of the newly arriving streaming media data.

2. The method of claim 1, further comprising: In response to reducing the resolution of newly arrived streaming media data by the first duration, the data volume threshold is reduced based on the predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data, wherein the first duration is greater than or equal to the predetermined maximum buffer time.

3. The method of claim 1, further comprising: In response to movement of the first device, and where the most recent resolution reduction was not in response to movement of the first device, the resolution of newly arriving streaming data is reduced; In response to movement of the first device, and having increased the resolution since the last time the resolution was reduced in response to movement of the first device, the resolution of the newly arriving streaming data is reduced; or In response to receiving a request to reduce the resolution from the second device, the resolution of newly arriving streaming media data is reduced.

4. The method of claim 1, further comprising: In response to newly arriving streaming media data having a resolution lower than the maximum resolution and a second duration elapsed since the first device initially sent streaming media data to the second device, the resolution of the newly arriving streaming media data is increased. In response to newly arriving streaming data having a resolution lower than the maximum resolution and after the second duration since the last decrease or increase in the resolution of the newly arriving streaming data, the resolution of the newly arriving streaming data is increased. In response to the arrival of new streaming data with a resolution lower than the maximum resolution and the second duration elapsed since the first device stopped moving, the resolution of the new streaming data is increased. or In response to receiving a request to increase resolution from the second device, the resolution of the newly arriving streaming media data is increased if the resolution of the newly arriving streaming media data is lower than the maximum resolution and the second duration has elapsed since the last resolution reduction or increase.

5. The method of claim 4, further comprising: In response to increasing the resolution of newly arriving streaming media data, the data volume threshold is increased based on the predetermined maximum buffer time and the maximum bitrate of the streaming media data corresponding to the resolution of the newly arriving streaming media data.

6. The method of claim 4, further comprising extending the second duration in response to at least one of the following: The newly arrived streaming data has reached its maximum resolution; and Within a third time period since the resolution of the newly arrived streaming data was increased, the first device moves or decreases the resolution of the newly arrived streaming data.

7. The method of claim 6, wherein, The second duration has an upper limit.

8. The method of claim 1, wherein, The size of the transmit buffer is set to be greater than or equal to the data volume threshold.

9. The method of claim 1, wherein, The data volume threshold is determined by multiplying the predetermined maximum buffer time by the maximum bitrate of the streaming media data corresponding to the resolution of the newly arrived streaming media data.

10. A device for transmitting streaming media data, comprising: At least one memory is configured to store program instructions; as well as One or more processors, said one or more processors coupled to said at least one memory, and configured to execute said program instructions to perform the method of any one of claims 1 to 6.

Images