A real-time audio stream playing method and related device

Abstract

The application discloses a real-time audio stream playing method and related device, and relates to the technical field of audio playing, which comprises the following steps: receiving a current audio data packet carrying an index; comparing the index of the current audio data packet with a maximum played index, if the index of the current audio data packet is greater than the maximum played index and is discontinuous with the maximum played index, storing the current audio data packet into a cache queue; sorting the audio data packets in the cache queue according to the index, finding a first discontinuous demarcation point in the index increasing direction for the sorted cache queue; if the demarcation point is found, playing the index continuous audio data packets before the demarcation point in the sorted cache queue, and continuing to cache the audio data packets after the demarcation point. The application effectively avoids the audio lag, jump or interruption caused by the packet disorder due to network jitter, and significantly improves the fluency and stability of real-time audio playing.

Description

Technical Field

[0001] This invention relates to the field of audio playback technology, and in particular to a real-time audio stream playback method and related apparatus. Background Technology

[0002] With the rapid development of technologies such as remote conferencing and live streaming, stable playback of real-time audio streams has become a core element of user experience. In demanding scenarios such as international conferences and large-scale government meetings, audio streams are often transmitted via the WebSocket protocol, requiring participants to receive multiple audio streams, including simultaneous interpretation and original audio, in real time.

[0003] However, due to the instability of network transmission, audio data packets inevitably arrive out of order during transmission. Current solutions to this problem are mainly the "all-wait" and "all-discard" schemes. The "all-wait" scheme waits for all out-of-order audio data packets to arrive before playing them in sequence. The "all-discard" scheme discards out-of-order audio data packets and only plays the ones that arrive in order.

[0004] The full wait scheme can guarantee playback continuity, but when the packet loss rate is high, it may lead to permanent waiting and extremely poor real-time performance. The full discard scheme has better real-time performance, but the high packet loss rate causes obvious audio stuttering. Summary of the Invention

[0005] In view of this, the present invention provides a real-time audio stream playback method and related apparatus to solve the problems of existing all-wait schemes which easily lead to permanent waiting and poor real-time performance when the packet loss rate is high, and all-drop schemes which easily lead to obvious audio stuttering and poor continuity when the packet loss rate is high. The technical solution is as follows:

[0006] The first aspect of this invention provides a real-time audio stream playback method, applied to a client, the method comprising:

[0007] Receive the current audio data packet from the server, wherein the current audio data packet carries an index, the index representing the temporal position of the current audio data packet in the global audio stream;

[0008] The index of the current audio data packet is compared with the maximum index of the played data packet. If the index of the current audio data packet is greater than the maximum index of the played data packet and is not contiguous with the maximum index of the played data packet, the current audio data packet is stored in the cache queue. The maximum index of the played data packet is the maximum index among the indices of the played audio data packets.

[0009] The audio data packets in the cache queue are sorted by index. For the sorted cache queue, the first point where the index is not continuous is found along the increasing direction of the index.

[0010] If the dividing point is found, the consecutive audio data packets with the index before the dividing point in the sorted cache queue are played, while the audio data packets after the dividing point continue to be cached and wait.

[0011] In one possible implementation, the real-time audio stream playback method further includes:

[0012] If the index of the current audio data packet is less than or equal to the maximum index that has been played, then the current audio data packet is discarded;

[0013] If the index of the current audio data packet is greater than the maximum index that has been played, and is consecutive to the maximum index that has been played, then the current audio data packet is played.

[0014] If the index of the current audio data packet is greater than the maximum index already played, and is not contiguous with the maximum index already played, the real-time audio stream playback method further includes:

[0015] If the boundary point is not found, then all audio data packets in the sorted cache queue will be played.

[0016] In one possible implementation, the index is a multi-level ordered index array;

[0017] The step of comparing the index of the current audio data packet with the maximum index that has been played includes:

[0018] The index of the current audio data packet is compared with the maximum index that has been played, level by level.

[0019] If the index value of the current audio data packet at the current level is less than the index value of the corresponding level of the maximum played index, then it is determined that the index of the current audio data packet is less than the maximum played index, and the comparison of subsequent levels is stopped.

[0020] If the index value of the current audio data packet at the current level is greater than the corresponding level index value of the maximum played index, then it is determined that the index of the current audio data packet is greater than the maximum played index, and the comparison of subsequent levels is stopped.

[0021] If the index of the current audio data packet is equal to the index value of the maximum played index at all levels, then the index of the current audio data packet is determined to be equal to the maximum played index.

[0022] In one possible implementation, the real-time audio stream playback method further includes:

[0023] Calculate the difference between the current time and the earliest arrival time among the arrival times of each audio data packet in the cache queue, and use this as the cache waiting time;

[0024] If the cache wait time exceeds a preset time threshold, the first consecutively indexed audio segment in the cache queue will be played. The consecutively indexed audio segment consists of consecutively indexed audio data packets.

[0025] In one possible implementation, the client maintains the maximum played index, which is updated after each audio data packet is played.

[0026] The real-time audio stream playback method further includes:

[0027] When the communication connection with the server is disconnected and then successfully reconnected, the server receives historical cached audio data packets pushed by the server. The historical cached audio data packets are audio data packets cached by the server during the disconnection period.

[0028] Determine whether the index of the historical cached audio data packet is less than or equal to the maximum index of the played data;

[0029] If the index of the historical cached audio data packet is less than or equal to the maximum index of the played data, then

[0030] Discard the historical cached audio data packets.

[0031] In one possible implementation, the real-time audio stream playback method further includes:

[0032] When the communication connection with the server is detected to be disconnected and then successfully reconnected, a reconnection flag is set.

[0033] If the historical cached audio data packet is the first audio data packet after the reconnection flag is set, and the index of the historical cached audio data packet is greater than the maximum index of the played data packet, then the historical cached audio data packet is played.

[0034] In one possible implementation, the real-time audio stream playback method further includes:

[0035] Obtain the current playback timestamp of the audio playback context object. The current playback timestamp is the real-time time value output by the internal clock of the audio playback context object, which is used to indicate the current playback progress of the audio hardware.

[0036] Obtain the estimated playback end time corresponding to the last audio data segment scheduled to the timeline of the audio playback context object. The estimated playback end time is the sum of the start playback time of the last audio data segment on the timeline and the duration of the last audio data segment.

[0037] The current playback duration is determined based on the estimated playback end time, the current playback timestamp, and the total duration of all audio data in the audio buffer that has not yet been scheduled to the audio playback context object.

[0038] The playback speed is determined based on the current playback duration, and subsequent audio data is played at the determined playback speed.

[0039] In one possible implementation, determining the playback speed based on the current playback duration includes:

[0040] If the current playback duration is less than or equal to the preset low latency threshold, then the playback speed is determined to be the standard speed.

[0041] If the current playback duration is greater than or equal to the preset high latency threshold, then the playback speed is determined to be the maximum allowed speed.

[0042] If the current playback duration is greater than the low latency threshold and less than the high latency threshold, then the playback speed is determined based on the ratio of the current playback duration between the low latency threshold and the high latency threshold.

[0043] In one possible implementation, the real-time audio stream playback method further includes:

[0044] Each time an audio scheduling refresh operation is performed, the current playback timestamp of the audio playback context object is recorded. The current playback timestamp is the real-time time value output by the internal clock of the audio playback context object, which is used to indicate the current playback progress of the audio hardware.

[0045] If the recorded playback timestamp does not change during multiple consecutive audio scheduling refresh operations, and the recorded playback timestamp is greater than zero, then the audio playback is determined to be interrupted.

[0046] After determining that audio playback is interrupted, the audio scheduling and refresh process is paused and a buffering waiting state is entered.

[0047] Once the amount of audio data to be played in the audio buffer reaches a preset buffer threshold, the audio scheduling refresh process is restarted to resume audio playback.

[0048] In one possible implementation, the real-time audio stream playback method further includes:

[0049] Calculate the current playback duration;

[0050] If the current playback duration exceeds the preset backlog threshold, the backlog of audio data to be played is cleared, and playback processing begins from the latest audio data to be played.

[0051] In one possible implementation, the real-time audio stream playback method further includes:

[0052] Monitor the visibility status of the page that hosts audio playback functionality;

[0053] When the page is switched to an invisible state, audio playback is paused, and the player user interface state is synchronously updated to the stopped state.

[0054] When the page becomes visible again, the player user interface is restored to the ready-to-play state, and the user is guided to manually trigger playback resumption.

[0055] In one possible implementation, the real-time audio stream playback method further includes:

[0056] Identify the application container that carries the page, and determine whether the identified application container is in the preset application container whitelist;

[0057] If the identified application container is within the application container whitelist, when the page is switched to an invisible state, audio playback is maintained, and when the page is restored to a visible state, an audio playback restoration operation is performed.

[0058] When the page is switched to an invisible state, audio playback is paused, and the player's user interface state is synchronously updated to the stopped state. When the page becomes visible again, the player's user interface state is restored to the ready-to-play state, and the user is guided to manually trigger playback resumption, including:

[0059] If the identified application container is not in the application container whitelist, when the page switches to an invisible state, audio playback is paused and the player user interface state is updated to a stopped state. When the page returns to a visible state, the player user interface state is restored to a ready-to-play state, and the user is guided to manually trigger playback resumption.

[0060] A second aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0061] The memory is used to store computer programs;

[0062] The processor is used to execute the computer program to enable the electronic device to implement any of the above-described real-time audio streaming methods.

[0063] A third aspect of this application provides a computer-readable storage medium storing computer instructions for causing an electronic device to implement any of the above-described real-time audio stream playback methods.

[0064] Using the above technical solution, the real-time audio stream playback method provided by the present invention, after receiving an audio data packet carrying a global temporal index, first compares its index with the maximum index that has been played. If the index of the current audio data packet is greater than the maximum index that has been played, and is not continuous with the maximum index that has been played, it indicates that the current audio data packet is out of order, and then stores it in a cache queue. Next, after sorting the audio data packets in the cache queue by index, the first dividing point where the index is not continuous is found along the increasing direction of the index, and only the audio data packets with continuous indexes before the dividing point are played. The audio data packets after the dividing point continue to be cached and wait. The real-time audio streaming method provided by this invention can dynamically identify consecutive audio data segments in the buffer queue and immediately play them without waiting for all out-of-order packets to arrive. This fundamentally avoids the problems of permanent waiting and real-time performance degradation caused by packet loss. At the same time, the received out-of-order packets are retained and participate in the sorting process. Compared with the solution of directly discarding out-of-order packets, this method can maximize the use of the arrived audio data packets, thereby improving playback continuity and effectively reducing audio stuttering. Furthermore, this invention only needs to perform a sequential lookup in the buffer queue to determine the playable consecutive audio data segments. The processing logic is simple, the computational overhead is small, and it can adapt to various unstable scenarios such as network out-of-order packets and packet loss, exhibiting good real-time performance and adaptability. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0066] Figure 1 A flowchart illustrating the real-time audio stream playback method provided in an embodiment of the present invention;

[0067] Figure 2 A flowchart illustrating the dynamic speed-up delay elimination scheme based on the internal clock of the audio playback context object provided in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the structure of a real-time audio stream playback device provided in an embodiment of the present invention. Detailed Implementation

[0069] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0070] The embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0071] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of the invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] like Figure 1 As shown, this embodiment provides a real-time audio streaming playback method. This method is applied to a client and is used to ensure the continuity and real-time performance of audio playback in complex network environments. The method may include:

[0073] Step S101: Receive the current audio data packet from the server.

[0074] The current audio data packet carries an index, which represents the temporal position of the current audio data packet in the global audio stream.

[0075] In this embodiment, the client continuously listens to and receives the audio data stream sent by the server through the network interface. As the core execution carrier of the multi-layer collaborative real-time audio stream intelligent playback system, the client is responsible for connecting to the audio stream data of the input layer.

[0076] In real-time network transmission environments, due to factors such as network jitter and differences in routing, the order in which audio data packets arrive at the client may differ from the order in which they are sent at the server, resulting in out-of-order delivery. To accurately reconstruct the original timing sequence of the audio on the client side, each audio data packet is assigned a unique identifier, i.e., an index, during encapsulation. This index strictly represents the temporal position of the audio data packet within the global audio stream.

[0077] Step S102: Compare the index of the current audio data packet with the maximum index that has been played.

[0078] Among them, the maximum index that has been played is the maximum index among the indices of all the audio data packets that have been played.

[0079] In this embodiment, the client maintains the maximum played index and updates it after each audio data packet is played. That is, the maximum played index is a dynamically updated variable that records the maximum index among the indices of each audio data packet that has been successfully submitted for playback or decoding.

[0080] When the current audio data packet is received, obtain the index carried by the current audio data packet and compare the index with the maximum index that has been played.

[0081] Step S103a: If the index of the current audio data packet is less than or equal to the maximum index that has been played, then discard the current audio data packet.

[0082] If the index of the current audio data packet is less than or equal to the maximum index that has been played, it means that the current audio data packet has been successfully submitted for playback or decoding, that is, it is a duplicate packet, and therefore, the current audio data packet is directly discarded.

[0083] In network retransmission mechanisms or weak network environments, the server may repeatedly send the same audio data packets. This step effectively prevents the audio content from being played repeatedly, avoiding echoes or overlapping sounds for the user, and also saves computational resources for subsequent decoding and rendering.

[0084] Step S103b: If the index of the current audio data packet is greater than the maximum index that has been played, and is consecutive to the maximum index that has been played, then the current audio data packet is immediately played.

[0085] If the index of the current audio data packet is greater than the maximum index that has been played and is consecutive to the maximum index that has been played, it means that the current audio data packet is exactly the next audio data packet that is expected to be received. That is, the current audio data packet is a consecutive packet, and the network transmission is in an ideal and orderly state. At this time, the current audio data packet can be played directly.

[0086] This step minimizes processing latency and ensures near-real-time performance when the network is good. Simultaneously, after playback completes, the index of the largest played audio data packet needs to be updated synchronously to prepare for the judgment of the next audio data packet.

[0087] Step S103c-1: If the index of the current audio data packet is greater than the maximum index that has been played, and is not contiguous with the maximum index that has been played, then store the current audio data packet in the buffer queue.

[0088] If the index of the current audio data packet is greater than the maximum index that has been played, and is not contiguous with the maximum index that has been played, it means that the current audio data packet belongs to the audio data packet that needs to be played in the future. That is, the current audio data packet is out of order. In this case, it is temporarily stored in the buffer queue to wait for the arrival of the missing data packet.

[0089] This step is the entry point for implementing the partial continuous playback strategy. It avoids putting all newly arriving data packets into the cache indiscriminately, thereby reducing unnecessary memory usage and processing latency.

[0090] Step S103c-2: Sort the audio data packets in the buffer queue by index, and for the sorted buffer queue, find the first non-contiguous boundary point along the increasing index direction.

[0091] Due to network jitter, data packets entering the buffer queue may arrive out of order; for example, data packets with indices 5, 3, and 4 may enter the buffer queue sequentially. Therefore, it is necessary to first sort the audio data packets in the buffer queue according to their index values ​​from smallest to largest to restore their logical order.

[0092] Considering that there may be duplicate packets in the cache queue, i.e. audio data packets with the same index, the cache queue can be deduplicated before sorting the audio data packets in the cache queue by index.

[0093] After sorting, for the sorted cache queue, in one possible implementation, the first non-contiguous index boundary can be found along the ascending index direction. In another possible implementation, the first non-contiguous index boundary in the sorted cache queue can be found starting from the next index adjacent to the highest playing index and along the ascending index direction. This boundary logically divides the cache queue into two parts: the part before the boundary is the playable segment with continuous indices, and the part after the boundary is the waiting segment that cannot be played temporarily due to missing intermediate data. It should be noted that the index of the audio data packets before the boundary is smaller than the index of the audio data packets after the boundary.

[0094] Step S103c-3a: If the first non-contiguous index boundary is found, the audio data packets with continuous indexes before the boundary in the sorted cache queue are played, and the audio data packets after the boundary continue to be cached and wait.

[0095] Unlike existing wait-all schemes (waiting for all out-of-order packets to arrive before playback), this invention does not require the entire buffer queue to be completely continuous before playback begins. Instead, it immediately retrieves and plays the first continuous segment of audio data as soon as it is identified. For example, if the maximum played index is 9, and the indices of the audio data packets in the sorted buffer queue are [10, 11, 12, 15, 16], then the dividing point is between 12 and 15. Therefore, the audio data packets with indices 10, 11, and 12 are retrieved for playback, while the audio data packets with indices 15 and 16 continue to wait in the buffer for the arrival of the audio data packets with indices 13 and 14. The advantage of this playback strategy is that it can tolerate a small amount of out-of-order data while continuously outputting audio data packets with consecutive indices. This avoids prolonged silence or stuttering caused by waiting for individual lost packets, and also avoids audio information loss caused by directly discarding out-of-order packets, thus achieving a better balance between continuity and real-time performance.

[0096] Step S103c-3b: If the first non-contiguous index boundary is not found, then play all audio data packets in the sorted cache queue.

[0097] If no first point of discontinuity in the index is found, it means that the missing audio data packets that were originally in the buffer queue have been filled, or the newly arrived audio data packets have filled the gaps, making the entire buffer queue a completely continuous sequence. At this point, there are no longer any breakpoints in the buffer queue that would hinder playback. Therefore, all audio data packets in the queue can be retrieved and played at once. After all playback is complete, the maximum index that has been played also needs to be updated to the maximum index value among these data packets.

[0098] The real-time audio stream playback method provided in this embodiment of the invention, after receiving an audio data packet carrying a global temporal index, first compares its index with the maximum index that has been played. If the index of the current audio data packet is less than or equal to the maximum index that has been played, it indicates that the current audio data packet is a duplicate packet, and the current audio data packet is discarded directly. If the index of the current audio data packet is greater than the maximum index that has been played and is continuous with the maximum index that has been played, it indicates that the current audio data packet is a desired continuous packet, and the current audio data packet is immediately played. If the index of the current audio data packet is greater than the maximum index that has been played and is not continuous with the maximum index that has been played, it indicates that the current audio data packet is an out-of-order packet, and it is stored in a cache queue. Then, the audio data packets in the cache queue are sorted by index, and the first non-continuous index boundary point is found along the index increment direction. Then, the continuous index audio data packets before the boundary point are played, while the audio data packets after the boundary point continue to be cached and wait. The real-time audio streaming playback method provided in this invention can dynamically identify consecutive audio data segments in the buffer queue and immediately play them without waiting for all out-of-order packets to arrive. This fundamentally avoids the problems of permanent waiting and real-time performance degradation caused by packet loss. At the same time, the received out-of-order packets are retained and participate in the sorting process. Compared with the scheme of directly discarding out-of-order packets, it can maximize the use of the arrived audio data packets, thereby improving playback continuity and effectively reducing audio stuttering. Furthermore, this invention only needs to perform a sequential lookup in the buffer queue to determine the playable consecutive audio data segments. The processing logic is simple, the computational overhead is small, and it can adapt to various unstable scenarios such as network out-of-order packets and packet loss, and has good real-time performance and adaptability.

[0099] Based on the above embodiments, this embodiment introduces the data structure of the index carried by the audio data packet and the index comparison logic.

[0100] In one possible implementation, the index carried by the audio data packet can be a multi-level ordered index array. Specifically, a multi-level ordered index array refers to a composite identifier composed of multiple fields with a fixed priority order, where each field can represent the temporal or logical position of the audio stream in different dimensions. Unlike the single integer index assumed in the above embodiments, a multi-level ordered index array can more accurately describe the positional relationships of data packets in complex business scenarios.

[0101] For example, in a multi-party conferencing scenario, the index can be represented as a binary tuple [sessionID, seq], where the first-level sessionID identifies the session number and the second-level seq identifies the frame sequence number within that session. As another example, in a multi-channel simultaneous interpretation scenario, the index can be represented as a triple [channel, frame, subframe], identifying the channel type, main frame number, and subframe number, respectively. It should be understood that the above examples are merely illustrative and not restrictive; the length and specific semantics of the multi-level ordered index array can be flexibly defined by the business protocol, and this application does not impose any limitations on this.

[0102] For a multi-level ordered index array, the process of comparing the index of the current audio data packet with the maximum played index involves comparing the index of the current audio data packet with the maximum played index level by level. This step-by-step comparison mechanism follows a strict priority order, starting with the highest priority. In specific execution, if the index value of the current audio data packet at the current level is less than the corresponding level index value of the maximum played index, then the current audio data packet index is determined to be less than the maximum played index, and the comparison at subsequent levels stops. If the index value of the current audio data packet at the current level is greater than the corresponding level index value of the maximum played index, then the current audio data packet index is determined to be greater than the maximum played index, and the comparison at subsequent levels stops. If the index value of the current audio data packet at the current level is equal to the corresponding level index value of the maximum played index, then the comparison at the next level continues. If the index value of the current audio data packet and the maximum played index are equal at all levels, then the current audio data packet index is determined to be equal to the maximum played index.

[0103] To illustrate the above step-by-step comparison logic more clearly, a specific example is provided below. Assume the maximum played index is [2, 3], and the current audio data packet's index is [2, 5]. First, the first level is compared. Both have an index value of 2 at the first level, so they are considered equal. The comparison then proceeds to the second level. At the second level, the current audio data packet's index value is 5, which is greater than the maximum played index's index value of 3 at that level. Therefore, the current audio data packet's index is determined to be greater than the maximum played index, and the comparison process terminates.

[0104] The comparison strategy described above makes the time complexity of the comparison operation depend only on the depth of the actual difference, rather than the total length of the index array, thereby maximizing computational efficiency while ensuring accuracy.

[0105] The synergistic effect of the aforementioned multi-level ordered index array and the hierarchical comparison and judgment logic enables this invention to automatically adapt to index structures with varying lengths and semantics under different business protocols without modifying the judgment code for each specific format. This design encapsulates the complexity of the index structure at the data definition level, maintaining a high degree of consistency in the processing logic, thereby significantly improving the scalability and cross-scenario reusability of the real-time audio streaming playback solution.

[0106] Based on the above embodiments, this embodiment introduces the timeout triggering mechanism of the cache queue to solve the head blocking problem that may occur in out-of-order scenarios.

[0107] In one possible implementation, the difference between the current time and the earliest arrival time among the arrival times of each audio data packet in the buffer queue can be calculated as the buffer waiting time. It is then determined whether the buffer waiting time is greater than a preset time threshold. If so, a partial continuous playback strategy is executed.

[0108] It should be noted that the buffer waiting time in this embodiment does not use a fixed time window, but is calculated based on the earliest arrival time among the arrival times of each audio data packet in the buffer queue. This earliest arrival time represents the enqueue time of the audio data packet with the longest retention time in the current buffer queue. Using this as the timing starting point can truly reflect the severity of the head-of-line congestion.

[0109] The preset duration threshold defines the upper limit of the allowed buffering time, and its setting is based on factors such as the sensitivity of audio applications to real-time performance and the average jitter level of the network environment. Optionally, the preset duration threshold can be set to 300ms. This value is not arbitrarily set, but is derived from the psychoacoustic characteristics of human hearing's perception of audio interruptions and the experiential balance of real-time interactive scenarios. Studies have shown that 300ms is the critical lower limit for human hearing's perception of audio stream interruptions or stutters. Short pauses below this threshold are usually smoothly processed by the auditory system and are not easily detected, while exceeding this threshold will produce a noticeable sense of discontinuity. Setting the timeout threshold to 300ms can reserve a sufficient buffer window for network jitter to reassemble out-of-order packets and ensure playback continuity, while also ensuring that, in the worst case, the additional delay perceived by the user will not exceed the auditory tolerance limit, thus achieving a balance between continuity and real-time performance. It should be understood that 300ms is only a preferred implementation. In practical applications, this threshold can also be adjusted according to specific business scenarios (such as live music streaming, voice calls, simultaneous interpretation, etc.). For example, in game voice scenarios with extremely high real-time requirements, it can be appropriately lowered to 200ms, while in music appreciation scenarios with higher continuity requirements, it can be appropriately increased to 500ms.

[0110] When the calculated buffer wait time exceeds the preset time threshold, it indicates that the earliest audio data packet in the buffer queue has been waiting for too long. Continuing to wait for subsequent audio data packets will introduce unacceptable delays, so a playback operation is triggered immediately. At this time, instead of simply playing all data packets in the buffer queue, a fine-grained process is performed. That is, the audio data packets in the buffer queue are sorted according to their index and scanned in the ascending direction of the index to find the first discontinuous index boundary. The continuous audio data packets with indices before the boundary point are extracted and played, while the audio data packets after the boundary point continue to be kept in the buffer to wait for the arrival of subsequent audio data packets or the next timeout trigger, thereby maximizing the continuity of audio playback while ensuring real-time performance.

[0111] The timeout triggering mechanism based on the earliest arrival time effectively solves the head blocking problem caused by the fixed time window mechanism's inability to accurately reflect the waiting time of the head of the queue in out-of-order arrival scenarios. It can ensure that the maximum waiting time of each audio data packet in the buffer is strictly limited to within the preset time threshold.

[0112] The timeout determination based on the earliest packet timestamp introduced in this embodiment is equivalent to installing a "safety valve" for the cache queue, ensuring that the playback progress can advance at a controllable pace under any abnormal network conditions, and preventing the infinite spread of delay.

[0113] Based on the above embodiments, this embodiment provides an intelligent deduplication and flag control strategy for the special scenario where the communication connection between the client and the server is disconnected and then successfully reconnected.

[0114] As mentioned in the above embodiments, the client maintains a maximum played index, which is updated after each audio data packet is played. This maximum played index serves not only as a benchmark for determining the age of data packets during playback but also as a core basis for identifying redundant historical data in reconnection scenarios. Because the maximum played index monotonically increases in real time with playback progress, it accurately records the position of the audio stream currently consumed by the client, thus providing a reliable state anchor for subsequent deduplication operations.

[0115] When the communication connection between the client and the server is lost and then successfully reconnected, the client receives historical cached audio data packets pushed by the server. These historical cached audio data packets are the audio data packets cached by the server during the connection loss period. Specifically, in real-time audio transmission systems, to prevent the client from losing critical audio data due to brief network fluctuations, the server typically maintains a sliding window cache of a fixed duration (e.g., 1 or 2 seconds). When the communication connection with the client is lost and then successfully reconnected, the server actively pushes the historical audio data packets in this cache window to the client. While this mechanism ensures data integrity, it also introduces the risk of duplicate playback, as the client may have already played part of this data before the connection was lost. Therefore, the audio data packets received after reconnection must be carefully selected.

[0116] After a successful reconnection, the client compares the index of the received historical cached audio data packet with the maintained maximum played index to determine if the index of the historical cached audio data packet is less than or equal to the maximum played index. If so, it means that the historical cached audio data packet is a duplicate packet. To avoid repeated playback of audio segments and interference with the user's listening experience, the historical cached audio data packet is discarded directly.

[0117] Clients and servers typically establish long-lived connections based on protocols such as WebSocket. In scenarios involving network fluctuations or switching, this connection may be briefly interrupted and then restored. To identify this state change, a reconnection flag can be introduced. Its initial state is unset by default. When the connection state is detected to have recovered from disconnected to connected, the setting logic is triggered to update the reconnection flag to set. The reconnection flag is specifically used to mark the special window period during which a reconnection recovery is currently taking place.

[0118] It should be noted that if the currently received historical cached audio data packet is the first audio data packet after the reconnection flag is set, out-of-order detection can be skipped (because the index of the first audio data packet after reconnection may not be consecutive), and only deduplication detection is performed. That is, if the currently received historical cached audio data packet is the first audio data packet after the reconnection flag is set, it is determined whether its index is less than or equal to the maximum index that has been played. If so, the audio data packet is directly discarded; otherwise, the audio data packet can be played. By accurately anchoring the reconnection time through the reconnection flag, it is possible to effectively distinguish between regular real-time audio data packets and historical retransmitted audio data packets after reconnection, thereby preventing duplicate playback and avoiding the accidental triggering of out-of-order caching logic due to the first frame index jumping.

[0119] In addition, for audio data packets that are not the first after reconnection, the normal real-time audio stream processing logic can be returned (i.e., the processing logic of steps S102 to S103 above).

[0120] Based on the above embodiments, this embodiment addresses the potential latency accumulation problem during real-time audio stream playback by providing a dynamic speed-up latency elimination scheme based on the internal clock of the audio playback context object, such as... Figure 2 As shown, the solution may include:

[0121] Step S201: Obtain the current playback timestamp of the audio playback context object.

[0122] It should be noted that the audio playback context object (such as AudioContext in the Web Audio API), as the core instance of the underlying audio processing interface, maintains a high-precision monotonic clock independent of the JavaScript main thread. This internal clock is directly driven by the underlying audio engine and can truly reflect the actual processing progress of the sound card or audio output device. Therefore, using it as the benchmark anchor point for latency measurement can fundamentally eliminate measurement errors caused by software layer scheduling uncertainties, ensuring that the data on which subsequent speed adjustment decisions are based is accurate and reliable.

[0123] The current playback timestamp of the audio playback context object is the real-time time value output by the internal clock of the audio playback context object. It is used to indicate the current playback progress of the audio hardware, that is, the precise position of the sound signal being played by the audio hardware on the global timeline. By calling the read-only property of the audio playback context object, this timestamp can be obtained at each scheduling time of the audio data packet or the triggering time of the monitoring timer, so as to capture the dynamic changes in the playback progress in real time.

[0124] Step S202: Obtain the estimated playback end time corresponding to the last audio data segment scheduled to the timeline of the audio playback context object.

[0125] The expected playback end time is the sum of the start playback time of the last audio data segment scheduled to the audio playback context object on the timeline and the duration of the last audio data segment.

[0126] In the Web Audio API-based architecture, after the audio data packets are decoded and converted into PCM (Pulse Code Modulation) data and filled into the audio buffer, an audio buffer source node is created and attached to the timeline of the audio playback context object, while specifying the specific start playback time.

[0127] By recording the start time and duration of the latest scheduled audio data packet, the estimated playback end time can be accurately calculated. Specifically, the estimated playback end time is obtained by adding the start time of the latest scheduled audio data segment to the duration of the audio data segment itself. This estimated playback end time represents the endpoint boundary of the audio stream that has been included in the playback plan but has not yet been output.

[0128] Step S203: Determine the current playback duration based on the expected playback end time, the current playback timestamp, and the total duration of audio data in the audio buffer that has not yet been scheduled to the audio playback context object.

[0129] Specifically, the current playback duration is a comprehensive latency quantification indicator, and its calculation formula is as follows:

[0130] D=scheduledEndTime - AudioContext.currentTime+bufferDuration (1).

[0131] Among them, scheduledEndTime represents the expected playback end time, AudioContext.currentTime represents the current playback timestamp of the audio playback context object, bufferDuration represents the total duration of all audio data in the audio buffer that has not yet been scheduled to the audio playback context object, and D represents the current playback duration.

[0132] The difference between scheduledEndTime and AudioContext.currentTime reflects the backlog at the hardware layer, while bufferDuration reflects the backlog at the software layer. By adding the two together, the total time span required from the current moment to the completion of playback of all data in the audio buffer can be precisely quantified. This allows for a comprehensive and accurate reflection of the lag in playback progress relative to real-time progress, thus providing an accurate basis for subsequent speed adjustment.

[0133] Step S204: Determine the playback speed based on the current playback duration, and play subsequent audio data at the determined playback speed.

[0134] This invention gradually digests accumulated delays by fine-tuning the playback speed, allowing the playback progress to smoothly return to the real-time baseline, rather than forcibly aligning by methods that disrupt continuity, such as dropping frames or skipping.

[0135] In one possible implementation, the process of determining the playback speed based on the current playback duration may include: if the current playback duration is less than or equal to a preset low latency threshold, then the playback speed is determined to be the standard speed; if the current playback duration is greater than or equal to a preset high latency threshold, then the playback speed is determined to be the maximum allowed speed; if the current playback duration is greater than the low latency threshold and less than the high latency threshold, then the playback speed is determined based on the ratio of the current playback duration between the low latency threshold and the high latency threshold.

[0136] The above three scenarios collectively define a piecewise linear speed-multiplication mapping function, assuming a low-latency threshold of... (e.g., 0.5 seconds), the high latency threshold is (For example, 3 seconds), the standard speed is 1.0, and the maximum permissible speed is... (e.g., 1.04): When When the current latency is determined to be within a normal fluctuation range imperceptible to the human ear, playback is set to standard speed to avoid unnecessary speed adjustments that introduce additional computational overhead or potential audio quality perturbations. This indicates that the backlog of delays has reached a point requiring active intervention, and the acceleration should be increased to its maximum. To maximize catching-up efficiency, while During the transition period, play at double speed. It can be dynamically calculated using the following formula:

[0137] (2).

[0138] The above formula establishes a linear mapping between playback speed and latency. As latency increases, the acceleration increases linearly; conversely, as latency is gradually absorbed, playback speed automatically and smoothly returns to the standard value. This design avoids the sudden speed changes caused by state switching boundaries in traditional dual-threshold switching schemes, eliminating the abrupt acceleration or deceleration that users may perceive, and ensuring natural and smooth audio output.

[0139] It should be noted that the above-mentioned maximum permissible speed is... The speed should not exceed the human ear's perception threshold for pitch changes (approximately 3% to 5%), ensuring that the user cannot perceive the acceleration. In simultaneous interpretation scenarios where extremely high sound quality is required, this can be achieved... Set it to a more conservative value (e.g., 1.02). Additionally, in practical applications, , and The specific value can be flexibly configured according to the business type, network environment, or user preferences.

[0140] The delay control strategy in this embodiment works closely with the out-of-order fault-tolerance strategy (partial continuous playback) in the above embodiment. The out-of-order fault-tolerance strategy ensures the continuity of audio playback, but introduces unavoidable buffering delay. The delay control strategy in this embodiment is specifically responsible for absorbing this cost. The two work together to gradually absorb the accumulated delay over time within the human ear's perception threshold, thereby achieving effective maintenance of real-time performance while ensuring continuity and sound quality.

[0141] In mobile operating system environments, audio playback is often forcibly suspended by the system when system-level events such as incoming calls, alarm rings, or switching to the background occur. Furthermore, the system may not send standard interruption event notifications to upper-layer applications. This creates the illusion that the front-end player's user interface is still playing audio, even though there is actually no sound output. To address this hidden anomaly of system-level playback pause on mobile devices, this embodiment provides an audio playback interruption detection and adaptive buffer recovery scheme. This scheme may include:

[0142] Step a1: Record the current playback timestamp of the audio playback context object each time an audio scheduling refresh operation is performed.

[0143] The audio scheduling refresh operation (usually called flush) is a periodic action that submits the audio data in the audio buffer to the underlying audio engine for rendering. The current playback timestamp recorded in this operation directly reflects the physical time point actually processed by the sound card or audio output device. In this embodiment, it is used as a reliable basis for judging whether the audio is actually being output.

[0144] Step a2: If the recorded playback timestamp does not change during multiple consecutive audio scheduling refresh operations, and the recorded playback timestamp is greater than zero, then the audio playback is determined to be interrupted.

[0145] This embodiment introduces a time-stamp-based interruption determination logic. By comparing the current playback timestamps obtained in multiple consecutive scheduling cycles, it determines whether the value remains static. At the same time, a condition greater than zero is introduced to eliminate interference from zero values ​​when the AudioContext has not been fully initialized or is in a reset state, ensuring that the determination only takes effect on the basis of an established valid playback stream (only when the timestamp has increased normally (i.e., greater than zero) and then stops, does it meet the characteristics of being unexpectedly interrupted during playback).

[0146] The judgment logic in this step can accurately identify system-level pause events caused by incoming calls, switching applications, or screen lock in complex mobile operating environments.

[0147] An interruption callback can be triggered when audio playback is interrupted, and the upper layer decides whether to clear the backlog of data or wait for recovery.

[0148] Step a3: After determining that the audio playback is interrupted, pause the audio scheduling and refresh process and enter the buffering and waiting state.

[0149] When an audio playback interruption is detected, blindly attempting to immediately resume playback or continue writing data to the audio hardware often leads to repeated stuttering and oscillations due to the exhaustion of the audio buffer, exacerbating auditory discomfort. Therefore, after confirming a playback interruption, the audio scheduling and refresh process can be paused, i.e., the submission of new audio scheduling tasks to the AudioContext is stopped, and the system is actively switched to a buffer waiting state. In this state, output operations are paused, and the system focuses on accumulating audio data. This strategy transforms passive response into proactive data storage, ensuring that playback attempts are only attempted when sufficient data reserves are available, thereby effectively avoiding the risk of stuttering when resuming playback from a breakpoint.

[0150] Step a4: After the amount of data in the audio buffer reaches the preset buffer threshold, restart the audio scheduling refresh process to restore audio playback.

[0151] After playback is interrupted and resumed (e.g., when a user hangs up a call and returns to the page), if playback is restarted immediately, the buffer may be empty or contain very little data. This can easily trigger stuttering or even a second interruption due to network fluctuations or decoding delays, creating a frustrating playback-pause loop. To avoid this, this embodiment introduces a preset buffer threshold. The preset buffer threshold is a criterion used to measure whether the amount of data accumulated in the audio buffer is sufficient to support a smooth playback recovery. After entering the buffering waiting state, the amount of audio data in the audio buffer is continuously monitored. Once the amount of audio data in the buffer reaches or exceeds the preset buffer threshold, it indicates that enough data reserves have been accumulated to resist network fluctuations. At this point, the pause state can be lifted, and the flush process can be restarted to resume audio playback.

[0152] The above strategy sacrifices a very short recovery time in exchange for long-term stability of the playback process, fundamentally solving the problem of repeated stuttering after interruption recovery. It should be understood that the specific value of the preset buffer threshold can be flexibly configured according to the business scenario. For example, a relatively low buffer threshold can be set in interactive live streaming scenarios with extremely high real-time requirements, while a relatively high buffer threshold can be set in audiobook playback scenarios with higher continuity requirements.

[0153] By coordinating playback interruption detection, pause buffering, and threshold recovery, the real-time audio streaming method ensures that it can maintain a high level of playback stability even when facing complex and ever-changing mobile operating environments.

[0154] In scenarios such as WebSocket reconnection and long-term background page operation, a large amount of audio data may accumulate in the playback queue. If only the above-mentioned dynamic speed-up strategy is used to catch up, it may take a very long time to clear the backlog. In view of this, this embodiment provides a backlog clearing strategy. Specifically, the current playback duration is obtained. If the current playback duration exceeds a preset backlog threshold, the backlog of audio data to be played is cleared, and playback processing starts from the latest obtained audio data to be played.

[0155] Taking the WebSocket reconnection scenario as an example, during the network disconnection, the client stops consuming audio data. When the connection is restored, the audio data waiting to be played may accumulate far beyond the normal level. At this time, the current waiting time can be calculated. If the current waiting time exceeds the preset backlog threshold, the backlog of audio data to be played is cleared, and playback processing starts from the latest obtained audio data to be played.

[0156] The preset backlog threshold serves as the threshold parameter for determining whether to trigger the emergency cleanup mechanism. Its setting is based on a trade-off between the delay elimination time and the user's tolerance limit. The backlog threshold can be set by considering the human ear's sensitivity to long wait times and the real-time requirements of the business scenario. For example, for scenarios extremely sensitive to latency, such as simultaneous interpretation or real-time conversations, the backlog threshold can be set to 10 seconds. For scenarios with lower real-time requirements, such as broadcast live streaming, this threshold can be appropriately relaxed to 15 or 20 seconds, etc.

[0157] It should be noted that the dynamic speed-up strategy described in the above embodiment gradually eliminates latency by fine-tuning the playback speed. While this method ensures a smooth listening experience and lossless sound quality, its latency elimination rate is limited. For example, at 1.04x speed, only 0.04 seconds of latency can be recovered per second. To eliminate a 10-second backlog, theoretically, continuous accelerated playback for about 250 seconds (more than 4 minutes) is required. In real-time conferencing or live streaming scenarios, it is unacceptable for users to endure a semi-real-time state for several minutes. Therefore, when the playback duration exceeds the preset backlog threshold (e.g., 10 seconds), the dynamic speed-up strategy is deemed ineffective, and an active discard strategy is adopted. This immediately clears the unplayed historical audio data and directly jumps the playback anchor point to the latest data to be played. Although this processing method results in the loss of audio content, it can instantly align the playback progress to the real-time state, ensuring the real-time nature of audio playback.

[0158] To more clearly illustrate the collaborative relationship between this embodiment and the aforementioned embodiments, it is necessary to specifically distinguish the differences in deduplication logic between these embodiments. Deduplication logic targets redundant data that has already been consumed, aiming to prevent repeated playback, while this embodiment cleans up valid data that is severely outdated relative to the real-time progress.

[0159] The extreme backlog clearing fallback mechanism provided in this embodiment can avoid getting stuck in a long-term, ineffective catch-up situation by decisively clearing the backlog of audio data to be played and starting playback directly from the latest obtained audio data when the current playback duration exceeds the preset backlog threshold.

[0160] The backlog clearing strategy in this embodiment complements the dynamic speed-up strategy in the above embodiment. The dynamic speed-up strategy is responsible for normal small latency fluctuations, while the backlog clearing strategy in this embodiment is responsible for handling extreme backlog situations that exceed the dynamic speed adjustment range. The two work together to ensure that the real-time performance and stability of playback can be maintained in the face of various network anomalies.

[0161] In a mobile browser environment, when a user locks the screen or switches the browser to the background, the operating system automatically suspends the browser's audio playback thread for resource management purposes. This behavior is a system-level restriction that web applications cannot interfere with or bypass. At this time, although audio output has stopped at the system level, the player's user interface still maintains a "playing" status, causing a discrepancy between the interface state and the actual audio playback state. Furthermore, some mobile devices require users to actively interact with the player's user interface (e.g., click) after the page becomes visible again to trigger audio playback again; web applications cannot automatically initiate playback when the page becomes visible again. To address these constraints, this embodiment provides the following solution:

[0162] Step b1: Monitor the visibility status of the page that hosts audio playback functionality.

[0163] Visibility state indicates whether the page currently hosting audio playback is within the user's view. In Web standards, this state change is typically detected in real-time by listening to the `visibilitychange` event. For example, when the user locks the screen or switches the browser to the background, the page becomes invisible; when the user unlocks the screen or switches the browser to the foreground, the page becomes visible again.

[0164] To conserve power and resources, mobile operating systems often forcibly suspend or pause the browser's audio playback thread when the page is not visible. This system-level pause behavior typically doesn't trigger the standard pause callback of the Web Audio API, meaning the application layer cannot directly perceive the loss of underlying playback capabilities. Therefore, actively monitoring page visibility becomes a crucial bridge connecting system behavior and application logic.

[0165] Step b2: When the page switches to an invisible state, pause audio playback and update the player user interface state to the stopped state.

[0166] When the page switches to an invisible state, audio playback is paused, and the player's user interface state is updated to show that it has stopped. Specifically, this processing logic aims to solve the technical problem of inconsistency between the player's user interface display and the actual listening experience.

[0167] In existing technologies, when a page is moved to the background and forcibly muted by the system, the front-end player's user interface still maintains a "playing" status indicator. This embodiment, by actively calling the player's pause interface the instant the invisible state is detected, and forcibly refreshing UI components (such as play buttons, progress bars, status indicators, etc.) to a "stopped" or "paused" style, ensures strict synchronization between visual feedback and auditory experience. Simultaneously, proactive pausing avoids the waste of computational resources and potential errors caused by ineffective background audio data scheduling. It should be understood that pausing audio playback here includes both stopping the submission of new data to the audio playback context object and the proper handling of scheduled but unplayed data to prevent audio overlap or misalignment upon resumption.

[0168] Through the above proactive intervention, it is ensured that the player's user interface state is strictly consistent with the actual playback behavior at the underlying operating system level, thus eliminating the display anomalies of fake playback.

[0169] Step b3: When the page returns to a visible state, restore the player user interface state to the ready-to-play state and guide the user to manually trigger playback resumption.

[0170] When the page becomes visible again, the player's user interface is restored to a "playback ready" state, guiding the user to manually trigger playback resumption. Specifically, this is a compatibility design to address the limitations of autoplay policies in common browser environments. Most mobile browsers, to prevent malicious web pages from abusing audio to interfere with users, prohibit automatic audio playback when the page becomes visible again; it must be activated by explicit user interaction (such as clicking or touching). Therefore, this embodiment does not attempt to force automatic playback resumption, but instead updates the UI state to "playback ready" (e.g., displaying a prominent play button or a notification overlay), clearly informing the user that the audio is ready but waiting to be triggered. While this guided resumption strategy adds an extra user step, it ensures stable and compliant audio resumption across all standards-compliant browser environments, significantly improving cross-device compatibility and user experience certainty.

[0171] Furthermore, in order to provide a better experience in special environments that support background playback, this embodiment also introduces an application container whitelist and a differentiated playback control strategy based on the application container whitelist.

[0172] Specifically, the system can identify the application container hosting the page and determine whether the identified application container is within a preset application container whitelist. If the identified application container is not within the application container whitelist, steps b2 and b3 above are executed. If the identified application container is within the application container whitelist, audio playback is maintained when the page is switched to an invisible state, and audio playback is restored when the page is restored to a visible state.

[0173] An application container refers to the host environment that carries and runs the current page. Examples include mobile operating system browsers, WeChat, QQ, and WebView components embedded in self-developed apps. Application containers can be identified, but are not limited to, parsing feature identifiers in the User-Agent string or calling the JavaScript Bridge interface provided by the container to query environment information.

[0174] The default application container whitelist is a configurable list that records the identifiers of application containers that have been verified as allowed to continuously play audio in the background or when the screen is locked. It's important to note that the application container whitelist is not a fixed, hard-coded list. In actual deployment, it can be hot-updated through cloud configuration distribution, local configuration files, or dynamic rule engines to promptly adapt to new container versions or newly supported container types, thus maintaining stability while preserving the ability to continuously evolve. Based on the application container whitelist, restricted and privileged environments can be accurately distinguished, enabling the implementation of differentiated playback control strategies.

[0175] If the identified application container is within the application container whitelist, audio playback continues when the page is switched to an invisible state, and audio playback is resumed when the page becomes visible again. Specifically, for application containers within the whitelist, they are determined to have background keep-alive capabilities. Therefore, no pause operation is performed when the page is invisible; instead, audio data scheduling and output continue, enabling advanced functions such as listening to audio on the lock screen and background audio playback. For application containers within the whitelist, although they can maintain playback in the background, playback may still be abnormal in some extreme cases. When the page becomes visible again, audio playback is resumed to ensure the stability of continuous playback.

[0176] If the identified application container is not in the application container whitelist, when the page switches to an invisible state, audio playback is paused and the player user interface state is updated to the stopped state. When the page returns to a visible state, the player user interface state is restored to the ready-to-play state, and the user is guided to manually trigger playback resumption.

[0177] By using page visibility awareness and differentiated control strategies based on application container whitelists, this embodiment constructs an environment adaptation defense line at the playback protection level, which forms a multi-dimensional synergy with the out-of-order fault tolerance, latency control, and interruption recovery mechanisms described in the above embodiments.

[0178] The real-time audio stream playback method provided by this invention will be described below with reference to a specific application scenario.

[0179] Users accessed a 3-hour international conference H5 page via their mobile browser. During the conference, the network environment fluctuated frequently, accompanied by various abnormal situations such as WiFi switching, incoming call interference, and background switching.

[0180] In the initial stage, the network connection is stable, and audio data packets arrive in order. The basic playback process is executed, which involves receiving audio data packets carrying indices, comparing them with the highest index already played, and playing them directly if the indices are consecutive. At this time, the internal clock of the audio playback context object advances smoothly, the playback duration is maintained within a low latency threshold (e.g., 0.5 seconds), and the playback speed remains at the standard 1.0, providing the user with a smooth and real-time listening experience.

[0181] As the meeting proceeded, a brief network jitter occurred, causing some audio data packets to arrive out of order. An out-of-order fault-tolerance strategy (partial continuous playback) was adopted to process the audio data packets. While this process ensured the continuity of audio output, it also introduced additional playback delays due to buffering and waiting.

[0182] When the delay caused by the above-mentioned disordered playback gradually accumulates, causing the current playback duration to reach 2 seconds, a dynamic speed-up strategy is introduced. The current playback duration is calculated, assuming that the current playback duration is between the low latency threshold (e.g., 0.5 seconds) and the high latency threshold (e.g., 3 seconds). The playback speed is determined to be 1.02 using the above formula (2). Subsequently, the audio playback context object renders the subsequent audio data at a speed of 1.02. Since this speed is within the threshold of human hearing perception, the user can hardly perceive any change in pitch, but the playback progress is silently catching up with the real-time baseline. After about 100 seconds of accelerated playback, the accumulated 2-second delay is smoothly eliminated, the playback duration drops back to below 0.5 seconds, and the speed automatically recovers to 1.0.

[0183] During the meeting, the user switched from a WiFi environment to a mobile data network, causing the WebSocket connection to drop and then reconnect successfully. At this time, a reconnection flag was set to skip the out-of-order detection of the first frame (because the index of the first frame may not be continuous after reconnection), and only deduplication detection was performed. This can prevent repeated playback and avoid the out-of-order caching logic being accidentally triggered due to the skipped index of the first audio data packet.

[0184] However, in some extremely weak network conditions, reconnection may result in severe data backlog. For example, if the calculated playback duration after a successful reconnection is as high as 12 seconds, far exceeding the preset backlog threshold (e.g., 10 seconds), the dynamic speed-up strategy will fail. In such cases, an extreme backlog cleanup strategy is introduced, which immediately clears all backlogged audio data and starts playback directly from the latest available audio data. This strategy can instantly align the playback progress to the real-time state.

[0185] During playback, a user answers a phone call. Assuming the application container hosting the conference page is not on the application container whitelist, the mobile operating system switches the page hosting audio playback to the background, making it invisible. Simultaneously, the system pauses the output of the audio playback context object. Since system-level pauses do not trigger standard Web API callbacks, the front-end interface still displays "Playing". At this point, if the playback timestamp recorded during multiple consecutive audio scheduling refresh operations remains unchanged and greater than zero, it is determined that audio playback has been interrupted. After determining that audio playback has been interrupted, data scheduling is paused and a buffering waiting state is entered. Simultaneously, the player's user interface state is synchronously updated to the stopped state to eliminate the anomaly of inconsistency between the displayed interface and the actual playback state.

[0186] Once the user hangs up the phone, the page becomes visible again, and the buffer data volume reaches the preset buffer threshold, the playback process enters a pending state. At this time, the player user interface is updated to a pending state, and the user is guided to manually click the play button to trigger playback resumption, thereby returning playback control to the user while ensuring compliance.

[0187] This embodiment provides a three-layer collaborative scheme. The first layer is out-of-order fault tolerance, which proactively introduces a controllable time delay as a trade-off while ensuring audio continuity. The second layer is delay control, which monitors the accumulation of this delay in real time and digests it through a gradual strategy, maintaining a dynamic balance between continuity, real-time performance, and sound quality. The third layer is playback assurance, providing fallback cleanup, adaptive recovery, and differentiated control strategies for scenarios such as reconnection, playback interruption, extreme backlog, and environment switching. This three-layer collaborative scheme enables the real-time audio streaming method provided by this invention to maintain an optimal balance among the three mutually constraining indicators of continuity, real-time performance, and sound quality in complex conference scenarios lasting several hours, significantly improving the user's listening experience.

[0188] This invention also provides a real-time audio stream playback method apparatus, which is applied to a client, such as... Figure 3 As shown, the device may include: an audio data packet receiving module 301, an index comparison module 302, a boundary point finding module 303, and an audio data packet processing module 304.

[0189] The audio data packet receiving module 301 is used to receive the current audio data packet from the server. The current audio data packet carries an index, which represents the temporal position of the current audio data packet in the global audio stream.

[0190] The index comparison module 302 is used to compare the index of the current audio data packet with the maximum index that has been played. If the index of the current audio data packet is greater than the maximum index that has been played and is not contiguous with the maximum index that has been played, the current audio data packet is stored in the buffer queue. The maximum index that has been played is the largest index among the indices of all the audio data packets that have been played.

[0191] The boundary point search module 303 is used to sort the audio data packets in the buffer queue by index, and for the sorted buffer queue, to find the first boundary point where the index is not continuous along the increasing direction of the index.

[0192] The audio data packet processing module 304 is used to play the consecutive audio data packets with the index before the boundary point in the sorted cache queue when the boundary point is found by the boundary point search module 303, and the audio data packets after the boundary point continue to be cached and wait.

[0193] In one possible implementation, the audio data packet processing module 304 is further configured to:

[0194] If the index of the current audio data packet is less than or equal to the maximum index that has been played, discard the current audio data packet.

[0195] If the index of the current audio data packet is greater than the maximum index that has been played, and is consecutive to the maximum index that has been played, then the current audio data packet is played.

[0196] The audio data packet processing module 304 is also used to perform playback processing on all audio data packets in the sorted buffer queue if the index of the current audio data packet is greater than the maximum index that has been played, and is not continuous with the maximum index that has been played, and no boundary point is found.

[0197] In one possible implementation, the index is a multi-level ordered index array;

[0198] When comparing the index of the current audio data packet with the maximum index that has been played, the index comparison module 302 is specifically used for:

[0199] The index of the current audio data packet is compared with the maximum index that has been played, level by level.

[0200] If the index value of the current audio data packet at the current level is less than the index value of the corresponding level of the maximum played index, then it is determined that the index of the current audio data packet is less than the maximum played index, and the comparison of subsequent levels is stopped.

[0201] If the index value of the current audio data packet at the current level is greater than the index value of the corresponding level of the maximum played index, then it is determined that the index of the current audio data packet is greater than the maximum played index, and the comparison of subsequent levels is stopped.

[0202] If the index of the current audio data packet is equal to the index of the maximum played index at all levels, then the index of the current audio data packet is determined to be equal to the maximum played index.

[0203] In one possible implementation, the real-time audio stream playback method apparatus may further include: a buffer wait time calculation module.

[0204] The cache wait time calculation module is used to calculate the difference between the current time and the earliest arrival time among the arrival times of each audio data packet in the cache queue, which is used as the cache wait time.

[0205] The audio data packet processing module 304 is also used to play the first consecutively indexed audio segment in the cache queue when the cache waiting time exceeds a preset time threshold. The consecutively indexed audio segment consists of consecutively indexed audio data packets.

[0206] In one possible implementation, the client maintains the maximum index of played data, which is updated after each audio data packet is played.

[0207] The audio data packet receiving module 301 is also used to receive historical cached audio data packets pushed by the server after the communication connection with the server is disconnected and then successfully reconnected. The historical cached audio data packets are audio data packets cached by the server during the disconnection period.

[0208] The index comparison module 302 is also used to determine whether the index of the historical cached audio data packet is less than or equal to the maximum index that has been played.

[0209] The audio data packet processing module 304 is also used to discard historical cached audio data packets when the index of the historical cached audio data packets is less than or equal to the maximum index that has been played.

[0210] In one possible implementation, the real-time audio stream playback method apparatus may further include: a reconnection flag setting module.

[0211] The reconnection flag setting module is used to set the reconnection flag when the communication connection between the detection and the server is lost and then successfully reconnected.

[0212] The audio data packet processing module 304 is also used to perform playback processing on the historical cached audio data packet when the index of the historical cached audio data packet is greater than the maximum index of the played audio data packet, and the historical cached audio data packet is the first audio data packet after the reconnection flag is set.

[0213] In one possible implementation, the real-time audio stream playback method apparatus may further include: a current playback timestamp acquisition module, an estimated playback end time acquisition module, a current playback duration determination module, and a playback speed determination module.

[0214] The current playback timestamp acquisition module is used to obtain the current playback timestamp of the audio playback context object. The current playback timestamp is the real-time time value output by the internal clock of the audio playback context object, which is used to indicate the current playback progress of the audio hardware.

[0215] The estimated playback end time acquisition module is used to obtain the estimated playback end time corresponding to the last audio data segment on the timeline that has been scheduled to the audio playback context object. The estimated playback end time is the sum of the start playback time of the last audio data segment on the timeline and the duration of the last audio data segment.

[0216] The current playback duration determination module is used to determine the current playback duration based on the expected playback end time, the current playback timestamp, and the total duration of all audio data in the audio buffer that has not yet been scheduled to the audio playback context object.

[0217] The playback speed determination module is used to determine the playback speed based on the current playback duration, so that subsequent audio data can be played at the determined playback speed.

[0218] In one possible implementation, the playback speed determination module, when determining the playback speed based on the current playback duration, is specifically used for:

[0219] If the current playback duration is less than or equal to the preset low latency threshold, the playback speed is set to the standard speed.

[0220] If the current playback duration is greater than or equal to the preset high latency threshold, the playback speed is determined to be the maximum allowed speed.

[0221] If the current playback duration is greater than or equal to the low latency threshold and less than the high latency threshold, the playback speed is determined based on the ratio of the current playback duration between the low latency threshold and the high latency threshold.

[0222] In one possible implementation, the real-time audio stream playback method apparatus may further include: a playback timestamp recording module, an audio playback interruption detection module, and an audio playback recovery module.

[0223] The playback timestamp recording module is used to record the current playback timestamp of the audio playback context object each time an audio scheduling refresh operation is performed. The current playback timestamp is the real-time time value output by the internal clock of the audio playback context object, which is used to indicate the current playback progress of the audio hardware.

[0224] The audio playback interruption detection module is used to determine that audio playback is interrupted if the recorded playback timestamp does not change when multiple audio scheduling refresh operations are performed consecutively, and the recorded playback timestamp is greater than zero.

[0225] The audio playback recovery module is used to pause the audio scheduling and refresh process and enter a buffer waiting state after determining that audio playback has been interrupted. Once the amount of audio data to be played in the audio buffer reaches a preset buffer threshold, the audio scheduling and refresh process is restarted to restore audio playback.

[0226] In one possible implementation, the real-time audio streaming method apparatus may further include a backlog clearing module.

[0227] The backlog clearing module is used to obtain the current playback duration. If the current playback duration exceeds the preset backlog threshold, the backlog of audio data to be played is cleared so that playback processing can start from the latest audio data to be played.

[0228] In one possible implementation, the real-time audio stream playback method apparatus may further include: a page visibility status monitoring module and a playback control module.

[0229] The page visibility status monitoring module is used to monitor the visibility status of the page that hosts audio playback functionality;

[0230] The playback control module is used to pause audio playback when the page is switched to an invisible state and to update the player user interface state to the stopped state. When the page is restored to a visible state, the player user interface state is restored to the ready-to-play state and the user is guided to manually trigger playback resumption.

[0231] In one possible implementation, the real-time audio stream playback method apparatus may further include: an application container identification module and a whitelist container discrimination module.

[0232] The application container identification module is used to identify the application container that hosts the page.

[0233] The whitelist container identification module is used to determine whether the identified application container is within the preset application container whitelist.

[0234] The playback control module, when the application container identified by the application container identification module is within the application container whitelist, maintains audio playback when the page is switched to an invisible state, and performs an audio playback resumption operation when the page returns to a visible state. When the application container identified by the application container identification module is not within the application container whitelist, when the page is switched to an invisible state, pauses audio playback and synchronously updates the player user interface state to the stopped state. When the page returns to a visible state, restores the player user interface state to the ready-to-play state and guides the user to manually trigger playback resumption.

[0235] The real-time audio streaming device provided by this invention can achieve a balance between audio playback continuity and real-time performance, significantly improving the user's listening experience.

[0236] This invention also provides an electronic device, which may include at least one processor and a memory connected to the processor.

[0237] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention; the memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage.

[0238] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the real-time audio stream playback method provided in the above embodiments.

[0239] This invention also provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is able to implement the real-time audio stream playback method provided in the above embodiments.

[0240] This invention also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the real-time audio stream playback method provided in the above embodiments.

[0241] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided by this invention, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0242] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0243] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0244] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for real-time audio stream playback, characterized in that, Applied to a client, the method includes: Receive the current audio data packet from the server, wherein the current audio data packet carries an index, the index representing the temporal position of the current audio data packet in the global audio stream; The index of the current audio data packet is compared with the maximum index of the played data packet. If the index of the current audio data packet is greater than the maximum index of the played data packet and is not contiguous with the maximum index of the played data packet, the current audio data packet is stored in the cache queue. The maximum index of the played data packet is the maximum index among the indices of the played audio data packets. The audio data packets in the cache queue are sorted by index. For the sorted cache queue, the first point where the index is not continuous is found along the increasing direction of the index. If the dividing point is found, the consecutive audio data packets with the index before the dividing point in the sorted cache queue are played, while the audio data packets after the dividing point continue to be cached and wait.

2. The real-time audio stream playback method according to claim 1, characterized in that, Also includes: If the index of the current audio data packet is less than or equal to the maximum index that has been played, then the current audio data packet is discarded; If the index of the current audio data packet is greater than the maximum index that has been played, and is consecutive to the maximum index that has been played, then the current audio data packet is played. If the index of the current audio data packet is greater than the maximum index already played, and is not contiguous with the maximum index already played, the real-time audio stream playback method further includes: If the boundary point is not found, then all audio data packets in the sorted cache queue will be played.

3. The real-time audio stream playback method according to claim 1, characterized in that, The index is a multi-level ordered index array; The step of comparing the index of the current audio data packet with the maximum index that has been played includes: The index of the current audio data packet is compared with the maximum index that has been played, level by level. If the index value of the current audio data packet at the current level is less than the index value of the corresponding level of the maximum played index, then it is determined that the index of the current audio data packet is less than the maximum played index, and the comparison of subsequent levels is stopped. If the index value of the current audio data packet at the current level is greater than the corresponding level index value of the maximum played index, then it is determined that the index of the current audio data packet is greater than the maximum played index, and the comparison of subsequent levels is stopped. If the index of the current audio data packet is equal to the index value of the maximum played index at all levels, then the index of the current audio data packet is determined to be equal to the maximum played index.

4. The real-time audio stream playback method according to claim 1, characterized in that, Also includes: Calculate the difference between the current time and the earliest arrival time among the arrival times of each audio data packet in the cache queue, and use this as the cache waiting time; If the cache wait time exceeds a preset time threshold, the first consecutively indexed audio segment in the cache queue will be played. The consecutively indexed audio segment consists of consecutively indexed audio data packets.

5. The real-time audio stream playback method according to claim 1, characterized in that, The client maintains the maximum played index, which is updated after each audio data packet is played. The real-time audio stream playback method further includes: When the communication connection with the server is disconnected and then successfully reconnected, the server receives historical cached audio data packets pushed by the server. The historical cached audio data packets are audio data packets cached by the server during the disconnection period. Determine whether the index of the historical cached audio data packet is less than or equal to the maximum index of the played data; If the index of the historical cached audio data packet is less than or equal to the maximum index of the played data, then Discard the historical cached audio data packets.

6. The real-time audio stream playback method according to claim 5, characterized in that, The method further includes: When a communication connection with the server is detected to have been lost and then successfully reconnected, a reconnection flag is set. If the historical cached audio data packet is the first audio data packet after the reconnection flag is set, and the index of the historical cached audio data packet is greater than the maximum index of the played data packet, then the historical cached audio data packet is played.

7. The real-time audio stream playback method according to claim 1, characterized in that, Also includes: Obtain the current playback timestamp of the audio playback context object. The current playback timestamp is the real-time time value output by the internal clock of the audio playback context object, which is used to indicate the current playback progress of the audio hardware. Obtain the estimated playback end time corresponding to the last audio data segment scheduled to the timeline of the audio playback context object. The estimated playback end time is the sum of the start playback time of the last audio data segment on the timeline and the duration of the last audio data segment. The current playback duration is determined based on the estimated playback end time, the current playback timestamp, and the total duration of all audio data in the audio buffer that has not yet been scheduled to the audio playback context object. The playback speed is determined based on the current playback duration, and subsequent audio data is played at the determined playback speed.

8. The real-time audio stream playback method according to claim 7, characterized in that, The step of determining the playback speed based on the current playback duration includes: If the current playback duration is less than or equal to the preset low latency threshold, then the playback speed is determined to be the standard speed. If the current playback duration is greater than or equal to the preset high latency threshold, then the playback speed is determined to be the maximum allowed speed. If the current playback duration is greater than the low latency threshold and less than the high latency threshold, the playback speed is determined based on the ratio of the current playback duration between the low latency threshold and the high latency threshold.

9. The real-time audio stream playback method according to claim 1, characterized in that, Also includes: Each time an audio scheduling refresh operation is performed, the current playback timestamp of the audio playback context object is recorded; If the recorded playback timestamp does not change during multiple consecutive audio scheduling refresh operations, and the recorded playback timestamp is greater than zero, then the audio playback is determined to be interrupted. After determining that audio playback is interrupted, the audio scheduling and refresh process is paused and a buffering waiting state is entered. Once the amount of audio data to be played in the audio buffer reaches a preset buffer threshold, the audio scheduling refresh process is restarted to resume audio playback.

10. The real-time audio stream playback method according to claim 1, characterized in that, Also includes: Calculate the current playback duration; If the current playback duration exceeds the preset backlog threshold, the backlog of audio data to be played is cleared, and playback processing begins from the latest audio data to be played.

11. The real-time audio stream playback method according to claim 1, characterized in that, Also includes: Monitor the visibility status of the page that hosts audio playback functionality; When the page is switched to an invisible state, audio playback is paused, and the state of the player user interface is synchronously updated to the stopped state. When the page becomes visible again, the player user interface is restored to the ready-to-play state, and the user is guided to manually trigger playback resumption.

12. The real-time audio stream playback method according to claim 11, characterized in that, Also includes: Identify the application container that carries the page, and determine whether the identified application container is in the preset application container whitelist; If the identified application container is within the application container whitelist, when the page is switched to an invisible state, audio playback is maintained, and when the page is restored to a visible state, an audio playback restoration operation is performed. When the page is switched to an invisible state, audio playback is paused, and the player's user interface state is synchronously updated to the stopped state. When the page becomes visible again, the player's user interface state is restored to the ready-to-play state, and the user is guided to manually trigger playback resumption, including: If the identified application container is not in the application container whitelist, when the page switches to an invisible state, audio playback is paused, and the player user interface state is synchronously updated to the stopped state. When the page returns to a visible state, the player user interface state is restored to the ready-to-play state, and the user is guided to manually trigger playback resumption.

13. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the real-time audio streaming playback method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing an electronic device to perform the real-time audio streaming playback method as described in any one of claims 1 to 12.

Images