An audio transmission control method

Abstract

This specification provides an audio transmission control method in some embodiments. The method includes: determining candidate relay nodes and their forwarding quotas based on resource status information of transmission nodes; determining a target relay node and a target transmission path based on transmission demand information, candidate relay nodes, forwarding quotas, and network topology; determining a target delay parameter of the transmission node based on the target transmission path; and transmitting target audio based on the target transmission path and the target delay parameter.

Description

Technical Field

[0001] This specification relates to the field of audio transmission technology, and in particular to an audio transmission control method. Background Technology

[0002] AoIP (Audio over IP) systems have been widely used in professional sound reinforcement, distributed audio, smart speakers, and networked audio devices. As applications continue to expand, the number of nodes, transmission channels, and network topology complexity in these systems are constantly increasing, placing higher demands on the stability, latency consistency, and system resource utilization efficiency of audio transmission.

[0003] In existing AoIP applications, a specific sending node typically handles the transmission of multi-channel audio streams. However, some sending devices are limited by the processing power of their Central Processing Unit (CPU), bandwidth resources, or channel capacity, resulting in an upper limit to the number of audio channels they can stably transmit. Meanwhile, existing AoIP systems primarily rely on manually configured fixed unicast or multicast channels, lacking dynamic adjustment capabilities for audio transmission paths. When some transmission nodes experience insufficient channel capacity due to resource constraints, the system cannot fully utilize other devices with AoIP transmission capabilities for offloading or relay transmission, leading to low resource utilization. Furthermore, existing systems lack automatic switching and replanning capabilities in the event of node failure, disconnection, or network fluctuations, easily causing audio interruptions and quality degradation, and making it difficult to maintain latency consistency in multi-node transmission scenarios.

[0004] Therefore, there is an urgent need for an audio transmission control method that can make fuller use of the transmission capabilities of each node, optimize the audio stream allocation method, and improve fault recovery capabilities. Summary of the Invention

[0005] This specification provides one or more embodiments of an audio transmission control method. The method includes: determining candidate relay nodes and their forwarding quotas based on resource status information of transmission nodes; determining a target relay node and a target transmission path based on transmission demand information, the candidate relay nodes, the forwarding quotas, and network topology; determining a target delay parameter of the transmission node based on the target transmission path; and transmitting target audio based on the target transmission path and the target delay parameter.

[0006] This specification provides an audio transmission control system according to one or more embodiments. The system includes: a parameter determination module and an audio transmission module; the parameter determination module is configured to: determine candidate relay nodes and their forwarding quotas based on resource status information of transmission nodes; determine a target relay node and a target transmission path based on transmission demand information, the candidate relay nodes, the forwarding quotas, and network topology; and determine a target delay parameter for the transmission node based on the target transmission path; the audio transmission module is configured to: transmit target audio based on the target transmission path and the target delay parameter.

[0007] This specification provides an audio transmission control device according to one or more embodiments. The device includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least a portion of the computer instructions to implement the audio transmission control method as described above.

[0008] One or more embodiments of this specification also provide a computer-readable storage medium that stores computer instructions that, when executed by a processor, implement the audio transmission control method as described above. Attached Figure Description

[0009] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0010] Figure 1 This is a schematic diagram illustrating an application scenario of the audio transmission control method according to some embodiments of this specification; Figure 2 This is an exemplary block diagram of an audio transmission control system according to some embodiments of this specification; Figure 3 This is an exemplary flowchart of an audio transmission control method according to some embodiments of this specification; Figure 4 This is an exemplary flowchart illustrating the update target delivery path according to some embodiments of this specification; and Figure 5 This is an exemplary schematic diagram illustrating a transition splicing according to some embodiments of this specification. Detailed Implementation

[0011] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0012] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0013] Unless the context clearly indicates an exception, words such as "a," "an," "a kind," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0014] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0015] Figure 1 This is a schematic diagram illustrating application scenarios of the audio transmission control system according to some embodiments of this specification. The audio transmission control system is applicable to various distributed AoIP scenarios, including public address systems in airports, shopping malls, and train stations; multi-room audio systems in homes; conference and educational sound reinforcement; stage performances and large-scale events; and industrial emergency communications. It can achieve highly reliable, low-latency, and automatically optimized audio transmission even under conditions of numerous nodes, varied topologies, and complex environments, significantly improving the robustness and flexibility of multi-node sound reinforcement networks.

[0016] like Figure 1 As shown, the application scenario 100 of the audio transmission control method (hereinafter referred to as application scenario 100) may include an audio input source 110, a server 120, a network 130, a storage device 140, a playback device 150, and a user 160.

[0017] Audio input source 110 refers to the starting point of the audio stream, a device responsible for acquiring, generating, or injecting raw audio data. In some embodiments, audio input source 110 may include a microphone, an audio interface box (AoIP box), a digital audio interface, a media player, etc. In some embodiments, audio input source 110 acts as the source node in a relay transmission path, responsible for converting the acquired analog or digital audio signals into data packets conforming to the AoIP protocol and injecting them into network 130.

[0018] Server 120 refers to the computing device that performs the function of a central controller in application scenario 100. In some embodiments, server 120 can be configured to: acquire resource status information of each transmission node in the system, determine candidate relay nodes and their forwarding quotas; determine the target transmission path and target delay parameters based on transmission requirements; and monitor stream quality and perform path replanning and transition splicing when nodes are abnormal. In some embodiments, server 120 can integrate multiple modules, and different modules can be configured to perform different steps. For example, server 120 can integrate a parameter determination module, an audio transmission module, etc.

[0019] In some embodiments, server 120 may be a standalone physical server, cloud server, cluster server, or a control module integrated into audio input source 110 or playback device 150.

[0020] Network 130 may include any suitable network capable of facilitating the exchange of audio data, control commands, and / or node status information. In some embodiments, at least one component of application scenario 100 (e.g., server 120, storage device 140, etc.) may exchange information and / or data with at least one other component in application scenario 100 via network 130.

[0021] In some embodiments, network 130 is an Internet Protocol (IP) network supporting Audio over Internet Protocol (AoIP) transmission, such as any one or more of wired networks (e.g., a local area network connected by a switch) or wireless networks. For example, network 130 may include cable networks, cable connections, or any combination thereof. Network connections between components may employ one or more of the methods described above. In some embodiments, the network may be a point-to-point, shared, centralized, or other topologies, or a combination of multiple topologies. In some embodiments, network 130 may include one or more network access points.

[0022] In some embodiments, network 130 can provide communication guarantees for multi-hop transmission of audio streams between source nodes, relay nodes and ordinary nodes.

[0023] Storage device 140 refers to a device used to store data, instructions, and / or information. In some embodiments, storage device 140 stores computer instructions that, when executed by a processor, implement the audio transmission control method as described in the embodiments of this specification. In some embodiments, storage device 140 may also store acoustic feature data such as the spectral characteristics, fundamental frequency characteristics, and energy envelope of historical audio streams, as well as resource status mapping tables and preset transition mode matching tables for each node, for real-time access by server 120 or nodes.

[0024] In some embodiments, storage device 140 may include a mass storage device, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), or any combination thereof. Storage device 140 may be implemented in the cloud or on a local server.

[0025] Playback device 150 refers to the audio output terminal in application scenario 100, which is responsible for receiving, processing, forwarding, and / or playing audio data. In some embodiments, playback device 150 may include speaker 150-1, headphones 150-2, audio 150-3, etc.

[0026] In some embodiments, depending on its role in the transmission path, the playback device 150 may be a relay node undertaking audio forwarding tasks (such as speaker B forwarding audio to speaker C), or a regular node at the end of the path (such as a terminal playback speaker). In some embodiments, some playback devices 150 have a hardware clock module that can perform audio data buffer alignment and synchronized playback according to the target delay parameters sent by the server 120.

[0027] User 160 refers to the operator or audio receiver in application scenario 100. In some embodiments, user 160 can send transmission request information to server 120 through a user terminal (such as a mobile phone or computer), such as specifying the start and end points of the audio, bandwidth requirements, etc.

[0028] In the exemplary application scenario 100, when the audio input source 110 generates an audio transmission request, the server 120 can plan one or more target transmission paths from the source node, the target relay node to the playback node, and calculate a stepped target delay parameter for each node on the path. The target audio is then transmitted through the network 130 along the target transmission path to the playback device 150 for playback, so that the user 160 can listen to it.

[0029] For more information on the above, please refer to [link / reference]. Figures 2 to 5 And its related descriptions.

[0030] It should be noted that application scenario 100 is provided for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can make various modifications or variations based on the description in this specification. For example, application scenario 100 can implement similar or different functions on other devices. However, these changes and modifications will not depart from the scope of this specification.

[0031] Figure 2 This is an exemplary block diagram of an audio transmission control system according to some embodiments of this specification.

[0032] In some embodiments, such as Figure 2 As shown, the audio transmission control system 200 (hereinafter referred to as system 200) includes: a parameter determination module 210 and an audio transmission module 220. In some embodiments, the parameter determination module 210 is configured to: determine candidate relay nodes and their forwarding quotas based on the resource status information of the transmission nodes; determine the target relay node and the target transmission path based on transmission demand information, candidate relay nodes, forwarding quotas, and network topology; and determine the target delay parameter of the transmission node based on the target transmission path. In some embodiments, the audio transmission module 220 is configured to: transmit the target audio based on the target transmission path and the target delay parameter.

[0033] The parameter determination module 210 is a module used to determine audio transmission-related parameters. In some embodiments, the parameter determination module 210 is configured to: determine candidate relay nodes and their forwarding quotas based on the resource status information of the transmission nodes; determine target relay nodes and target transmission paths based on transmission demand information, candidate relay nodes, forwarding quotas, and network topology; and determine target delay parameters of the transmission nodes based on the target transmission paths.

[0034] In some embodiments, the parameter determination module 210 is further configured to: determine the basic delay parameters of the transmission node based on the target transmission path; determine the dynamic compensation amount of the transmission node based on the transmission status information; and determine the target delay parameters of the transmission node based on the basic delay parameters and the dynamic compensation amount of the transmission node.

[0035] In some embodiments, the parameter determination module 210 is further configured to: generate a transition waveform based on the spectral characteristics, fundamental frequency characteristics, and energy envelope of the historical audio stream; determine alternative relay nodes and alternative transmission paths based on transmission demand information, candidate relay nodes, and forwarding quotas, and use the alternative transmission paths as the updated target transmission paths.

[0036] In some embodiments, the parameter determination module 210 is further configured to: determine the fault type and expected switching duration of the target relay node based on the quality detection information of the target transmission path and / or the node response information of the transmission node; and determine the transition mode based on the fault type, the expected switching duration, and the buffer level of the normal node.

[0037] The audio transmission module 220 refers to a module responsible for transmitting audio streams. In some embodiments, the audio transmission module 220 is configured to transmit target audio based on the target transmission path and target delay parameters.

[0038] In some embodiments, the audio transmission module 220 is further configured to: when switching the current transmission path to the updated target transmission path, perform a transitional splicing of the historical audio stream and the updated audio stream based on a transition waveform; and transmit the updated audio stream based on the updated target transmission path. In some embodiments, the audio transmission module 220 is further configured to: perform a transitional splicing of the historical audio stream and the updated audio stream based on a transition mode and a transition waveform.

[0039] For further explanation of the above content, please refer to [link / reference]. Figures 3-5 And its corresponding descriptions.

[0040] In some embodiments, the parameter determination module 210 and the audio transmission module 220 may be partially or entirely configured in the aforementioned server 120. For more information about the server 120, please refer to [link to relevant documentation]. Figure 1 And its related descriptions.

[0041] It should be noted that the above description of the audio transmission control system and its modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. In some embodiments, Figure 2 The parameter determination module 210 and audio transmission module 220 disclosed herein can be different modules within a system, or a single module can implement the functions of two or more of the aforementioned modules. For example, the modules can share a single storage module, or each module can have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0042] Figure 3 This is an exemplary flowchart illustrating an audio transmission control method according to some embodiments of this specification. Figure 3 As shown, process 300 includes steps 310 to 340. In some embodiments, process 300 may be executed by server 120 or audio transmission control system 200.

[0043] Step 310: Based on the resource status information of the transmission nodes, determine the candidate relay nodes and their forwarding quotas.

[0044] A transmission node is an independent processing unit that participates in audio stream transmission in a multi-device audio sharing system (i.e., an AoIP system). In some embodiments, a transmission node has the ability to receive, process, and / or forward audio data. For example, a transmission node may include a source node, a relay node, or a regular node.

[0045] A source node is a node that serves as the starting point of an audio stream. In some embodiments, the source node is responsible for acquiring, generating, or injecting raw audio data into the network for subsequent relay transmission. For example, a source node may include a microphone, an AoIP box, a digital audio interface, etc.

[0046] A relay node is an intermediate forwarding unit in the relay transmission path of an audio stream. In some embodiments, a relay node is not only responsible for receiving and outputting (e.g., for local playback) audio, but also for forwarding the audio stream to downstream nodes in the network. For example, through relay transmission by relay nodes, the transmission load concentrated at the source node can be distributed across multiple nodes. For instance, a relay node may include a speaker, loudspeaker, or other playback device with network transmission or forwarding capabilities.

[0047] In some embodiments, a relay node may include candidate relay nodes, alternative relay nodes, etc. For more information on alternative relay nodes, please refer to [link to relevant documentation]. Figure 4 And its related descriptions.

[0048] A regular node refers to the receiving end of an audio stream in a multi-device audio sharing system. In some embodiments, a regular node is responsible for receiving and outputting audio, such as playing, storing, or further processing the audio. It is understood that a regular node typically exists as the tail-end playback node in a relay transmission path. For example, a regular node may include terminal devices such as speakers, headphones, and recording devices that do not currently undertake or are not currently undertaking subsequent forwarding tasks.

[0049] Resource status information refers to information used to characterize the current processing capacity, available resource status, and operating load of a transmission node. In some embodiments, resource status information may include one or more combinations of Central Processing Unit (CPU) utilization, cache (memory) utilization, remaining network bandwidth, and network bandwidth utilization. For example, the resource status information of a speaker node may reflect that its current CPU utilization is 20% and its remaining network bandwidth is 100Mbps. In some embodiments, resource status information may be periodically reported to server 120 by each transmission node, or it may be obtained by server 120 actively querying each transmission node.

[0050] Candidate relay nodes refer to nodes selected from transmission nodes that possess relay forwarding capabilities. In some embodiments, server 120 can evaluate based on the resource status information of each transmission node to determine candidate relay nodes. For example, a transmission node can be identified as a candidate relay node when its CPU utilization is lower than a utilization threshold, its memory utilization is lower than a memory threshold, and its remaining network bandwidth is higher than a bandwidth threshold. The utilization threshold, memory threshold, and bandwidth threshold can be preset.

[0051] Forwarding quota refers to the maximum forwarding task allocated to each candidate relay node. In some embodiments, forwarding quota can be used to limit the additional forwarding load that a candidate relay node can bear while undertaking local audio playback tasks. In some embodiments, forwarding quota can be characterized by the number of audio streams allowed to be forwarded and / or network bandwidth usage. It is understood that by setting forwarding quota, it can be ensured that the local playback of a candidate relay node is not affected when playback and forwarding are performed simultaneously, that is, the node's own audio decoding and digital-to-analog conversion processes will not experience processing delays and output distortion due to resource contention. For example, server 120 can allocate a forwarding quota of "up to 5 additional audio streams can be forwarded" or "up to 10Mbps of additional bandwidth usage" to a node based on its resource status information.

[0052] In some embodiments, after determining candidate relay nodes, server 120 can further allocate forwarding quotas to each candidate relay node. Server 120 can determine forwarding quotas based on various methods. For example, server 120 can determine forwarding quotas by querying a first preset table. The first preset table records the mapping relationship between resource status information and forwarding quotas. Server 120 can match candidate relay nodes in the preset table based on their resource status information; the better the resource status information (e.g., lower processor utilization, lower memory usage, and more sufficient remaining bandwidth), the higher the forwarding quota allocated by server 120.

[0053] For example, server 120 can also quantitatively evaluate the resource status information of each transmission node based on preset scoring rules, calculate the relay capability score of each transmission node, and determine the final candidate relay nodes and their corresponding maximum forwarding quotas based on the relay capability scores. For example, a node with a higher relay capability score indicates that it has greater current performance redundancy and a stronger ability to carry additional forwarding tasks, and the forwarding quota allocated to it by server 120 will also increase accordingly.

[0054] In some embodiments, server 120 is further configured to update the forwarding quota corresponding to the candidate relay node by adjusting the model based on the resource status information of the candidate relay node, and the model is adjusted to a machine learning model.

[0055] The adjustment model refers to the model used to obtain dynamically updated forwarding quotas. In some embodiments, the adjustment model is a machine learning model, such as any one or a combination of Extreme Gradient Boosting (XGBoost), Deep Neural Networks (DNN), or other custom model structures.

[0056] In some embodiments, the input to the adjustment model may include resource status information of candidate relay nodes (such as CPU utilization, remaining bandwidth, and memory utilization); the output may include the adjusted forwarding quota.

[0057] In some embodiments, the model adjustment can be obtained through training based on at least one set of first training samples and their corresponding first labels. In some embodiments, the first training samples can be constructed based on historical data, which can be obtained from storage device 140. The first training samples may include the sample CPU utilization rate, sample remaining bandwidth, and sample memory utilization rate of the sample relay nodes. The first label corresponding to the first training sample may be the optimal forwarding quota corresponding to that sample. In some embodiments, the first label can be determined based on historical running data, obtained by manual annotation, rule preset, or simulation optimization. For example, the forwarding quota value that has been historically tested through simulation and can guarantee smooth local playback and maximize resource utilization can be marked as the label corresponding to the sample.

[0058] In some embodiments, the server 120 can input multiple first training samples with first labels into the initial adjustment model, construct a first loss function through the first labels and the output of the initial adjustment model, and iteratively update the parameters of the initial adjustment model based on the first loss function through methods such as gradient descent. When the preset conditions are met, the model training is completed, and the trained adjustment model is obtained.

[0059] In some embodiments, the preset conditions may include, but are not limited to, the convergence of the first loss function and the reaching of a threshold during the training period.

[0060] In some embodiments, the server 120 inputs the resource status information of the candidate relay nodes into the trained adjustment model, and directly outputs the adjusted forwarding quota through the adjustment model; based on the adjusted forwarding quota, the server dynamically adjusts and updates the forwarding quota corresponding to the candidate relay node in real time.

[0061] In some embodiments of this specification, the forwarding quota is dynamically adjusted based on resource status information, and the relay node is dynamically adjusted or replaced in combination with the joint constraints of node depth and target delay parameters. This can effectively suppress the accumulation of delay and buffer pressure caused by excessive relay path depth, avoid delay parameter exceeding limits, and thus significantly improve the real-time performance and stability of audio transmission.

[0062] Step 320: Based on transmission demand information, candidate relay nodes, forwarding quotas, and network topology, determine the target relay node and the target transmission path.

[0063] Transmission requirement information refers to information describing the constraints and objectives that need to be met for this audio transmission task. In some embodiments, transmission requirement information may include the start point, end point, and bandwidth requirements for audio transmission. For example, user 160 needs to send audio data collected by a microphone (start point) to multiple speakers (end points) in a specific conference room, and requires the transmission bandwidth to meet a specific high-fidelity audio quality standard; these conditions constitute the transmission requirement information for this task. In some embodiments, transmission requirement information may be obtained by server 120 based on playback instructions sent by user 160 using a terminal, or automatically generated by the system according to a preset scheduled broadcast task.

[0064] Network topology refers to the physical connection structure or logical connection relationship formed between various transmission nodes (such as audio devices, switches, servers, etc.) through communication links. For example, a simple AoIP network topology can include a source node S, relay node R1, relay node R2, and ordinary node D. The source node S is connected to both relay nodes R1 and R2, and relay nodes R1 and R2 can be connected to ordinary node D respectively, thus forming two independent logical paths: S→R1→D and S→R2→D. In some embodiments, the network topology can be obtained by the server through network discovery protocols (such as Link Layer Discovery Protocol), neighbor node information periodically reported by each transmission node, or pre-configuration operations by the network administrator.

[0065] In some embodiments, network topology can be represented using a network topology graph. A network topology graph is a structural model that represents network topology in graph form (i.e., nodes and edges). In some embodiments, a network topology graph can be represented as a combination of nodes and edges, where "nodes" can correspond to various transmission nodes in a multi-device audio sharing system (such as source nodes, candidate relay nodes, ordinary nodes, etc.), and "edges" correspond to communication links that are physically or logically reachable between devices.

[0066] The target relay node refers to the relay node selected based on the already selected candidate relay nodes that meets the corresponding transmission requirements. In some embodiments, the server 120 can select a combination of transmission nodes from the candidate relay nodes that have a forwarding quota greater than the quota threshold, good link quality, and the ability to connect the origin to the destination as the target relay node. The quota threshold is a threshold used to filter the forwarding capacity of the target relay node. In some embodiments, the quota threshold can be preset based on historical experience or actual needs.

[0067] For example, in a large conference room application scenario containing ten speakers with network transmission capabilities, server 120 can initially screen five speakers with redundant computing power and bandwidth as candidate relay nodes based on real-time resource status information; when there is a need to transmit the audio stream of a microphone source node to the farthest ordinary node, server 120 can combine network topology and forwarding quota to ultimately select only the two speakers with the best physical location and current link quality to actually perform the data reception and re-forwarding tasks.

[0068] A target transmission path refers to the transmission path topology from the source node to the target node for the target audio stream. In some embodiments, the source node is the original sender of the audio stream in the AoIP system, the starting point of the relay transmission path, responsible for injecting audio data into the network and initiating the relay transmission; the target node is the final receiver of the audio stream in the AoIP system, the end point of the relay transmission path, responsible for receiving and outputting (e.g., playing, storing, or further processing) the audio. For example, a complete target transmission path can be represented as: Source Node → Target Relay Node A → Target Relay Node B → Target Node.

[0069] In some embodiments, server 120 can determine basic transmission paths based on network topology using the K Shortest Paths (KSP) algorithm; and determine target transmission paths based on the basic transmission paths. For example, server 120 can use the source node of the target audio, the target node, and the network topology as input parameters to the KSP algorithm, and set the required number of shortest paths (e.g., K=3). The KSP algorithm will output a corresponding list of basic transmission paths (e.g., containing paths P1, P2, and P3).

[0070] In some embodiments, after obtaining the list of basic transmission paths, the server 120 can determine whether the forwarding quota of each candidate relay node on each basic transmission path has exceeded the limit in order to determine the target transmission path.

[0071] For example, server 120 can obtain the resource amount required for the target audio transmission demand (e.g., the number of new audio streams or the required network bandwidth) and compare this resource amount with the current forwarding quota of each candidate relay node on the basic transmission path. If the resource amount required by the basic transmission path is greater than the forwarding quota of any candidate relay node on its path, the basic transmission path is determined to be out of limit, and server 120 will then remove the out-of-limit basic transmission path. Conversely, if the resource amount required by the basic transmission path does not exceed the forwarding quota limit of all candidate relay nodes on its path, the basic transmission path is determined not to be out of limit. Server 120 can determine a complete route connecting the source node to the target node that is not out of limit as the target transmission path, and determine the candidate relay nodes included in the target transmission path as the target relay nodes.

[0072] For example, if the current transmission requirement is to add one audio stream, and a selected basic transmission path includes candidate relay node A and candidate relay node B, where node A has a remaining forwarding quota of 3 streams and node B has a remaining forwarding quota of 0 streams, the server will determine that the basic transmission path exceeds the limit and remove it because node B's quota cannot meet the resource requirement.

[0073] Step 330: Determine the target delay parameters of the transmission nodes based on the target transmission path.

[0074] The target delay parameter refers to the audio playback delay control parameter set for the transmission node. In some embodiments, the target delay parameter may include the target delay for the transmission node to buffer, align, and output the received audio data. For example, by setting different target delay parameters for different nodes, the waiting time of each node after receiving data can be controlled to achieve a synchronous effect of all nodes playing the same audio at the same time.

[0075] In some embodiments, server 120 may determine the node depth of each transmission node on the target transmission path based on the target transmission path, and assign a corresponding target delay parameter to each transmission node based on the node depth of each transmission node. Here, node depth refers to the number of hops traversed from the source node.

[0076] For example, in the target transmission path "source node S → target relay node A → target relay node B → target node D", the depth of source node S is 0, the depth of relay node A is 1, the depth of relay node B is 2, and the depth of target node D is 3. In some embodiments, the shallower the node depth (i.e., the closer it is to the source node), the earlier it receives the target audio, and therefore the longer the target delay parameter (such as target delay) is assigned to it. Conversely, the deeper the node depth, the shorter the target delay parameter is assigned to it.

[0077] Continuing the previous example, source node S can send data immediately after acquiring audio. Target relay node A, being shallower, receives data earlier, so its target delay parameter is set to 20ms, meaning it needs to wait a relatively long 20ms after receiving data before playing it and simultaneously forwarding it. Target relay node B receives data slightly later, so its target delay parameter is set to 10ms, meaning it only needs to wait 10ms before playing and forwarding it. Target node D, located at the end and deepest point, receives data last and has the shortest allocated waiting time; its target delay parameter can be set to 0ms, meaning it plays immediately after receiving data. Through this depth-based, tiered delay backward calculation and compensation mechanism, transmission nodes at different levels can output audio at a consistent absolute time point, thus achieving extremely low time difference synchronous playback in multi-node collaborative scenarios. The target delay parameters corresponding to transmission nodes at different depths can be obtained based on historical data statistics.

[0078] In some embodiments, the server 120 is further configured to: determine the basic delay parameters of the transmission node based on the target transmission path; determine the dynamic compensation amount of the transmission node based on transmission status information; and determine the target delay parameters of the transmission node based on the basic delay parameters and the dynamic compensation amount of the transmission node.

[0079] The base delay parameter refers to a determined initial latency reference value. In some embodiments, the base delay parameter can be obtained based on node depth using the aforementioned method, that is, the server 120 allocates the corresponding target latency as the base delay parameter according to the depth of the transmission node in the target transmission path.

[0080] Transmission status information refers to information reflecting the real-time status of the current audio stream transmission. In some embodiments, transmission status information may include link jitter, node processing latency, buffer level changes, etc.

[0081] Link jitter refers to the fluctuation in the end-to-end transmission delay of audio data packets during transmission in a network communication link. In some embodiments, the server 120 can perform pairwise calculations for all transmission nodes in the system, obtaining the link jitter value of a link by calculating the standard deviation of the arrival time interval of audio data packets between two adjacent nodes. For example, the calculated link jitter of a certain adjacent link may be 2ms.

[0082] Node processing latency refers to the time consumed by a relay node after receiving an audio data packet, performing internal processing, and completing forwarding. In some embodiments, the node processing latency can be obtained only for the relay node. In some embodiments, the server 120 can select a preset time window (e.g., 1 second), calculate the difference between the arrival timestamp and the forwarding completion timestamp of the audio data packet in each small sampling time segment (e.g., every 10 ms), and then average all the differences within the preset time window to obtain the node processing latency of the relay node.

[0083] Buffer level change refers to the dynamic fluctuation of data occupancy in the audio playback buffer within a transmission node. The buffer level reflects the current data occupancy status of the audio playback buffers of all transmission nodes in the system, representing how much audio data remains in the buffer for continuous playback. In some embodiments, the server 120 can calculate the buffer level change by subtracting the historical buffer level value from the current buffer level value of a transmission node. For example, the calculation result might show that the buffer level of a certain node has decreased by 15ms.

[0084] The dynamic compensation amount refers to the adjustment value used to correct the target latency parameter. In some embodiments, the server 120 can obtain the dynamic compensation amount by querying a second preset table based on the real-time acquired transmission status information. The second preset table records the mapping relationship between link jitter, node processing latency, buffer level changes, and compensation amount.

[0085] In some embodiments, in the relay transmission path of the AoIP system, link jitter reflects the transmission uncertainty at the network communication link level, node processing latency reflects the internal computing power and processing constraints at the relay node device level, and buffer level changes reflect the dynamic balance of supply and demand at the data level within the audio output buffer. These three transmission status information collectively constitute a complete picture of the audio stream transmission quality. Understandably, relying on a single indicator alone cannot comprehensively assess the current transmission status, which can easily lead to insufficient latency compensation (causing audio playback stuttering) or overcompensation (causing unnecessary overall playback delay). However, by combining the above three factors to comprehensively determine the dynamic compensation amount, it can accurately cope with the dual fluctuations of network link and node processing, and through linkage with the forwarding quota allocation mechanism of the aforementioned candidate relay nodes, it can ensure the consistency of synchronous playback across multiple nodes while achieving overall system load balancing.

[0086] In some embodiments, considering that node processing latency usually only exists for relay nodes that undertake forwarding tasks, for non-relay nodes (e.g., ordinary nodes that only receive and do not forward, or source nodes that only send and do not receive), the server can directly set the dynamic compensation amount of non-relay nodes to a preset value, such as 0.

[0087] In some embodiments, the server 120 may also determine the dynamic compensation amount based on a prediction model. The prediction model refers to a model used to obtain the dynamic compensation amount of the transmission node. In some embodiments, the prediction model may be a Long Short-Term Memory (LSTM) network model, etc.

[0088] In some embodiments, the input to the prediction model may include a sequence of transmission state information within a preset sliding time window. This sequence may include link jitter over the last K time windows, node processing latency (or average processing latency) over the last K time windows, and buffer water level changes over the last K time windows. For example, assuming K=10 and each time window is 100ms long, the input sequence of transmission state information for the prediction model would be a feature sequence (or a 10×3 feature matrix) containing 10 time steps, each with 3-dimensional features (i.e., link jitter, node processing latency, and buffer water level changes), to preserve the evolutionary characteristics of the transmission state information over time.

[0089] In some embodiments, the output of the prediction model may include the amount of dynamic compensation (in milliseconds, ms) required by the transmission node in the next time period.

[0090] In some embodiments, the prediction model can be obtained through training based on at least one set of second training samples and their corresponding second labels. In some embodiments, the second training samples can be constructed by collecting historical operational data of transmission nodes under different network conditions. For example, each second training sample may include the link jitter, node processing latency, and buffer level changes of the node over the past K time windows. In some embodiments, the second label corresponding to the second training sample is the optimal compensation amount actually needed. For example, in order to ensure transmission quality and avoid resource waste, the optimal compensation amount (i.e., the second label) may be: the minimum compensation amount value that enables subsequent audio transmission quality to meet preset requirements (e.g., satisfying conditions such as packet loss rate less than 1% and latency jitter less than 5ms).

[0091] The method for obtaining a preset model by training based on the second training sample is similar to the aforementioned method for obtaining an adjusted model by training based on the first training sample, and will not be repeated here.

[0092] In some embodiments, after obtaining the basic delay parameters and dynamic compensation amount of the transmission node, the server 120 can directly add the basic delay parameters and the dynamic compensation amount, and use the sum of the two as the final target delay parameter of the transmission node.

[0093] In some embodiments of this specification, the target delay parameter is determined by combining a basic delay parameter with a dynamic compensation amount. This ensures both the baseline delay determined by the path depth and the ability to dynamically adjust compensation based on real-time transmission conditions (such as jitter, processing latency, and buffer changes). This mechanism effectively improves the real-time performance and anti-interference capability of audio transmission, avoiding playback asynchrony or stuttering caused by fixed delays.

[0094] Step 340: Transmit the target audio based on the target transmission path and target delay parameters.

[0095] Target audio refers to audio data that needs to be transmitted from a source node to a specific target node. For example, target audio could be a speaker's voice stream captured in real time by a conference room microphone, or high-fidelity background music data output by a media player. In some embodiments, target audio can be acquired by the source node (such as an audio input source) through its real-time recording interface or a sensor, or it can be acquired by the source node reading a pre-recorded audio file from storage device 140.

[0096] In some embodiments, after determining the target transmission path, target relay nodes, and target delay parameters, server 120 can control the source node to encapsulate the target audio into audio data packets conforming to the network transmission protocol and send them sequentially downstream along the target transmission path. Upon receiving the audio data packets, each target relay node along the path relays the audio data packets to the next hop node, extracts the target delay parameters assigned by the server, and places the received audio data packets in a local buffer for a duration corresponding to the target delay parameters. After the waiting time, the target relay nodes decode and convert the audio data locally for playback. Finally, through the coordinated efforts of the target relay nodes along the target transmission path and the end ordinary nodes, the target audio is stably transmitted to all target devices, enabling all participating nodes to synchronously output the target audio at the same physical moment.

[0097] The audio transmission control method provided in this specification breaks through the performance bottleneck of a single sending node in traditional AoIP systems by filtering resources in real time and utilizing multiple relay nodes to participate in the relay transmission of audio streams, while rationally allocating forwarding quotas to relay nodes. Furthermore, by combining transmission demand information to determine the target transmission path and target delay parameters, more reasonable resource allocation and load balancing can be achieved among multiple nodes with transmission capabilities. This effectively improves the overall transmission efficiency of the AoIP system, the real-time synchronization of multi-node playback, and system stability, and significantly reduces the risk of channel congestion and audio transmission interruption caused by limited resources of a single node.

[0098] In some embodiments, during the actual operation of an AoIP system, the network environment and device status are often dynamically changing. When the target relay node responsible for relaying fails or goes offline, or when severe congestion occurs in the network link leading to a decline in transmission quality, the original target transmission path may no longer be able to guarantee stable audio transmission. At this time, the server 120 needs to promptly replan the transmission path and smoothly switch the audio stream when it detects an anomaly in the transmission path. To this end, embodiments of this specification provide an update mechanism when an anomaly occurs in the target transmission path.

[0099] Figure 4 This is an exemplary flowchart illustrating the update target delivery path according to some embodiments of this specification. Figure 4 As shown, process 400 includes steps 410 to 440. In some embodiments, process 400 may be executed by server 120 or audio transmission control system 200.

[0100] In some embodiments, such as Figure 4As shown, in response to the target delivery path meeting the preset conditions, server 120 is also configured to perform the following steps 410 to 440.

[0101] Preset conditions refer to thresholds or status standards used to trigger the system to update the target transmission path and perform audio transition splicing. In some embodiments, preset conditions may include abnormal states such as node downtime, network connection interruption, or network latency exceeding the tolerance limit in the target transmission path. In some embodiments, the preset conditions may be pre-configured by the system administrator or automatically learned and set by the server 120 based on historical network operating status. For example, the server 120 can continuously monitor the connectivity and packet arrival rate of each transmission node in the network topology to determine whether the preset conditions are currently met.

[0102] In some embodiments, the preset conditions include the transmission score of the target transmission path being lower than a preset threshold. In some embodiments, the server 120 is further configured to: determine the path quality score of the target transmission path and the health score of the target relay node based on the resource status information of the target transmission path and the target relay node; and determine the transmission score of the target path based on the path quality score and the health score.

[0103] A preset threshold refers to the lowest acceptable score for the allowed audio transmission quality. In some embodiments, when the transmission score is lower than the preset threshold, it indicates that the quality of the current transmission path can no longer guarantee smooth audio playback, and a path switch must be performed. For example, the preset threshold can be set to 60 points (out of 100). In some embodiments, the preset threshold can be manually set by the user 160 through a user terminal, or it can be adaptively obtained by the server 120 based on the fault tolerance of different audio scenarios (such as conference voice, high-fidelity music, etc.).

[0104] Path quality score is an indicator used to quantitatively evaluate the transmission stability and efficiency of a target transmission path. In some embodiments, the server can obtain the path quality score by weighted calculation based on the resource status information (such as packet loss rate, jitter, etc.) of each link segment on the target transmission path. For example, the path quality score can be determined using the following formula (1): Path quality score = w1×(1-normalized jitter)+w2×(1-normalized packet loss rate)(1).

[0105] The weights (w1, w2) for the path quality score can be preset by the system, such as setting w1=0.6 and w2=0.4. Alternatively, server 120 can dynamically adjust the above weight parameters according to the actual application scenario.

[0106] Normalized jitter refers to a dimensionless value that maps the actual measured link jitter to a preset interval (e.g., [0, 1]). In some embodiments, the server can calculate the normalized jitter by obtaining the actual link jitter of the current link and dividing it by the system's preset maximum tolerable jitter threshold. If the calculated result is greater than 1, it can be truncated to 1.

[0107] Normalized packet loss rate (NPR) refers to a dimensionless value that maps the actual measured packet loss rate to a preset interval (e.g., [0, 1]). In some embodiments, server 120 can calculate the NPR by statistically analyzing the actual packet loss rate of the current link within a preset time window (e.g., the ratio of the number of lost audio data packets to the total number of packets sent) and dividing it by the system's preset maximum tolerable packet loss threshold. If the calculated result is greater than 1, it can also be truncated to 1.

[0108] A node's health score is a quantitative indicator that comprehensively reflects the current operating status of a transmission node. In some embodiments, the server can obtain the node's health score through weighted calculation based on the resource status information of the target relay node. For example, the health score can be determined based on the following formula (2): Health score = k1 × (1 - CPU utilization) + k2 × cache water level occupancy ratio (2).

[0109] The weights (k1, k2) for the node's health score can be preset by the system, for example, k1=0.7 and k2=0.3. The buffer occupancy rate refers to the percentage of currently stored audio data in the audio output buffer of a transmission node (such as a target relay node or a regular node) relative to the total capacity of that buffer. In some embodiments, the server 120 or the transmission node itself can calculate this buffer occupancy rate by reading the current data volume of the audio output buffer in real time and dividing it by the total capacity of the buffer.

[0110] The transmission score is a final quantitative assessment of the overall availability of the current target delivery path. In some embodiments, the transmission score can be obtained by weighted summation of the path quality score and health score mentioned above. For example, the transmission score can be determined based on the following formula (3): Transmission score = α × path quality score + β × health score (3).

[0111] The weights α and β can be determined manually, for example, α is set to 0.7 and β is set to 0.3. In some embodiments, when there are multiple target relay nodes on the target transmission path, the server can first take the average of the health scores of the multiple nodes, and then substitute it into the above formula (3) to complete the calculation of the transmission score.

[0112] In some embodiments of this specification, a comprehensive transmission score is determined by combining path quality scoring and health scoring, and this score is used as a preset condition to trigger path updates. This enables three-dimensional monitoring of the transmission status of a multi-device audio sharing system. This mechanism effectively avoids misjudgments caused by a single network indicator (such as only looking at packet loss rate) or a single device indicator (such as only looking at CPU utilization rate). It allows the system to predict transmission risks in advance and smoothly initiate the switching of backup paths before the link is completely disconnected or the node completely crashes, thereby significantly improving the robustness and disaster recovery capability of audio relay transmission.

[0113] In some embodiments, when the transmission score of the target transmission path is lower than a preset threshold, the server 120 is configured to perform the following steps:

[0114] Step 410: Generate a transition waveform based on the spectral characteristics, fundamental frequency characteristics, and energy envelope of the historical audio stream.

[0115] Historical audio streams refer to audio data being transmitted through the target relay node before a failure or anomaly occurs in the current transmission path. For example, the last segment of audio data successfully received and cached by the target relay node before an anomaly is detected. In some embodiments, server 120 can obtain historical audio streams by reading audio stream data from a local audio playback buffer for the last period before the failure.

[0116] Spectral features refer to the frequency distribution characteristics of an audio signal in the frequency domain. For example, the spectral envelope used to reflect the timbre characteristics of a sound. In some embodiments, the server 120 can perform a Short-Time Fourier Transform (STFT) on the extracted historical audio stream (i.e., reference sample) to calculate and extract the spectral envelope of the historical audio stream, and use it as a spectral feature.

[0117] Fundamental frequency characteristics refer to the fundamental frequency of an audio signal, used to reflect the basic pitch pattern of sound. For example, the fundamental frequency of a speaker's voice or the pitch of a musical note being played. In some embodiments, server 120 can use fundamental frequency detection algorithms such as autocorrelation to calculate and analyze historical audio streams, thereby obtaining and extracting the corresponding fundamental frequency characteristics.

[0118] The energy envelope refers to the contour trajectory of an audio signal's amplitude changing over time. For example, it can reflect the onset, decay, sustain, and release contours that reflect the volume changes from low to high and then back to low. In some embodiments, the server 120 can extract and fit the energy values ​​of each data sampling point or data frame in a historical audio stream sequence to obtain the signal's energy envelope, and then discretize or quantize this contour trajectory into a series of time-varying energy envelope coefficients (e.g., a coefficient sequence used to characterize amplitude proportions). For example, the sequence of energy envelope coefficients extracted from a historical audio stream could be [0.2, 0.5, 0.8, 1.0, 0.7, 0.3].

[0119] Transition waveforms refer to short predicted audio signals generated before the network layer path switching is complete. These waveforms fill the gaps in the flow during path switching to mask the abrupt changes or pops caused by the flow interruption. For example, a synthesized sound with a similar timbre and volume variation pattern to real sound, generated according to the expected switching duration (e.g., 50ms).

[0120] In some embodiments, server 120 or candidate relay nodes (such as processor units within candidate relay nodes) can first synthesize an initial synthesized waveform using a sine model based on extracted spectral features (such as spectral envelope) and fundamental frequency features. This initial synthesized waveform is similar to the spectrum of historical audio stream data samples. The extracted energy envelope coefficients are then applied point-by-point to this initial synthesized waveform based on the corresponding sampling times, ensuring that the amplitude changes of the synthesized waveform naturally transition with the real sound. Finally, a transition waveform of the corresponding length is generated according to the expected switching duration. For more information on the expected switching duration, please refer to [link to relevant documentation]. Figure 5 And its related descriptions.

[0121] In some embodiments, the extracted energy envelope is applied to the initial synthesized waveform by multiplying the sampling points of the synthesized waveform with the energy envelope coefficients point by point. For example, suppose the sequence of energy envelope coefficients extracted from the historical audio stream is [0.2, 0.5, 0.8, 1.0, 0.7, 0.3], which reflects the physical process of the sound volume increasing and then decreasing. Simultaneously, suppose the initial synthesized waveform generated based on spectral and fundamental frequency characteristics contains a corresponding sequence of audio sampling points [s1, s2, s3, s4, s5, s6]. Each element in the audio sampling point sequence (such as s1, s2, etc.) represents the instantaneous amplitude (or instantaneous signal strength) at the sampling time corresponding to that segment of the waveform.

[0122] In some embodiments, the server 120 can multiply the initial synthesized waveform (such as the audio sampling point sequence mentioned above) point by point with the energy envelope coefficients. For example, at time t1, the value of the final waveform is 0.2×s1; at time t2, the value of the final waveform is 0.5×s2; at time t3, the value of the final waveform is 0.8×s3; and so on. Through the above processing, the original synthesized waveform, which originally had a constant volume change (i.e., appeared smooth without fluctuations), will show a trend of increasing and then decreasing in volume after the application of the energy envelope, thereby highly restoring the dynamic listening experience of real sound and allowing the transition waveform to naturally connect the gaps in the audio.

[0123] In some embodiments, the generated transition waveform is temporarily stored in the local audio output buffer of the transmission node and seamlessly inserted between the historical audio stream and the updated audio stream during subsequent transition splicing for continuous playback output, thus allowing the transition waveform to naturally bridge the gaps in the stream. It is understood that not all transmission nodes need to store the transition waveform. For example, transmission nodes that need to store and play the transition waveform may include: relay nodes in the updated target transmission path that perform audio playback tasks, and ordinary nodes that act as audio playback terminals.

[0124] Step 420: Based on transmission demand information, candidate relay nodes and forwarding quotas, determine alternative relay nodes and alternative transmission paths, and use the alternative transmission paths as the updated target transmission paths.

[0125] For more information on transmission demand, candidate relay nodes, and forwarding quotas, please refer to [link / reference]. Figure 3 And its related descriptions.

[0126] Alternate relay nodes are transmission nodes that are re-evaluated and selected from the original candidate relay nodes when a failure or anomaly occurs in the current target transmission path. In essence, alternate relay nodes can replace the faulty nodes and continue to handle the relay forwarding of the target audio.

[0127] Alternative delivery paths refer to the alternative transmission routes identified for the current delivery path (i.e., the target delivery path where the anomaly has occurred).

[0128] In some embodiments, server 120 can exclude target relay nodes that have already experienced anomalies, and subsequently exclude relay nodes that are currently experiencing resource constraints or insufficient remaining forwarding quotas. For example, suppose the current target transmission path is "source node S → relay node A → relay node B → target node", where relay node B has experienced anomalies and needs to be switched. Server 120 can filter from previous candidate relay nodes (assuming they include transmission nodes A, C, E, and F), excluding the abnormal transmission node B, and simultaneously excluding transmission node E, whose forwarding quota is currently full, retaining transmission nodes C and F that can connect S to D. At this point, transmission nodes C and F are obtained and determined as candidate relay nodes.

[0129] In some embodiments, server 120 can, based on a determined pool of candidate relay nodes and in conjunction with the current network topology, use the KSP algorithm to search for all available paths from the start point to the end point of the audio stream, where intermediate nodes only contain candidate relay nodes. After filtering out physically or logically unreachable paths, the remaining connected paths are determined as candidate transmission paths. For more information on the KSP algorithm, please refer to [link to relevant documentation]. Figure 3 And its related descriptions.

[0130] For example, continuing the previous example, the starting point is S, the ending point is D, and the identified candidate relay nodes are C and F. Based on the current network topology, transmission node S connects to A and C, transmission node C connects to D, and transmission node F connects to D. Server 120, using the KSP algorithm, searches all possible paths and finds that path 1 "S→C→D" is connected; however, for path 2 "S→F→D," due to network topology limitations such as VLAN isolation, port disabling, or temporary physical link failures, there is no physically or logically available link between the source node S and the transmission node F (i.e., source node S cannot directly connect to transmission node F), making this path unreachable under the current network conditions. Therefore, the server filters out the unreachable path 2 and ultimately obtains and determines path "S→C→D" as the candidate transmission path.

[0131] Step 430: When switching the current transmission path to the updated target transmission path, the historical audio stream and the updated audio stream are spliced ​​together based on the transition waveform.

[0132] An updated audio stream refers to audio data transmitted via the updated target transmission path after a transmission path switch is triggered. For example, after the original target relay node fails, subsequent conference voice data sent from the audio input source via a newly established network route. In some embodiments, a playback device (such as a regular node) can receive and acquire this updated audio stream from a newly identified alternative relay node through its network communication interface.

[0133] Transition splicing refers to the process of smoothly connecting the historical audio stream retained before the interruption with the updated audio stream arriving after reconnection using a generated transition waveform, in order to fill the gap during the path switching. In some embodiments, the server 120 can use the generated transition waveform as an intermediate bridge, and before and after the splicing points of the historical audio stream and the transition waveform, as well as the splicing points of the transition waveform and the updated audio stream, weightedly superimpose the two adjacent waveforms (i.e., the historical audio stream and the transition waveform, and the transition waveform and the updated audio stream), and achieve the transition splicing of the historical audio stream, the transition waveform, and the updated audio stream through smooth transition methods such as cross-fade-in and fade-out, and switching near the zero crossing point.

[0134] In some embodiments, if a phase discontinuity is detected during the splicing process, the server 120 can also fine-tune the waveform and perform phase compensation to completely mask the abrupt change caused by the interruption and reduce the impact of the interruption on the speaker output. The final spliced ​​continuous audio stream will be sent to the local audio output buffer of the transmission node (such as a relay node or ordinary node with audio output function), and then the continuous audio will be played and output, thereby ensuring a smooth and uninterrupted listening experience for the user.

[0135] In some embodiments, server 120 may also be configured to: determine a transition mode based on quality detection information of the target transmission path and / or node response information of the transmission nodes; and perform transition splicing on the historical audio stream and the updated audio stream based on the transition mode and the transition waveform. For more information on this, please refer to [link to relevant documentation]. Figure 5 And its related descriptions.

[0136] Step 440: Transmit the updated audio stream based on the updated target delivery path.

[0137] In some embodiments, once the updated target transmission path is established and the new audio stream arrives, the server 120 can formally switch the original target transmission path to the updated target transmission path. Subsequently, the source node can continue to encapsulate the subsequent target audio into data packets and hand them over to the various alternative relay nodes on the updated target transmission path for relay forwarding, ultimately transmitting them stably to the end ordinary node for decoding and playback.

[0138] In some embodiments of this specification, a pre-switching mechanism is triggered by preset conditions. When the target transmission path is abnormal or performs poorly, a transition waveform based on historical audio characteristics is generated in advance, and alternative relay nodes and alternative transmission paths are dynamically determined, achieving smooth switching in fault conditions. By using transition waveforms for transition splicing during the switching process, this method effectively avoids audio interruptions and noise caused by node failures, greatly improving the user's listening experience while ensuring transmission continuity.

[0139] In some embodiments, during the switching of the aforementioned transmission path, network rerouting and backup link reconstruction inevitably consume a certain amount of time. Simply relying on network layer switching often fails to completely mask the interruption, potentially causing perceptible abrupt changes or popping sounds. To reduce the impact of the interruption on audio output and improve the smoothness of the listening experience, server 120 needs to adopt targeted acoustic repair and transition strategies. Therefore, embodiments of this specification further provide an implementation method for transition splicing.

[0140] Figure 5 This is an exemplary schematic diagram illustrating a transition splicing according to some embodiments of this specification.

[0141] In some embodiments, server 120 is further configured to: determine the fault type 531 and estimated switching duration 532 of target relay node 530 based on quality detection information 511 of target transmission path 510 and / or node response information 521 of transmission node 520; determine transition mode 550 based on fault type 531, estimated switching duration 532 and buffer level 540 of normal node; and perform transition splicing on historical audio stream 570 and updated audio stream 580 based on transition mode 550 and transition waveform 560.

[0142] For more information on transition waveforms, historical audio streams, updated audio streams, and spliced ​​audio, please refer to [link to relevant documentation]. Figure 4 And its related descriptions.

[0143] Quality detection information 511 refers to data information reflecting the quality status of the audio stream during transmission over the network communication link. In some embodiments, quality detection information 511 may include measured values ​​of indicators such as the packet loss rate, transmission delay, and jitter amplitude of audio data packets. In some embodiments, quality detection information 511 may be measured in real time by the transmission node acting as the receiving end when receiving audio data, and periodically reported to the central controller (such as server 120) for acquisition.

[0144] Node response information 521 refers to information reflecting the response status of a transmission node to control commands or probe requests issued by the server. For example, node response information 521 may include system heartbeat status, response status to probe requests, and measured node response time, signaling interaction latency, and node readiness latency. In some embodiments, node response information 521 may be periodically and proactively reported to the server 120 by each transmission node, or obtained by the server 120 proactively sending query commands to each node.

[0145] Fault type 531 refers to the classification of anomalies that occur when the target relay node performs audio forwarding tasks. For example, fault type 531 can include node failure, transmission anomaly, and path degradation.

[0146] In some embodiments, server 120 can determine the fault type 531 based on the received quality detection information 511, node response information 521, and preset rules. For example, the preset rules may include: if a node heartbeat loss is detected and the response times out, it is determined as a node failure; if a packet loss rate exceeds a preset threshold but the node can still respond normally, it is determined as a transmission anomaly; if latency jitter exceeds a preset threshold, it is determined as a path degradation fault. The preset rules can be obtained from historical experience or preset according to actual needs.

[0147] The estimated handover duration of 532 refers to the time required from the moment the system detects a node failure to the point of switching to the updated target delivery path and resuming audio playback. For example, the estimated handover duration of 532 can be the total duration obtained by adding the signaling interaction delay to the node readiness delay.

[0148] Signaling interaction latency refers to the time elapsed from when server 120 issues a path switching command to when it receives an acknowledgment (ACK) response from the relay node. For example, if server 120 issues a path switching signaling command at time T1 and receives an acknowledgment from the relay node at time T2, then the signaling interaction latency is the difference between T1 and T2.

[0149] In some embodiments, the server 120 can obtain the signaling interaction delay in real time by recording a first local timestamp when sending a path switching instruction and a second local timestamp when receiving a response message from a candidate relay node.

[0150] The node readiness latency refers to the time required for a candidate relay node to complete internal forwarding environment preparation (such as audio buffer allocation, encoding / decoding task initialization, network port binding, etc.) from the moment it receives the path switching instruction. For example, if a speaker device acting as a relay node needs to spend 10ms to allocate memory for the forwarding task and load the audio stream decapsulation module after receiving the instruction, then this 10ms is the node readiness latency.

[0151] In some embodiments, the server can extract the ready latency from the node response information 521, such as by having the candidate relay node carry the node's current estimated ready latency in its returned response message, or by having the server estimate it based on the node's historical performance and its current resource status information (such as CPU utilization).

[0152] The buffer level 540 refers to the percentage of data occupied in the real-time audio output buffer of the transmission node. For example, the buffer level 540 can be expressed as the percentage of cached audio data in the buffer that accounts for the total capacity of the buffer (such as 60% or 30%). In some embodiments, a normal node can calculate the buffer level 540 by reading the current amount of data used in its own audio output buffer in real time and dividing it by the total capacity of the buffer, and then report it to the server 120.

[0153] Transition mode 550 refers to different levels of audio patching or splicing strategies employed to mask audio interruptions and abrupt transitions. For example, transition mode 550 may include modes such as direct switching, short-time smooth transition, and transition waveform generation.

[0154] In some embodiments, server 120 may determine the corresponding transition mode 550 by querying a third preset table based on a determined fault type 531, an estimated switching time 532, and a cache level 540 of ordinary nodes. The third preset table stores the mapping relationship between fault type 531, estimated switching time 532, cache level 540, and transition mode 550.

[0155] For example, when fault type 531 is path degradation, the expected handover duration 532 is short (e.g., less than 30ms), and the buffer level 540 is high (e.g., greater than 60%), the transition mode 550 is determined to be a direct handover mode. When fault type 531 is transmission anomaly, the expected handover duration 532 is in a medium range (e.g., 30ms to 100ms), and the buffer level 540 is moderate (e.g., 30% to 60%), the transition mode 550 is determined to be a short-term smooth transition mode. When fault type 531 is node failure, the expected handover duration 532 is long (e.g., greater than 100ms), and the buffer level 540 is low (e.g., less than 30%), the transition mode 550 is determined to be a transition waveform generation mode. For other combinations, the transition mode 550 can default to the short-term smooth transition mode.

[0156] In some embodiments, after determining the transition mode 550, the server 120 can perform transition splicing on the historical audio stream 570 and the updated audio stream 580 based on the transition mode 550 and the transition waveform 560.

[0157] For example, when the determined transition mode 550 is direct switching, the server 120 may not use the transition waveform 560 for patching. Instead, in response to the successful establishment of the updated target transmission path, it may immediately stop playing the historical audio stream 570 and switch to the updated audio stream 580 for playback. In this mode, no waveform splicing operation is performed.

[0158] For example, when the determined transition mode 550 is a short-duration smooth transition, the server 120 may not introduce the transition waveform 560, but instead perform a short-duration crossfade process at the end of the historical audio stream 570 and the beginning of the updated audio stream 580 (e.g., setting the crossfade time range to 5ms to 20ms). During this process, the server 120 can apply a fade-out gain to the end of the historical audio stream 570 and a fade-in gain to the beginning of the updated audio stream 580, and then overlap and add them together to achieve a smooth transition of sound energy.

[0159] For example, when the determined transition mode 550 is a transition waveform generation, the server 120 can use the generated transition waveform 560 as a bridge to connect the historical audio stream 570 and the updated audio stream 580, and sequentially perform alignment timing, weighted superposition, and phase compensation operations. At this time, the server 120 performs timing alignment, that is, based on the precise time of the fault occurrence and the expected switching duration 532, determines the insertion start and end points of the transition waveform 560 on the time axis, ensuring that the transition waveform 560 can seamlessly fill the gap between the end of the historical audio stream 570 and the start of the updated audio stream 580; then, it performs weighted superposition, applying weighted window functions at the junctions of the transition waveform 560 and the historical audio stream 570, and at the junctions of the transition waveform 560 and the updated audio stream 580. For example, by setting a certain overlap area (such as 5ms to 10ms) before and after the splicing point, the two waveforms are weighted and averaged to make the amplitude of the waveforms exhibit a smooth linear or non-linear evolution at the junction; finally, phase compensation is performed. By detecting the phase difference between the end of the transition waveform 560 and the beginning of the updated audio stream 580, if the waveform phase is found to be discontinuous (such as a phase jump), the phase of the sampling point of the transition waveform 560 is finely adjusted or a very short phase smoothing component is inserted to make the phase of the two waveforms at the splicing position tend to be consistent, so as to maximize the continuity of audio output and the smoothness of the listening experience.

[0160] In some embodiments, server 120 is further configured to: determine the transition duration based on the cache water level 540 of a normal node and the expected switching duration 532; and perform transition splicing within the transition duration.

[0161] Transition duration refers to the length of the time window allocated by the server for performing the connection operation when switching transmission paths. For example, when using the short-time smooth transition mode, server 120 can allocate a time window of 40ms as the transition duration to gradually decrease the volume of the historical audio stream and simultaneously increase the volume of the updated audio stream, thereby completing the smooth transition between waveforms. As another example, when using the transition waveform generation mode, the transition duration determines the length of the transition waveform generated by server 120.

[0162] In some embodiments, server 120 can query a fourth preset table based on the current cache level 540 of a normal node and the estimated switching duration 532 to obtain the transition duration. The fourth preset table stores the mapping relationship between the cache level 540, the estimated switching duration 532, and the transition duration.

[0163] For example, when the cache level of a normal node is high (e.g., greater than 60%) and the expected switching time is short (e.g., less than 30ms), the transition time determined by looking up the table is equal to the expected switching time. When the cache level is high (e.g., greater than 60%) and the expected switching time is long (e.g., greater than 100ms), the transition time can be slightly shortened based on the expected switching time of 532 (e.g., the transition time can be 80% of the expected switching time of 532). When the cache level is low (less than 30%) and the expected switching time is short (e.g., less than 30ms), the transition time can be moderately shortened based on the expected switching time of 532 (e.g., the transition time can be 50% of the expected switching time of 532). When the cache level is low (less than 30%) and the expected switching time is long (e.g., greater than 100ms), the transition time can be significantly shortened based on the expected switching time of 532 (e.g., the transition time can be 30% of the expected switching time of 532).

[0164] In some embodiments, when the server 120 performs the corresponding transition splicing operation within a determined transition period, it may adopt methods such as fast switching, standard filling, and smooth switching.

[0165] For example, when the transition duration is short (e.g., a 20ms crossfade), a fast switching scheme can be used. In a fast switching scheme, if the updated audio stream has not arrived by the end of the transition duration, silence or repeating audio is used to fill the remaining gap.

[0166] For example, when the transition duration equals the expected switching duration (e.g., generating a 50ms transition waveform), a standard padding scheme can be used. In the standard padding scheme, the transition waveform can exactly fill the gap duration during the entire switching period.

[0167] For example, if the transition duration is longer than the expected switching duration (e.g., using an 80ms crossfade), a smooth transition scheme can be used. In a smooth transition scheme, the fade processing not only covers the switching period but also extends after the arrival of the updated audio stream, thus creating a longer overlapping transition between the historical and updated audio streams and achieving a smoother listening experience.

[0168] In some embodiments of this specification, the transition duration is dynamically and adaptively adjusted by linking it to the device's real-time caching capabilities and the estimated fault recovery time. This mechanism effectively solves the problems of buffer drying out and interruption that may occur when using a fixed transition duration, or the clicking sound left behind due to overly hasty splicing. Thus, under the strict premise of ensuring that the underlying audio data stream is not interrupted, the smoothness of the top-level user's listening experience is maximized.

[0169] In some embodiments of this specification, a multi-level, intelligent fault tolerance and acoustic repair mechanism is constructed for multi-device audio sharing systems by comprehensively determining the transition mode and performing transition splicing based on fault type, expected switching duration, and buffer level. By accurately assessing network and transmission node anomalies at different levels and matching differentiated transition strategies (such as direct switching, short-term smooth transition, or transition waveform generation), it avoids the waste of computing power caused by over-repairing minor network fluctuations and provides powerful waveform-level masking for severe node downtime faults. This tailored transition processing architecture, while considering system computational overhead, greatly enhances the robustness, self-healing ability, and continuous playback experience of the audio relay transmission network under complex fault scenarios.

[0170] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0171] Furthermore, unless expressly stated in the claims, the order of elements and sequences, the use of numbers and letters, or other names in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on an existing server or mobile device.

[0172] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0173] Finally, it should be understood that the embodiments in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments in this specification are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments in this specification are not limited to those explicitly described and illustrated herein.

Claims

1. An audio transmission control method, characterized in that, The method includes: Based on the resource status information of the transmission nodes, candidate relay nodes and their forwarding quotas are determined. Based on transmission demand information, the candidate relay nodes, the forwarding quota, and the network topology, the target relay node and the target transmission path are determined. Based on the target transmission path, determine the target delay parameter of the transmission node; The target audio is transmitted based on the target transmission path and the target delay parameters.

2. The method according to claim 1, characterized in that, Determining the target delay parameter of the transmission node based on the target transmission path includes: Based on the target transmission path, determine the basic delay parameters of the transmission node; Based on the transmission status information, the dynamic compensation amount of the transmission node is determined; The target delay parameter of the transmission node is determined based on the basic delay parameter and the dynamic compensation amount of the transmission node.

3. The method according to claim 1, characterized in that, The method further includes: In response to the target transmission path satisfying a preset condition: Transition waveforms are generated based on the spectral characteristics, fundamental frequency characteristics, and energy envelope of historical audio streams; Based on the transmission demand information, the candidate relay nodes, and the forwarding quota, alternative relay nodes and alternative transmission paths are determined, and the alternative transmission paths are used as the updated target transmission paths. When switching the current transmission path to the updated target transmission path, the historical audio stream and the updated audio stream are transitionally spliced ​​based on the transition waveform; and The updated audio stream is transmitted based on the updated target delivery path.

4. The method according to claim 3, characterized in that, The step of performing a transitional splicing of the historical audio stream and the updated audio stream based on the transition waveform includes: Based on the quality detection information of the target transmission path and / or the node response information of the transmission node, determine the fault type and expected switching time of the target relay node; The transition mode is determined based on the fault type, the expected switching duration, and the cache level of ordinary nodes; Based on the transition mode and the transition waveform, the historical audio stream and the updated audio stream are spliced ​​together using the transition.

5. An audio transmission control system, characterized in that, The system includes: a parameter determination module and an audio transmission module; The parameter determination module is configured as follows: Based on the resource status information of the transmission nodes, candidate relay nodes and their forwarding quotas are determined. Based on transmission demand information, the candidate relay nodes, the forwarding quota, and the network topology, the target relay node and the target transmission path are determined. Based on the target transmission path, determine the target delay parameter of the transmission node; The audio transmission module is configured as follows: The target audio is transmitted based on the target transmission path and the target delay parameters.

6. The system according to claim 5, characterized in that, The parameter determination module is also configured to: Based on the target transmission path, determine the basic delay parameters of the transmission node; Based on the transmission status information, the dynamic compensation amount of the transmission node is determined; The target delay parameter of the transmission node is determined based on the basic delay parameter and the dynamic compensation amount of the transmission node.

7. The system according to claim 5, characterized in that, In response to the target transmission path meeting preset conditions. The parameter determination module is also configured to: Transition waveforms are generated based on the spectral characteristics, fundamental frequency characteristics, and energy envelope of historical audio streams; Based on the transmission demand information, the candidate relay nodes, and the forwarding quota, alternative relay nodes and alternative transmission paths are determined, and the alternative transmission paths are used as the updated target transmission paths. The audio transmission module is also configured to: When the current transmission path is switched to the updated target transmission path, the historical audio stream and the updated audio stream are spliced ​​together based on the transition waveform. as well as The updated audio stream is transmitted based on the updated target delivery path.

8. The system according to claim 7, characterized in that, The parameter determination module is also configured to: Based on the quality detection information of the target transmission path and / or the node response information of the transmission node, determine the fault type and expected switching time of the target relay node; The transition mode is determined based on the fault type, the expected switching duration, and the cache level of ordinary nodes; The audio transmission module is also configured to: Based on the transition mode and the transition waveform, the historical audio stream and the updated audio stream are transitionally spliced.

9. An audio transmission control device, characterized in that, The device includes at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is configured to execute at least a portion of the computer instructions to implement the audio transmission control method as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the audio transmission control method as described in any one of claims 1-4.

Images