Multichannel audio synchronization method, device and equipment of conference terminal equipment and storage medium

Abstract

The invention relates to the technical field of multi-channel audio synchronization, and discloses a multi-channel audio synchronization method and device for conference terminal equipment, equipment and a storage medium, and the method comprises the steps: collecting an audio data stream of each conference terminal equipment, and obtaining an embedded audio data stream; performing time-frequency domain conversion and peak detection to obtain an audio feature vector; calculating a cross-correlation function between the terminal devices to obtain a global time relation graph; constructing a two-layer cache structure, and adjusting the residence time of the audio data in the cache to obtain a buffer strategy; inputting the buffer strategy as a state into a multi-agent reinforcement learning model for communication resource allocation to obtain a resource allocation strategy; a global synchronization optimization problem is constructed based on a resource allocation strategy, an optimal synchronization parameter is solved through a gradient descent algorithm, time axis adjustment and content alignment are carried out on an embedded audio data stream, and a plurality of paths of synchronized audios are obtained, so that the audio quality and user experience of a multi-party teleconference are improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-channel audio synchronization, and in particular to a multi-channel audio synchronization method, device, equipment and storage medium for conference terminal equipment. Background Art

[0002] In conference scenarios involving multiple terminal devices, achieving accurate synchronization of multi-channel audio still faces huge challenges due to network latency, device performance differences and environmental factors. Traditional audio synchronization methods often rely on simple timestamp comparison or fixed buffer strategies, which are difficult to adapt to complex and changing network environments and dynamic conference scenarios.

[0003] These methods often cause audio desynchronization, echo, and crosstalk when dealing with multi-terminal and long-term meetings, which seriously affects the user's meeting experience and communication efficiency. In addition, existing technologies are also unable to cope with environmental noise, network jitter, and device heterogeneity, and it is difficult to ensure the quality of audio synchronization in various complex situations. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for synchronizing multiple audio channels of a conference terminal device, thereby improving the audio quality and user experience of a multi-party remote conference.

[0005] A first aspect of the present application provides a multi-channel audio synchronization method for a conference terminal device, the multi-channel audio synchronization method for a conference terminal device comprising: The audio data stream of each conference terminal device is collected, and the environmental audio information is encoded into a frequency domain feature and embedded into the audio data stream to obtain an embedded audio data stream; Performing time-frequency domain conversion and peak detection on the embedded audio data stream, mapping the peak point into a binary string by using a local sensitive hashing algorithm, and obtaining an audio feature vector; Calculate the cross-correlation function between terminal devices based on the audio feature vector, determine the time delay through multi-scale analysis and median filtering, and obtain a global time relationship diagram; A two-layer cache structure is constructed according to the global time relationship graph, a network state is predicted by a Kalman filter, and the residence time of the audio data in the cache is adjusted to obtain a buffering strategy; The buffer strategy is input as a state into a multi-agent reinforcement learning model, and the communication resource allocation is optimized through a dual deep Q network to obtain a resource allocation strategy; A global synchronization optimization problem is constructed based on the resource allocation strategy, the optimal synchronization parameters are solved by a gradient descent algorithm, the time axis of the embedded audio data stream is adjusted and the content is aligned to obtain synchronized multi-channel audio.

[0006] A second aspect of the present application provides a multi-channel audio synchronization device for a conference terminal device, the multi-channel audio synchronization device for a conference terminal device comprising: A collection module is used to collect the audio data stream of each conference terminal device, and encode the environmental audio information into frequency domain features and embed them into the audio data stream to obtain an embedded audio data stream; A mapping module, used to perform time-frequency domain conversion and peak detection on the embedded audio data stream, and map the peak point into a binary string through a local sensitive hashing algorithm to obtain an audio feature vector; A calculation module, used to calculate the cross-correlation function between terminal devices based on the audio feature vector, determine the time delay through multi-scale analysis and median filtering, and obtain a global time relationship diagram; A construction module is used to construct a two-layer cache structure according to the global time relationship graph, predict the network state through a Kalman filter, adjust the residence time of the audio data in the cache, and obtain a buffering strategy; An allocation module, configured to input the buffer strategy as a state into a multi-agent reinforcement learning model, optimize the communication resource allocation through a dual deep Q network, and obtain a resource allocation strategy; The synchronization module is used to construct a global synchronization optimization problem based on the resource allocation strategy, solve the optimal synchronization parameters through a gradient descent algorithm, adjust the time axis and align the content of the embedded audio data stream, and obtain synchronized multi-channel audio.

[0007] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned multi-channel audio synchronization method of the conference terminal device.

[0008] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned multi-channel audio synchronization method for a conference terminal device.

[0009] Compared with the prior art, the present application has the following beneficial effects: by encoding the environmental audio information into frequency domain features and embedding them into the audio data stream, the present invention enhances the robustness of audio synchronization and can better cope with background noise interference in different conference environments. The local sensitive hashing algorithm is used to map the peak points into binary strings, which greatly reduces the storage and transmission overhead of audio features while maintaining the distinguishability of the features. The introduction of multi-scale analysis and median filtering technology improves the accuracy and stability of time delay estimation and effectively overcomes the influence of network jitter and burst noise. By constructing a two-layer cache structure and combining the Kalman filter to predict the network state, the dynamic adjustment of the audio data cache is realized, and the synchronization accuracy and system delay are effectively balanced. The multi-agent reinforcement learning model and the dual-depth Q network are used to optimize the allocation of communication resources, so that the system can adaptively cope with complex and changeable network environments and improve resource utilization efficiency. By constructing a global synchronization optimization problem and solving it using the gradient descent algorithm, the precise time axis adjustment and content alignment of multi-channel audio are achieved, and the synchronization of audio is guaranteed to the greatest extent. The present invention significantly improves the audio quality and user experience of multi-party remote conferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0012] Figure 1 It is a flowchart of a multi-channel audio synchronization method for a conference terminal device provided by an embodiment of the present invention; Figure 2 It is a schematic block diagram of the structure of a multi-channel audio synchronization device for a conference terminal device provided by an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0015] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0016] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 , an embodiment of the multi-channel audio synchronization method of a conference terminal device in the embodiment of the present application includes: Step 100: collect the audio data stream of each conference terminal device, and encode the environmental audio information into frequency domain features and embed them into the audio data stream to obtain an embedded audio data stream; It is understandable that the execution subject of the present application can be a multi-channel audio synchronization device of a conference terminal device, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0017] Specifically, the sound of the environment around each conference terminal device is collected to obtain the real environmental background sound information. The collected sound signal is preprocessed by a high-pass filter to filter out low-frequency interference, retain the high-frequency components in the environment, and obtain the environmental background audio. The processed environmental background audio is subjected to spectrum analysis, and the frequency components that are stable and have obvious characteristics in time are selected by analyzing the energy aggregation of different frequency components. These frequency components represent the significant features in the environmental sound and can describe the characteristics of the environment, thereby obtaining the characteristic environmental audio. The characteristic environmental audio is subjected to discrete cosine transformation, and the characteristic environmental audio is converted from the time domain signal to the frequency domain space to obtain the frequency domain environmental audio characteristics. At the same time, the audio input signal of the conference terminal device is sampled and quantized, and the collected analog audio signal is converted into a digital signal by an analog-to-digital converter to obtain an audio data stream for processing. The audio data stream is framed, and the continuous audio data is divided into audio frames of fixed length to obtain framed audio data. The framed audio data is subjected to fast Fourier transformation, and the framed audio data in the time domain is converted to the frequency domain space to obtain frequency domain audio data. In the frequency domain space, based on the frequency domain environmental audio characteristics, the mid-high frequency bands of the frequency domain audio data are modulated, and the environmental audio information is embedded into the audio data stream to obtain embedded frequency domain audio data. The embedded frequency domain audio data is inversely fast Fourier transformed, and the embedded frequency domain audio data is converted back to the time domain space to obtain an embedded audio data stream.

[0018] Step 200: performing time-frequency domain conversion and peak detection on the embedded audio data stream, mapping the peak point to a binary string through a local sensitive hashing algorithm, and obtaining an audio feature vector; Specifically, the embedded audio data stream is segmented, and the continuous embedded audio data stream is divided into time segments of fixed length to obtain an audio segment sequence, so as to facilitate feature extraction and synchronization operations for each segment. Each audio segment in the audio segment sequence is transformed, and the time domain signal is converted into a time-frequency representation to obtain a time-frequency spectrum, which reflects the energy distribution of each frequency component of the audio signal at different time points and helps to identify the characteristics of the signal at different times and frequencies. Energy calculation is performed on the time-frequency spectrum to quantify the energy distribution of each time-frequency unit. The energy value of the time-frequency unit is calculated by an integral method to obtain an energy distribution matrix. The energy distribution matrix is ​​used to represent the signal strength at each time and frequency combination in the time-frequency spectrum. Based on the energy distribution matrix, peak detection is performed on each time-frequency unit to obtain a set of peak points in the time-frequency spectrum, and representative important frequency components in the time-frequency spectrum are identified. Coordinate quantization is performed on the peak point set, and the time-frequency coordinates are mapped to a discrete grid to obtain quantized peak points. Coordinate quantization simplifies the representation of time-frequency coordinates, making it easier to compare and match peak points between different audio segments. The feature descriptor is constructed based on the quantized peak points. The difference vector is obtained by calculating the time-frequency difference between adjacent peak points. The difference vector is used to describe the relationship between adjacent peak points. The difference vector is input into the local sensitive hash function, and the hash value is generated by random projection and threshold comparison to obtain a binary feature string. Local sensitive hashing can maintain the distance relationship between similar data in high-dimensional space, map similar features to similar hash values, and simplify the matching process between audio clips. Binary feature strings are grouped and merged. Multiple feature strings are combined into a fixed-length vector through bit operations to obtain an audio feature vector.

[0019] Step 300: Calculate the cross-correlation function between terminal devices based on the audio feature vector, determine the time delay through multi-scale analysis and median filtering, and obtain a global time relationship diagram; It should be noted that the audio feature vector is regarded as time series data, and the time series is decomposed. By resampling the audio feature vector according to different time scales, multiple subsequences with different resolutions are generated. These subsequences represent the information of the audio feature vector at multiple preset time scales, forming a multi-scale feature sequence. Based on the multi-scale feature sequence, the feature sequence of each pair of conference terminal devices is cross-correlated. A time window of a fixed size is selected, and the time window is simultaneously slid on the multi-scale feature sequence of the two conference terminal devices. At each position, the Pearson correlation coefficient of the data in the time window is calculated, and the correlation coefficient values ​​under different time offsets are recorded to form a cross-correlation function curve. The cross-correlation function curve represents the similarity of the feature sequences between the two devices under different time offsets, and the peak point on the curve represents the optimal synchronization time delay between the two devices. Peak detection is performed on the cross-correlation function curve, and the local maximum point is searched on the cross-correlation function curve. For each local maximum point, two points on the left and right are taken, a total of five points of data, and a quadratic curve is fitted using these points. Through quadratic curve fitting, the vertex position of the curve is calculated, and the vertex position is used as the precise position of the peak to obtain the candidate value of the initial time delay. A multi-scale time delay matrix is ​​constructed based on the initial time delay candidate values. Each row of the multi-scale time delay matrix represents a time scale, and each column represents the delay between a pair of devices. Through the analysis of different time scales, various time-varying information contained in the audio feature vector is captured. On this basis, each column in the multi-scale time delay matrix is ​​weighted averaged to obtain a fused time delay value, which represents the comprehensive delay estimation at multiple time scales. The fused time delay value is median filtered, an odd-sized sliding window is set, and the sliding window is slid on the time delay sequence. Each time it slides, all values ​​in the sliding window are taken and sorted by size, and the middle value is selected as the output of the current position to obtain a smooth time delay sequence. Median filtering effectively suppresses noise in the sequence and removes abnormal delays caused by network jitter or sudden interference, making the final time delay sequence smoother and more stable. Based on the smoothed time delay sequence, a time delay graph is constructed. Each conference terminal device is regarded as a node in the time delay graph, the time delay between devices is used as the weight of the edge, and then the Kruskal algorithm is used to construct the minimum spanning tree. The goal of the Kruskal algorithm is to select the edge set with the minimum total delay to obtain the topology between devices. This topology represents the interdependence of each terminal device in terms of time synchronization to ensure that the global delay is minimized. When the constructed device topology is globally optimized, a relative time coordinate system with the reference device as the origin is established. The time delay between each pair of adjacent devices is regarded as a constraint condition, and a least squares optimization problem is constructed.By solving the least squares optimization problem, the position of each conference terminal device in the unified time coordinate system is obtained, and the consistent time coordinate is obtained. Based on the consistent time coordinate, the position of each conference terminal device in the unified time coordinate system is mapped to a two-dimensional plane, where the distance between the devices reflects the time synchronization relationship between them. The positions of these nodes are adjusted using a force-directed algorithm to obtain a global time relationship graph. The force-directed algorithm simulates the interaction force between nodes, so that the nodes present a more balanced layout on the plane that conforms to the time synchronization relationship. The final global time relationship graph intuitively shows the synchronization status between each conference terminal device.

[0020] Step 400: construct a two-layer cache structure according to the global time relationship graph, predict the network state through the Kalman filter, adjust the residence time of the audio data in the cache, and obtain a buffering strategy; Specifically, based on the global time relationship graph, an appropriate amount of cache space is allocated to each conference terminal device, and the cache space is divided into two layers: fast cache and deep cache. The fast cache is used to store the recently received audio data, mainly with low latency as the goal, so that the latest audio data can respond quickly and enter the processing flow, while the deep cache is used to store the processed historical audio data to provide a buffer for audio data with a longer time span. Through the two-layer cache structure, the real-time and stability of the audio stream are more effectively balanced, and the synchronization effect of the audio data is improved. The data in the two-layer cache structure is analyzed by timestamp to obtain the time law of the arrival of the audio data. By calculating the time interval between adjacent audio frames, the time series of the arrival of the audio data is obtained, and the state space model of the Kalman filter is constructed. In this model, the arrival time of the audio data is regarded as the observation value, and the network transmission delay is used as the state variable. The model is used to describe the time-varying characteristics and delay fluctuations in network transmission. The Kalman filter is a tool for estimating the state of a time-varying system. By dynamically adjusting the observation value, the system state is best estimated, and it is suitable for predicting network delay. In order to enable the Kalman filter to effectively predict the network state, the parameters of the network state prediction model are initialized. By analyzing the historical audio data, the initial state estimate and error covariance matrix are calculated to obtain the initial parameters of the Kalman filter. On this basis, the state of the newly arrived audio data is predicted and updated based on the initial parameters of the Kalman filter. The network state at the next moment is predicted by the time update equation, and then the prediction result is corrected by the measurement update equation to obtain the optimal estimate of the current network state. The Kalman filter makes the estimation of the network state have good accuracy and real-time performance through the cycle of prediction and correction. Based on the optimal estimate of the current network state, the ideal residence time of the audio data in the cache is calculated. The target value of cache adjustment is obtained by comparing the predicted network transmission delay with the target playback delay. If the network delay is large, the residence time of the audio data in the cache needs to be extended to avoid data incoherence during playback; if the network delay is small, the cache time is reduced to reduce the impact of the delay. The cache adjustment target value is obtained by comparing the network state predicted by the Kalman filter with the target playback requirement. Based on the cache adjustment target value, the data in the two-layer cache structure is reorganized to achieve the optimal cache distribution. By adjusting the data allocation ratio between the fast cache and the deep cache, the buffering time of the audio data can be finely controlled. For the audio data that arrives ahead of time, its processing is appropriately delayed to match the rhythm of the playback; for the audio data that arrives late, the processing is accelerated to synchronize it with the audio data stream of other devices as much as possible. Through the flexible deployment of the fast cache and the deep cache, the audio data can be synchronized in time, the difference in playback delay between different devices can be reduced, and the synchronization quality of the overall conference audio can be improved.The performance of the cache strategy is evaluated to verify its effectiveness and adaptability under the current network status. The performance indicators of the buffer strategy are obtained by calculating indicators such as buffer utilization, packet loss rate, and audio quality. Buffer utilization reflects the use of cache resources. If the utilization is too high, it will lead to excessive delay, while if the utilization is too low, it will not be able to fully cope with network fluctuations; packet loss rate is an important indicator to measure the reliability of network transmission. If the packet loss rate is high, it means that the redundancy of the cache needs to be improved to reduce the impact of packet loss; audio quality indicators are important standards that directly reflect the user experience, including audio continuity and clarity. According to these performance indicators, the current buffer strategy is adjusted as necessary to ensure that the buffer strategy can adapt to the current network conditions and finally obtain the best cache management solution.

[0021] Step 500: Input the buffer strategy as a state into the multi-agent reinforcement learning model, optimize the communication resource allocation through the dual-depth Q network, and obtain the resource allocation strategy; Specifically, the cache strategy is feature extracted to extract the buffer state feature vector, including information such as cache utilization, audio stream delay, and network state fluctuation. Based on the buffer state feature vector, a reinforcement learning environment is constructed, in which each conference terminal device is regarded as an independent agent, and the environmental state of each agent is represented by its buffer state feature vector. At the same time, the action space is set as an optional communication frequency band and transmission power, thus forming a multi-agent learning environment. Each agent selects a suitable action based on its own state to optimize its communication and resource utilization. In the multi-agent learning environment, a reward function is designed. The role of the reward function is to guide each agent to make the most favorable choice for the overall goal of the system. The design of the reward function comprehensively considers three factors: audio synchronization, transmission delay, and channel utilization. Audio synchronization is used to measure the quality of audio data synchronization between each conference terminal device, transmission delay reflects the delay of data transmission from one device to another, and channel utilization measures the efficiency of communication resource utilization. By combining these factors in the form of weighted sum, the optimization target of reinforcement learning is constructed. A dual deep Q network model is constructed based on the optimization target of reinforcement learning. The dual-depth Q network consists of an online network and a target network. The two networks have the same structure and consist of multiple fully connected layers. The input is the current state vector and the output is the Q value of each possible action. In the initialization stage, all parameters of the dual-depth Q network are randomly initialized. The structure of the network enables it to effectively map complex environmental states and provide the best action selection basis for each intelligent agent. In order to enable the dual-depth Q network to learn effectively, an experience replay buffer is generated by random sampling. The buffer is used to store the experience data of each intelligent agent interacting with the environment, including information such as state, action, reward and next state. The existence of the experience pool enables the network to be trained from past experience, avoiding the problem of training instability caused by data correlation. Based on the experience pool, the dual-depth Q network is trained. By randomly extracting a batch of experience data from the experience pool, the mean square error between the target Q value and the current Q value is calculated, and the error is used as the loss function. Then, the back propagation algorithm is used to update the parameters of the online network. Through multiple iterations and training, the online network gradually learns how to make the best action selection from the current environmental state, and a trained dual-depth Q network is obtained. Based on the trained dual-depth Q network, the network is used to select actions for each agent, and the ε-greedy strategy is used to strike a balance between exploration and utilization. The core of the ε-greedy strategy is to select random actions (exploration) with a certain probability, and to select the action with the largest current Q value (utilization) with the remaining probability. This strategy enables both finding new optimal strategies and making full use of the learned knowledge during the training process, thereby selecting the optimal communication frequency band and transmission power for each agent and forming an initial resource allocation strategy.The initial allocation strategy is globally coordinated to ensure that there is no conflict in resource allocation between all agents in the system. For example, multiple agents choose the same communication frequency band or too high a transmission power, which leads to channel conflicts or power overload problems. In order to resolve resource conflicts, a centralized resource allocator is built to collect the state information and action selections of all agents, and coordinate and adjust the allocation strategies of each agent based on this information. By reasonably adjusting the initial allocation strategy of each agent, resource conflicts are eliminated and the globally optimal resource allocation strategy is obtained.

[0022] Step 600: construct a global synchronization optimization problem based on the resource allocation strategy, solve the optimal synchronization parameters through the gradient descent algorithm, adjust the time axis and align the content of the embedded audio data stream, and obtain synchronized multi-channel audio.

[0023] Specifically, based on the resource allocation strategy, the communication parameters of each conference terminal device are quantified, and the communication parameters such as frequency band selection and transmission power are converted into numerical representations to construct a communication resource allocation matrix. The communication resource allocation matrix can intuitively reflect the communication resources allocated to each device. Feature extraction is performed on the communication resource allocation matrix. By calculating the channel capacity and transmission delay between the conference terminal devices, the network performance feature vector is obtained to characterize the performance of the network state, including transmission bandwidth, delay, and communication stability. Based on the network performance feature vector and the global time relationship graph, a global synchronization optimization objective function is constructed. The global synchronization optimization objective function uses time synchronization error and network resource utilization as optimization indicators, where the time synchronization error measures the synchronization accuracy of audio data between devices, and the network resource utilization reflects the efficiency of communication resource utilization. In order to ensure that the synchronization effect meets the established requirements, constraints are set for the global synchronization optimization objective function, and a constrained optimization problem is constructed by limiting the maximum allowable synchronization error and the minimum necessary transmission rate. The maximum allowable synchronization error ensures that the synchronization quality of audio between terminals does not exceed the acceptable range, while the minimum necessary transmission rate ensures that the communication bandwidth is sufficient to support the stable transmission of audio data. Based on the constrained optimization problem, the Lagrangian function is constructed. By introducing the Lagrangian multiplier, the constraint conditions are integrated into the global synchronization optimization objective function, and the original constrained problem is converted into an unconstrained optimization problem. The advantage of the unconstrained optimization problem is that the optimal synchronization parameters can be solved by classical optimization algorithms, such as the gradient descent method. By iteratively calculating the gradient of the Lagrangian function and gradually updating the optimization variables, the optimal solution of the global synchronization optimization objective function is converged to obtain the optimal synchronization parameters. The gradient descent method adjusts the error of each iteration so that the synchronization error gradually decreases and eventually reaches the optimal synchronization state. Based on the obtained optimal synchronization parameters, the time axis of the embedded audio data stream is adjusted. By interpolating the audio frames, the audio time axis of each conference terminal device is aligned to a unified reference time axis to obtain a time-aligned audio stream. The interpolation operation is achieved by inserting appropriate sampling points between adjacent audio frames. The interpolated audio stream eliminates the time difference between different devices, so that the audio data of each terminal device is synchronized on a unified time axis. Content alignment is performed on time-aligned audio streams to ensure that the audio content is consistent across all devices. By analyzing the similarity of audio feature vectors, duplicate or missing audio segments in the audio stream are identified. By analyzing the similarity of audio feature vectors, duplicate or missing segments are accurately located and appropriate measures are taken to deal with them. For duplicate audio segments, the audio stream is simplified by deleting redundant parts; for missing audio segments, they are supplemented by interpolation or using audio data from adjacent devices to ensure the integrity and consistency of the audio stream. Through dual alignment of the timeline and content, synchronized multi-channel audio is finally obtained.

[0024] In the embodiment of the present application, by encoding the ambient audio information into frequency domain features and embedding it into the audio data stream, the present invention enhances the robustness of audio synchronization and can better cope with background noise interference in different conference environments. The local sensitive hashing algorithm is used to map the peak point into a binary string, which greatly reduces the storage and transmission overhead of the audio features while maintaining the distinguishability of the features. The introduction of multi-scale analysis and median filtering technology improves the accuracy and stability of time delay estimation and effectively overcomes the influence of network jitter and burst noise. A two-layer cache structure is constructed and combined with the Kalman filter to predict the network state, which realizes the dynamic adjustment of the audio data cache and effectively balances the synchronization accuracy and system delay. The multi-agent reinforcement learning model and the dual-depth Q network are used to optimize the allocation of communication resources, so that the system can adaptively cope with complex and changeable network environments and improve resource utilization efficiency. By constructing a global synchronization optimization problem and solving it using a gradient descent algorithm, accurate time axis adjustment and content alignment of multi-channel audio are achieved, and the synchronization of audio is guaranteed to the greatest extent. The present invention significantly improves the audio quality and user experience of multi-party remote conferences.

[0025] In a specific embodiment, the process of executing step 100 may specifically include the following steps: The sound of the environment of each conference terminal device is collected, and the collected sound signals are preprocessed through a high-pass filter to obtain the environmental background audio. The environmental background audio is then subjected to spectrum analysis. The characteristic environmental audio is obtained by calculating the energy concentration and selecting the continuous, stable and characteristic frequency components. Perform discrete cosine transform on the characteristic environment audio, convert the characteristic environment audio into frequency domain space, obtain frequency domain environment audio features, sample and quantize the audio input of the conference terminal device, convert the analog signal into a digital signal through an analog-to-digital converter, and obtain an audio data stream; Performing frame processing on the audio data stream, dividing the continuous audio data into audio frames of fixed length to obtain framed audio data, and performing fast Fourier transform on the framed audio data to convert the framed audio data into frequency domain space to obtain frequency domain audio data; Based on the frequency domain environmental audio characteristics, the mid- and high-frequency bands of the frequency domain audio data are modulated, and the environmental audio information is embedded into the audio data stream to obtain embedded frequency domain audio data. The embedded frequency domain audio data is then inverse fast Fourier transformed and converted back to the time domain space to obtain an embedded audio data stream.

[0026] Specifically, the sound of the environment of each conference terminal device is collected, and the sound signals in the environment are captured using the built-in microphone of the device or other external audio sensors. The environmental sound signal contains audio information from many different sources, including the environmental noise of the device, background music, speech, etc. In order to extract useful environmental features from it, the collected sound signal is preprocessed and a high-pass filter is used to eliminate low-frequency noise. The main function of the high-pass filter is to remove low-frequency interference (such as power hum, wind noise, etc.) and retain the parts with higher energy characteristics in the mid- and high-frequency bands, which can be specifically expressed as: ; in, is the output signal after being processed by the high-pass filter. is the original ambient sound signal, is the attenuation coefficient of the filter, which is used to control the degree of removal of low-frequency components. After preprocessing with a high-pass filter, the ambient background audio is obtained. The ambient background audio processed by the high-pass filter is subjected to spectrum analysis to extract the main frequency components in the signal. By performing spectrum analysis on the audio signal, the energy distribution at each frequency is quantified. Energy concentration calculation is used to determine which frequency components are stable and have significant characteristics over time. The energy at each frequency component is calculated.

[0027] ; in, Indicates frequency The average energy on is the time window for analysis, For frequency In time The spectrum amplitude on . In this way, frequency components with significant energy characteristics and stable in time are selected as characteristic environmental audio. The characteristic environmental audio contains relatively important sound characteristics in the device environment, such as the frequency components of background noise. The characteristic environmental audio is further processed and converted into the frequency domain space using discrete cosine transform to obtain the frequency domain environmental audio features. The function of discrete cosine transform is to convert the time domain signal into frequency domain representation, and retain the core information of the signal by removing high-frequency noise and compressing energy. The formula of discrete cosine transform is: ; in, is the frequency domain coefficient after transformation, is the input signal in the time domain, is the total number of samples. Through discrete cosine transform, the time domain information of the characteristic ambient audio is converted into frequency domain coefficients , these frequency domain coefficients can compactly represent the environmental characteristics. At the same time, the audio input of the conference terminal device is sampled and quantized, and the analog signal is converted into a digital signal through an analog-to-digital converter to obtain an audio data stream. The audio data stream is the result of digitizing the input analog audio and is represented as a series of discrete audio samples. The audio data stream is framed, and the continuous audio data is divided into multiple audio frames of fixed length. Each frame contains several continuous audio sampling points. The audio data is locally analyzed and processed through framing processing to improve the real-time processing. The framed audio data is fast Fourier transformed to convert the audio data from the time domain to the frequency domain space to obtain frequency domain audio data. The frequency domain audio data contains the amplitude and phase information of the audio signal at each frequency. Based on the frequency domain environmental audio characteristics, the frequency domain representation of the audio data stream is modulated to embed the environmental audio information into the audio data stream. The mid-high frequency bands in the frequency domain audio data are selected for modulation operations. These frequency bands are usually the parts where the energy of the audio signal is relatively concentrated, and are also the parts where the human ear is most sensitive to audio quality. By modulating the characteristic information of the ambient audio onto these frequency bands, the audio features are effectively embedded while ensuring the naturalness and coherence of the original audio signal. , the environmental characteristic coefficient Instead of amplitude modulation, get embedded frequency domain audio data: ; in, is the original frequency domain audio data at frequency The amplitude of is a factor that controls the embedding strength, is the frequency domain coefficient of the characteristic ambient audio. Through modulation, the ambient audio features are embedded into the original audio signal, so that the audio stream can carry the ambient information. The embedded frequency domain audio data is inversely fast Fourier transformed to convert it from the frequency domain back to the time domain to obtain the embedded audio data stream.

[0028] In a specific embodiment, the process of executing step 200 may specifically include the following steps: Segmenting the embedded audio data stream, dividing the embedded audio data stream into time segments of fixed length to obtain an audio segment sequence, and transforming each audio segment in the audio segment sequence to convert the time domain signal into a time-frequency representation to obtain a time-frequency spectrum. Perform energy calculation on the time-frequency spectrum, calculate the energy value of each time-frequency unit by the integration method, obtain the energy distribution matrix, and perform peak detection on each time-frequency unit based on the energy distribution matrix to obtain the peak point set; The coordinates of the peak point set are quantized, and the time-frequency coordinates are mapped to a discrete grid to obtain quantized peak points. Feature descriptors are constructed based on the quantized peak points, and the difference vector is obtained by calculating the time-frequency difference between adjacent peak points. The difference vector is input into the local sensitive hash function, and the hash value is generated through random projection and threshold comparison to obtain a binary feature string. The binary feature strings are grouped and merged, and multiple feature strings are combined into a vector of fixed length through bit operations to obtain an audio feature vector.

[0029] Specifically, the embedded audio data stream is divided into time segments of fixed length to obtain an audio segment sequence. Suppose the audio data stream is , and divide it into segments to obtain a length of Time fragment: ; in, Indicates audio clips, is the length of each segment, is a time variable. The entire audio data stream is divided into Each segment contains a part of the audio information. Each audio segment is transformed from a time domain signal to a time-frequency representation to obtain a time-frequency spectrum. Through the short-time Fourier transform, the local characteristics of the signal are provided simultaneously in time and frequency: ; in, Indicates Audio clips at frequency and time The time-frequency representation of is the input signal, is a window function, usually a Hanning window or a Gaussian window, which is used to limit the local range of the signal in time. Represents an imaginary unit. This conversion reveals both the time characteristics and frequency characteristics of the time domain signal to form a time-frequency spectrum. Energy calculation is performed on the time-frequency spectrum, and the energy value of each time-frequency unit is calculated by the integration method to obtain an energy distribution matrix. The purpose of energy calculation is to determine the intensity of the audio signal at different times and frequencies. The specific formula is: ; in, Indicates at time and frequency The energy on In this way, the energy distribution matrix is ​​obtained: , describing the energy of each small unit of the signal in the time-frequency space. Based on the energy distribution matrix, peak detection is performed to determine at which time and frequency the audio signal has significant characteristics. By comparing each time-frequency unit, the local energy peak is found and a set of peak points is obtained, which represents the important frequency components and events in the signal. The peak point set is quantized and its time-frequency coordinates are mapped to a discrete grid to obtain the quantized peak point. Assume that the coordinates of the peak point are , by changing the frequency and time Quantized into discrete grid coordinates: ; in, and are the quantized frequency and time coordinates, and are the quantization steps of frequency and time respectively, and round represents the rounding operation. The purpose of quantization is to map continuous time-frequency coordinates to discrete grid points, reduce the amount of data and improve the matching ability of features. The feature descriptor is constructed based on the quantized peak points, and the difference vector is obtained by calculating the time-frequency difference between adjacent peak points. The difference vector is used to describe the relationship between adjacent important events in the signal, thereby capturing the global structural characteristics of the signal. Suppose the coordinates of two adjacent peak points are and , then the difference vector between them is expressed as: ; The difference vector describes the change pattern of the audio signal in the time-frequency space and is important information for feature matching and synchronization. In order to improve the operability and compactness of audio features, the difference vector is input into the local sensitive hash function, and the hash value is generated by random projection and threshold comparison to obtain a binary feature string. The role of local sensitive hashing is to accelerate similarity search by reducing the dimension of high-dimensional data while maintaining the similarity between similar data. With a set of random vectors Perform the inner product and compare it with the threshold to generate a binary hash value: ; in, Represents the generated binary hash value, is a random vector, and · represents the inner product operation. In this way, the difference vector is mapped into a binary string of fixed length. The obtained binary feature strings are grouped and merged, and multiple feature strings are combined into a vector of fixed length through bit operations to obtain an audio feature vector.

[0030] In a specific embodiment, the process of executing step 300 may specifically include the following steps: Performing time series decomposition on the audio feature vector, by resampling the audio feature vector according to different time scales, generating multiple subsequences with different resolutions, each subsequence containing information of the audio feature vector at a preset time scale, and obtaining a multi-scale feature sequence; Based on the multi-scale feature sequence, the cross-correlation calculation is performed on the feature sequence of each pair of conference terminal devices, specifically: a fixed-size time window is selected, and the time window is simultaneously slid on the multi-scale feature sequences of the two conference terminal devices, the Pearson correlation coefficient is calculated for the data in the time window, and the correlation coefficient values ​​under different time offsets are recorded to form a cross-correlation function curve; Perform peak detection on the cross-correlation function curve, by searching for local maximum points on the cross-correlation function curve, taking two points on the left and right of each local maximum point, fitting a quadratic curve using the data of these five points, calculating the vertex position of the quadratic curve as the precise peak position, and obtaining the initial time delay candidate value; A multi-scale time delay matrix is ​​constructed based on the initial time delay candidate values, and the time delay candidate values ​​obtained at different time scales are organized into a matrix form. Each row of the multi-scale time delay matrix represents a time scale, and each column represents the delay between a pair of devices. Then, each column in the multi-scale time delay matrix is ​​weighted averaged to obtain a fused time delay value. Perform median filtering on the fused time delay value, set a sliding window of odd size, slide the sliding window on the time delay sequence, take all the values ​​in the sliding window each time, sort them by size, and select the middle value as the output of the current position to obtain a smooth time delay sequence; A time delay graph is constructed based on the smoothed time delay sequence. Each conference terminal device is regarded as a node in the graph, and the time delay between conference terminal devices is used as the weight of the edge. Then, the minimum spanning tree is constructed using the Kruskal algorithm. The edge set with the minimum total delay is selected to obtain the topological structure between devices. Perform global optimization on the topological structure between devices, establish a relative time coordinate system with the reference device as the origin, regard the time delay between each pair of adjacent devices as a constraint, construct a least squares optimization problem, solve the least squares optimization problem to obtain the position of each conference terminal device in the unified time coordinate system, and obtain the consistent time coordinate; Based on the consistent time coordinate, the position of each conference terminal device in the unified time coordinate system is mapped to a two-dimensional plane. The distance between the conference terminal devices reflects their time synchronization relationship. Then, the force-directed algorithm is used to adjust the node position to obtain a global time relationship graph.

[0031] Specifically, the audio feature vector is decomposed into time series, and multiple subsequences with different resolutions are generated by resampling the audio feature vector according to different time scales. Multi-scale analysis can effectively capture the characteristics of the signal at different time scales, thereby improving the robustness and accuracy of audio synchronization. Suppose the original audio feature vector is , by resampling it, a multi-scale feature sequence is generated ,in Indicates different time scales, for example, the time scales from short to long are . Through decomposition, each subsequence contains the information of the audio feature vector at a preset time scale. The obtained multi-scale feature sequence can characterize the dynamic characteristics of the audio signal at each time level. Based on the multi-scale feature sequence, the feature sequence of each pair of conference terminal devices is cross-correlated to determine the time synchronization deviation between the devices. A fixed-size time window is selected for each pair of devices. , slide the time window simultaneously on the multi-scale feature sequences of the two conference terminal devices, and calculate the Pearson correlation coefficient in each time window. The calculation formula is as follows: ; in, and are the feature vector values ​​of the two devices in the time window, and is the mean of the corresponding eigenvector, is the number of samples in the time window. In this way, the correlation coefficients of the two feature sequences are calculated at different time offsets to obtain the cross-correlation function curve. The cross-correlation function curve describes the similarity of the signals between the two devices at different time offsets, and the peak represents the best time synchronization point. In order to accurately determine the peak position, the cross-correlation function curve is peak detected, and the local maximum point is searched on the cross-correlation function curve. Two points are taken on the left and right of each local maximum point, a total of five points of data, and these points are used to fit a quadratic curve. The fitted quadratic curve is expressed as: ; in, , and are the coefficients of the quadratic curve, is the time offset, is the cross-correlation value. The exact position of the peak is obtained by solving the vertex position of the quadratic curve. The formula is: ; Through the formula, the precise time offset position of each local peak is obtained, thereby determining the candidate value of the initial time delay. Based on the candidate value of the initial time delay, a multi-scale time delay matrix is ​​constructed. The candidate values ​​of the time delay obtained at different time scales are organized into a matrix form, where each row of the matrix represents a time scale and each column represents the delay between a pair of devices. Let the time delay matrix be , whose elements Indicated in The time scale For the time delay between devices. In order to obtain a fused time delay value, a weighted average is performed on each column in the time delay matrix to obtain the fused time delay value: ; in, For the Delay in fusion time of the device, For the The weights of the time scales are set to better balance the contributions of different time scales in synchronization calculations through appropriate weight selection. Median filtering is performed on the fused time delay value to eliminate abnormal delays introduced by network fluctuations or other interference. Set an odd-sized sliding window , slide the sliding window on the time delay sequence, take all the values ​​in the sliding window each time and sort them by size, then select the middle value as the output of the current position to obtain a smooth time delay sequence. The median filtering method can effectively remove burst noise, making the time delay sequence smoother and more stable. Based on the smoothed time delay sequence, a time delay graph is constructed. Each conference terminal device is regarded as a node in the graph, and the time delay between devices is used as the weight of the edge between the nodes in the graph. Then use the Kruskal algorithm to construct a minimum spanning tree, from which the edge set with the minimum total delay is selected to obtain the topological structure between devices. The construction of the minimum spanning tree ensures that the lowest delay connection is established between each device to achieve global time synchronization. The topological structure between devices is globally optimized, and a relative time coordinate system with the reference device as the origin is established. The time delay between each pair of adjacent devices is regarded as a constraint condition, and a least squares optimization problem is constructed. Assume that the time position of each device is , the goal is to minimize the following objective function: ; in, To optimize the objective function, Indicates the device and equipment There is an edge between them. is the time delay between the two devices. By solving the least squares optimization problem, the relative position of each device in the unified time coordinate system is obtained, and the consistent time coordinate is obtained. Based on the consistent time coordinate, the position of each conference terminal device in the unified time coordinate system is mapped to a two-dimensional plane. This mapping reflects the time synchronization relationship between each device. In order to optimize the distribution of nodes on the plane, the force-directed algorithm is used to adjust the position of the nodes. The force-directed algorithm simulates the attraction and repulsion between nodes to make the node distribution more balanced, while maintaining the relative distance between each node, and obtain a global time relationship diagram.

[0032] In a specific embodiment, the process of executing step 400 may specifically include the following steps: Based on the global time relationship graph, cache space is allocated to each conference terminal device, and the cache space is divided into two layers: fast cache and deep cache. The fast cache is used to store the most recently received audio data, and the deep cache is used to store the processed historical audio data, thus obtaining a two-layer cache structure. Perform timestamp analysis on the data in the two-layer cache structure, and obtain the time series of audio data arrival by calculating the time interval between adjacent audio frames. Based on the time series of audio data arrival, build a state space model of the Kalman filter, take the arrival time of audio data as the observation value, and the network transmission delay as the state variable, and obtain the network state prediction model. Initialize the parameters of the network state prediction model, calculate the initial state estimate and error covariance matrix through historical data, obtain the initial parameters of the Kalman filter, and based on the initial parameters of the Kalman filter, predict and update the state of the newly arrived audio data, predict the network state at the next moment through the time update equation, and then use the measurement update equation to correct the prediction result to obtain the optimal estimate of the current network state; Based on the optimal estimate of the network status, the ideal residence time of the audio data in the cache is calculated, and the cache adjustment target value is obtained by subtracting the predicted network transmission delay from the target playback delay; Based on the cache adjustment target value, the data in the two-layer cache structure is reorganized, and the data allocation ratio between the fast cache and the deep cache is adjusted to delay the processing of the advanced audio data and accelerate the processing of the lagging audio data to obtain the cache data distribution; The cache data distribution is evaluated for performance. The performance indicators of the buffer strategy are obtained by calculating the buffer utilization, packet loss rate and audio quality indicators. The buffer strategy is adjusted according to the performance indicators, and finally a buffer strategy that adapts to the current network status is obtained.

[0033] Specifically, cache space is allocated to each conference terminal device and divided into a two-layer cache structure, namely a fast cache and a deep cache. The fast cache is used to store recently received audio data, which are the latest audio packets that have not yet been deeply processed. The role of the fast cache is to achieve low-latency instant response to ensure real-time playback of audio data. Relatively speaking, the deep cache is used to store historical audio data that has been further processed to ensure the continuity and overall quality of the entire audio stream. Through the two-layer cache structure, a balance can be found between real-time and stability. The data in the two-layer cache structure is timestamped. By recording the arrival time of each audio frame, the time interval between adjacent audio frames is calculated to obtain the time series of audio data arrival. Suppose the audio frame arrival time series is ,in Indicates The arrival time of audio frames, the time interval It is expressed as: ; By calculating the time interval, we can understand the delay and jitter characteristics of audio data in network transmission. Based on the time series of audio data arrival, we build a state space model of the Kalman filter, taking the arrival time of the audio data as the observation value and the network transmission delay as the state variable to obtain a network state prediction model. The state space model of the Kalman filter is described as: ; ; in, Indicates the state of the system, that is, the network transmission delay, is the state transition matrix, describing the evolution of delay over time, is the process noise, is the observed value, i.e. the arrival time of the audio data, is the observation matrix, is the observation noise. Through the state space model, the transmission state of the network is effectively modeled and predicted. In order to enable the Kalman filter to accurately predict the network state, its parameters are initialized, and the initial state estimate and error covariance matrix are calculated. Initial state estimate Estimation is based on historical data, and the error covariance matrix Used to describe the uncertainty of the initial state. Assume that the historical arrival time of the audio data is , the initial state estimate is expressed as: ; in, is the number of historical data samples, Indicates The arrival time of historical samples, is the mean of historical samples. Through the initialization method, the Kalman filter better reflects the initial conditions of the network state. After initialization, the Kalman filter enters the prediction and update stage to predict and update the state of the newly arrived audio data. The network state at the next moment is predicted through the time update equation: ; The prediction result is then corrected using the measurement update equation to obtain the optimal estimate of the current network state: ; in, is the Kalman gain, which determines the degree of influence of the observation value on the state estimation correction. Through the iterative prediction and update process, the Kalman filter gradually obtains an accurate estimate of the network transmission delay. Based on the obtained optimal estimate of the network state, the ideal residence time of the audio data in the cache is calculated. The predicted network transmission delay is compared with the target playback delay. Let the target playback delay be , and the network delay predicted by the Kalman filter is , then the cache adjustment target value It is expressed as: ; Cache adjustment target value Indicates the amount of adjustment that should be made to the audio data in the cache to ensure the smoothness and synchronization of audio playback. Based on the adjustment target value, the data in the two-layer cache structure is reorganized by adjusting the data allocation ratio between the fast cache and the deep cache. For example, for audio data that arrives ahead of time, it is stored in the deep cache and delayed appropriately, while for audio data that arrives late, its processing is accelerated and it is directly stored in the fast cache to minimize the impact of delay on audio quality. By reasonably adjusting the allocation between the fast cache and the deep cache, the optimized distribution of cached data is achieved, thereby ensuring the synchronization and stability of audio under different network conditions. The performance of the cached data distribution is evaluated, and the effectiveness of the current buffering strategy is measured by calculating the buffer utilization, packet loss rate, and audio quality indicators. Buffer utilization Defined as: ; in, Indicates the used cache space. is the total cache capacity. By evaluating the buffer utilization, we can understand the usage of cache resources and determine whether the cache strategy needs to be adjusted. Packet loss rate It is used to reflect the audio data loss caused by insufficient buffer or unstable transmission during the network transmission process. Audio quality indicators such as signal-to-noise ratio directly measure the final audio output quality. In the performance evaluation, the current buffering strategy is adaptively adjusted according to the obtained performance indicators. When the buffer utilization is high and the packet loss rate increases, the capacity of the deep buffer is increased to store more historical audio data, thereby reducing packet loss. When the audio quality indicator decreases, the ratio of fast cache to deep cache is adjusted to ensure that key audio frames can be processed and played in a timely manner. Through dynamic adjustment, a buffering strategy that adapts to the current network status is finally obtained.

[0034] In a specific embodiment, the process of executing step 500 may specifically include the following steps: The buffer strategy is feature extracted to obtain the buffer state feature vector, and a reinforcement learning environment is constructed based on the buffer state feature vector. Each conference terminal device is regarded as an intelligent agent, the buffer state feature vector is used as the environment state, and the action space is set as the selectable communication frequency band and transmission power to obtain a multi-agent learning environment. Design a reward function for the multi-agent learning environment. By comprehensively considering audio synchronization, transmission delay, and channel utilization, a weighted sum reward function is constructed to obtain the optimization goal of reinforcement learning. A dual-depth Q network model is constructed based on the optimization objective of reinforcement learning, including an online network and a target network. The two networks have the same structure and consist of multiple fully connected layers. The input is the state vector and the output is the Q value of each action. The initialized dual-depth Q network is obtained. Initialize the parameters of the initialized dual-depth Q network, generate an experience replay buffer through a random sampling method, store the experience data of the interaction between the agent and the environment, and obtain an experience pool; The dual-depth Q network is trained based on the experience pool. By randomly extracting batch data from the experience pool, the mean square error between the target Q value and the current Q value is calculated as the loss function, and the back propagation algorithm is used to update the online network parameters to obtain the trained dual-depth Q network. The trained dual-depth Q network is used to select actions, and the ε-greedy strategy is used to strike a balance between exploration and utilization. The optimal communication frequency band and transmission power are selected for each agent to obtain the initial allocation strategy. The initial allocation strategy is globally coordinated by building a centralized resource allocator to collect the state information and action choices of all agents, resolve possible resource conflicts, and obtain a resource allocation strategy.

[0035] Specifically, the buffer strategy is feature extracted to obtain a buffer state feature vector. The buffer state feature vector describes the current state of the system, including information such as the cache utilization of each device, the delay and arrival of audio data, etc. Suppose the buffer state feature of each device is ,in Indicates the device number, feature vector Contains the following information: Cache utilization ,Delay , channel conditions By extracting these features, a complete feature vector describing the current state of each conference terminal device is obtained. Based on the buffer state feature vector, a reinforcement learning environment is constructed. In the reinforcement learning environment, each conference terminal device is regarded as an intelligent agent, and the buffer state feature vector is used as the state of the environment. The action space of the intelligent agent is set to the selectable communication frequency band and transmission power. Assume that the communication frequency band is and the transmission power is , then the action It is expressed as: ; By defining the action space, each agent selects the communication frequency band and transmission power suitable for the current network conditions to achieve the best communication effect and improve the stability and quality of audio synchronization, thus obtaining a multi-agent learning environment in which all devices interact as independent agents. In order to guide each agent to choose the appropriate action, a reward function is designed for the multi-agent learning environment. The reward function comprehensively considers several important factors: audio synchronization, transmission delay, and channel utilization. Assume that the synchronization degree is , the transmission delay is , the channel utilization is , then the reward function Expressed as a weighted sum: ; in, , and are the weight coefficients of each factor, which are used to balance the influence of synchronization, delay and channel utilization. Through the design of the reward function, each agent is prompted to ensure the synchronization quality of the audio while reducing the delay as much as possible when choosing actions, while making full use of network channel resources. Based on the above reward function and reinforcement learning objectives, a dual deep Q network (DQN) model is constructed. The dual deep Q network consists of two networks: one is the online network (OnlineNetwork) and the other is the target network (Target Network). The two networks have the same structure and consist of multiple fully connected layers. The input is the state vector and the output is the Q value of each action. The Q value is used to measure the expected benefit of taking an action in the current state. Suppose the input state is , the output of the Q network is for each possible action Q value: ; in, Indicates status Next action The Q value, is the parameter of the network. Through this structure, the dual-depth Q network learns how to select the best communication strategy for each device in different states. After the network is built, the parameters of the initialized dual-depth Q network are initialized. In order to conduct effective training, the experience replay buffer is generated by random sampling to store the experience data of the interaction between the agent and the environment. The experience data includes the current state ,action ,award and the next state . By accumulating these experience data, an experience pool is obtained for subsequent Q network training. The dual-depth Q network is trained based on the experience pool. By randomly extracting a batch of experience data from the experience pool, the mean square error between the target Q value and the current Q value is calculated and used as the loss function. The calculation formula for the target Q value is as follows: ; in, is the target Q value, is the reward for the current action, is the discount factor, Represents the parameters of the target network. The mean square error loss function is: ; By minimizing the loss function , use the back propagation algorithm to update the parameters of the online network, thereby gradually improving the performance of the network and obtaining the trained dual-depth Q network. After the network training is completed, the trained dual-depth Q network is used for action selection. In order to strike a balance between exploration and utilization, the - Greedy strategy. With probability Randomly select actions (exploration), with probability Select the action with the largest current Q value (utilization). With this strategy, each agent selects the optimal communication frequency band and transmission power in most cases, but can also maintain a certain amount of exploration to discover new possible optimal strategies and obtain the initial communication resource allocation strategy. Globally coordinate the initial allocation strategy of each agent to resolve resource conflicts. Build a centralized resource allocator to collect the status information and action selections of all agents, and adjust the resource allocation of each device based on this information. The goal of the centralized resource allocator is to resolve problems such as frequency band conflicts and power allocation overloads to ensure that each device can communicate without interfering with other devices. Through global coordination, an optimized resource allocation strategy is obtained.

[0036] In a specific embodiment, the process of executing step 600 may specifically include the following steps: Based on the resource allocation strategy, the communication parameters of each conference terminal device are quantified. By converting the frequency band selection and transmission power into numerical representation, the communication resource allocation matrix is ​​obtained. The communication resource allocation matrix is ​​then feature extracted. By calculating the channel capacity and transmission delay between conference terminal devices, the network performance feature vector is obtained. Based on the network performance characteristic vector and the global time relationship graph, a global synchronization optimization objective function is constructed. By taking the time synchronization error and network resource utilization as optimization indicators and setting constraints on the global synchronization optimization objective function, a constrained optimization problem is obtained by limiting the maximum allowable synchronization error and the minimum necessary transmission rate. Based on the constrained optimization problem, a Lagrangian function is constructed. By introducing Lagrangian multipliers, the constraints are integrated into the global synchronization optimization objective function to obtain an unconstrained optimization problem. Based on the unconstrained optimization problem, the gradient of the global synchronization optimization objective function is iteratively calculated and the optimization variables are updated to obtain the optimal synchronization parameters. Based on the optimal synchronization parameters, the time axis of the embedded audio data stream is adjusted. By interpolating the audio frames, the audio time axis of each conference terminal device is aligned to a unified reference time axis to obtain a time-aligned audio stream. The content of the time-aligned audio stream is aligned. By analyzing the similarity of the audio feature vectors, repeated or missing audio clips are identified and eliminated, and finally synchronized multi-channel audio is obtained.

[0037] Specifically, based on the resource allocation strategy, the communication parameters of each conference terminal device are quantified. The communication frequency band selection and transmission power of each conference terminal device are converted into numerical form. Assume that the communication frequency band is , the transmission power is , then the quantized communication parameters are represented as a numerical vector: ; in, Indicates The communication parameters of each terminal device are quantified. By quantifying the communication parameters of all devices, a communication resource allocation matrix is ​​constructed. , among which Row represents Communication resource allocation for each device. Feature extraction is performed on the communication resource allocation matrix to calculate the channel capacity and transmission delay between each device. Channel capacity The Shannon formula is used to measure the maximum communication data rate between two devices. The formula is as follows: ; in, represents the channel bandwidth, is the transmit power, is the channel gain, reflecting the device and equipment The signal propagation effect between is the noise power density. In this way, the performance index of communication resource allocation is obtained, and then the transmission delay between conference terminal devices is calculated. , describing the time delay of audio data transmission from one device to another. Based on the channel capacity and transmission delay, the network performance feature vector is obtained to describe the communication status of the entire network. Based on the network performance feature vector and the global time relationship graph, the global synchronization optimization objective function is constructed. The global synchronization optimization objective function uses time synchronization error and network resource utilization as optimization indicators. Assume that the time synchronization error is , the network resource utilization is , then the optimization objective function is expressed as: ; in, Optimize the objective function for global synchronization, and are weight coefficients, which control the relative importance of synchronization error and resource utilization in the optimization. In this way, the utilization efficiency of network resources is improved while ensuring the accuracy of time synchronization. In order to limit the optimization problem, constraints are set for the global synchronization optimization objective function, including limiting the maximum allowable synchronization error and the minimum necessary transmission rate. Assume that the maximum allowable synchronization error is , the minimum necessary transmission rate is , then the constraint condition is expressed as: ; The constraints ensure that the optimization process meets the basic requirements in terms of synchronization error and transmission rate, thus achieving high-quality audio synchronization. Based on the optimization problem with constraints, the Lagrangian function is constructed. By introducing the Lagrangian multiplier, the constraints are integrated into the global synchronization optimization objective function to obtain the unconstrained optimization problem. Let the Lagrangian multiplier be and , then the Lagrangian function It is expressed as: ; in, and is a non-negative Lagrangian multiplier used to penalize the violation of constraints. By solving the unconstrained Lagrangian function, the original complex constrained optimization problem is transformed into an equivalent unconstrained problem, which can be solved using numerical optimization methods such as gradient descent. By iteratively calculating the Lagrangian function, solving its gradient and updating the optimization variable, the optimal synchronization parameter is gradually found. Let the synchronization parameter of the current iteration step be , then the next step of parameter update is expressed as: ; in, is the learning rate, which controls the update amplitude of each step, Represents the gradient of the synchronization parameter. Through multiple iterations, the optimal synchronization parameter that minimizes the Lagrangian function is found. Based on the optimal synchronization parameter, the embedded audio data stream is adjusted in time. In order to synchronize the audio of all conference terminal devices on a unified reference time axis, the audio frames are interpolated to align the audio data in time. By interpolation, the audio time axis of each device is aligned to the reference time axis to obtain a time-aligned audio stream. Assume that the device The audio frame is , and the aligned audio frames are obtained by interpolation: ; in, is the interpolation coefficient, is the time synchronization error. The purpose of the interpolation operation is to synchronize the audio streams of each device at the same time point, thereby avoiding audio misalignment caused by asynchrony. After completing the time alignment, the content of the time-aligned audio streams is aligned to ensure the consistency of the audio content. By analyzing the similarity of the audio feature vectors, the repeated or missing segments in the audio stream are identified. Suppose the feature vectors of the two audio segments are and , then their similarity is calculated by cosine similarity: ; in, represents the similarity between audio clips, represents the vector dot product, Represents the modulus of the vector. By calculating the similarity, duplicate segments are identified and eliminated, or missing segments are completed, so as to finally obtain synchronized multi-channel audio streams.

[0038] The above describes the multi-channel audio synchronization method of the conference terminal device in the embodiment of the present application. The following describes the multi-channel audio synchronization device 10 of the conference terminal device in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a multi-channel audio synchronization device 10 for a conference terminal device includes: The acquisition module 11 is used to acquire the audio data stream of each conference terminal device, and encode the environmental audio information into frequency domain features and embed them into the audio data stream to obtain an embedded audio data stream; A mapping module 12 is used to perform time-frequency domain conversion and peak detection on the embedded audio data stream, and map the peak point to a binary string through a local sensitive hashing algorithm to obtain an audio feature vector; A calculation module 13 is used to calculate the cross-correlation function between terminal devices based on the audio feature vector, determine the time delay through multi-scale analysis and median filtering, and obtain a global time relationship diagram; A construction module 14 is used to construct a two-layer cache structure according to the global time relationship graph, predict the network state through a Kalman filter, adjust the residence time of the audio data in the cache, and obtain a buffering strategy; An allocation module 15 is used to input the buffer strategy as a state into the multi-agent reinforcement learning model, optimize the communication resource allocation through the dual deep Q network, and obtain the resource allocation strategy; The synchronization module 16 is used to construct a global synchronization optimization problem based on the resource allocation strategy, solve the optimal synchronization parameters through the gradient descent algorithm, adjust the time axis and align the content of the embedded audio data stream, and obtain synchronized multi-channel audio.

[0039] Through the collaborative cooperation of the above-mentioned components, by encoding the environmental audio information into frequency domain features and embedding it into the audio data stream, the present invention enhances the robustness of audio synchronization and can better cope with background noise interference in different conference environments. The local sensitive hashing algorithm is used to map the peak point into a binary string, which greatly reduces the storage and transmission overhead of the audio features while maintaining the distinguishability of the features. The introduction of multi-scale analysis and median filtering technology improves the accuracy and stability of time delay estimation and effectively overcomes the influence of network jitter and burst noise. A two-layer cache structure is constructed and combined with the Kalman filter to predict the network state, which realizes the dynamic adjustment of the audio data cache and effectively balances the synchronization accuracy and system delay. The multi-agent reinforcement learning model and the dual-depth Q network are used to optimize the allocation of communication resources, so that the system can adaptively cope with complex and changeable network environments and improve resource utilization efficiency. By constructing a global synchronization optimization problem and solving it using the gradient descent algorithm, accurate time axis adjustment and content alignment of multi-channel audio are achieved, and the synchronization of audio is guaranteed to the greatest extent. The present invention significantly improves the audio quality and user experience of multi-party remote conferences.

[0040] See also Figure 3 , Figure 3 This is a schematic block diagram of the structure of an electronic device 300 provided in an embodiment of the present application. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are connected via a device bus 203, wherein the memory 302 may include a non-volatile storage medium and an internal memory.

[0041] The non-volatile storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 301, the processor 301 can execute any of the above-mentioned multi-channel audio synchronization methods of conference terminal equipment.

[0042] The processor 301 is used to provide computing and control capabilities to support the operation of the entire electronic device 300 .

[0043] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 301, the processor 301 can execute any of the above-mentioned multi-channel audio synchronization methods of the conference terminal device.

[0044] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the electronic device 300 involved in the present application scheme. The specific electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0045] It should be understood that the processor 301 may be a central processing unit (CPU), and the processor 301 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0046] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the electronic device 300 described above can refer to the corresponding process of the aforementioned conference terminal device multi-channel audio synchronization method, which will not be repeated here.

[0047] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the multi-channel audio synchronization method of a conference terminal device provided in an embodiment of the present application.

[0048] The computer-readable storage medium may be an internal storage unit of the electronic device 300 in the aforementioned embodiment, such as a hard disk or memory of the electronic device 300. The computer-readable storage medium may also be an external storage device of the electronic device 300, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the electronic device 300.

[0049] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0050] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0051] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-channel audio synchronization method for conference terminal equipment, characterized in that: The method comprises: The audio data stream of each conference terminal device is collected, and the environmental audio information is encoded into a frequency domain feature and embedded into the audio data stream to obtain an embedded audio data stream; Performing time-frequency domain conversion and peak detection on the embedded audio data stream, mapping the peak point into a binary string by using a local sensitive hashing algorithm, and obtaining an audio feature vector; Calculate the cross-correlation function between terminal devices based on the audio feature vector, determine the time delay through multi-scale analysis and median filtering, and obtain a global time relationship diagram; A two-layer cache structure is constructed according to the global time relationship graph, a network state is predicted by a Kalman filter, and the residence time of the audio data in the cache is adjusted to obtain a buffering strategy; The buffer strategy is input as a state into a multi-agent reinforcement learning model, and the communication resource allocation is optimized through a dual deep Q network to obtain a resource allocation strategy; A global synchronization optimization problem is constructed based on the resource allocation strategy, the optimal synchronization parameters are solved by a gradient descent algorithm, the time axis of the embedded audio data stream is adjusted and the content is aligned to obtain synchronized multi-channel audio.

2. The multi-channel audio synchronization method of conference terminal equipment according to claim 1, characterized in that: The method collects the audio data stream of each conference terminal device, and encodes the ambient audio information into frequency domain features and embeds them into the audio data stream to obtain an embedded audio data stream, including: The sound of the environment of each conference terminal device is collected, and the collected sound signal is preprocessed through a high-pass filter to obtain the environmental background audio, and the spectrum analysis of the environmental background audio is performed to obtain the characteristic environmental audio by calculating the energy concentration and selecting the continuous, stable and characteristic frequency components; Performing discrete cosine transform on the characteristic environment audio, converting the characteristic environment audio into frequency domain space to obtain frequency domain environment audio features, sampling and quantizing the audio input of the conference terminal device, converting the analog signal into a digital signal through an analog-to-digital converter to obtain an audio data stream; Performing frame processing on the audio data stream, dividing the continuous audio data into audio frames of fixed length to obtain framed audio data, and performing fast Fourier transform on the framed audio data to convert the framed audio data into frequency domain space to obtain frequency domain audio data; Based on the frequency domain environmental audio characteristics, the mid-high frequency bands of the frequency domain audio data are modulated, the environmental audio information is embedded into the audio data stream to obtain embedded frequency domain audio data, and the embedded frequency domain audio data is inverse fast Fourier transformed to convert the embedded frequency domain audio data back to the time domain space to obtain an embedded audio data stream.

3. The multi-channel audio synchronization method of conference terminal equipment according to claim 2, characterized in that: The embedded audio data stream is subjected to time-frequency domain conversion and peak detection, and the peak point is mapped into a binary string by a local sensitive hashing algorithm to obtain an audio feature vector, including: Segmenting the embedded audio data stream, dividing the embedded audio data stream into time segments of fixed length to obtain an audio segment sequence, and transforming each audio segment in the audio segment sequence to convert the time domain signal into a time-frequency representation to obtain a time-frequency spectrum; Performing energy calculation on the time-frequency spectrum, calculating the energy value of each time-frequency unit by an integration method to obtain an energy distribution matrix, and performing peak detection on each time-frequency unit based on the energy distribution matrix to obtain a peak point set; Quantizing the coordinates of the peak point set, mapping the time-frequency coordinates to a discrete grid to obtain quantized peak points, constructing feature descriptors based on the quantized peak points, and obtaining a difference vector by calculating the time-frequency difference between adjacent peak points; The difference vector is input into a local sensitive hash function, a hash value is generated through random projection and threshold comparison to obtain a binary feature string, and the binary feature strings are grouped and merged, and multiple feature strings are combined into a vector of fixed length through bit operations to obtain an audio feature vector.

4. The multi-channel audio synchronization method of conference terminal equipment according to claim 3, characterized in that: The method of calculating the cross-correlation function between the terminal devices based on the audio feature vector, determining the time delay through multi-scale analysis and median filtering, and obtaining a global time relationship diagram includes: Performing time series decomposition on the audio feature vector, generating a plurality of subsequences with different resolutions by resampling the audio feature vector according to different time scales, each subsequence containing information of the audio feature vector at a preset time scale, and obtaining a multi-scale feature sequence; Based on the multi-scale feature sequence, a cross-correlation calculation is performed on the feature sequence of each pair of conference terminal devices, specifically: a time window of a fixed size is selected, the time window is simultaneously slid on the multi-scale feature sequences of the two conference terminal devices, the Pearson correlation coefficient is calculated for the data in the time window, and the correlation coefficient values ​​under different time offsets are recorded to form a cross-correlation function curve; Perform peak detection on the cross-correlation function curve, search for local maximum points on the cross-correlation function curve, select two points on the left and right of each local maximum point, use the data of these five points to fit a quadratic curve, calculate the vertex position of the quadratic curve as the precise peak position, and obtain an initial time delay candidate value; Constructing a multi-scale time delay matrix based on the initial time delay candidate value, organizing the time delay candidate values ​​obtained at different time scales into a matrix form, wherein each row of the multi-scale time delay matrix represents a time scale, and each column represents a delay between a pair of devices, and then performing weighted averaging on each column in the multi-scale time delay matrix to obtain a fused time delay value; Perform median filtering on the fused time delay value, set a sliding window of an odd size, slide the sliding window on the time delay sequence, take all the values ​​in the sliding window each time and sort them by size, then select the middle value as the output of the current position, to obtain a smoothed time delay sequence; A time delay graph is constructed based on the smoothed time delay sequence, each conference terminal device is regarded as a node in the graph, the time delay between conference terminal devices is used as the weight of the edge, and then a minimum spanning tree is constructed using the Kruskal algorithm, and the edge set with the minimum total delay is selected to obtain the topological structure between the devices; The topological structure between the devices is globally optimized, a relative time coordinate system with the reference device as the origin is established, the time delay between each pair of adjacent devices is regarded as a constraint condition, a least squares optimization problem is constructed, and the least squares optimization problem is solved to obtain the position of each conference terminal device in the unified time coordinate system to obtain a consistent time coordinate; Based on the consistent time coordinate, the position of each conference terminal device in the unified time coordinate system is mapped to a two-dimensional plane. The distance between the conference terminal devices reflects their time synchronization relationship. Then, the node position is adjusted using a force-directed algorithm to obtain a global time relationship diagram.

5. The multi-channel audio synchronization method of conference terminal equipment according to claim 4, characterized in that: The two-layer cache structure is constructed according to the global time relationship graph, the network state is predicted by the Kalman filter, the residence time of the audio data in the cache is adjusted, and the buffering strategy is obtained, including: Based on the global time relationship graph, cache space is allocated to each conference terminal device, and the cache space is divided into two layers: a fast cache and a deep cache, wherein the fast cache is used to store recently received audio data, and the deep cache is used to store processed historical audio data, thereby obtaining a two-layer cache structure; Performing timestamp analysis on the data in the two-layer cache structure, obtaining a time series of audio data arrival by calculating the time interval between adjacent audio frames, and constructing a state space model of a Kalman filter based on the time series of audio data arrival, taking the arrival time of the audio data as an observation value and the network transmission delay as a state variable, to obtain a network state prediction model; Initializing the parameters of the network state prediction model, calculating the initial state estimate and the error covariance matrix through historical data, obtaining the initial parameters of the Kalman filter, and based on the initial parameters of the Kalman filter, predicting and updating the state of the newly arrived audio data, predicting the network state at the next moment through the time update equation, and then correcting the prediction result using the measurement update equation to obtain the optimal estimate of the current network state; Calculating an ideal residence time of the audio data in the cache based on the optimal estimate of the network state, and obtaining a cache adjustment target value by subtracting the predicted network transmission delay from the target playback delay; Based on the cache adjustment target value, the data in the two-layer cache structure is reorganized, and the data allocation ratio between the fast cache and the deep cache is adjusted to delay the processing of the advanced audio data and accelerate the processing of the lagging audio data to obtain the cache data distribution; A performance evaluation is performed on the cache data distribution, and a performance indicator of the buffer strategy is obtained by calculating the buffer utilization, packet loss rate and audio quality indicator. The buffer strategy is adjusted according to the performance indicator to finally obtain a buffer strategy that adapts to the current network status.

6. The multi-channel audio synchronization method of conference terminal equipment according to claim 5, characterized in that: The buffer strategy is used as a state input into a multi-agent reinforcement learning model, and communication resource allocation is optimized through a dual-depth Q network to obtain a resource allocation strategy, including: Extract features of the buffer strategy to obtain a buffer state feature vector, and build a reinforcement learning environment based on the buffer state feature vector, regard each conference terminal device as an intelligent agent, use the buffer state feature vector as the environment state, set the action space as an optional communication frequency band and transmission power, and obtain a multi-agent learning environment; Designing a reward function for the multi-agent learning environment, constructing a reward function in the form of a weighted sum by comprehensively considering audio synchronization, transmission delay, and channel utilization, and obtaining an optimization target for reinforcement learning; A dual-depth Q network model is constructed based on the optimization objective of the reinforcement learning, including an online network and a target network, the two networks have the same structure, are composed of multiple fully connected layers, the input is a state vector, and the output is a Q value of each action, to obtain an initialized dual-depth Q network; Initializing parameters of the initialized dual-depth Q network, generating an experience replay buffer by a random sampling method, storing experience data of the interaction between the agent and the environment, and obtaining an experience pool; The dual-depth Q network is trained based on the experience pool, batch data is randomly extracted from the experience pool, a mean square error between a target Q value and a current Q value is calculated as a loss function, and an online network parameter is updated using a back propagation algorithm to obtain a trained dual-depth Q network; The trained dual-depth Q network is used to select actions, and an ε-greedy strategy is used to strike a balance between exploration and utilization, and an optimal communication frequency band and transmission power are selected for each intelligent agent to obtain an initial allocation strategy; The initial allocation strategy is globally coordinated, and a centralized resource allocator is constructed to collect the state information and action selections of all agents, resolve possible resource conflicts, and obtain a resource allocation strategy.

7. The multi-channel audio synchronization method of conference terminal equipment according to claim 6, characterized in that: The method of constructing a global synchronization optimization problem based on the resource allocation strategy, solving the optimal synchronization parameters by using a gradient descent algorithm, adjusting the time axis and aligning the content of the embedded audio data stream, and obtaining synchronized multi-channel audio includes: Based on the resource allocation strategy, the communication parameters of each conference terminal device are quantified, and the communication resource allocation matrix is ​​obtained by converting the frequency band selection and the transmission power into numerical representations, and the communication resource allocation matrix is ​​feature extracted, and the network performance feature vector is obtained by calculating the channel capacity and transmission delay between the conference terminal devices; Based on the network performance characteristic vector and the global time relationship graph, a global synchronization optimization objective function is constructed, and by taking the time synchronization error and the network resource utilization rate as optimization indicators, and setting constraints on the global synchronization optimization objective function, a constrained optimization problem is obtained by limiting the maximum allowable synchronization error and the minimum necessary transmission rate; Based on the constrained optimization problem, a Lagrangian function is constructed, and the constraints are integrated into the global synchronization optimization objective function by introducing Lagrangian multipliers to obtain an unconstrained optimization problem. According to the unconstrained optimization problem, the gradient of the global synchronization optimization objective function is iteratively calculated and the optimization variables are updated to obtain the optimal synchronization parameters. Based on the optimal synchronization parameters, the embedded audio data stream is adjusted in time axis, and the audio time axis of each conference terminal device is aligned to a unified reference time axis by interpolating the audio frames to obtain a time-aligned audio stream, and the time-aligned audio stream is content aligned. By analyzing the similarity of audio feature vectors, repeated or missing audio segments are identified and eliminated, and finally synchronized multi-channel audio is obtained.

8. A multi-channel audio synchronization device for conference terminal equipment, characterized in that: The multi-channel audio synchronization device of the conference terminal equipment comprises: A collection module, used to collect the audio data stream of each conference terminal device, and encode the ambient audio information into frequency domain features and embed them into the audio data stream to obtain an embedded audio data stream; A mapping module, used to perform time-frequency domain conversion and peak detection on the embedded audio data stream, and map the peak point into a binary string through a local sensitive hashing algorithm to obtain an audio feature vector; A calculation module, used to calculate the cross-correlation function between terminal devices based on the audio feature vector, determine the time delay through multi-scale analysis and median filtering, and obtain a global time relationship diagram; A construction module is used to construct a two-layer cache structure according to the global time relationship graph, predict the network state through a Kalman filter, adjust the residence time of the audio data in the cache, and obtain a buffering strategy; An allocation module, configured to input the buffer strategy as a state into a multi-agent reinforcement learning model, optimize the communication resource allocation through a dual deep Q network, and obtain a resource allocation strategy; The synchronization module is used to construct a global synchronization optimization problem based on the resource allocation strategy, solve the optimal synchronization parameters through a gradient descent algorithm, adjust the time axis and align the content of the embedded audio data stream, and obtain synchronized multi-channel audio.

9. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instruction in the memory so that the electronic device executes the multi-channel audio synchronization method of a conference terminal device as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, a multi-channel audio synchronization method for a conference terminal device as described in any one of claims 1 to 7 is implemented.

Images