Audio processing system control method and device, and storage medium

Abstract

The invention discloses a control method and device of an audio processing system and a storage medium, and relates to the technical field of audio data processing, and the method comprises the steps: obtaining environment information and device state data collected by a sensor, and inputting the environment information and the device state data into a clock compensation module; the method comprises the following steps: generating a corresponding compensation parameter according to a preset sampling rate, performing a compensation action on an audio clock based on the compensation parameter, determining a source sampling rate corresponding to at least one audio source according to the compensated audio clock, finally detecting the source sampling rate through a sampling rate conversion module, and when the source sampling rate is different from a preset sampling rate, outputting the audio source. And adjusting the source sampling rate based on the preset sampling rate. According to the application, the audio clock is compensated, and the compensated sampling rate is uniformly converted through the sampling rate conversion module, so that clock synchronization among multi-source audio input and sampling rate unification of the multi-source audio input are realized, and the processing efficiency of the audio data is improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio data processing, and in particular to a control method, device and storage medium of an audio processing system. Background Art

[0002] Traditional audio processing solutions usually use fixed sampling rates and clock synchronization mechanisms. When processing audio signals, they mainly rely on preset sampling rates and fixed clock sources. Specifically, it is generally assumed that the sampling rates of all audio sources are the same, and a central clock is used to synchronize all audio processing operations. In practical applications, different audio sources may have different sampling rates, and clock drift of devices is inevitable. Due to the lack of flexible clock compensation and sampling rate conversion mechanisms, traditional solutions make the clocks between multi-source audio inputs asynchronous, and the sampling rates of multi-source audio inputs are not uniform, which increases the burden on the system. This results in low efficiency of traditional solutions when processing audio data.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The present application provides a control method, device and storage medium of an audio processing system, aiming to solve the problem of low efficiency of traditional solutions in processing audio data.

[0005] To achieve the above-mentioned purpose, the present application provides a control method of an audio processing system, which is applied to the audio processing system. The audio processing system includes a clock compensation module and a sampling rate conversion module. The control method of the audio processing system includes the following steps: Obtain environmental information and equipment status data collected by sensors; Inputting the environment information and the device status data into the clock compensation module, generating corresponding compensation parameters, and performing compensation actions on the audio clock based on the compensation parameters; Determine a source sampling rate corresponding to at least one audio source according to the compensated audio clock; The source sampling rate is detected by the sampling rate conversion module, and when the source sampling rate is different from a preset sampling rate, the source sampling rate is adjusted based on the preset sampling rate.

[0006] In one embodiment, before the step of acquiring the environmental information and device status data collected by the sensor, the step further includes: Obtaining an initial timestamp based on the system clock record through the clock compensation module, and determining a time difference based on the current clock value; Determine an audio source sampling rate corresponding to the initial timestamp, and determine the number of audio samples corresponding to the audio source sampling rate according to the time difference; The actual number of samples corresponding to the buffer is obtained, and the actual number of samples is compared with the number of audio samples. If they are different, a corresponding compensation action is performed.

[0007] In one embodiment, the step of performing the corresponding compensation action includes: Determining a ratio of the audio source sampling rate to the actual sampling rate according to the audio source sampling rate and the actual sampling rate corresponding to the actual number of samples; Determining a target reading speed according to the ratio and the number of samples corresponding to the reading speed; The target reading speed is adjusted to the current reading speed.

[0008] In one embodiment, the step of detecting the source sampling rate by the sampling rate conversion module, and when the source sampling rate is different from a preset sampling rate, adjusting the source sampling rate based on the preset sampling rate comprises: Obtaining the preset sampling rate; Comparing the source sampling rate with the preset sampling rate by the sampling rate conversion module; When the source sampling rate is less than the preset sampling rate, determining a proportional relationship according to the source sampling rate and the preset sampling rate; The sample points that need to be inserted into the source sampling rate are determined according to the proportional relationship, and the sample points are inserted into the source sampling rate based on a preset interpolation type.

[0009] In one embodiment, after the step of determining the sample points to be inserted into the source sampling rate according to the proportional relationship and inserting the sample points into the source sampling rate based on a preset interpolation type, the method further includes: When the source sampling rate is greater than the preset sampling rate, determining a proportional relationship between the source sampling rate and the preset sampling rate; Downsampling the source sampling rate according to the proportional relationship In one embodiment, after the step of detecting the source sampling rate by the sampling rate conversion module and adjusting the source sampling rate based on the preset sampling rate when the source sampling rate is different from the preset sampling rate, the step further includes: Detect system resource information through the audio delay control module; When the system resource information is greater than or equal to a preset threshold, setting the reading and writing speed of the audio data to a first preset speed to reduce the reading and writing rhythm of the audio data; When the system resource information is less than the preset threshold, the reading and writing speed of the audio data is set to a second preset speed.

[0010] In one embodiment, after the step of detecting the system resource information by the audio delay control module, the method further includes: Obtaining the pending audio tasks in the pending queue and determining the priority of the pending audio tasks; When the system resource information is greater than or equal to a preset threshold, determining a target audio task whose priority is higher than a preset priority among the audio tasks to be processed; The target audio task is placed in a priority processing position in the queue to be processed.

[0011] In one embodiment, before the step of detecting the system resource information by the audio delay control module, the method further includes: Collecting the audio delay, buffer size and remaining data amount in the audio processing process based on the audio delay control module; Inputting the audio processing information and resource usage information into a predictive read / write scheduling model, and determining a prediction result output by the predictive read / write scheduling model; An audio processing strategy is generated according to the prediction result, and the audio processing process is optimized based on the audio processing strategy.

[0012] In addition, to achieve the above-mentioned purpose, the present application also provides a control device of an audio processing system, the control device of the audio processing system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method of the audio processing system as described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the control method of the audio processing system as described above are implemented.

[0014] The present application provides a control method for an audio processing system, a control device for an audio processing system, and a storage medium. The method obtains environmental information and device status data collected by a sensor, and then inputs the environmental information and the device status data into the clock compensation module to generate corresponding compensation parameters, and performs compensation actions on the audio clock based on the compensation parameters, and then determines the source sampling rate corresponding to at least one audio source according to the compensated audio clock. Finally, the source sampling rate is detected by the sampling rate conversion module. When the source sampling rate is different from the preset sampling rate, the source sampling rate is adjusted based on the preset sampling rate. The present application compensates the audio clock through the clock compensation module, and uniformly converts the compensated sampling rate through the sampling rate conversion module to improve the processing efficiency of audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 A schematic flow chart of a first embodiment of a control method for an audio processing system of the present application; Figure 2 A schematic flow chart of a second embodiment of a control method for an audio processing system of the present application; Figure 3 A schematic flow chart of a third embodiment of a control method of an audio processing system of the present application; Figure 4 This is a schematic diagram of the architecture of the hardware operating environment of the control device of the audio processing system involved in the embodiment of the present application.

[0018] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0021] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of the present application is: obtaining environmental information and device status data collected by sensors; inputting the environmental information and the device status data into the clock compensation module, generating corresponding compensation parameters, and performing compensation actions on the audio clock based on the compensation parameters; determining a source sampling rate corresponding to at least one audio source according to the compensated audio clock; detecting the source sampling rate through the sampling rate conversion module, and when the source sampling rate is different from the preset sampling rate, adjusting the source sampling rate based on the preset sampling rate.

[0023] Traditional audio processing solutions usually use fixed sampling rates and clock synchronization mechanisms. When processing audio signals, they mainly rely on preset sampling rates and fixed clock sources. Specifically, it is generally assumed that the sampling rates of all audio sources are the same, and a central clock is used to synchronize all audio processing operations. In practical applications, different audio sources may have different sampling rates, and clock drift of devices is inevitable. Due to the lack of flexible clock compensation and sampling rate conversion mechanisms, traditional solutions make the clocks between multi-source audio inputs asynchronous, and the sampling rates of multi-source audio inputs are not uniform, which increases the burden on the system. This results in low efficiency of traditional solutions when processing audio data.

[0024] By acquiring environmental information and device status data collected by sensors, and then inputting the environmental information and the device status data into the clock compensation module, corresponding compensation parameters are generated, and compensation actions are performed on the audio clock based on the compensation parameters, and then the source sampling rate corresponding to at least one audio source is determined according to the compensated audio clock, and finally the source sampling rate is detected by the sampling rate conversion module. When the source sampling rate is different from the preset sampling rate, the source sampling rate is adjusted based on the preset sampling rate. The present application compensates the audio clock through the clock compensation module, and uniformly converts the compensated sampling rate through the sampling rate conversion module to improve the processing efficiency of audio data.

[0025] It should be noted that the execution subject of this embodiment may be an audio processing system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a control device of an audio processing system capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the audio processing system as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0026] Embodiment 1 Based on this, the present application embodiment provides a control method for an audio processing system, referring to Figure 1 , Figure 1This is a flow chart of a first embodiment of a control method of an audio processing system of the present application. The control method of the audio processing system includes steps S10 to S40: Step S10: Acquire environmental information and device status data collected by sensors.

[0027] In this embodiment, the processing action is performed by an audio processing system, and the audio processing system includes a clock compensation module and a sampling rate conversion module. The sensor includes a temperature sensor, a humidity sensor, and a pressure sensor, and the sensor is used to detect and collect environmental information and device status data of the audio processor, wherein the audio processing system is deployed in the audio processor. The environmental information includes ambient temperature, humidity, and air pressure, and the device status data includes device load, such as CPU usage, memory occupancy, etc., and also includes device running time.

[0028] After the sensor collects environmental information and device status data, it transmits the data to the audio processing system through the Internet of Things technology, that is, through the sensor network.

[0029] Specifically, the sensor converts the collected physical quantity into a measurable electrical signal, and performs preliminary processing and data formatting through the internal processing unit. The processed data is then prepared for transmission through the communication unit and sent to the network layer. According to the application scenario and requirements, select appropriate wireless communication technologies such as WiFi, Bluetooth, Zigbee, LoRaWAN, NB-IoT, etc. The sensor node then sends the data to the fusion network of the communication network and the Internet in the network layer through the selected wireless communication technology. Receiving devices in the network layer, such as routers and base stations, receive data from sensor nodes and forward it to the audio processing system. The audio processing system receives data from the network layer through the network interface, thereby obtaining environmental information collected by the sensor and device status data.

[0030] Step S20: input the environment information and the device status data into the clock compensation module, generate corresponding compensation parameters, and perform compensation actions on the audio clock based on the compensation parameters.

[0031] In this embodiment, the clock compensation module is a module used to correct the audio clock asynchronization problem in the audio processing system, and can adjust the audio clock according to the input relevant data to reduce the impact of the clock asynchronization. The audio clock refers to the clock signal used to control the sampling, playback and other operations of the audio signal in the audio processing system, and its accuracy directly affects the audio quality. By inputting the environmental information and the device status data into the clock compensation module, the compensation parameters are calculated and generated based on the clock compensation module, and finally the compensation action is performed on the audio clock based on the compensation parameters, that is, the audio clock is adjusted to ensure the synchronization of the audio signal.

[0032] Specifically, the audio processing system inputs the environmental information and device status data collected by the sensor into the clock compensation module. The clock compensation module calculates the clock offset value based on these data. The system adjusts the audio clock according to the clock offset value. For example, assuming that the system detects that the audio clock has an offset, and the calculated offset value is 0.1 seconds. The clock compensation module adjusts the audio clock according to the offset value.

[0033] As an optional implementation method for calculating compensation parameters, compensation parameters can be output based on environmental information and device status data through a preset training model. The preset training model is based on a neural network to learn the impact of external environmental changes on the audio clock. The input of the neural network can be sensor data such as ambient temperature and humidity (these factors may affect the performance of electronic components of the audio device, and thus affect the clock accuracy), as well as device load (such as CPU usage, memory occupancy, etc.) and other information. During the training stage, different environmental conditions and device load conditions are artificially created, and the corresponding audio clock deviation is recorded at the same time, and these data are used as training samples. And when the ambient temperature rises, the crystal oscillator frequency of the audio device may change slightly, resulting in clock deviation. By learning these samples, the neural network establishes a relationship model between the environment and device status and the clock deviation, that is, the preset training model.

[0034] In actual operation, when new environment and device status data are received, the neural network can quickly predict the clock deviation based on the learned model and generate corresponding compensation parameters. In addition, the clock compensation strategy can be optimized by defining a reward mechanism through a reinforcement learning algorithm. The agent (representing the compensation strategy) can take different actions, such as adjusting the clock frequency, changing the buffer size, etc. By constantly trying different action combinations and learning based on reward feedback, the reinforcement learning algorithm can find the optimal compensation strategy.

[0035] As another optional implementation method for generating compensation parameters, calculation can be performed by spline interpolation. First, collect clock offset data at known time points in the database. These data points should be distributed as evenly as possible to ensure the accuracy of the interpolation results. Then select a spline function based on the characteristics and requirements of the data. Common spline functions include linear splines, quadratic splines, and cubic splines, among which cubic spline interpolation has higher smoothness and accuracy, and cubic spline interpolation is usually used as a spline function. Then, a spline curve is constructed using known data points and spline functions. Finally, at the time point that needs to be predicted, the constructed spline curve is used to estimate the clock offset. The calculation of the clock offset is thereby achieved by calculating the function value of the spline curve at that time point.

[0036] Step S30: determining a source sampling rate corresponding to at least one audio source according to the compensated audio clock.

[0037] In this embodiment, the source sampling rate refers to the original sampling rate of the audio source, that is, the sampling frequency of the audio signal when it is collected. The compensated audio clock can be used to determine the source sampling rate of the audio source, and the source sampling rate of the audio source can be determined by the frequency of the audio clock. There is at least one audio source, and by using an ARM chip as an audio processing chip, 36 audio inputs can be supported, including 16 analog, 16 Dante / network, and 4 USB.

[0038] Specifically, the audio processing system analyzes the spectrum of the audio signal according to the compensated audio clock to determine the source sampling rate. The frequency components of the audio signal can be analyzed using techniques such as Fourier transform to determine the source sampling rate. Audio signals are usually continuous analog signals. In order to be processed in a computer, they need to be sampled and converted into discrete signals. The selection of the sampling frequency is crucial and must satisfy the Nyquist sampling theorem, that is, the sampling frequency should be greater than twice the highest frequency of the signal to avoid aliasing. The sampled signal is preprocessed, such as removing the DC component and normalizing, to improve the accuracy of subsequent analysis. The audio signal in the time domain is converted into a frequency domain representation using Fourier transform technology. This step is the core of spectrum analysis, which allows us to observe the energy distribution of the signal at different frequencies. The frequency domain signal after Fourier transform is analyzed to identify the individual frequency components in the signal. These components are usually represented in complex form and contain amplitude and phase information. By analyzing the peaks, valleys and other features in the spectrum graph, the frequency components, harmonic structure and so on of the signal can be understood. Based on the results of spectrum analysis, combined with known audio signal processing algorithms and features, the original sampling rate can be inferred.

[0039] Step S40: detecting the source sampling rate through the sampling rate conversion module, and when the source sampling rate is different from the preset sampling rate, adjusting the source sampling rate based on the preset sampling rate.

[0040] In this embodiment, the sampling rate conversion module is used to convert the audio signal from one sampling rate to another sampling rate. Through interpolation and filtering technology, the quality of the audio signal can be ensured during the conversion process. Specifically, the sampling rate can be converted by sinc interpolation, anti-aliasing filter, etc. The audio processing system detects the source sampling rate of the audio source through the sampling rate conversion module. If the source sampling rate is different from the preset sampling rate, the audio processing system uses the sampling rate conversion module to make adjustments.

[0041] Specifically, digital signal processing algorithms such as interpolation and filtering are used according to the preset sampling rate. When the source sampling rate is lower than the preset sampling rate, the sample points are supplemented by interpolation to improve the audio resolution; when it is higher than the preset sampling rate, the sampling is reasonably downsampled and the redundant high-frequency components are filtered out to avoid aliasing, so that all audio streams flow at a uniform sampling rate.

[0042] Optionally, in this embodiment, the step of detecting the source sampling rate by the sampling rate conversion module, and when the source sampling rate is different from a preset sampling rate, adjusting the source sampling rate based on the preset sampling rate includes: Obtain the preset sampling rate; compare the source sampling rate and the preset sampling rate through the sampling rate conversion module; when the source sampling rate is less than the preset sampling rate, determine the proportional relationship according to the source sampling rate and the preset sampling rate; determine the sample points that need to be inserted into the source sampling rate according to the proportional relationship, and insert the sample points into the source sampling rate based on the preset interpolation type.

[0043] Specifically, the preset sampling rate is a sampling frequency pre-set in the audio processing system, which is determined based on factors such as the system's playback device and audio processing requirements, and is used to ensure uniform processing and high-quality output of audio signals in the system. During the configuration phase of the audio processing system, a suitable sampling rate is determined as the preset sampling rate based on the target application scenario and device performance, and is stored in the system's parameter settings. The sampling rate conversion module first reads the source sampling rate of the audio source and then compares it with the preset sampling rate stored in the audio processing system. This comparison process usually involves a simple numerical comparison to determine the relationship between the two sampling rates, such as whether they are equal, which is higher or lower. When the sampling rate conversion module determines that the source sampling rate is less than the preset sampling rate, the ratio of the two sampling rates is calculated. For example, if the source sampling rate is 32kHz and the preset sampling rate is 44.1kHz, the ratio relationship is 44.1 / 32. This ratio will be used in subsequent sample point insertion operations to ensure that the number and position of the inserted sample points can smoothly transition the source sampling rate to the preset sampling rate.

[0044] Interpolation is a mathematical method used to estimate the value of unknown data points between known data points. The preset interpolation type refers to the interpolation algorithm pre-selected during the sampling rate conversion process, such as linear interpolation, spline interpolation, etc., which is used to generate new sample points. According to the proportional relationship calculated in the above steps, the number of sample points that need to be inserted between each sampling point of the source sampling rate is determined. Then, the preset interpolation type (such as linear interpolation) is used to calculate the values ​​of these new sample points. For example, if the proportional relationship is 1.378125, it may be necessary to insert 0.378125 sample points between every two source sampling points, and calculate the specific values ​​of these new sample points through the interpolation algorithm, thereby completing the insertion of the sample points into the source sampling rate based on the preset interpolation type.

[0045] Further, in this embodiment, after the step of determining the sample points to be inserted into the source sampling rate according to the proportional relationship and inserting the sample points into the source sampling rate based on a preset interpolation type, the method further includes: When the source sampling rate is greater than the preset sampling rate, a proportional relationship between the source sampling rate and the preset sampling rate is determined; and the source sampling rate is downsampled according to the proportional relationship.

[0046] Specifically, when the sampling rate conversion module detects that the source sampling rate is higher than the preset sampling rate, the ratio of the two sampling rates is calculated. This ratio will be used in subsequent downsampling operations to determine how many sample points need to be discarded or how to adjust the interval between sample points. Downsampling refers to the process of reducing the sampling rate of an audio signal, usually by discarding certain sample points or using filters to reduce the number of sample points while trying to maintain the quality of the audio signal. According to the determined proportional relationship, the sampling rate conversion module will downsample the source sampling rate. If the proportional relationship is an integer, some sample points can be directly discarded every certain number of sample points; if the proportional relationship is a decimal, filtering and interpolation techniques are used to smoothly reduce the sample points.

[0047] In addition, the quality of the audio signal needs to be considered. Low-pass filters are usually used to prevent aliasing effects and ensure that the downsampled audio signal still maintains good quality. For example, in the downsampling process from 96kHz to 48kHz, the sampling rate conversion module will first apply a low-pass filter with a cutoff frequency set at 24kHz (half of 48kHz) to filter out frequency components above 24kHz, and then discard the sample points. This can effectively reduce the audio distortion that may occur during the downsampling process.

[0048] Optionally, for speech audio sources, since the frequency range of speech signals is relatively narrow, more attention can be paid to preserving the clarity and intelligibility of speech when performing sampling rate conversion. When mapping a speech audio source from a lower sampling rate to a target sampling rate, a linear interpolation method can be used to fill in new sampling points to avoid excessive harmonic distortion. If the target sampling rate is an integer multiple of the source sampling rate, a new sampling point can be simply inserted between every two original sampling points, and its value can be calculated by linear interpolation. At the same time, since speech signals are sensitive to phase changes, care should be taken to maintain the phase continuity of the speech signal during the sampling rate conversion process to ensure the naturalness of the speech.

[0049] Optionally, the music audio source has rich harmonic components, and when performing sampling rate conversion, more attention needs to be paid to preserving the timbre and rhythm of the music. Using a high-quality sinc-function-based interpolation algorithm, new sampling points can be calculated by weighted summing the original audio signal, where the weights are determined by the sinc-function. This algorithm can better handle high-frequency components in the music signal and reduce the audio quality degradation caused by sampling rate conversion, especially for instrumental music with complex timbre.

[0050] In the technical solution provided in this embodiment, by acquiring environmental information and device status data collected by sensors, the environmental information and device status data are then input into the clock compensation module to generate corresponding compensation parameters, and based on the compensation parameters, a compensation action is performed on the audio clock, and then the source sampling rate corresponding to at least one audio source is determined according to the compensated audio clock, and finally the source sampling rate is detected by the sampling rate conversion module, and when the source sampling rate is different from the preset sampling rate, the source sampling rate is adjusted based on the preset sampling rate. The solution of this embodiment compensates the audio clock through the clock compensation module, and uniformly converts the compensated sampling rate through the sampling rate conversion module, so that the clocks between multiple source audio inputs are synchronized and the sampling rates of multiple source audio inputs are unified, so as to improve the processing efficiency of audio data.

[0051] Embodiment 2 Please refer to Figure 2 In the second embodiment, before the step S10, steps S50 to S70 are also included: Step S50: obtaining an initial timestamp based on the system clock record through the clock compensation module, and determining the time difference according to the current clock value.

[0052] In this embodiment, the initial timestamp refers to a time reference point recorded according to the system clock at the beginning of audio processing, which is used for subsequent time calculation and synchronization. The time difference refers to the time interval between the current clock value and the initial timestamp. At the beginning of audio processing, the clock compensation module obtains the current value of the system clock and records it as the initial timestamp. During the audio processing process, the clock compensation module continuously detects the change of the system clock, obtains the current clock value, and determines the time difference based on the current clock value and the initial timestamp. This time difference will be used for the subsequent calculation of the number of audio samples and the execution of compensation actions.

[0053] Step S60: determining the audio source sampling rate corresponding to the initial timestamp, and determining the number of audio samples corresponding to the audio source sampling rate according to the time difference.

[0054] In this embodiment, the audio source sampling rate refers to the sampling frequency used by the audio source (such as an audio file, microphone input, etc.) when collecting or storing audio signals, which determines the number of sampling points of the audio signal per unit time. The number of audio samples refers to the number of sampling points of the audio signal calculated according to the audio source sampling rate within a given time interval. This parameter is very important for audio playback, processing and synchronization.

[0055] Specifically, the sampling rate of the audio source at that time is determined based on the initial timestamp. This is usually a parameter that is already known at the beginning of audio processing, or can be obtained through the metadata of the audio file. Using the calculated time difference and the audio source sampling rate, the number of audio samples that should theoretically be generated within the time difference is calculated using the formula "number of audio samples = time difference × audio source sampling rate".

[0056] Step S70: obtaining the actual number of samples corresponding to the buffer, comparing the actual number of samples with the number of audio samples, and performing corresponding compensation actions if they are different.

[0057] In this embodiment, the buffer refers to an area in the audio processing system for temporarily storing audio data, which plays a role of buffering and synchronization in the process of reading, processing and playing audio. The actual number of samples refers to the number of audio samples actually stored in the buffer, which may be inconsistent with the theoretically calculated number of audio samples due to factors such as clock deviation and processing delay. Compensation action refers to a series of adjustment measures taken to correct problems such as clock deviation or inconsistent number of samples in the audio processing process, such as inserting or deleting sample points, adjusting the playback speed, etc., to ensure accurate playback and synchronization of audio.

[0058] Get the current number of audio samples actually stored in the buffer of the audio processing system. Compare the actual number of samples with the calculated theoretical number of audio samples. If the two are different, perform corresponding compensation actions according to the difference. If the actual number of samples is less than the theoretical number of samples, you need to insert sample points to supplement; if the actual number of samples is more than the theoretical number of samples, you need to delete sample points or adjust the playback speed to keep the audio playback consistent with the theoretical timing.

[0059] Optionally, use the system's high-precision clock (CLOCK_MONOTONIC) to record the initial timestamp as the "time zero" of the audio source. Subsequently, the current clock value is obtained again at regular intervals, and the elapsed time is calculated by the time difference. Based on the known sampling rate of the audio source, the lost time is converted into the number of audio samples that should be collected in theory. Compared with the number of samples in the actual buffer, if there is a deviation (due to factors such as differences in device clock accuracy and transmission delay fluctuations), the index, write position, etc. of the subsequent data processing links are dynamically adjusted to ensure that the audio streams are aligned in the time dimension and compensate for the time misalignment caused by hardware differences.

[0060] Optionally, in this embodiment, the step of performing a corresponding compensation action includes: Determine a ratio of the audio source sampling rate to the actual sampling rate according to the audio source sampling rate and the actual sampling rate corresponding to the actual number of samples; determine a target reading speed according to the ratio and the number of samples corresponding to the reading speed; and adjust the target reading speed to the current reading speed.

[0061] Specifically, according to the audio source sampling rate and the actual sampling rate corresponding to the actual number of samples, the ratio of the audio source sampling rate to the actual sampling rate is determined, and then according to the ratio and the number of samples corresponding to the reading speed, the target reading speed is determined, and the target reading speed is adjusted to the current reading speed.

[0062] For example, the characteristic sampling rate of the audio source is detected in real time through an intelligent algorithm. When the system receives an audio source with a sampling rate of 48kHz, the algorithm will record this parameter. If the sampling rate of the audio source is later changed to 44.1kHz, the algorithm can calculate the compensation parameter according to the pre-set rules. The reading speed of the audio buffer is adjusted by calculating the ratio of the two (44.1 / 48). If the original reading speed is to read 100 audio samples per second, it is now adjusted to read 100*(44.1 / 48)≈92 audio samples per second to achieve audio clock synchronization.

[0063] In the technical solution provided in this embodiment, the clock compensation module obtains the initial timestamp recorded based on the system clock, determines the time difference according to the current clock value, and then determines the audio source sampling rate corresponding to the initial timestamp, and determines the number of audio samples corresponding to the audio source sampling rate according to the time difference, and then obtains the actual number of samples corresponding to the buffer, and compares the actual number of samples with the number of audio samples. If they are different, the corresponding compensation action is performed. The solution of this embodiment determines the time difference based on the initial timestamp recorded by the system clock through the clock compensation module, and then compensates the audio clock according to the determined number of audio samples to achieve synchronization of the audio clock.

[0064] Embodiment 3 Please refer to Figure 3 In the third embodiment, after the step of detecting the source sampling rate by the sampling rate conversion module and adjusting the source sampling rate based on the preset sampling rate when the source sampling rate is different from the preset sampling rate, the method further includes steps S80 to S100: Step S80: Detecting system resource information through the audio delay control module.

[0065] In this embodiment, the audio delay control module is used to manage and control the audio signal delay, can detect the usage of system resources, and adjust the audio processing parameters according to the resource status to optimize the audio performance and system stability. System resource information refers to the resource data available in the computer or audio processing system, such as CPU usage, memory usage, disk I / O speed, etc., which reflects the current operating status and processing capacity of the system.

[0066] Specifically, the audio delay control module is started and initialized to prepare to detect system resources. The system resource information such as CPU usage, memory usage, disk I / O speed, etc. is obtained from the system resource manager or related interface in real time. The obtained system resource information is stored in an internal data structure.

[0067] You can also build an ALSA (Advanced Linux Sound Architecture) read and write scheduling mechanism that is linked to system load perception, and detect CPU usage, remaining memory, and the busyness of other processes in real time. When system resources are sufficient, increase the ALSA read and write frequency, speed up the speed of audio data in and out, reduce the time that data is accumulated in the buffer, and reduce latency; when the system is busy, intelligently reduce the read and write beats to avoid excessive resource grabbing that causes system freezes and audio interruptions, and optimize latency characteristics while ensuring overall stability. Assign priorities to different application scenarios.

[0068] Step S90: When the system resource information is greater than or equal to a preset threshold, the reading and writing speed of the audio data is set to a first preset speed to reduce the reading and writing rhythm of the audio data.

[0069] In this embodiment, before increasing the read and write speed, the audio processing system will pre-read a certain amount of audio data into the buffer to ensure the smoothness of the audio playback. The preset threshold refers to a preset critical value of the system resource utilization rate. When the system resource information exceeds this value, it indicates that the system may be in a high-load state, and measures need to be taken to reduce the processing speed of the audio data to avoid system overload. The first preset speed refers to a slower read and write speed set to reduce the read and write rhythm of the audio data when the system resources are tight. The purpose is to reduce the occupation of system resources by audio processing and ensure the stable operation of the system. The read and write rhythm of audio data refers to the time interval or frequency of the audio data during the reading and writing process. The faster the read and write rhythm, the larger the amount of audio data processed per unit time, and the higher the requirement for system resources.

[0070] Specifically, the audio delay control module compares the acquired system resource information with a preset threshold. If the system resource information is greater than or equal to the preset threshold, it indicates that the system resources are tight, and the module adjusts the reading and writing speed of the audio data to a first preset speed. By adjusting the reading and writing speed, the reading and writing rhythm of the audio data is reduced, and the occupation of system resources by audio processing is reduced, thereby alleviating the load pressure of the system.

[0071] Step S100: When the system resource information is less than the preset threshold, the reading and writing speed of the audio data is set to a second preset speed.

[0072] In this embodiment, the second preset speed refers to a faster read / write speed set when system resources are sufficient in order to improve audio processing efficiency and performance, with the purpose of making full use of system resources while ensuring stable operation of the system.

[0073] Specifically, the audio delay control module continuously detects the system resource information and compares it with the preset threshold. If the system resource information is less than the preset threshold, it indicates that the system resources are sufficient, and the module adjusts the reading and writing speed of the audio data to the second preset speed. By increasing the reading and writing speed, the processing speed of the audio data is accelerated, the fluency and response speed of the audio playback are improved, and the system resources are fully utilized to improve the overall performance.

[0074] Optionally, in this embodiment, after the step of detecting the system resource information by the audio delay control module, the method further includes: Obtain the pending audio tasks in the pending queue and determine the priority of the pending audio tasks; when the system resource information is greater than or equal to a preset threshold, determine a target audio task whose priority is higher than the preset priority among the pending audio tasks; and place the target audio task in a priority processing position in the pending queue.

[0075] Specifically, the priority is adjusted dynamically according to the real-time status and importance of the application. For example, during a live performance, if a certain audio special effects processing task is found to be critical to the performance effect and the task begins to have the risk of audio interruption, the priority of this task can be temporarily increased to ensure that it can obtain resources and complete the processing in time. For high-real-time scenarios, such as audio processing for live performances, the highest priority is assigned. When system resources are tight, the operating system's scheduler will first ensure the execution of high-priority tasks.

[0076] For example, in a system that runs multiple audio applications simultaneously (including live performance audio processing, music playback, and voice recording), the live performance audio processing is given the highest priority. When CPU and memory resources are tight, the scheduler will suspend low-priority tasks such as music playback and voice recording, and prioritize resources for live performance audio processing, ensuring that the live performance audio can be processed and played in real time and with high quality.

[0077] In addition, by utilizing a multi-core processor architecture, independent audio processing subtasks (such as audio filtering of different channels and multi-band sound effect adjustment) are assigned to different cores for parallel execution. At the same time, a pipelined processing flow is designed to allow data processed in the previous link to flow seamlessly into the next link. For example, after audio acquisition, it immediately enters the low-latency ASRC (asynchronous sampling rate conversion) link, and immediately enters the mixing process after the conversion is completed, eliminating idle waiting time throughout the process, minimizing the overall processing delay, and improving the real-time response capability of audio.

[0078] Delay detection probes are implanted in each node of ALSA initialization, reading and writing, and audio processing to capture the time spent on audio data staying and processing at each stage in real time, and aggregate them into a complete delay analysis graph. This can accurately locate the delay bottleneck, whether it is caused by improper device initialization configuration, poor reading and writing scheduling, or slow algorithm processing, and provide data support for targeted optimization.

[0079] Optionally, in this embodiment, before the step of detecting the system resource information by the audio delay control module, the step further includes: Based on the audio delay control module, the audio delay, buffer size and remaining data volume in the audio processing process are collected; the audio processing information and resource usage information are input into a predictive read-write scheduling model, and the prediction result output by the predictive read-write scheduling model is determined; an audio processing strategy is generated according to the prediction result, and the audio processing process is optimized based on the audio processing strategy.

[0080] Specifically, a predictive scheduling model is built based on the historical transmission rules of audio data and the characteristics of application scenarios (such as the periodicity of music rhythm and the intermittent nature of voice calls). The amount of audio data and transmission requirements for the next stage are estimated in advance, and ALSA pre-reading / pre-writing operations are scheduled in advance during low-load periods to prepare audio data in advance and fill potential processing gaps. This ensures that there is sufficient data to respond in a timely manner at moments of high real-time demand, such as when music is played to complex rhythm sections or when multiple parties in a call speak at the same time, effectively controlling the sudden increase in delay.

[0081] Based on the feedback data from the detection tool, an adaptive adjustment mechanism is built. Once the delay of a certain link is found to be excessive, the corresponding optimization strategy is automatically triggered, such as adjusting the ALSA buffer parameters, switching to the lightweight algorithm mode, or rebalancing the read and write scheduling priorities. Continuous closed-loop iterative optimization ensures that the audio delay is always maintained at a controllable and ideal low-latency level in the complex and changeable audio processing system operating environment and various audio application scenarios, achieving efficient and stable system operation and excellent audio experience.

[0082] For example, during an online meeting, various relevant data are collected as model inputs, such as audio delay, buffer size and remaining data volume, device CPU and memory usage, etc. Machine learning algorithms, such as long short-term memory networks (LSTMs), are used to build delay prediction models, learn patterns in historical data, and predict future delay trends. During the training phase, the audio delay, buffer size, and remaining data volume in the past few minutes are used as training samples to allow LSTM to learn the relationship between these data. After the model training is completed, the current audio delay, buffer size, and remaining data volume are input into the model. The model can predict the delay in the future (such as the next 5 seconds) and take measures in advance to ensure audio output.

[0083] In the technical solution provided in this embodiment, the system resource information is detected by the audio delay control module, and when the system resource information is greater than or equal to a preset threshold, the reading and writing speed of the audio data is set to a first preset speed to reduce the reading and writing rhythm of the audio data, and when the system resource information is less than the preset threshold, the reading and writing speed of the audio data is set to a second preset speed, thereby ensuring the high real-time performance of audio signal processing.

[0084] Since the system introduced in the embodiment of the present application is a system used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the embodiment of the present application belong to the scope of protection of this application.

[0085] The present application provides a control device for an audio processing system, the control device for the audio processing system comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the control method for the audio processing system in the above-mentioned embodiment 1.

[0086] Reference below Figure 4 , which shows a schematic diagram of the structure of a control device suitable for implementing the audio processing system of the embodiment of the present application. The control device of the audio processing system in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDA, Personal Digital Assistant), tablet computers (PAD, Portable Application Description), portable multimedia players (PMP, Portable Media Player), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The control device of the audio processing system shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0087] like Figure 4As shown, the control device of the audio processing system may include a processing device 1001 (e.g., a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of the control device of the audio processing system are also stored in the RAM 1004. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the control device of the audio processing system to communicate wirelessly or wired with other devices to exchange data. Although the control device of the audio processing system with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0088] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0089] The control device of the audio processing system provided by the present application adopts the control method of the audio processing system in the above embodiment, which can solve the technical problem of low efficiency of the traditional solution when processing audio data. Compared with the prior art, the beneficial effects of the control device of the audio processing system provided by the present application are the same as the beneficial effects of the control method of the audio processing system provided by the above embodiment, and the other technical features of the control device of the audio processing system are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0090] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0091] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0092] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the control method of the audio processing system in the above-mentioned embodiment.

[0093] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM, CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequencies (RF, Radio Frequency), etc., or any suitable combination of the above.

[0094] The computer-readable storage medium may be included in the control device of the audio processing system; or may exist independently without being assembled into the control device of the audio processing system.

[0095] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the control device of the audio processing system, the control device of the audio processing system: obtains environmental information and device status data collected by sensors; inputs the environmental information and the device status data into the clock compensation module, generates corresponding compensation parameters, and performs compensation actions on the audio clock based on the compensation parameters; determines a source sampling rate corresponding to at least one audio source according to the compensated audio clock; detects the source sampling rate through the sampling rate conversion module, and when the source sampling rate is different from the preset sampling rate, adjusts the source sampling rate based on the preset sampling rate.

[0096] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0099] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the control method of the above-mentioned audio processing system, and can solve the technical problem of low efficiency in processing audio data in the traditional solution. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the control method of the audio processing system provided by the above-mentioned embodiment, and will not be described in detail here.

[0100] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the control method of the audio processing system as described above when the computer program is executed by a processor.

[0101] The computer program product provided by the present application can solve the technical problem that the traditional solution has low efficiency in processing audio data. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the control method of the audio processing system provided by the above embodiment, which will not be repeated here.

[0102] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A control method for an audio processing system, characterized in that: Applied to an audio processing system, the audio processing system includes a clock compensation module and a sampling rate conversion module, and the control method of the audio processing system includes the following steps: Obtain environmental information and equipment status data collected by sensors; Inputting the environment information and the device status data into the clock compensation module, generating corresponding compensation parameters, and performing compensation actions on the audio clock based on the compensation parameters; Determine a source sampling rate corresponding to at least one audio source according to the compensated audio clock; The source sampling rate is detected by the sampling rate conversion module, and when the source sampling rate is different from a preset sampling rate, the source sampling rate is adjusted based on the preset sampling rate.

2. The method according to claim 1, characterized in that Before the step of obtaining the environmental information and device status data collected by the sensor, the method further includes: Obtaining an initial timestamp based on the system clock record through the clock compensation module, and determining a time difference based on the current clock value; Determine an audio source sampling rate corresponding to the initial timestamp, and determine the number of audio samples corresponding to the audio source sampling rate according to the time difference; The actual number of samples corresponding to the buffer is obtained, and the actual number of samples is compared with the number of audio samples. If they are different, a corresponding compensation action is performed.

3. The method according to claim 2, characterized in that The step of performing the corresponding compensation action comprises: Determining a ratio of the audio source sampling rate to the actual sampling rate according to the audio source sampling rate and the actual sampling rate corresponding to the actual number of samples; Determining a target reading speed according to the ratio and the number of samples corresponding to the reading speed; The target reading speed is adjusted to the current reading speed.

4. The method according to claim 1, characterized in that The step of detecting the source sampling rate by the sampling rate conversion module and adjusting the source sampling rate based on the preset sampling rate when the source sampling rate is different from the preset sampling rate comprises: Obtaining the preset sampling rate; Comparing the source sampling rate with the preset sampling rate by the sampling rate conversion module; When the source sampling rate is less than the preset sampling rate, determining a proportional relationship according to the source sampling rate and the preset sampling rate; The sample points that need to be inserted into the source sampling rate are determined according to the proportional relationship, and the sample points are inserted into the source sampling rate based on a preset interpolation type.

5. The method according to claim 4, characterized in that After the step of determining the sample points to be inserted into the source sampling rate according to the proportional relationship and inserting the sample points into the source sampling rate based on a preset interpolation type, the method further includes: When the source sampling rate is greater than the preset sampling rate, determining a proportional relationship between the source sampling rate and the preset sampling rate; The source sampling rate is downsampled according to the proportional relationship.

6. The method according to claim 1, characterized in that After the step of detecting the source sampling rate by the sampling rate conversion module and adjusting the source sampling rate based on the preset sampling rate when the source sampling rate is different from the preset sampling rate, the method further includes: Detect system resource information through the audio delay control module; When the system resource information is greater than or equal to a preset threshold, setting the reading and writing speed of the audio data to a first preset speed to reduce the reading and writing rhythm of the audio data; When the system resource information is less than the preset threshold, the reading and writing speed of the audio data is set to a second preset speed.

7. The method according to claim 6, characterized in that After the step of detecting the system resource information by the audio delay control module, the method further includes: Obtaining the pending audio tasks in the pending queue and determining the priority of the pending audio tasks; When the system resource information is greater than or equal to a preset threshold, determining a target audio task whose priority is higher than a preset priority among the audio tasks to be processed; The target audio task is placed in a priority processing position in the queue to be processed.

8. The method according to claim 6, characterized in that Before the step of detecting the system resource information by the audio delay control module, the method further includes: Collecting the audio delay, buffer size and remaining data amount in the audio processing process based on the audio delay control module; Inputting the audio processing information and resource usage information into a predictive read / write scheduling model, and determining a prediction result output by the predictive read / write scheduling model; An audio processing strategy is generated according to the prediction result, and the audio processing process is optimized based on the audio processing strategy.

9. A control device for an audio processing system, characterized in that: The control device of the audio processing system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the control method of the audio processing system as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the control method of the audio processing system according to any one of claims 1 to 8 are implemented.

Images