Real-time audio processing method and device based on three-core heterogeneous SoC

Abstract

The invention relates to the technical field of data processing, and provides a real-time audio processing method and device based on a three-core heterogeneous SoC, and the method comprises the steps: collecting an original audio signal in real time through collection equipment, carrying out the analysis of the original audio signal through employing a medium-performance core, and obtaining and generating an optimized audio signal and a multi-stage recursion time sequence graph; performing dynamic time sequence adjustment on the optimized audio signal by using a multi-stage recursive time sequence graph to obtain a high-precision audio data frame; and distributing the high-precision audio data frame to the high-performance core, generating a target audio output signal, distributing the multi-stage recursive time sequence atlas to the low-power-consumption core, and generating an audio playing synchronous control signal. Through cooperative work of the high-performance core, the medium-performance core and the low-power-consumption core, the efficiency and precision of audio processing are improved, high efficiency and low delay of real-time audio processing are realized, and the problems of insufficient processing performance, high power consumption, processing delay and the like when complex real-time audio signals are processed are solved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a real-time audio processing method and device based on a triple-core heterogeneous SoC. Background Art

[0002] With the rapid development of the Internet of Things and artificial intelligence technologies, the audio processing requirements of smart devices are becoming more and more complex. In particular, various smart speakers, smart headphones, and other devices with voice interaction functions have increasingly higher requirements for real-time audio processing. These devices not only require high-quality audio output, but also need to have real-time voice recognition and processing capabilities to provide a more intelligent and convenient user experience.

[0003] In the related technical means, traditional single-core or dual-core processors are used for audio processing. These processors usually implement audio processing through steps such as audio signal acquisition, analysis, feature extraction and signal optimization. For example, the audio processor can implement frequency domain analysis of audio signals through fast Fourier transform, thereby extracting audio feature parameters and performing signal optimization and enhancement processing.

[0004] Regarding the above technical solutions, although basic audio processing functions can be implemented through single-core or dual-core processors, there are problems such as insufficient processing performance, high power consumption and processing delay when processing complex real-time audio signals. Summary of the invention

[0005] In order to improve the problems of insufficient processing performance, high power consumption and processing delay when processing complex real-time audio signals, the present application provides a real-time audio processing method and device based on a triple-core heterogeneous SoC.

[0006] The present invention provides a real-time audio processing method based on a three-core heterogeneous SoC, which is applied to a three-core heterogeneous SoC. The three-core heterogeneous SoC includes a high-performance core, a medium-performance core and a low-power consumption core. The audio processing method includes: real-time acquisition of original audio signals by an audio acquisition device, parsing the original audio signals by using the medium-performance core to obtain audio feature parameters and timing information parameters; performing time-frequency domain joint transformation processing on the audio feature parameters to obtain speech feature vectors, performing deep mapping analysis on the speech feature vectors to generate speech mode mapping data; generating an optimized audio signal based on the speech mode mapping data, constructing a multi-level recursive timing map according to the timing information parameters, and dynamically adjusting the timing of the optimized audio signal by using the multi-level recursive timing map to obtain a high-precision audio data frame; allocating the high-precision audio data frame to the high-performance core to generate a target audio output signal, and allocating the multi-level recursive timing map to the low-power consumption core to generate an audio playback synchronization control signal.

[0007] As a preferred solution, the step of using the medium-performance core to parse the original audio signal to obtain audio feature parameters and timing information parameters includes: in the medium-performance core, performing time domain preprocessing on the original audio signal to obtain short-time waveform data and energy distribution data, performing adaptive filtering on the short-time waveform data to obtain noise-suppressed audio data, and calculating signal dynamic range parameters using the noise-suppressed audio data and the energy distribution data; performing logarithmic scaling on the noise-suppressed audio data according to the signal dynamic range parameters to obtain normalized audio feature data, performing short-time Fourier transform on the normalized audio feature data to obtain short-time spectrum data, and extracting audio feature parameters based on the short-time spectrum data; inputting the short-time spectrum data and the signal dynamic range parameters into a timing analysis module, calculating the frame-level energy change trend to obtain timing change data, constructing frame-level timing segments according to the timing change data, and extracting local transition information to obtain timing information parameters.

[0008] As a preferred solution, the steps of performing a joint transformation process in the time-frequency domain on the audio feature parameters to obtain a speech feature vector, performing a deep mapping analysis on the speech feature vector, and generating speech pattern mapping data include: performing a multi-scale discrete wavelet transform on the audio feature parameters to obtain multi-resolution feature data, and calculating a local frequency change rate based on the multi-resolution feature data, constructing a sub-band energy distribution matrix according to the local frequency change rate, and performing a sub-band comparison analysis on the sub-band energy distribution matrix to obtain spectrum enhancement data and non-stationarity compensation data; adjusting the energy weight of the spectrum enhancement data through a time-frequency masking mechanism to obtain weighted speech feature data, and calculating a time alignment parameter using the weighted speech feature data in combination with the non-stationarity compensation data; inputting the weighted speech feature data and the time alignment parameter into a preset nonlinear mapping model to calculate the time offset distribution and obtain a speech feature vector; constructing a time sequence pattern mapping matrix according to the speech feature vector, and performing adaptive matching using the time sequence pattern mapping matrix in combination with the time sequence information parameters to generate matching mapping data; performing a graph structure transformation on the matching mapping data, calculating a global similarity weight, and obtaining speech pattern mapping data.

[0009] As a preferred scheme, the step of performing graph structure transformation on the matching mapping data, calculating the global similarity weight, and obtaining the speech pattern mapping data includes: performing graph processing on the matching mapping data, constructing a graph structure representation of the audio features, and calculating the similarity measure between the nodes based on the graph structure representation; analyzing the similarity measure through a graph convolutional network to obtain feature node data, and constructing a speech pattern graph based on the feature node data; using the speech pattern graph to calculate the contribution of each node to obtain the global similarity weight, and generating the speech pattern mapping data based on the global similarity weight.

[0010] As a preferred scheme, the step of generating an optimized audio signal based on the speech pattern mapping data, constructing a multi-level recursive timing spectrum according to the timing information parameters, and using the multi-level recursive timing spectrum to dynamically adjust the timing of the optimized audio signal to obtain a high-precision audio data frame includes: performing nonlinear feature transformation on the speech pattern mapping data, extracting local audio feature data and global spectrum distribution data, calculating frame-level feature offset parameters based on the global spectrum distribution data, adaptively correcting the local audio feature data through the frame-level feature offset parameters, calculating the time domain feature change rate based on the correction result and generating a feature-enhanced audio signal; constructing a timing segment mapping matrix according to the time domain feature change rate, and calculating the timing consistency parameter according to the timing segment mapping matrix. number, dynamically adjust the feature-enhanced audio signal through the timing consistency parameter, generate an optimized audio signal based on the adjustment result and calculate the inter-frame time offset data; construct a multi-level recursive timing graph based on the inter-frame time offset data, extract the timing correlation features through a multi-layer mapping mechanism, obtain a timing correlation matrix, perform nonlinear time alignment on the optimized audio signal according to the timing correlation matrix, and obtain a timing-aligned audio signal and a local energy compensation parameter; reconstruct the timing-aligned audio signal in the frequency domain according to the local energy compensation parameter to generate a frequency-compensated audio signal, and calculate the short-time power adjustment parameter through the frequency-compensated audio signal; power normalize the frequency-compensated audio signal through the short-time power adjustment parameter to obtain a high-precision audio data frame.

[0011] As a preferred scheme, the step of allocating the high-precision audio data frame to a high-performance core to generate a target audio output signal includes: in the high-performance core, dynamically performing amplitude equalization on the high-precision audio data frame to obtain an amplitude compensated audio data frame and spectrum equalization parameters, calculating a multi-channel spectrum deviation according to the spectrum equalization parameters, and constructing a frequency compensation matrix based on the multi-channel spectrum deviation; using the frequency compensation matrix to perform frequency adjustment on the amplitude compensated audio data frame to obtain a spectrum optimized audio data frame, and performing short-time time domain reconstruction on the spectrum optimized audio data frame to obtain a target audio output signal.

[0012] As a preferred scheme, the step of assigning the multi-level recursive timing map to the low-power core and generating an audio playback synchronization control signal includes: calculating frame-level timing offset parameters according to the multi-level recursive timing map, and constructing a global time offset matrix based on the frame-level timing offset parameters; calculating playback time prediction data through the global time offset matrix, and calculating playback timing correction parameters through the playback time prediction data, and performing time axis matching on the multi-level recursive timing map through the playback timing correction parameters to obtain a playback time adjustment matrix; calculating current energy consumption data using the playback time adjustment matrix, and calculating power consumption optimization adjustment parameters based on the current energy consumption data, and dynamically adjusting the playback time adjustment matrix according to the power consumption optimization adjustment parameters to obtain a synchronization correction timing map; calculating playback synchronization control data through the synchronization correction timing map, and calculating synchronization control compensation parameters using the playback synchronization control data combined with the current energy consumption data; adjusting the playback synchronization control data according to the synchronization control compensation parameters to obtain an audio playback synchronization control signal.

[0013] The present application also provides a real-time audio processing device based on a three-core heterogeneous SoC, which is applied to a three-core heterogeneous SoC, wherein the three-core heterogeneous SoC includes a high-performance core, a medium-performance core and a low-power core, and the audio processing device includes: a parsing module, which is used to collect original audio signals in real time through an audio acquisition device, and use the medium-performance core to parse the original audio signals to obtain audio feature parameters and timing information parameters; a processing module, which is used to perform a joint transformation process in the time-frequency domain on the audio feature parameters to obtain a speech feature vector, perform a deep mapping analysis on the speech feature vector, and generate speech mode mapping data; an adjustment module, which is used to generate an optimized audio signal based on the speech mode mapping data, construct a multi-level recursive timing map according to the timing information parameters, and use the multi-level recursive timing map to dynamically adjust the timing of the optimized audio signal to obtain a high-precision audio data frame; a generation module, which is used to allocate the high-precision audio data frame to the high-performance core to generate a target audio output signal, allocate the multi-level recursive timing map to the low-power core, and generate an audio playback synchronization control signal.

[0014] Compared with the prior art, the present application has the following beneficial effects: low processing power consumption and high processing efficiency. The original audio signal is collected in real time by an audio acquisition device, and the original audio signal is parsed by a medium-performance core to obtain audio feature parameters and timing information parameters. The audio feature parameters are processed by a joint transformation in the time and frequency domain to obtain a speech feature vector, and the speech feature vector is deeply mapped and analyzed to generate speech mode mapping data, and an optimized audio signal is generated based on the speech mode mapping data, and a multi-level recursive timing map is constructed according to the timing information parameters, and the optimized audio signal is dynamically adjusted by using the multi-level recursive timing map to obtain a high-precision audio data frame, and the high-precision audio data frame is allocated to the high-performance core to generate a target audio output signal, and the multi-level recursive timing map is allocated to the low-power core to generate an audio playback synchronization control signal, and the efficiency and accuracy of audio processing are improved through the collaborative work of the high-performance core, the medium-performance core and the low-power core, so as to achieve high efficiency and low latency of real-time audio processing, and improve the problems of insufficient processing performance, high power consumption and processing delay when processing complex real-time audio signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0017] Figure 1 It is a flowchart of a real-time audio processing method based on a triple-core heterogeneous SoC provided by an embodiment of the present invention; Figure 2 It is a schematic block diagram of the structure of a real-time audio processing device based on a triple-core heterogeneous SoC provided in an embodiment of the present invention.

[0018] Description of reference numerals: 10. Real-time audio processing device based on three-core heterogeneous SoC; 11. Analysis module; 12. Processing module; 13. Adjustment module; 14. Generation module. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be performed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual order of execution may change according to actual conditions.

[0021] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0022] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0024] Embodiment 1: like Figure 1 As shown, the present application also provides a real-time audio processing method based on a three-core heterogeneous SoC, which is applied to the three-core heterogeneous SoC. The three-core heterogeneous SoC includes a high-performance core, a medium-performance core and a low-power core. The audio processing method includes steps S100 to S400.

[0025] Step S100: collect original audio signals in real time through an audio collection device, and analyze the original audio signals using a medium-performance core to obtain audio feature parameters and timing information parameters.

[0026] In this step, the original audio signal is collected in real time by an audio acquisition device and transmitted to the medium-performance core for analysis; specifically, the medium-performance core performs fast Fourier transform (FFT) processing on the original audio signal to extract audio feature parameters and timing information parameters.

[0027] For example, the original audio signal collected by the microphone can be amplified and digitized by the front end and converted into a digital audio signal that can be processed by the medium-performance core.

[0028] Step S200: Perform a joint transformation process in the time-frequency domain on the audio feature parameters to obtain a speech feature vector, perform a deep mapping analysis on the speech feature vector, and generate speech pattern mapping data.

[0029] In this step, the speech feature vector is extracted by performing short-time Fourier transform (STFT) and Mel-frequency cepstral coefficient (MFCC) processing on the audio feature parameters; specifically, the audio feature parameters are input into a deep neural network (DNN) for deep mapping analysis to generate speech pattern mapping data.

[0030] For example, the audio feature parameters processed by the medium-performance core can be used to extract higher-level speech features through a convolutional neural network (CNN) and perform deep learning model training to generate more accurate speech pattern mapping data.

[0031] Step S300: Generate an optimized audio signal based on the speech pattern mapping data, construct a multi-level recursive timing graph according to the timing information parameters, and use the multi-level recursive timing graph to dynamically adjust the timing of the optimized audio signal to obtain a high-precision audio data frame.

[0032] In this step, an optimized audio signal is generated through speech pattern mapping data, and a multi-level recursive timing graph is constructed according to timing information parameters; specifically, a recursive neural network (RNN) is used to dynamically adjust the timing of the optimized audio signal to obtain a high-precision audio data frame.

[0033] For example, the optimized audio signal can be further time-series analyzed and adjusted through a long short-term memory network (LSTM) to ensure high accuracy and real-time performance of the audio signal.

[0034] Step S400: distribute the high-precision audio data frame to the high-performance core to generate the target audio output signal, distribute the multi-level recursive timing graph to the low-power core to generate the audio playback synchronization control signal.

[0035] In this step, the high-precision audio data frame is allocated to the high-performance core to generate the target audio output signal, and the multi-level recursive timing graph is allocated to the low-power core to generate the audio playback synchronization control signal; specifically, the high-performance core is responsible for the final processing and output of the audio signal, and the low-power core is responsible for the synchronization control of the audio playback.

[0036] For example, the high-performance core can use the digital signal processor (DSP) to further optimize the audio signal and output high-quality audio signals through the audio amplifier, while the low-power core can achieve precise synchronization of audio playback through the timing controller.

[0037] In this embodiment, the original audio signal is collected in real time by an audio acquisition device, and the original audio signal is parsed using a medium-performance core to obtain audio feature parameters and timing information parameters. Then, the audio feature parameters are processed by a joint transformation in the time-frequency domain to obtain a speech feature vector, and the speech feature vector is deeply mapped and analyzed to generate speech mode mapping data. An optimized audio signal is generated based on the speech mode mapping data, and a multi-level recursive timing map is constructed according to the timing information parameters. The optimized audio signal is dynamically adjusted using the multi-level recursive timing map to obtain a high-precision audio data frame. Finally, the high-precision audio data frame is assigned to the high-performance core to generate a target audio output signal, and the multi-level recursive timing map is assigned to the low-power core to generate an audio playback synchronization control signal.

[0038] By using the collaborative work of high-performance cores, medium-performance cores, and low-power cores, the efficiency and accuracy of audio processing are improved, and high efficiency and low latency of real-time audio processing are achieved, which improves the problems of insufficient processing performance, high power consumption, and processing delay when processing complex real-time audio signals. At the same time, through dynamic timing adjustment and deep mapping analysis, the quality of audio signals is optimized, and the clarity and consistency of audio output are significantly improved. In addition, the construction and application of multi-level recursive timing graphs effectively reduce the power consumption of the system, extend the battery life of the device, and meet the needs of modern smart devices for efficient, low-power real-time audio processing.

[0039] Embodiment 2: In step S100, in the medium performance core, the original audio signal is preprocessed in the time domain to obtain short-time waveform data and energy distribution data, the short-time waveform data is adaptively filtered to obtain noise-suppressed audio data, and the signal dynamic range parameters are calculated using the noise-suppressed audio data and energy distribution data.

[0040] By preprocessing the original audio signal in the time domain, the specific expression of the audio signal in the time domain can be obtained; specifically, the audio signal is divided into multiple short-time waveform data segments through time domain preprocessing, and the energy distribution of each segment is analyzed, so that the dynamic range of the signal can be better adjusted in subsequent processing. In the process of adaptive filtering of short-time waveform data, environmental noise interference can be effectively suppressed, useful audio information can be retained, and the dynamic range parameters of the signal can be calculated in combination with the energy distribution data.

[0041] For example, through the adaptive filter algorithm, the filter parameters can be adjusted in real time to achieve the best noise suppression effect in different noise environments, thereby obtaining purer audio data.

[0042] According to the signal dynamic range parameters, the audio data after noise suppression is logarithmically scaled to obtain normalized audio feature data, the normalized audio feature data is short-time Fourier transformed to obtain short-time spectrum data, and the audio feature parameters are extracted based on the short-time spectrum data.

[0043] By performing logarithmic scaling on the audio data after noise suppression, the dynamic range of the audio signal can be adjusted to a suitable range for subsequent feature extraction; specifically, through logarithmic scaling, the amplitude value of the audio signal is nonlinearly converted to make the amplitude change of the signal more uniform, and the normalized audio feature data is subjected to short-time Fourier transform (STFT) to obtain the representation of the audio signal in the frequency domain. By analyzing the short-time spectrum data, the audio feature parameters are extracted for subsequent processing and analysis.

[0044] For example, through logarithmic scaling, the part of the audio signal with large changes in intensity can be made smoother, making it easier to perform spectrum analysis and feature extraction.

[0045] The short-time spectrum data and signal dynamic range parameters are input into the timing analysis module, the frame-level energy change trend is calculated, the timing change data is obtained, the frame-level timing fragment is constructed according to the timing change data, and the local transition information is extracted to obtain the timing information parameters.

[0046] By inputting the short-time spectrum data and signal dynamic range parameters into the timing analysis module, the energy change trend of the audio signal can be analyzed in detail; specifically, by calculating the frame-level energy change trend, the energy distribution of the audio signal at different times can be obtained, and based on these data, frame-level timing segments can be constructed to extract the local transition information of the audio signal, and finally the timing information parameters can be obtained for subsequent timing adjustment and optimization.

[0047] For example, through the timing analysis module, the energy changes of the audio signal between different frames can be calculated, so as to accurately evaluate the timing characteristics of the audio signal and extract the key transition points and change trends of the signal.

[0048] In step S200, a multi-scale discrete wavelet transform is performed on the audio feature parameters to obtain multi-resolution feature data, and the local frequency change rate is calculated based on the multi-resolution feature data. According to the local frequency change rate, a sub-band energy distribution matrix is ​​constructed, and a sub-band comparison analysis is performed on the sub-band energy distribution matrix to obtain spectrum enhancement data and non-stationarity compensation data.

[0049] By performing multi-scale discrete wavelet transform on audio feature parameters, the frequency characteristics of audio signals at different scales can be extracted; specifically, by performing multi-scale discrete wavelet transform, the audio signal is decomposed into feature data of different resolutions, and the local frequency change rate of these data is calculated to construct a sub-band energy distribution matrix. By performing sub-band comparative analysis on the sub-band energy distribution matrix, the spectral performance of the audio signal can be effectively enhanced, and non-stationarity can be compensated, thereby obtaining spectral enhancement data and non-stationarity compensation data.

[0050] For example, through multi-scale discrete wavelet transform, the audio signal can be decomposed into high-frequency and low-frequency components, and their energy distribution can be calculated separately, so as to better analyze the frequency characteristics of the audio signal.

[0051] The energy weight of the spectrum enhancement data is adjusted through the time-frequency masking mechanism to obtain weighted speech feature data, and the time alignment parameters are calculated using the weighted speech feature data combined with the non-stationary compensation data.

[0052] By adjusting the energy weight of the spectrum enhancement data through the time-frequency masking mechanism, the spectral performance of the audio signal can be optimized; specifically, by weighting the energy of the spectrum enhancement data, more accurate speech feature data can be obtained, and combined with the non-stationary compensation data, the time alignment parameters can be calculated for subsequent speech signal processing and analysis.

[0053] For example, through the time-frequency masking mechanism, the noise component in the audio signal can be suppressed, while the useful speech features can be enhanced, so as to obtain weighted speech feature data and calculate the time alignment parameters to improve the processing accuracy of the speech signal.

[0054] The weighted speech feature data and time alignment parameters are input into a preset nonlinear mapping model to calculate the time offset distribution and obtain a speech feature vector.

[0055] By inputting weighted speech feature data and time alignment parameters into a nonlinear mapping model, the time offset distribution of the audio signal can be calculated; specifically, the weighted speech feature data and time alignment parameters are processed through a preset nonlinear mapping model to obtain a speech feature vector, thereby further improving the representation capability of the audio signal.

[0056] For example, through the nonlinear mapping model, the time offset of the audio signal can be analyzed in detail and the corresponding speech feature vector can be generated for subsequent signal processing and optimization.

[0057] According to the speech feature vector, a temporal pattern mapping matrix is ​​constructed, and the temporal pattern mapping matrix is ​​used in combination with the temporal information parameters for adaptive matching to generate matching mapping data.

[0058] By constructing a timing pattern mapping matrix based on speech feature vectors, the timing characteristics of the audio signal can be better represented; specifically, by adaptively matching the speech feature vectors with timing information parameters, matching mapping data is generated for subsequent timing adjustment and optimization processing.

[0059] For example, by constructing a timing pattern mapping matrix, the timing changes of an audio signal can be accurately modeled, and matching mapping data can be generated through an adaptive matching algorithm to better optimize the timing performance of the audio signal.

[0060] The matching mapping data is transformed into a graph structure, and the global similarity weight is calculated to obtain the speech pattern mapping data.

[0061] By performing graph structure transformation on the matching mapping data, the global characteristics of the audio signal can be represented more intuitively. Specifically, by performing graph processing on the matching mapping data, a graph structure representation of the audio features is constructed, and the similarity measure between the nodes is calculated based on the graph structure representation. The global similarity weight is further calculated to obtain the speech pattern mapping data.

[0062] For example, through graph structure transformation, the characteristic nodes of the audio signal and their mutual relationships can be represented as a graph structure, and by calculating the similarity measure, the similarity weights between each node can be obtained to generate speech pattern mapping data to better represent the global characteristics of the audio signal.

[0063] Among them, the steps of performing graph structure transformation on the matching mapping data, calculating the global similarity weight, and obtaining the speech pattern mapping data include: performing graph processing on the matching mapping data, constructing a graph structure representation of the audio features, and calculating the similarity measure between nodes based on the graph structure representation.

[0064] By graphing the matching mapping data, the audio feature data can be represented as a graph structure; specifically, through graphing, the feature nodes in the audio signal and their mutual relationships are represented as a graph structure, and the similarity measure between the nodes is calculated based on the graph structure to better represent the global characteristics of the audio signal.

[0065] For example, through graph processing, the spectrum data of the audio signal can be represented as a graph structure, and by calculating the similarity measure between nodes, the global similarity weight of the audio signal can be obtained to form speech pattern mapping data.

[0066] The similarity measurement is analyzed through a graph convolutional network to obtain feature node data, and a speech pattern graph is constructed based on the feature node data.

[0067] By using a graph convolutional network to analyze similarity metrics, the characteristic node data of the audio signal can be effectively extracted; specifically, the graph structure representation is processed using a graph convolutional network to obtain characteristic node data, and a speech pattern graph is constructed based on this data.

[0068] For example, through graph convolutional networks, we can perform deep learning on the spectral data of audio signals, extract representative feature nodes, and construct a more expressive speech pattern graph.

[0069] The contribution of each node is calculated using the speech pattern graph to obtain the global similarity weight, and the speech pattern mapping data is generated based on the global similarity weight.

[0070] By using the speech pattern graph to calculate the contribution of each node, the global similarity weight of the audio signal can be obtained more accurately; specifically, the contribution of each node in the global characteristics is calculated using the data of each feature node in the speech pattern graph, and the global similarity weight is calculated based on these contributions to generate speech pattern mapping data.

[0071] For example, by calculating the contribution of each node in the speech pattern graph, the importance of each feature node in the audio signal can be evaluated, thereby generating a more accurate global similarity weight for subsequent audio processing and optimization.

[0072] In step S300, nonlinear feature transformation is performed on the speech pattern mapping data, local audio feature data and global spectrum distribution data are extracted, frame-level feature offset parameters are calculated based on the global spectrum distribution data, local audio feature data are adaptively corrected using the frame-level feature offset parameters, and the time domain feature change rate is calculated based on the correction result and a feature-enhanced audio signal is generated.

[0073] By performing nonlinear feature transformation on the speech pattern mapping data, the local feature data and global spectrum distribution data of the audio signal can be extracted; specifically, by calculating the frame-level feature offset parameters of the global spectrum distribution data, and using the parameters to adaptively correct the local feature data of the audio, the time domain feature change rate is finally calculated based on the correction result, and a feature-enhanced audio signal is generated.

[0074] For example, the frequency characteristics of the audio signal can be analyzed in detail through a feature transformation algorithm, and the frequency characteristic data can be corrected through frame-level feature offset parameters to generate a more realistic feature-enhanced audio signal.

[0075] According to the rate of change of time domain features, a temporal segment mapping matrix is ​​constructed, and the timing consistency parameters are calculated based on the temporal segment mapping matrix. The feature enhanced audio signal is dynamically adjusted through the timing consistency parameters, and an optimized audio signal is generated based on the adjustment results and the inter-frame time offset data is calculated.

[0076] By constructing a timing segment mapping matrix based on the rate of change of time domain features, the feature-enhanced audio signal can be adjusted more accurately in timing; specifically, by calculating the timing consistency parameter and using the parameter to dynamically adjust the feature-enhanced audio signal, an optimized audio signal is finally generated, and the inter-frame time offset data is calculated.

[0077] For example, by calculating the timing segment mapping matrix, the timing consistency of the audio signal can be evaluated, and the timing of the audio signal can be adjusted based on the evaluation result, thereby generating an optimized audio signal.

[0078] A multi-level recursive timing graph is constructed based on the inter-frame time offset data, and the timing correlation features are extracted through a multi-layer mapping mechanism to obtain a timing correlation matrix. According to the timing correlation matrix, nonlinear time alignment is performed on the optimized audio signal to obtain a timing-aligned audio signal and local energy compensation parameters.

[0079] By constructing a multi-level recursive timing graph based on the inter-frame time offset data, the timing correlation characteristics of the audio signal can be extracted; specifically, the timing correlation characteristics are analyzed through a multi-layer mapping mechanism to generate a timing correlation matrix, and the optimized audio signal is nonlinearly time-aligned according to the timing correlation matrix, finally obtaining a timing-aligned audio signal and local energy compensation parameters.

[0080] For example, by processing the inter-frame time offset data through a recursive neural network (RNN) model, a multi-level recursive timing graph can be constructed, and the audio signals can be time-aligned through these graphs to generate more accurate time-aligned audio signals.

[0081] The time-aligned audio signal is reconstructed in frequency domain according to the local energy compensation parameter to generate a frequency compensated audio signal, and the short-time power adjustment parameter is calculated through the frequency compensated audio signal.

[0082] By reconstructing the time-aligned audio signal in the frequency domain according to the local energy compensation parameters, the quality of the audio signal can be further improved; specifically, through the frequency domain reconstruction, a frequency compensated audio signal is generated, and the short-time power adjustment parameters are calculated for subsequent audio signal processing.

[0083] For example, through frequency domain reconstruction, the spectrum of the audio signal can be optimized and short-term power adjustment parameters can be calculated to generate a higher quality frequency compensated audio signal.

[0084] The power of the frequency compensated audio signal is normalized by adjusting the short-time power parameter to obtain a high-precision audio data frame.

[0085] By using short-time power adjustment parameters to perform power normalization on the frequency compensated audio signal, the power consistency and accuracy of the audio signal can be ensured; specifically, through power normalization processing, a high-precision audio data frame is obtained to meet the needs of high-quality audio output.

[0086] For example, through power normalization processing, the power of audio signals in different frequency bands can be balanced, and high-precision audio data frames can be generated to provide support for subsequent processing of the high-performance core.

[0087] In step S400, in the high-performance core, dynamic amplitude equalization is performed on the high-precision audio data frame to obtain amplitude compensated audio data frame and spectrum equalization parameters, multi-channel spectrum deviation is calculated according to the spectrum equalization parameters, and a frequency compensation matrix is ​​constructed based on the multi-channel spectrum deviation.

[0088] By performing dynamic amplitude equalization on high-precision audio data frames in a high-performance core, the quality of the audio signal can be further improved; specifically, through dynamic amplitude equalization, amplitude compensated audio data frames and spectrum equalization parameters are obtained, and multi-channel spectrum deviations are calculated based on these parameters.

[0089] For example, through dynamic amplitude equalization processing, the various frequency bands in the audio signal can be balanced to obtain amplitude compensated audio data frames, and the spectrum deviation between the various frequency bands can be calculated to provide data support for subsequent frequency compensation.

[0090] The frequency compensation matrix is ​​used to adjust the frequency of the amplitude compensated audio data frame to obtain a spectrum optimized audio data frame, and the spectrum optimized audio data frame is reconstructed in a short time domain to obtain a target audio output signal.

[0091] By using a frequency compensation matrix to adjust the frequency of the amplitude compensated audio data frame, the spectrum of the audio signal can be further optimized; specifically, through frequency adjustment, a spectrum optimized audio data frame is obtained, and through short-time time domain reconstruction processing, a target audio output signal is generated.

[0092] For example, through frequency adjustment, the spectrum of the audio signal can be finely adjusted to optimize the spectrum distribution of each frequency band, thereby generating a clearer and higher-quality target audio output signal.

[0093] Among them, the step of allocating the multi-level recursive timing graph to the low-power core to generate the audio playback synchronization control signal includes: calculating the frame-level timing offset parameters according to the multi-level recursive timing graph, and constructing a global time offset matrix based on the frame-level timing offset parameters.

[0094] By assigning multi-level recursive timing graphs to low-power cores for processing, audio playback synchronization control signals can be effectively generated; specifically, by calculating frame-level timing offset parameters and building a global time offset matrix based on these parameters, the synchronization of audio playback can be controlled.

[0095] For example, by calculating the global time offset matrix, the timing of audio playback can be precisely controlled to ensure the synchronization and stability of audio playback.

[0096] The playback time prediction data is calculated through the global time offset matrix, and the playback timing correction parameters are calculated through the playback time prediction data. The multi-level recursive timing graph is matched with the time axis through the playback timing correction parameters to obtain the playback time adjustment matrix.

[0097] By calculating the playback time prediction data through the global time offset matrix, the time axis changes of audio playback can be predicted; specifically, by calculating the playback timing correction parameters and performing time axis matching on the multi-level recursive timing graph, a playback time adjustment matrix is ​​generated for synchronous control of audio playback.

[0098] For example, by calculating the playback time prediction data, the time axis changes of the audio playback can be predicted in advance, so that the time axis can be dynamically adjusted to ensure the synchronization of the audio playback.

[0099] The current energy consumption data is calculated using the playback time adjustment matrix, and the power consumption optimization adjustment parameters are calculated based on the current energy consumption data. The playback time adjustment matrix is ​​dynamically adjusted according to the power consumption optimization adjustment parameters to obtain a synchronous correction timing map.

[0100] By using the playback time adjustment matrix to calculate the current energy consumption data, the energy consumption of audio playback can be optimized; specifically, by calculating the power consumption optimization adjustment parameters and dynamically adjusting the playback time adjustment matrix, a synchronous correction timing diagram is generated to improve the efficiency of audio playback.

[0101] For example, through power consumption optimization and adjustment, the system power consumption can be reduced and the device usage time can be extended while ensuring the audio playback quality.

[0102] The playback synchronization control data is calculated through the synchronization correction timing diagram, and the synchronization control compensation parameters are calculated by combining the playback synchronization control data with the current energy consumption data.

[0103] By calculating the playback synchronization control data by synchronously correcting the timing graph, the synchronization and stability of audio playback can be further improved; specifically, by calculating the synchronization control compensation parameters in combination with the current energy consumption data, high efficiency and low energy consumption of audio playback can be ensured.

[0104] For example, by synchronously controlling the calculation of compensation parameters, the synchronization of audio playback can be adjusted in real time to ensure the stability and high quality of playback.

[0105] The playback synchronization control data is adjusted according to the synchronization control compensation parameter to obtain an audio playback synchronization control signal.

[0106] By adjusting the playback synchronization control data according to the synchronization control compensation parameters, a final audio playback synchronization control signal can be generated; specifically, by adjusting the synchronization control compensation parameters, the synchronization and accuracy of the audio playback are ensured, and finally an audio playback synchronization control signal is obtained.

[0107] For example, by synchronously controlling the adjustment of compensation parameters, the timeline and energy consumption of audio playback can be optimized to ensure stable and efficient audio playback.

[0108] In this embodiment, the original audio signal is preprocessed in the time domain to obtain short-time waveform data and energy distribution data, and the short-time waveform data is adaptively filtered to obtain audio data after noise suppression, and the signal dynamic range parameters are calculated using the audio data after noise suppression and the energy distribution data. Next, the audio data after noise suppression is logarithmically scaled to obtain normalized audio feature data, and the normalized audio feature data is short-time Fourier transformed to obtain short-time spectrum data, and the audio feature parameters are extracted based on the short-time spectrum data. Then, the short-time spectrum data and the signal dynamic range parameters are input into the timing analysis module, the frame-level energy change trend is calculated, the timing change data is obtained, the frame-level timing fragment is constructed according to the timing change data, and the local transition information is extracted to obtain the timing information parameters. In the process of processing the audio feature parameters, a multi-scale discrete wavelet transform is used to obtain multi-resolution feature data, and the local frequency change rate is calculated based on the multi-resolution feature data, and the sub-band energy distribution matrix is ​​constructed. Through sub-band contrast analysis, spectrum enhancement data and non-stationarity compensation data are obtained.

[0109] The energy weight of the spectrum enhancement data is adjusted by using the time-frequency masking mechanism to obtain weighted speech feature data, and the time alignment parameters are calculated in combination with the non-stationary compensation data. Then, the time offset distribution is calculated by a preset nonlinear mapping model to obtain a speech feature vector. According to the speech feature vector, a temporal pattern mapping matrix is ​​constructed, and adaptive matching is performed in combination with the temporal information parameters to generate matching mapping data, and the matching mapping data is transformed into a graph structure to calculate the global similarity weight to obtain speech pattern mapping data. During the processing, the similarity measure is analyzed by a graph convolutional network to obtain feature node data, construct a speech pattern graph, and calculate the contribution of each node according to the feature node data to generate a global similarity weight, thereby obtaining speech pattern mapping data. Through the above steps, the generated speech pattern mapping data undergoes nonlinear feature transformation, extracts audio local feature data and global spectrum distribution data, calculates frame-level feature offset parameters, and adaptively corrects the audio local feature data to generate a feature-enhanced audio signal. Finally, through frequency domain reconstruction, power normalization and other steps, high-precision audio data frames are obtained, and dynamic amplitude equalization, spectrum optimization and other processing are performed in the high-performance core to generate the target audio output signal, and the audio playback synchronization control signal is generated through a multi-level recursive timing graph in the low-power core to ensure high efficiency, low power consumption and high quality of audio playback.

[0110] Embodiment 3: like Figure 2 As shown, the present application also proposes a real-time audio processing device 10 based on a three-core heterogeneous SoC, which is applied to the three-core heterogeneous SoC. The three-core heterogeneous SoC includes a high-performance core, a medium-performance core and a low-power core. The audio processing device includes a parsing module 11, a processing module 12, an adjustment module 13 and a generation module 14.

[0111] The analysis module 11 is mainly used to collect original audio signals in real time through an audio acquisition device, and analyze the original audio signals using a medium-performance core to obtain audio feature parameters and timing information parameters.

[0112] The processing module 12 is mainly used to perform a joint transformation process in the time-frequency domain on the audio feature parameters to obtain a speech feature vector, perform a deep mapping analysis on the speech feature vector, and generate speech pattern mapping data.

[0113] The adjustment module 13 is mainly used to generate an optimized audio signal based on the speech pattern mapping data, construct a multi-level recursive timing map according to the timing information parameters, and use the multi-level recursive timing map to dynamically adjust the timing of the optimized audio signal to obtain a high-precision audio data frame.

[0114] The generation module 14 is mainly used to distribute high-precision audio data frames to the high-performance core, generate target audio output signals, distribute multi-level recursive timing graphs to the low-power core, and generate audio playback synchronization control signals.

[0115] In this embodiment, through the design of the parsing module 11, the original audio signal is collected in real time using an audio acquisition device, and parsed through a medium-performance core to obtain audio feature parameters and timing information parameters. The processing module 12 performs a joint transformation process in the time-frequency domain on the audio feature parameters to obtain a speech feature vector, and performs a deep mapping analysis on the speech feature vector to generate speech mode mapping data. The adjustment module 13 generates an optimized audio signal based on the speech mode mapping data, and constructs a multi-level recursive timing map according to the timing information parameters, and uses the multi-level recursive timing map to dynamically adjust the optimized audio signal to obtain a high-precision audio data frame. The generation module 14 allocates the high-precision audio data frame to the high-performance core to generate a target audio output signal, and at the same time allocates the multi-level recursive timing map to the low-power core to generate an audio playback synchronization control signal. Through the collaborative work of the above modules, the efficiency and accuracy of audio processing are significantly improved, and efficient, low-latency real-time audio processing is achieved. At the same time, dynamic timing adjustment and deep mapping analysis optimize the quality of the audio signal and improve the clarity and consistency of the audio output. The construction and application of multi-level recursive timing graphs effectively reduces the power consumption of the system, extends the battery life of the device, and meets the needs of modern smart devices for efficient, low-power real-time audio processing. This design based on a three-core heterogeneous SoC fully utilizes the respective advantages of high-performance cores, medium-performance cores, and low-power cores, ensuring the stability and excellent performance of the audio processing device in various complex scenarios.

[0116] It should be noted that technicians in the relevant technical field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the real-time audio processing method based on a three-core heterogeneous SoC, and will not be repeated here.

[0117] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0118] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time audio processing method based on a three-core heterogeneous SoC, applied to a three-core heterogeneous SoC, wherein the three-core heterogeneous SoC includes a high-performance core, a medium-performance core, and a low-power core, characterized in that: Audio processing methods include: Collecting original audio signals in real time through an audio acquisition device, parsing the original audio signals using the medium performance core to obtain audio feature parameters and timing information parameters; Performing a time-frequency domain joint transformation process on the audio feature parameters to obtain a speech feature vector, performing a deep mapping analysis on the speech feature vector to generate speech pattern mapping data; Generate an optimized audio signal based on the speech mode mapping data, construct a multi-level recursive timing graph according to the timing information parameters, and use the multi-level recursive timing graph to dynamically adjust the timing of the optimized audio signal to obtain a high-precision audio data frame; The high-precision audio data frame is distributed to the high-performance core to generate a target audio output signal, and the multi-level recursive timing graph is distributed to the low-power core to generate an audio playback synchronization control signal.

2. The real-time audio processing method based on a triple-core heterogeneous SoC according to claim 1, characterized in that: The step of using the medium performance core to parse the original audio signal to obtain audio feature parameters and timing information parameters includes: In the medium performance core, the original audio signal is preprocessed in the time domain to obtain short-time waveform data and energy distribution data, the short-time waveform data is adaptively filtered to obtain noise-suppressed audio data, and a signal dynamic range parameter is calculated using the noise-suppressed audio data and the energy distribution data; According to the signal dynamic range parameter, logarithmically scale the audio data after noise suppression to obtain normalized audio feature data, perform short-time Fourier transform on the normalized audio feature data to obtain short-time spectrum data, and extract audio feature parameters based on the short-time spectrum data; The short-time spectrum data and the signal dynamic range parameters are input into a timing analysis module, the frame-level energy change trend is calculated to obtain timing change data, frame-level timing segments are constructed according to the timing change data, and local transition information is extracted to obtain timing information parameters.

3. The real-time audio processing method based on a triple-core heterogeneous SoC according to claim 2, characterized in that: The step of performing a time-frequency domain joint transformation process on the audio feature parameters to obtain a speech feature vector, performing a deep mapping analysis on the speech feature vector, and generating speech mode mapping data comprises: Performing a multi-scale discrete wavelet transform on the audio feature parameters to obtain multi-resolution feature data, calculating a local frequency change rate based on the multi-resolution feature data, constructing a sub-band energy distribution matrix according to the local frequency change rate, performing a sub-band comparison analysis on the sub-band energy distribution matrix, and obtaining spectrum enhancement data and non-stationarity compensation data; The energy weight of the spectrum enhancement data is adjusted by a time-frequency masking mechanism to obtain weighted speech feature data, and the time alignment parameter is calculated by combining the weighted speech feature data with the non-stationarity compensation data; Inputting the weighted speech feature data and the time alignment parameter into a preset nonlinear mapping model to calculate the time offset distribution, thereby obtaining a speech feature vector; According to the speech feature vector, a time sequence pattern mapping matrix is ​​constructed, and the time sequence pattern mapping matrix is ​​used in combination with the time sequence information parameters to perform adaptive matching to generate matching mapping data; The matching mapping data is subjected to graph structure transformation, and a global similarity weight is calculated to obtain speech pattern mapping data.

4. The real-time audio processing method based on a triple-core heterogeneous SoC according to claim 3, characterized in that: The step of performing graph structure transformation on the matching mapping data, calculating the global similarity weight, and obtaining the speech mode mapping data comprises: Performing graph processing on the matching mapping data to construct a graph structure representation of the audio features, and calculating a similarity measure between nodes based on the graph structure representation; Analyzing the similarity measure through a graph convolutional network to obtain feature node data, and constructing a speech pattern graph according to the feature node data; The contribution of each node is calculated using the speech pattern graph to obtain a global similarity weight, and speech pattern mapping data is generated based on the global similarity weight.

5. The real-time audio processing method based on a triple-core heterogeneous SoC according to claim 1, characterized in that: The step of generating an optimized audio signal based on the speech mode mapping data, constructing a multi-level recursive timing graph according to the timing information parameters, and dynamically adjusting the timing of the optimized audio signal using the multi-level recursive timing graph to obtain a high-precision audio data frame includes: Performing nonlinear feature transformation on the speech pattern mapping data, extracting audio local feature data and global spectrum distribution data, calculating frame-level feature offset parameters based on the global spectrum distribution data, adaptively correcting the audio local feature data using the frame-level feature offset parameters, calculating the time domain feature change rate based on the correction result, and generating a feature-enhanced audio signal; According to the time domain feature change rate, a time segment mapping matrix is ​​constructed, and a time consistency parameter is calculated according to the time segment mapping matrix, the feature enhanced audio signal is dynamically adjusted by the time consistency parameter, an optimized audio signal is generated based on the adjustment result, and inter-frame time offset data is calculated; A multi-level recursive time map is constructed based on the inter-frame time offset data, and time correlation features are extracted through a multi-layer mapping mechanism to obtain a time correlation matrix, and nonlinear time alignment is performed on the optimized audio signal according to the time correlation matrix to obtain a time-aligned audio signal and a local energy compensation parameter; Reconstructing the time-aligned audio signal in the frequency domain according to the local energy compensation parameter to generate a frequency compensated audio signal, and calculating a short-time power adjustment parameter through the frequency compensated audio signal; The power of the frequency compensated audio signal is normalized by using the short-time power adjustment parameter to obtain a high-precision audio data frame.

6. The real-time audio processing method based on a triple-core heterogeneous SoC according to claim 1, characterized in that: The step of allocating the high-precision audio data frame to the high-performance core to generate a target audio output signal includes: In the high-performance core, the high-precision audio data frame is dynamically amplitude-equalized to obtain an amplitude-compensated audio data frame and a spectrum equalization parameter, a multi-channel spectrum deviation is calculated according to the spectrum equalization parameter, and a frequency compensation matrix is ​​constructed based on the multi-channel spectrum deviation; The frequency compensation matrix is ​​used to perform frequency adjustment on the amplitude compensated audio data frame to obtain a spectrum optimized audio data frame, and the spectrum optimized audio data frame is reconstructed in a short-term time domain to obtain a target audio output signal.

7. The real-time audio processing method based on a triple-core heterogeneous SoC according to claim 1, characterized in that: The step of allocating the multi-level recursive timing graph to the low-power core to generate an audio playback synchronization control signal includes: Calculating frame-level timing offset parameters according to the multi-level recursive timing graph, and constructing a global time offset matrix based on the frame-level timing offset parameters; Calculating the playback time prediction data through the global time offset matrix, and calculating the playback timing correction parameters through the playback time prediction data, and performing time axis matching on the multi-level recursive timing graph through the playback timing correction parameters to obtain a playback time adjustment matrix; Calculating current energy consumption data using the playback time adjustment matrix, and calculating power consumption optimization adjustment parameters based on the current energy consumption data, dynamically adjusting the playback time adjustment matrix according to the power consumption optimization adjustment parameters, and obtaining a synchronous correction timing diagram; Calculate the playback synchronization control data through the synchronization correction time series graph, and calculate the synchronization control compensation parameters by combining the playback synchronization control data with the current energy consumption data; The playback synchronization control data is adjusted according to the synchronization control compensation parameter to obtain an audio playback synchronization control signal.

8. A real-time audio processing device based on a three-core heterogeneous SoC, applied to a three-core heterogeneous SoC, wherein the three-core heterogeneous SoC includes a high-performance core, a medium-performance core, and a low-power core, characterized in that: The audio processing device comprises: A parsing module, used for collecting original audio signals in real time through an audio collection device, and parsing the original audio signals using the medium performance core to obtain audio feature parameters and timing information parameters; A processing module, configured to perform a time-frequency domain joint transformation process on the audio feature parameters to obtain a speech feature vector, perform a deep mapping analysis on the speech feature vector, and generate speech pattern mapping data; An adjustment module, configured to generate an optimized audio signal based on the speech mode mapping data, construct a multi-level recursive timing graph according to the timing information parameters, and dynamically adjust the timing of the optimized audio signal using the multi-level recursive timing graph to obtain a high-precision audio data frame; A generation module is used to distribute the high-precision audio data frame to the high-performance core to generate a target audio output signal, distribute the multi-level recursive timing graph to the low-power core, and generate an audio playback synchronization control signal.

Images