Audio data processing method, system, equipment and medium

Abstract

The invention relates to the technical field of computers, and discloses an audio data processing method, system and device and a medium, and the method comprises the steps: obtaining to-be-processed audio data and a service demand thereof, and carrying out the adaptive adjustment of an operation parameter of a linear operator according to the to-be-processed audio data and the service demand; the operation parameters comprise a row scale, a column scale and a bit width, and the linear operation operator comprises a multiplier and a logic arithmetic unit; converting the to-be-processed audio data from a time domain to a frequency domain, and calculating the gain of the converted to-be-processed audio data; the frequency spectrum of the converted audio data to be processed is calculated through the adjusted linear operator, and the gain is applied to the frequency spectrum; converting the frequency spectrum after gain application to a time domain to obtain processed audio data; and an operation resource ratio is changed by dynamically configuring operation parameters, so that optimal matching of power consumption and time delay of audio signal processing at any time section is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an audio data processing method, system, device and medium. Background Art

[0002] In audio signal processing, power consumption and latency are two important issues that have a direct impact on the performance of audio processing devices and user experience. Although there are technologies such as low-power processors and optimized algorithms, parallel processing and hardware acceleration, sampling rate control, power management strategies, buffering and pipeline design to improve power consumption and latency, there is still a challenge of balancing performance and power consumption.

[0003] It can be seen that how to solve the balance problem between power consumption and delay in the audio data processing process has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0004] The present invention provides an audio data processing method, system, device and medium to solve the problem of balancing power consumption and delay in the audio data processing process.

[0005] In order to solve the above technical problems, the present invention provides an audio data processing method in a first aspect, comprising:

[0006] Acquire the audio data to be processed and its business requirements, and adaptively adjust the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator;

[0007] Convert the audio data to be processed from the time domain to the frequency domain, and calculate the gain of the converted audio data to be processed;

[0008] Calculate the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator, and apply the gain to the frequency spectrum;

[0009] The spectrum after the gain is applied is converted to the time domain to obtain processed audio data.

[0010] As one of the preferred solutions, after obtaining the audio data to be processed and its business requirements, the method further includes:

[0011] The audio data to be processed is preprocessed, and the preprocessing includes but is not limited to denoising and filtering.

[0012] As one of the preferred solutions, the adaptively adjusting the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements includes:

[0013] configuring the operating voltage and clock frequency of the audio data to be processed according to the business requirements, and determining the maximum value of the column size according to the clock frequency;

[0014] quantifying the delay of the audio data to be processed according to the scale of available resources;

[0015] The power consumption of the audio data to be processed is quantified according to the operating voltage and the clock frequency, and within the range of the column scale and the callable resource scale, the row scale and the column scale of the linear operation operator are adjusted with the goal of minimizing the product of the row scale and the column scale, the power consumption and the delay.

[0016] As one of the preferred solutions, the adaptively adjusting the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements also includes:

[0017] configuring the operating voltage and clock main frequency of the audio data to be processed according to the service requirements, and quantifying the power consumption of the audio data to be processed according to the operating voltage and clock main frequency;

[0018] The operation precision of the audio data to be processed is determined according to the business requirements, and within the scope of the available resource scale, the bit width of the linear operation operator is adjusted with the goal of maximizing the operation precision and minimizing the power consumption.

[0019] As one preferred solution, the step of calculating the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator includes:

[0020] Initialize the adjusted path address of the linear operation operator to obtain an application gain topology circuit;

[0021] The frequency spectrum is obtained by respectively calculating the converted positive frequency expression and negative frequency expression of the to-be-processed audio data through the applied gain topology circuit.

[0022] As one of the preferred solutions, the positive frequency expression of the audio data to be processed after calculation by applying the gain topology circuit comprises:

[0023] Calculating the difference between the spectrum array corresponding to the positive frequency and a preset linear threshold value through a logic operator in the first column of the linear operation operator of the applied gain topology circuit to obtain a calculation result represented by a Boolean value;

[0024] By using the linear operation operator of the second column to the third to last column of the applied gain topology circuit, the base value element corresponding to the Boolean value is retrieved from the base value array to update the spectrum array;

[0025] Based on the gain difference and the Boolean value, the gain of the frequency point corresponding to the updated spectrum array is calculated by the multiplier in the second last column linear operation operator of the applied gain topology circuit; the gain difference is the difference between the gain applied when the updated spectrum array exceeds the preset linear threshold and the gain applied when it is lower than the preset linear threshold;

[0026] The updated spectrum array and the gains of the corresponding frequency points are combined by the tail-column linear operation operator of the applied gain topology circuit to obtain the positive frequency expression.

[0027] As one of the preferred solutions, the calculation of the updated gain of the frequency point corresponding to the spectrum array based on the gain difference and the Boolean value through the multiplier in the second last column linear operation operator of the applied gain topology circuit includes:

[0028] The Boolean value and the audio data to be processed are respectively converted into binary to obtain first input data and second input data, and are input into a multiplier in the second last column operation operator of the applied gain topology circuit; the first input data is the converted Boolean value, and the second input data is the audio data to be processed with the bit width;

[0029] Performing a logical AND operation on the first input data and the second input data bit by bit in sequence to form a first shape data block having the bit width row, and performing effective mapping on the first shape data block to form a second shape data block having the bit width row;

[0030] Rearranging the data in the second shape data block and performing addition operations step by step to gradually reduce the number of rows of the calculation result, repeating the rearrangement and addition operation steps until the number of rows of the calculation result reaches a minimum value, and obtaining a final calculation result outputted from high to low;

[0031] Based on the gain difference and according to the final calculation result, the updated gain of the frequency point corresponding to the frequency spectrum array is calculated.

[0032] A second aspect of the present invention provides an audio data processing system, comprising:

[0033] A parameter adjustment module, used to obtain the audio data to be processed and its business requirements, and adaptively adjust the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator;

[0034] A gain calculation module, used for converting the audio data to be processed from the time domain to the frequency domain, and calculating the gain of the converted audio data to be processed;

[0035] A gain application module, used for calculating the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator, and applying the gain to the frequency spectrum;

[0036] The spectrum conversion module is used to convert the spectrum after the gain is applied to the time domain to obtain processed audio data.

[0037] A third aspect of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the audio data processing method as described above when executing the computer program.

[0038] A fourth aspect of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the audio data processing method as described above is implemented.

[0039] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0040] The present invention discloses an audio data processing method, system, device and medium, the method comprising obtaining audio data to be processed and its business requirements, and adaptively adjusting the operation parameters of a linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator; converting the audio data to be processed from the time domain to the frequency domain, and calculating the gain of the converted audio data to be processed; calculating the spectrum of the converted audio data to be processed by the adjusted linear operation operator, and applying the gain to the spectrum; converting the spectrum after the gain is applied to the time domain to obtain the processed audio data; the application dynamically configures the most suitable operation parameters for the linear operation operator according to the specific business requirements of the audio data to be processed, and then dynamically changes the operation resource ratio and performs effective calculation to achieve the best matching of power consumption and delay at any time section of audio signal processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flowchart of an audio data processing method provided by a certain embodiment of the present invention;

[0043] Figure 2 is a structural diagram of a linear operation operator provided by an embodiment of the present invention;

[0044] Figure 3 is a diagram of an adaptive adjustment process of operation parameters provided by a certain embodiment of the present invention;

[0045] Figure 4 is a diagram of a process of adaptively adjusting operation parameters provided by another embodiment of the present invention;

[0046] Figure 5 is a spectrum calculation process diagram provided by a certain embodiment of the present invention;

[0047] Figure 6 is a structural diagram of an application gain topology circuit provided by a certain embodiment of the present invention;

[0048] Figure 7 is a diagram of a calculation process of a positive frequency expression provided by an embodiment of the present invention;

[0049] Figure 8 is an internal logic diagram of a multiplier provided by a certain embodiment of the present invention;

[0050] Fig. 9 is a flowchart of step S323 provided by an embodiment of the present invention;

[0051] Fig.10 It is a data logic calculation process diagram provided by a certain embodiment of the present invention;

[0052] Fig.11 is a diagram of a first shape data block structure provided by an embodiment of the present invention;

[0053] Fig.12 is a diagram of a second shape data block structure provided by an embodiment of the present invention;

[0054] Fig.13 is a rearrangement process diagram provided by a certain embodiment of the present invention;

[0055] Fig.14 is a system diagram of an audio data processing system provided by a certain embodiment of the present invention;

[0056] Fig.15 It is a structural diagram of an electronic device provided by a certain embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0059] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two components. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the system or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0060] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0061] At present, Quantum architecture FPGAs have been involved in fields such as communications and video signal processing, but are almost blank in the field of audio signal processing, and how to reduce and balance power consumption and delay in audio signal processing is still a major problem. Based on this, this application proposes an audio data processing solution for Quantum architecture FPGAs, which is also an adaptive adjustment solution for hardware. By taking advantage of the characteristics of its logic part and routing part as non-fixed architecture, it balances the power consumption and operation delay in the audio processing process, and fills the gap in the existing Quantum architecture FPGA in the field of audio signal processing, especially in the field of audio signal processing of wearable devices, to achieve a hardware architecture that can dynamically and adaptively adjust the operation scale, operation path, and operation accuracy, thereby greatly optimizing the operation power consumption and operation delay.

[0062] In one embodiment, if Figure 1 As shown, the first aspect of the present invention provides an audio data processing method, comprising:

[0063] S1. Acquire the audio data to be processed and its business requirements, and adaptively adjust the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator;

[0064] Specifically, the present application adjusts the operation parameters of the linear operation operator by the audio data to be processed and its business requirements, that is, the specific situation and requirements of the audio data; wherein the business requirements include audio recognition, synthesis, compression, optimization, noise suppression, etc. The structure of the linear operation operator cell_func is as follows Figure 2 As shown, the number is rows multiplied by columns. Each linear operator includes a set of multipliers and a set of logic operators Math_logic. Figure 2 There are 4 linear operators in it, namely 2 times 2.

[0065] In one embodiment, after obtaining the audio data to be processed and its service requirements, the method further includes:

[0066] The audio data to be processed is preprocessed, and the preprocessing includes but is not limited to denoising and filtering; specifically, preprocessing the audio data to be processed can improve data quality, improve processing effects, and reduce later calculation complexity, thereby improving user experience.

[0067] In one embodiment, the adaptive adjustment process of the operation parameters is as follows: Figure 3 As shown, including:

[0068] S111, configuring the operating voltage and clock main frequency of the audio data to be processed according to the business requirements, and determining the maximum value of the column size according to the clock main frequency;

[0069] Specifically, when the demand is audio synthesis, since the service has high requirements on the computing power of the processor, a higher clock frequency should be selected to provide faster processing speed to ensure the real-time and smoothness of the synthesis process. While maintaining an acceptable range of heat dissipation and power consumption, the operating voltage should be appropriately increased to support a higher clock frequency, but attention should be paid to power consumption and temperature management; when the demand is audio recognition, since the service has high requirements on the response speed and accuracy of the processor, a moderate clock frequency should be selected to balance the processing speed and power consumption to ensure the real-time and accuracy of the recognition process, and the operating voltage should be adjusted according to the selected processor architecture and power consumption characteristics to optimize the power consumption and performance ratio; that is, this application configures the specific operating voltage and clock frequency according to the characteristics of the business needs, and the specific configuration plan is adjusted according to the business needs, which will not be elaborated here. The clock frequency F is related to the maximum value N of the column size. max Inversely proportional, that is, N max =k 1 / F,k 1 is a constant.

[0070] S112, quantifying the delay of the audio data to be processed according to the scale of available resources; wherein the delay is V lm / M*N,V lm is the size of available resources, M is the row size, and N is the column size.

[0071] S113, quantifying the power consumption of the audio data to be processed according to the operating voltage and the clock main frequency, and adjusting the row scale and column scale of the linear operation operator within the range of the column scale and the callable resource scale to minimize the product of the row scale and the column scale, the power consumption and the delay;

[0072] Specifically, since the power consumption in the audio processing process is generated by the charging and discharging of the equivalent capacitor in the circuit due to the jump of data during operation, the power consumption P = CV 2 F=V 2F*εS / d, C is capacitance, ε is dielectric constant, d is equivalent capacitance spacing, S is equivalent capacitance area, S is proportional to M, d is proportional to N, that is, the area of ​​linear operation operator, equivalent capacitance spacing is proportional to column scale, when working voltage and working main frequency are given, operation dynamic power consumption is proportional to equivalent capacitance in operation circuit, proportional to row scale, and inversely proportional to column scale, that is to say, in order to reduce operation power consumption, in theory, column scale should be increased as much as possible and row scale should be reduced if resources permit, but reducing row scale means that fewer operation resources are available for one clock cycle, which will affect computing power, therefore, this application aims to minimize the product of row scale and column scale, power consumption and delay, and adjusts row scale and column scale of linear operation operator, which can fully utilize resources while reducing power consumption and delay, and can balance power consumption and delay well.

[0073] In another embodiment, the adaptive adjustment process of the operation parameters is as follows: Figure 4 As shown, it also includes:

[0074] S121, configuring the operating voltage and clock main frequency of the audio data to be processed according to the business requirements, and quantifying the power consumption of the audio data to be processed according to the operating voltage and clock main frequency;

[0075] S122. Determine the operation accuracy of the audio data to be processed according to the business requirements, and adjust the bit width of the linear operation operator within the scope of the available resource scale with the goal of maximizing the operation accuracy and minimizing the power consumption.

[0076] Specifically, when processing audio data, determining the calculation accuracy is a key step to ensure that audio applications meet specific business requirements. For example, when the requirement is audio synthesis, the generated audio signal should be as close to reality as possible or have a specific style. It is necessary to finely control various parameters of the sound, such as frequency, phase, amplitude, etc., and usually use a sampling rate of 44.1kHz or higher and a bit depth of 16 bits or higher; when the requirement is audio recognition, there are high requirements for the clarity and noise suppression of the voice signal. Generally, a sampling rate of 16kHz and a bit depth of 16 bits are used, and pre-processing technologies such as noise suppression and echo cancellation are used. The specific accuracy determination method will not be elaborated in detail. Since the higher the operation bit width, the higher the operation accuracy, the larger the equivalent total capacitance C, and the higher the power consumption, the present application selects the operation accuracy based on business needs, and adjusts the bit width of the linear operation operator within the range of the available resource scale, with the goal of maximizing the effect that can be achieved with the selected accuracy and minimizing the corresponding power consumption. This can ensure that the processing results achieve the best effect while meeting business needs, reduce quantization errors, and improve the accuracy of the operation results, so as to improve the overall effect, avoid unnecessary waste of resources, and improve energy efficiency.

[0077] S2. Convert the audio data to be processed from the time domain to the frequency domain, and calculate the gain of the converted audio data to be processed; specifically, the present application converts the audio data to be processed from the time domain to the frequency domain through Fourier transform, and uses a conventional gain function to calculate the gain of the converted audio data to be processed, such as a linear gain or a constant gain function, or a dynamic gain adjustment function based on the audio content, which is not specifically limited here.

[0078] S3, calculating the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator, and applying the gain to the frequency spectrum;

[0079] In one embodiment, the spectrum calculation process is as follows: Figure 5 As shown, including:

[0080] S31, initializing the adjusted path address of the linear operation operator to obtain an application gain topology circuit;

[0081] S32. Calculate the converted positive frequency expression and negative frequency expression of the to-be-processed audio data respectively through the applied gain topology circuit to obtain the frequency spectrum.

[0082] Specifically, the gain topology circuit structure is as follows: Figure 6 As shown, the present application uses the application gain topology circuit formed by the path address of the linear operation operator after initialization to calculate the frequency spectrum of the audio data to be processed, which can maximize the use of resources and improve processing efficiency while balancing power consumption and delay; by calculating the positive frequency and negative frequency expressions separately, different parts of the audio data can be accurately controlled to achieve a more refined audio processing effect.

[0083] In one embodiment, the process of calculating the positive frequency expression of the audio data to be processed is as follows: Figure 7 As shown, including:

[0084] S321, calculating the difference between the spectrum array corresponding to the positive frequency and a preset linear threshold value through a logic operator in the first column of the linear operation operator of the applied gain topology circuit, and obtaining a calculation result represented by a Boolean value;

[0085] S322, by applying the linear operation operator of the second column to the third to last column of the gain topology circuit, retrieve the base value element corresponding to the Boolean value from the base value array to update the spectrum array;

[0086] S323, based on the gain difference and the Boolean value, calculating the gain of the frequency point corresponding to the updated spectrum array through the multiplier in the second last column linear operation operator of the applied gain topology circuit; the gain difference is the difference between the gain applied when the updated spectrum array exceeds the preset linear threshold and the gain applied when it is lower than the preset linear threshold;

[0087] S324. Combining the updated spectrum array and the gains of the corresponding frequency points thereof through the tail-column linear operation operator of the applied gain topology circuit to obtain the positive frequency expression.

[0088] Specifically, Figure 6 The application gain topology circuit (16 rows × 4 columns) shown in the figure is used as an example to describe the calculation process of the positive frequency expression of the audio data to be processed. This configuration can calculate 16 elements in spectrum[k] in parallel each time:

[0089] Assume that the frequency domain signal spectrum(k) has a total of 256 frequency points. First, the comparison function is calculated through the first column of linear operators, that is, the absolute value of each spectrum[i] is calculated and multiplied by 1, and then subtracted from the preset linear threshold (threshold) to obtain the difference. The index, that is, the Boolean value, is determined based on the highest bit (i.e., the sign bit) of the difference. If the highest bit is 1, it means that the absolute value of spectrum[i] is greater than the threshold; if it is 0, it means that the absolute value of spectrum[i] is less than or equal to the threshold. In essence, this step can be simplified to directly passing the absolute value of spectrum[i].

[0090] Then, the calculation of spectrum[k]+base is directly performed through the second column of linear operators, and the result is passed to the next column for processing; where baseArray is a two-dimensional array, for each element, index = 0, base returns 0, index = 1, base returns threshold;

[0091] Then the multiplier in the third column of linear operators, the internal logic of the multiplier is as follows Figure 8As shown, the index calculated in the first column is used as input to select or calculate the gain gain; wherein gainArray is a two-dimensional array, and for each gain, index=0 returns gain_below_linear, index=1 returns gain_above_linear, and gain=gainArray(index); that is, this column multiplier uses index as a selection condition or index, and directly selects the gain object based on the gain difference gain_delta_liner=gain_above_linear-gain_below_liner to calculate the gain of the frequency point corresponding to the updated spectrum array;

[0092] Finally, the fourth column of linear operation operators is used to calculate the spectrum gain spectrum_gain(k) after considering the base value and gain adjustment, that is, the result of the second column (spectrum[k]+base) is multiplied by the gain calculated in the third column to obtain the positive frequency expression of the audio data to be processed spectrum_gain(k)=(spectrum[k]+base)*gain.

[0093] The present application realizes complex processing of the audio spectrum through a series of multiplication and addition operations of linear operation operators. By setting the bit width of the multiplier 1 to 1 bit, not only will the computational complexity not be increased, but also the multiplication operation will not become a performance bottleneck, thereby efficiently utilizing the multiplier resources and reducing the need for additional comparators; the threshold-based gain adjustment realizes flexible gain control, and can automatically adjust the gain according to different parts of the spectrum, thereby optimizing the dynamic range of the audio signal or emphasizing specific frequency components; the base value addition operation can perform global or local offset adjustment on the spectrum, which helps to improve the sound quality of the audio signal; the spectrum gain calculation combines the effects of base value addition and gain adjustment, which is crucial for applications such as audio signal enhancement, compression, and equalization; the present application divides the processing flow into multiple columns (or stages), each column performs a specific task (such as comparison, base value addition, gain calculation, and spectrum gain calculation), making the entire system more modular and scalable. Since each column performs a relatively independent task and the output can be used as the input of the subsequent column, the solution can be easily integrated into a larger audio processing system or signal processing framework.

[0094] In one embodiment, step S323 is as follows: Fig. 9 As shown, including:

[0095] S3231, performing binary conversion on the Boolean value and the audio data to be processed respectively to obtain first input data and second input data, and inputting the data into a multiplier in the second last column operation operator of the applied gain topology circuit; the first input data is the converted Boolean value, and the second input data is the audio data to be processed having the bit width;

[0096] S3232, performing a logical AND operation on the first input data and the second input data bit by bit in sequence to form a first shape data block having the bit width row, and performing effective mapping on the first shape data block to form a second shape data block having the bit width row;

[0097] S3233, rearranging the data in the second shape data block, and performing addition operations step by step, so as to gradually reduce the number of rows of the calculation result, repeating the rearranging and addition operation steps until the number of rows of the calculation result reaches a minimum value, and obtaining a final calculation result outputted from high to low;

[0098] S3234. Based on the gain difference and according to the final calculation result, calculate the updated gain of the frequency point corresponding to the frequency spectrum array.

[0099] Specifically, each multipier includes four modules: serial_and, array_mapping, serial_reorder, and serial_adder. This application uses a 4-bit × 4-bit multiplier to illustrate the calculation process of the multiplier in the linear operation operator:

[0100] First, the numerical value returned by the Boolean value and the audio data to be processed are converted into binary respectively to obtain the first input data DCBA and the second input data dcba as the input of the multiplier;

[0101] Next, the eight letters DCBA and dcba represent 0 or 1 in binary respectively, and the data logic calculation process is as follows Fig.10 As shown, the multiplication of two 4-bit binary numbers DCBA and dcba can be simplified to the corresponding bit AND of them, and then the corresponding shift and then addition. This application uses the numbers 3, 2, 1, 0 to replace D, C, B, A or d, c, b, a. When the bit width of the two multipliers is 4, the serial_and module is responsible for performing logical AND operations on the sum of the two input multipliers in sequence, that is, the leftmost rectangle [3 0] in the first row represents mul2[3]&

[0102] mul1[0], corresponding to Fig.10 Da, the rightmost rectangle [0 0] in row 1 represents mul2[0]&mul1[0], corresponding to Fig.10 Aa, the leftmost rectangle [3 3] in the 4th row represents mul2[3]&mul1[3, corresponding to Fig.10 Dd, the rightmost rectangle [0 3] in the 4th row represents mul2[0]&mul1[3], corresponding to Fig.10 aD, and so on..., finally forming a first shape data block with 4 rows. The structure of the first shape data block is as follows Fig.11 Then, the Array_mapping module is responsible for converting Fig.11 The parallelogram data block in is effectively mapped to form a second shape data block, whose structure is as follows: Fig.12 The triangle shown.

[0103] Then, the serial_reorder module will Fig.12 After the numbers of the triangle data blocks in are rearranged, the serial_adder module performs addition operations step by step. The rearrangement process is as follows Fig.13 As shown, the process is repeated until the number of rows of the calculation result reaches the minimum value 1 to obtain the final result; wherein, HS_*_* represents the value obtained after the two numbers are passed through the 1-bit half adder, FS_*_* represents the value obtained after the two numbers are passed through the 1-bit full adder, and FC_*_* represents the carry obtained after the two numbers are passed through the 1-bit full adder. Since the bit width in this example is 4, the reordering will go through 4 stackings. The first stacking: after serial_reorder reordering, serial_adder finds: {[1 3], [1 2]}+{[0 3]}, the result is: {FC_2_2, FS_2_2, HS_2_3}; the second stacking: after serial_reorder reordering, serial_adder finds: {[2 3], [2 2], [2 1], [1 1]}+{FC_2_2, FS_2_2, HS_2_1, [0 2]}, the result is: {FC_1_1, FS_1_1, FS_1_2, FS_1_3, HS_1_4}; the third stacking: after serial_reorder reordering, serial_adder finds: {[3 3], [3 2], [3 1], [3 0], [2 0], [1 0]}+{FC_1_1, FS_1_1, FS_1_2, FS_1_3, HS_1_4, [0 1]} results in: {FC_0_0, FS_0_0, FS_0_1, FS_0_2, FS_0_3, FS_0_4, HS_0_5}; after the fourth stacking, serial_reorder reordering, the results are directly output in the order of reordering from low to high.

[0104] Finally, based on the gain difference, the gain of the corresponding frequency point of the updated spectrum array can be calculated according to the final calculation result. It is not difficult to see from the entire multiplication process that if you want to realize the multiplication of two multipliers with a bit width of n, you need to consume n^2 AND gates and n! 1-bit adders. This application dynamically configures the bit width of the linear operation operator so that the audio processing process can more reasonably and effectively call logic resources dynamically according to different operation accuracy requirements, and other idle logic resources are used for wiring, so as to optimize the operation power consumption.

[0105] In addition, the present application improves the data processing rate and efficiency through efficient data processing - binary conversion and logical AND operation; the effective mapping of shape data blocks can achieve more precise control of the gain of audio data, and realizes efficient accumulation and aggregation of audio data by performing addition operations step by step and reducing the number of rows of data blocks, which not only reduces the amount of calculation, but also makes the gain adjustment smoother and continuous; the calculation based on gain difference ensures the accuracy of gain adjustment, and can be dynamically adjusted according to the characteristics of the audio signal; and the various steps in the scheme are independent of each other, making the entire processing flow clearer and easier to manage, and also convenient for expansion or modification when needed, providing a powerful solution for the field of audio processing.

[0106] In addition, the expression of negative frequency is spectrum_gain(p)=(spectrum[p]+base)*gain(p=256-k+1), and its calculation process is similar to that of positive frequency, which will not be elaborated here. After combining the expression of positive frequency with the expression of negative frequency, the spectrum of the audio data to be processed can be obtained, and the calculated gain is applied to the spectrum accordingly.

[0107] S4. Convert the spectrum after the gain is applied to the time domain to obtain processed audio data. Specifically, the spectrum after the gain is applied can be converted to the time domain by inverse Fourier transform to obtain processed audio data.

[0108] In the embodiment of the present application, based on the balance problem between power consumption and delay in the audio data processing process, an audio data processing method is designed, which realizes the acquisition of audio data to be processed and its business needs, and adaptively adjusts the operation parameters of the linear operation operator according to the audio data to be processed and the business needs; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator; the audio data to be processed is converted from the time domain to the frequency domain, and the gain of the converted audio data to be processed is calculated; the spectrum of the converted audio data to be processed is calculated by the adjusted linear operation operator, and the gain is applied to the spectrum; the spectrum after the gain is applied is converted to the time domain to obtain the technical solution of the processed audio data; the operation resource ratio is changed by dynamically configuring the operation parameters to achieve the best matching of power consumption and delay of audio signal processing at any time section.

[0109] It should be noted that although the steps in the above flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders.

[0110] In another embodiment, if Fig.14 As shown, the second aspect of the present invention provides an audio data processing system, comprising:

[0111] A parameter adjustment module 10 is used to obtain the audio data to be processed and its business requirements, and adaptively adjust the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator;

[0112] A gain calculation module 20, used for converting the audio data to be processed from the time domain to the frequency domain, and calculating the gain of the converted audio data to be processed;

[0113] A gain application module 30, configured to calculate the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator, and apply the gain to the frequency spectrum;

[0114] The spectrum conversion module 40 is used to convert the spectrum after the gain is applied to the time domain to obtain processed audio data.

[0115] It should be noted that the various modules in the above-mentioned audio data processing system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. For the specific definition of an audio data processing system, please refer to the definition of an audio data processing method above. The two have the same functions and effects, which will not be repeated here.

[0116] A third aspect of the present invention provides an electronic device, the electronic device comprising:

[0117] processor, memory, and bus;

[0118] The bus is used to connect the processor and the memory;

[0119] The memory is used to store operation instructions;

[0120] The processor is used to call the operation instruction, and the executable instruction enables the processor to perform an operation corresponding to an audio data processing method as shown in the first aspect of the present application.

[0121] In an alternative embodiment, an electronic device is provided, such as Fig.15 As shown, Fig.15 The electronic device 5000 shown includes: a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, such as through a bus 5002. Optionally, the electronic device 5000 may also include a transceiver 5004. It should be noted that in actual applications, the transceiver 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation on the embodiments of the present application.

[0122] Processor 5001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 5001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0123] The bus 5002 may include a path to transmit information between the above components. The bus 5002 may be a PCI bus or an EISA bus, etc. The bus 5002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0124] The memory 5003 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compressed optical disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0125] The memory 5003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 5001. The processor 5001 is used to execute the application code stored in the memory 5003 to implement the content shown in any of the above method embodiments.

[0126] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0127] A fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, an audio data processing method shown in the first aspect of the present application is implemented.

[0128] Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0129] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented.

[0130] In summary, the present invention relates to the field of computer technology, and discloses an audio data processing method, system, device and medium, the method comprising obtaining audio data to be processed and its business requirements, and adaptively adjusting the operation parameters of a linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator; converting the audio data to be processed from the time domain to the frequency domain, and calculating the gain of the converted audio data to be processed; calculating the spectrum of the converted audio data to be processed by the adjusted linear operation operator, and applying the gain to the spectrum; converting the spectrum after the gain is applied to the time domain to obtain processed audio data; changing the operation resource ratio by dynamically configuring the operation parameters to achieve the best matching of power consumption and delay of audio signal processing at any time section.

[0131] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. An audio data processing method, characterized in that: include: Acquire the audio data to be processed and its business requirements, and adaptively adjust the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator; Convert the audio data to be processed from the time domain to the frequency domain, and calculate the gain of the converted audio data to be processed; Calculate the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator, and apply the gain to the frequency spectrum; The spectrum after the gain is applied is converted to the time domain to obtain processed audio data.

2. The audio data processing method according to claim 1, characterized in that: After obtaining the audio data to be processed and its business requirements, the method further includes: The audio data to be processed is preprocessed, and the preprocessing includes but is not limited to denoising and filtering.

3. The audio data processing method according to claim 1, characterized in that: The adaptively adjusting the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements includes: configuring the operating voltage and clock frequency of the audio data to be processed according to the business requirements, and determining the maximum value of the column size according to the clock frequency; quantifying the delay of the audio data to be processed according to the scale of available resources; The power consumption of the audio data to be processed is quantified according to the operating voltage and the clock frequency, and within the range of the column scale and the callable resource scale, the row scale and the column scale of the linear operation operator are adjusted with the goal of minimizing the product of the row scale and the column scale, the power consumption and the delay.

4. The audio data processing method according to claim 3, characterized in that: The adaptively adjusting the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements also includes: configuring the operating voltage and clock main frequency of the audio data to be processed according to the service requirements, and quantifying the power consumption of the audio data to be processed according to the operating voltage and clock main frequency; The operation precision of the audio data to be processed is determined according to the business requirements, and within the scope of the available resource scale, the bit width of the linear operation operator is adjusted with the goal of maximizing the operation precision and minimizing the power consumption.

5. The audio data processing method according to claim 1, characterized in that: The step of calculating the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator comprises: Initialize the adjusted path address of the linear operation operator to obtain an application gain topology circuit; The frequency spectrum is obtained by respectively calculating the converted positive frequency expression and negative frequency expression of the to-be-processed audio data through the applied gain topology circuit.

6. The audio data processing method according to claim 5, characterized in that: The step of calculating the positive frequency expression of the converted audio data to be processed by applying the gain topology circuit comprises: Calculating the difference between the spectrum array corresponding to the positive frequency and a preset linear threshold value through a logic operator in the first column of the linear operation operator of the applied gain topology circuit to obtain a calculation result represented by a Boolean value; By using the linear operation operator of the second column to the third to last column of the applied gain topology circuit, the base value element corresponding to the Boolean value is retrieved from the base value array to update the spectrum array; Based on the gain difference and the Boolean value, the gain of the frequency point corresponding to the updated spectrum array is calculated by the multiplier in the second last column linear operation operator of the applied gain topology circuit; the gain difference is the difference between the gain applied when the updated spectrum array exceeds the preset linear threshold and the gain applied when it is lower than the preset linear threshold; The updated spectrum array and the gains of the corresponding frequency points are combined by the tail-column linear operation operator of the applied gain topology circuit to obtain the positive frequency expression.

7. The audio data processing method according to claim 6, characterized in that: The step of calculating the updated gain of the frequency point corresponding to the spectrum array based on the gain difference and the Boolean value through a multiplier in the second last column linear operation operator of the applied gain topology circuit comprises: The Boolean value and the audio data to be processed are respectively converted into binary to obtain first input data and second input data, and are input into a multiplier in the second last column operation operator of the applied gain topology circuit; the first input data is the converted Boolean value, and the second input data is the audio data to be processed with the bit width; Performing a logical AND operation on the first input data and the second input data bit by bit in sequence to form a first shape data block having the bit width row, and performing effective mapping on the first shape data block to form a second shape data block having the bit width row; Rearranging the data in the second shape data block and performing addition operations step by step to gradually reduce the number of rows of the calculation result, repeating the rearrangement and addition operation steps until the number of rows of the calculation result reaches a minimum value, and obtaining a final calculation result outputted from high to low; Based on the gain difference and according to the final calculation result, the updated gain of the frequency point corresponding to the frequency spectrum array is calculated.

8. An audio data processing system, characterized in that: include: A parameter adjustment module, used to obtain the audio data to be processed and its business requirements, and adaptively adjust the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator; A gain calculation module, used for converting the audio data to be processed from the time domain to the frequency domain, and calculating the gain of the converted audio data to be processed; A gain application module, used for calculating the converted frequency spectrum of the to-be-processed audio data by using the adjusted linear operation operator, and applying the gain to the frequency spectrum; The spectrum conversion module is used to convert the spectrum after the gain is applied to the time domain to obtain processed audio data.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the audio data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the audio data processing method according to any one of claims 1 to 7 is implemented.

Images