An audio data processing method, system, device and medium

Abstract

The application relates to the technical field of computers and discloses an audio data processing method, system, device and medium, the method comprising the following steps: acquiring to-be-processed audio data and service requirements thereof, and adaptively adjusting operation parameters of a linear operation operator according to the to-be-processed audio data and the service requirements; the operation parameters comprise a row scale, a column scale and a bit width, and the linear operation operator comprises a multiplier and a logic operator; converting the to-be-processed audio data from a time domain to a frequency domain, and calculating the gain of the to-be-processed audio data after conversion; calculating the spectrum of the to-be-processed audio data after conversion through the linear operation operator after adjustment, and applying the gain to the spectrum; converting the spectrum after the gain is applied to the time domain to obtain processed audio data; and changing operation resource allocation by dynamically configuring operation parameters, so that the best matching between audio signal processing and time section power consumption and delay is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an audio data processing method, system, device and medium. BACKGROUND

[0002] In audio signal processing, power consumption and latency are two important issues, which have a direct impact on the performance of audio processing devices and user experience; although there are technologies such as low-power processors and optimized algorithms, parallel processing and hardware acceleration, sampling rate control, power management strategies, buffer and pipeline design to improve power consumption and latency, but there are still performance and power consumption balance challenges.

[0003] Therefore, how to solve the balance problem of power consumption and latency in the process of audio data processing has become a technical problem to be solved by those skilled in the art. SUMMARY

[0004] The present application provides an audio data processing method, system, device and medium, which solves the balance problem of power consumption and latency in the process of audio data processing.

[0005] To solve the above technical problems, the first aspect of the present application provides an audio data processing method, comprising:

[0006] Obtaining audio data to be processed and its business requirements, and adaptively adjusting the operation parameters of a linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator;

[0007] Converting the audio data to be processed from time domain to frequency domain, and calculating the gain of the converted audio data to be processed;

[0008] Calculating the spectrum of the converted audio data to be processed by the adjusted linear operation operator, and applying the gain to the spectrum;

[0009] Converting the spectrum after applying the gain to the time domain to obtain processed audio data.

[0010] As one of the preferred schemes, after obtaining the audio data to be processed and its business requirements, the method further comprises:

[0011] Pretreating the audio data to be processed, which includes but is not limited to denoising and filtering.

[0012] As one of the preferred schemes, the adaptive adjustment of the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements comprises:

[0013] configuring a working voltage and a clock frequency of the to-be-processed audio data according to the service requirement, and determining a maximum value of the column scale according to the clock frequency;

[0014] quantifying a delay of the to-be-processed audio data according to a callable resource scale;

[0015] quantifying a power consumption of the to-be-processed audio data according to the working voltage and the clock frequency, and adjusting a row scale and the column scale of the linear operation operator in a range of the column scale and the callable resource scale, with a target of minimizing a product of the row scale and the column scale, the power consumption and the delay.

[0016] As one of the preferred solutions, the self-adaptive adjustment of the operation parameters of the linear operation operator according to the to-be-processed audio data and the service requirement further comprises:

[0017] configuring a working voltage and a clock frequency of the to-be-processed audio data according to the service requirement, and quantifying a power consumption of the to-be-processed audio data according to the working voltage and the clock frequency;

[0018] determining an operation precision of the to-be-processed audio data according to the service requirement, and adjusting a bit width of the linear operation operator in a range of the callable resource scale, with a target of maximizing the operation precision and minimizing the power consumption.

[0019] As one of the preferred solutions, the calculation of the spectrum of the converted to-be-processed audio data by the adjusted linear operation operator comprises:

[0020] initializing a path address of the adjusted linear operation operator to obtain an application gain topology circuit;

[0021] calculating a positive frequency expression and a negative frequency expression of the converted to-be-processed audio data respectively by the application gain topology circuit to obtain the spectrum.

[0022] As one of the preferred solutions, the calculation of the positive frequency expression of the converted to-be-processed audio data by the application gain topology circuit comprises:

[0023] calculating, by a logic operator in a first column linear operation operator of the application gain topology circuit, a difference between a spectrum array corresponding to the positive frequency and a preset linear threshold to obtain a calculation result represented by a Boolean value;

[0024] updating, by a second column linear operation operator to a third from last column linear operation operator of the application gain topology circuit, the spectrum array from a base value element corresponding to the Boolean value in a base value array;

[0025] Based on the gain difference and the Boolean value, the gain of the corresponding frequency point of the updated spectrum array is calculated by the multiplier in the next tail column linear operation operator of the application gain topology circuit; the gain difference is the difference between the gain applied when the updated spectrum array exceeds the preset linear threshold and the gain applied when the updated spectrum array is lower than the preset linear threshold.

[0026] The updated spectrum array and the gain of the corresponding frequency point thereof are combined by the tail column linear operation operator of the application gain topology circuit to obtain the positive frequency expression.

[0027] As one of the preferred schemes, based on the gain difference and the Boolean value, the gain of the corresponding frequency point of the updated spectrum array is calculated by the multiplier in the next tail column linear operation operator of the application gain topology circuit, comprising:

[0028] The Boolean value and the to-be-processed audio data are respectively converted into binary to obtain first input data and second input data, and are input to the multiplier in the next tail column linear operation operator of the application gain topology circuit; the first input data is the converted Boolean value, and the second input data is the to-be-processed audio data with the bit width bit;

[0029] The first input data and the second input data are sequentially subjected to logical AND operation bit by bit to form a first shape data block with the bit width row, and the first shape data block is subjected to effective mapping to form a second shape data block with the bit width row;

[0030] The data in the second shape data block is rearranged, and is subjected to addition operation step by step to gradually reduce the number of rows of the calculation result, and the rearrangement and addition operation steps are repeated until the number of rows of the calculation result reaches a minimum value, so that a final calculation result output from high to low is obtained;

[0031] Based on the gain difference, the gain of the corresponding frequency point of the updated spectrum array is calculated according to the final calculation result.

[0032] The second aspect of the application provides an audio data processing system, comprising:

[0033] A parameter adjustment module is configured to obtain to-be-processed audio data and service requirements thereof, and to adaptively adjust operation parameters of a linear operation operator according to the to-be-processed audio data and the service requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator.

[0034] a gain calculation module, configured to convert the to-be-processed audio data from a time domain to a frequency domain, and calculate a gain of the converted to-be-processed audio data;

[0035] a gain application module, configured to calculate a spectrum of the converted to-be-processed audio data by using the adjusted linear operation operator, and apply the gain to the spectrum;

[0036] a spectrum conversion module, configured to convert the spectrum to which the gain is applied from the frequency domain to the time domain, to obtain processed audio data.

[0037] The third aspect of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the audio data processing method as described above when executing the computer program.

[0038] The fourth aspect of the present application provides a computer-readable storage medium, comprising a stored computer program, wherein the device in which the computer-readable storage medium is located implements the audio data processing method as described above when executing the computer program.

[0039] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0040] The present application discloses an audio data processing method, system, device and medium, the method comprising obtaining to-be-processed audio data and its service requirement, and adaptively adjusting operation parameters of a linear operation operator according to the to-be-processed audio data and the service requirement; the operation parameters comprise a row scale, a column scale and a bit width, and the linear operation operator comprises a multiplier and a logic operator; the to-be-processed audio data is converted from a time domain to a frequency domain, and a gain of the converted to-be-processed audio data is calculated; a spectrum of the converted to-be-processed audio data is calculated by using the adjusted linear operation operator, and the gain is applied to the spectrum; the spectrum to which the gain is applied is converted from the frequency domain to the time domain, to obtain processed audio data; the present application dynamically configures the most suitable operation parameters for the linear operation operator according to the specific service requirement of the to-be-processed audio data, and then dynamically changes the operation resource ratio and performs effective calculation, so as to achieve the best matching of power consumption and delay in any time section of audio signal processing. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0042] Figure 1 is a flowchart of an audio data processing method according to an embodiment of the present application;

[0043] Figure 2 is a structure diagram of a linear operation operator according to an embodiment of the present application;

[0044] Figure 3 is a flowchart of an adaptive adjustment process of operation parameters according to an embodiment of the present application;

[0045] Figure 4 is a flowchart of an adaptive adjustment process of operation parameters according to another embodiment of the present application;

[0046] Figure 5 is a flowchart of a spectrum calculation process according to an embodiment of the present application;

[0047] Figure 6 is a structure diagram of an application gain topology circuit according to an embodiment of the present application;

[0048] Figure 7 is a flowchart of a positive frequency expression calculation process according to an embodiment of the present application;

[0049] Figure 8 is an internal logic diagram of a multiplier according to an embodiment of the present application;

[0050] Figure 9 is a flowchart of step S323 according to an embodiment of the present application;

[0051] Figure 10 is a flowchart of a data logic calculation process according to an embodiment of the present application;

[0052] Figure 11 is a structure diagram of a first shape data block according to an embodiment of the present application;

[0053] Figure 12 is a structure diagram of a second shape data block according to an embodiment of the present application;

[0054] Figure 13 is a flowchart of a rearrangement process according to an embodiment of the present application;

[0055] Figure 14 is a system diagram of an audio data processing system according to an embodiment of the present application;

[0056] Figure 15 is a structure diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] With reference to the drawings and embodiments, the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. The purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0058] In the description of the present application, the terms "first", "second", "third" and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0059] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are only for the purpose of description, and cannot be understood as indicating or implying that the system or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood in specific cases.

[0060] In the description of the present application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by those skilled in the art of the present technology. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood in specific cases.

[0061] At present, the FPGA of Quantum architecture has been involved in the fields of communication and video signal processing, but is almost blank in the field of audio signal processing, and how to weaken and balance the power consumption and delay in audio signal processing is still a big problem. Based on this, the application provides an audio data processing scheme for the FPGA of Quantum architecture, which is also a self-adaptive adjustment scheme for hardware. By means of the characteristics of the logic part and the routing part for the non-fixed architecture, the power consumption and operation delay in the audio processing process are balanced, and the blank of the FPGA of Quantum architecture in the field of audio signal processing, especially in the field of audio signal processing of wearable devices, is filled, so as to realize a hardware architecture which can dynamically and adaptively adjust the operation scale, operation path and operation precision, and greatly optimize the operation power consumption and operation delay.

[0062] In an embodiment, as shown in Figure 1 , the first aspect of the application provides an audio data processing method, comprising:

[0063] S1, obtaining to-be-processed audio data and its service demand, and adaptively adjusting operation parameters of a linear operation operator according to the to-be-processed audio data and the service demand; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logic operator;

[0064] Specifically, the application adjusts the operation parameters of the linear operation operator through the to-be-processed audio data and its service demand, i.e. the specific situation and demand of the audio data; wherein the service demand includes audio recognition, synthesis, compression, optimization, noise suppression, etc., and the structure of the linear operation operator cell_func is as shown in Figure 2 , the number of which is the product of rows and columns, and each linear operation operator contains a set of multipliers multiplier and a set of logic operators Math_logic. Figure 2 In the embodiment, there are four, i.e. two linear operation operators.

[0065] In an embodiment, after obtaining the to-be-processed audio data and its service demand, the method further comprises:

[0066] preprocessing the to-be-processed audio data, which includes but is not limited to denoising and filtering; specifically, preprocessing the to-be-processed audio data can improve the data quality, improve the processing effect, reduce the complexity of later calculation, and thus improve the user experience.

[0067] In an embodiment, the adaptive adjustment process of the operation parameters is as shown in Figure 3 , comprising:

[0068] S111. Configure the operating voltage and clock frequency of the audio data to be processed according to the business requirements, and determine the maximum value of the column size according to the clock frequency;

[0069] Specifically, when the requirement is audio synthesis, since this service has high requirements for processor computing power, a higher clock frequency should be selected to provide faster processing speed and ensure the real-time performance and smoothness of the synthesis process. While maintaining acceptable heat dissipation and power consumption, the operating voltage should be appropriately increased to support a higher clock frequency, but attention must be paid to power consumption and temperature management. When the requirement is audio recognition, since this service has high requirements for processor response speed and accuracy, a moderate clock frequency should be selected to balance processing speed and power consumption, ensuring the real-time performance and accuracy of the recognition process. The operating voltage should be adjusted according to the selected processor architecture and power consumption characteristics to optimize the power consumption and performance ratio. In other words, this application configures specific operating voltages and clock frequencies according to the characteristics of the business requirements. Specific configuration schemes are adjusted according to business requirements and will not be elaborated further here. The clock frequency F and the maximum value N of the column size... max Inversely proportional, i.e., N max = k1 / F, where k1 is a constant.

[0070] S112. Quantize the latency of the audio data to be processed according to the available resource scale; wherein, the latency is V. lm / M*N,V lm M represents the size of the callable resources, where M is the row size and N is the column size.

[0071] S113. Quantize the power consumption of the audio data to be processed according to the operating voltage and clock frequency, and within the range of the column size and the available resource size, adjust the row size and column size of the linear operation operator with the goal of minimizing the product of the row size and the column size, the power consumption and the delay.

[0072] Specifically, since the power consumption during audio processing is generated by the charging and discharging of the equivalent capacitance in the circuit due to data transitions during computation, the power consumption P = CV. 2 F = V 2F*εS / d, C is a capacitance, ε is a dielectric constant, d is an equivalent capacitance spacing, S is an equivalent capacitance area, S is proportional to M, and d is proportional to N, that is, the area of the linear operation operator, the equivalent capacitance spacing is proportional to the column size, under the given working voltage and working frequency, the operation dynamic power consumption is proportional to the equivalent capacitance in the operation circuit, proportional to the row size, and inversely proportional to the column size, that is, in order to reduce the operation power consumption, theoretically, the column size should be increased as much as possible and the row size should be reduced as much as possible under the condition of resource permission, but reducing the row size means that the operation resources available in a clock cycle are less, which will affect the computing power, therefore, the application takes minimizing the product of the row size and the column size, power consumption and delay as the target, adjusts the row size and the column size of the linear operation operator, can fully utilize the resources while reducing the power consumption and the delay, and can well balance the power consumption and the delay.

[0073] In another embodiment, the adaptive adjustment process of the operation parameter as shown in Figure 4 includes the following steps:

[0074] S121, configuring a working voltage and a clock frequency of the to-be-processed audio data according to the business requirement, and quantifying power consumption of the to-be-processed audio data according to the working voltage and the clock frequency;

[0075] S122, determining operation precision of the to-be-processed audio data according to the business requirement, and adjusting a bit width of the linear operation operator in the range of the invocable resource size, taking maximizing the operation precision and minimizing the power consumption as the target.

[0076] Specifically, in processing the audio data, determining the operation precision is a key step to ensure that the audio application meets the specific business requirement. For example, when the requirement is audio synthesis, the generated audio signal is as close as possible to the real one or has a specific style in hearing, and each parameter of the sound such as frequency, phase and amplitude needs to be controlled finely, usually using a sampling rate of 44.1 kHz or higher and using a bit depth of 16 bits or higher; for example, when the requirement is audio recognition, there is a higher requirement for the intelligibility of the voice signal and noise suppression, and generally a sampling rate of 16 kHz and a bit depth of 16 bits are adopted, and noise suppression, echo cancellation and other preprocessing technologies are adopted, and the specific precision determination method is not described in detail. Since the higher the operation bit width is, the higher the operation precision is, the larger the equivalent total capacitance C is, and the higher the power consumption is, the application selects the operation precision according to the business requirement, takes the effect that can be achieved by the selected precision as the maximum and the required power consumption as the minimum as the target in the range of the invocable resource size, to adjust the bit width of the linear operation operator, which can ensure that the processing result reaches the best effect under the premise of meeting the business requirement, reduces the quantization error, improves the precision of the operation result, improves the overall effect, avoids unnecessary resource waste, and improves the energy efficiency ratio.

[0077] S2, converting the to-be-processed audio data from time domain to frequency domain, and calculating the gain of the converted to-be-processed audio data; specifically, the present application converts the to-be-processed audio data from time domain to frequency domain through Fourier transform, and adopts a conventional gain function to calculate the gain of the converted to-be-processed audio data, such as a linear gain or a constant gain function, or a dynamic gain adjustment function based on audio content, which is not specifically limited herein.

[0078] S3, calculating the frequency spectrum of the converted to-be-processed audio data through the adjusted linear operation operator, and applying the gain to the frequency spectrum;

[0079] In an embodiment, the frequency spectrum calculation process is as shown in Figure 5

[0080] S31, initializing the path address of the adjusted linear operation operator to obtain an application gain topology circuit;

[0081] S32, calculating the positive frequency expression and the negative frequency expression of the converted to-be-processed audio data through the application gain topology circuit respectively to obtain the frequency spectrum.

[0082] Specifically, the application gain topology circuit structure is as shown in Figure 6 The present application adopts the application gain topology circuit formed by the initialized path address of the linear operation operator to calculate the frequency spectrum of the to-be-processed audio data, which can maximize the use of resources while balancing power consumption and delay, and improve processing efficiency; by calculating the positive frequency expression and the negative frequency expression respectively, different parts of the audio data can be accurately controlled, and more fine audio processing effect can be achieved.

[0083] In an embodiment, the process of calculating the positive frequency expression of the to-be-processed audio data is as shown in Figure 7

[0084] S321, calculating the difference between the frequency spectrum array corresponding to the positive frequency and the preset linear threshold value through the logic operation unit in the first column of linear operation operators of the application gain topology circuit to obtain a calculation result represented by a Boolean value;

[0085] S322, calling the base value element corresponding to the Boolean value from the base value array through the second column to the third from last column of linear operation operators of the application gain topology circuit to update the frequency spectrum array;

[0086] ​​S323, calculating the gain of the updated spectrum array corresponding to the frequency point by the multiplier in the next tail column linear operation operator of the application gain topology circuit based on the gain difference and the Boolean value; the gain difference is the difference between the gain applied when the updated spectrum array exceeds the preset linear threshold and the gain applied when the updated spectrum array is lower than the preset linear threshold;

[0087] S324, combining the updated spectrum array and the gain of the corresponding frequency point by the tail column linear operation operator of the application gain topology circuit to obtain the positive frequency expression.

[0088] Specifically, taking the application gain topology circuit (16 rows x 4 columns) shown in Figure 6 as an example to describe the positive frequency expression calculation process of the audio data to be processed, this configuration can calculate 16 elements in spectrum[k] in parallel each time:

[0089] Assuming that the frequency domain signal specturm(k) has a total of 256 frequency points, first, the first column linear operation operator is used to calculate the comparison function, that is, the absolute value of each spectrum[i] is multiplied by 1, and the difference with the preset linear threshold (threshold) is calculated, and the index, that is, the Boolean value, is determined based on the highest bit (that is, the sign bit) of the difference, if the highest bit is 1, it indicates that the absolute value of spectrum[i] is greater than threshold; if it is 0, it indicates that the absolute value of spectrum[i] is less than or equal to threshold, in essence, this step can be simplified as directly passing the absolute value of spectrum[i];

[0090] Then, the second column linear operation operator directly performs the calculation of spectrum[k]+base, and the result is passed to the next column for processing; wherein, baseArray is a two-dimensional array, for each element, index=0, base returns 0, index=1, base returns threshold;

[0091] Then, the multiplier in the third column linear operation operator, the internal logic of the multiplier is as follows Figure 8As shown, the index calculated in the first column is used as input to select or calculate the gain; where gainArray is a two-dimensional array, for each gain, index=0 returns gain below linear, index=1 returns gain above linear, and gain=gainArray(index); that is, this column multiplier uses index as a selection condition or index to directly select the gain object based on the gain difference gain_delta_liner=gain above linear-gain below linear to calculate the updated gain of the frequency point corresponding to the spectrum array;

[0092] Finally, the fourth column linear operation operator is used to calculate the spectrum gain spectrum gain(k) considering the base value and the gain adjustment, that is, the result of the second column (spectrum[k]+base) is multiplied by the gain calculated in the third column to obtain the positive frequency expression spectrum gain(k)=(spectrum[k]+base)*gain of the audio data to be processed.

[0093] The present application realizes complex processing of audio spectrum through a series of multiplication and addition operations of linear operation operators. By setting the bit width of the multiplier 1 to 1 bit, the calculation complexity is not increased, and the multiplication operation is ensured not to become a performance bottleneck, thereby efficiently utilizing the multiplier resources and reducing the demand for additional comparators. The threshold-based gain adjustment realizes flexible gain control, which can automatically adjust the gain according to different parts of the spectrum, thereby optimizing the dynamic range of the audio signal or emphasizing specific frequency components. The base value addition operation can adjust the global or local offset of the spectrum, which helps to improve the sound quality of the audio signal. The spectrum gain calculation combines the effects of base value addition and gain adjustment, which is crucial for the enhancement, compression, equalization, etc. of the audio signal. The present application divides the processing flow into multiple columns (or stages), each column performs a specific task (such as comparison, base value addition, gain calculation, and spectrum gain calculation), making the entire system more modular and scalable. Since each column performs a relatively independent task, and the output can be used as the input of the subsequent column, this scheme can be easily integrated into a larger audio processing system or signal processing framework.

[0094] In an embodiment, step S323 includes, as shown in Figure 9

[0095] ​S3231, convert the Boolean value and the to-be-processed audio data into binary respectively to obtain first input data and second input data, and input the first input data and the second input data into a multiplier in the next tail column operation operator of the application gain topology circuit; the first input data is the converted Boolean value, and the second input data is the to-be-processed audio data with the bit width bit;

[0096] S3232, sequentially perform logical and operation on the first input data and the second input data by bit to form a first shape data block with the bit width row, and perform effective mapping on the first shape data block to form a second shape data block with the bit width row;

[0097] S3233, rearrange data in the second shape data block, and perform addition operation step by step to gradually reduce the number of rows of a calculation result, and repeat the rearranging and addition operation steps until the number of rows of the calculation result reaches a minimum value, to obtain a final calculation result output from high to low;

[0098] S3234, based on the gain difference, and according to the final calculation result, calculate the gain of the updated frequency point corresponding to the frequency spectrum array.

[0099] Specifically, each multiplier includes four modules of serial_and, array_mapping, serial_reorder and serial_adder, and the application takes a 4-bit*4-bit multiplier to illustrate the calculation process of the multiplier in the linear operation operator:

[0100] First, convert the returned value of the Boolean value and the to-be-processed audio data into binary respectively to obtain first input data DCBA and second input data dcba as inputs of the multiplier;

[0101] Next, the eight letters DCBA and dcba represent 0 or 1 in binary respectively, and the data logic calculation process is as shown in Figure 10 As shown in the figure, the multiplication of two 4-bit binary numbers DCBA and dcba can be simplified as the logical and of corresponding bits, and then after corresponding shift, addition is performed. In the application, the numbers 3, 2, 1 and 0 are used to replace D, C, B and A or d, c, b and a. When the bit width of the two multipliers is 4, the serial_and module is responsible for performing logical and operation on the two input multipliers sequentially, that is, the leftmost rectangle [3 0] in the first row represents mul2[3]&mul1[0], which corresponds to Da, and the rightmost rectangle [0 0] in the first row represents mul2[0]&mul1[0], which corresponds to

[0102] Figure 10 Da.​Figure 10 Aa, the long rectangle [3 3] in the leftmost column of the 4th row represents mul2[3]&mul1[3], corresponding to Figure 10 Dd, the long rectangle [0 3] in the rightmost column of the 4th row represents mul2[0]&mul1[3], corresponding to Figure 10 aD, and so on, finally forming a first shape data block with 4 rows, the structure of which is shown in Figure 11 , i.e. a parallelogram. Then, the Array_mapping module is responsible for effectively mapping the parallelogram data block in Figure 11 to form a second shape data block, the structure of which is shown in Figure 12 , i.e. a triangle.

[0103] In turn, the serial_reorder module rearranges the numbers of the triangle data block in Figure 12 , and the serial_adder module performs addition step by step, the rearrangement process is shown in Figure 13 , and the process is repeated until the number of rows of the calculation result reaches the minimum value 1, and the final result is obtained; wherein, HS_*_* represents the value of two numbers after 1 bit half-adder, FS_*_* represents the value of two numbers after 1 bit full-adder, and FC_*_* represents the carry of two numbers after 1 bit full-adder. Since the bit width described in this example is 4, the rearrangement will go through 4 times of stacking. The first time of stacking: after serial_reorder rearrangement, serial_adder calculates the result of {[1 3], [1 2]}+{[0 3]}: {FC_2_2, FS_2_2, HS_2_3}. The second time of stacking: after serial_reorder rearrangement, serial_adder calculates the result of {[2 3], [2 2], [2 1], [1 1]}+{FC_2_2, FS_2_2, HS_2_1, [0 2]}: {FC_1_1, FS_1_1, FS_1_2, FS_1_3, HS_1_4}. The third time of stacking: after serial_reorder rearrangement, serial_adder calculates the result of {[3 3], [3 2], [3 1],

[30] , [2 0], [1 0]}+{FC_1_1, FS_1_1, FS_1_2, FS_1_3, HS_1_4, [0 1]}: {FC_0_0, FS_0_0, FS_0_1, FS_0_2, FS_0_3, FS_0_4, HS_0_5}. The fourth time of stacking: after serial_reorder rearrangement, the result is directly output in the order from low to high according to the rearrangement.

[0104] Finally, based on the gain difference, the gain of the updated frequency point of the spectrum array can be calculated according to the final calculation result. It can be seen from the entire multiplication process that if the multiplication of two multipliers with a bit width of n is to be implemented, n^2 AND gates and n! 1-bit adders are needed. The bit width of the linear operation operator is dynamically configured in the present application, so that the audio processing process can dynamically call logical resources more reasonably and effectively according to different operation accuracy requirements, and other idle logical resources are used for wiring, so as to optimize the operation power consumption.

[0105] In addition, the present application improves the data processing rate and efficiency through efficient data processing-binary conversion and logical AND operation; the effective mapping of the shape data block can control the gain of the audio data more finely, and through the step-by-step addition operation and the reduction of the number of data block rows, efficient accumulation and aggregation of the audio data are realized, which not only reduces the calculation amount, but also makes the gain adjustment more smooth and continuous; based on the gain difference calculation, the accuracy of the gain adjustment is ensured, and the gain adjustment can be dynamically adjusted according to the characteristics of the audio signal; and each step in the scheme is independent, so that the entire processing flow is clearer and easier to manage, and it is also convenient to expand or modify when needed, thereby providing a powerful solution for the field of audio processing.

[0106] In addition, the expression of the negative frequency is spectrum_gain(p)=(spectrum[p]+base)*gain(p=256-k+1), and its calculation process is similar to that of the positive frequency, which will not be described in detail here; after combining the expression of the positive frequency and the expression of the negative frequency, the spectrum of the audio data to be processed can be obtained, and the calculated gain is applied to the spectrum.

[0107] S4, converting the spectrum after applying the gain to the time domain to obtain processed audio data; specifically, the spectrum after applying the gain can be converted to the time domain through inverse Fourier transform to obtain processed audio data.

[0108] This application proposes an audio data processing method based on the balance between power consumption and latency in audio data processing. The method acquires the audio data to be processed and its business requirements, and adaptively adjusts the operational parameters of a linear operator according to these requirements. The operational parameters include row size, column size, and bit width. The linear operator includes a multiplier and a logic unit. The method converts the audio data to be processed from the time domain to the frequency domain and calculates the gain of the converted audio data. It then calculates the spectrum of the converted audio data using the adjusted linear operator and applies the gain to the spectrum. Finally, it converts the spectrum with the applied gain back to the time domain to obtain the processed audio data. By dynamically configuring the operational parameters to change the allocation of computational resources, the method achieves the optimal match between power consumption and latency in audio signal processing at any time point.

[0109] It should be noted that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0110] In another embodiment, such as Figure 14 As shown, a second aspect of the present invention provides an audio data processing system, comprising:

[0111] The parameter adjustment module 10 is used to acquire the audio data to be processed and its business requirements, and to adaptively adjust the operation parameters of the linear operation operator according to the audio data to be processed and the business requirements; the operation parameters include row size, column size and bit width, and the linear operation operator includes a multiplier and a logic unit;

[0112] The gain calculation module 20 is used to convert the audio data to be processed from the time domain to the frequency domain and calculate the gain of the converted audio data to be processed.

[0113] The gain application module 30 is used to calculate the spectrum of the converted audio data to be processed through the adjusted linear operation operator, and apply the gain to the spectrum.

[0114] The spectrum conversion module 40 is used to convert the spectrum after gain application to the time domain to obtain processed audio data.

[0115] It should be noted that the above various modules of the audio data processing system can be realized by software, hardware and combinations thereof. The modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules. For the specific definition of the audio data processing system, refer to the definition of the audio data processing method, which has the same functions and effects, and will not be repeated here.

[0116] The third aspect of the application provides an electronic device, comprising:

[0117] a processor, a memory and a bus;

[0118] the bus is used to connect the processor and the memory;

[0119] the memory is used to store operation instructions;

[0120] the processor is used to execute the operations corresponding to the audio data processing method of the first aspect of the application by calling the operation instructions and executable instructions.

[0121] In an optional embodiment, an electronic device is provided, as shown in Figure 15 The electronic device 5000 shown in Figure 15 The electronic device 5000 shown in comprises a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, such as through a bus 5002. Optionally, the electronic device 5000 can also comprise a transceiver 5004. It should be noted that in actual application, the transceiver 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation on the embodiments of the application.

[0122] The processor 5001 can be a CPU, a general processor, a DSP, an ASIC, an FPGA or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 5001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0123] The bus 5002 can include a channel for transmitting information between the above components. The bus 5002 can be a PCI bus or an EISA bus, etc. The bus 5002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 15 only one thick line is used, but it does not mean that there is only one bus or one type of bus.

[0124] The memory 5003 can be a ROM, or other type of static storage device that can store static information and instructions; a RAM, or other type of dynamic storage device that can store information and instructions; an EEPROM, CD-ROM or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program codes in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0125] The memory 5003 is configured to store application program codes for implementing the solutions of the present application, and the processor 5001 is configured to control the execution. The processor 5001 is configured to execute the application program codes stored in the memory 5003 to implement the content shown in any of the foregoing method embodiments.

[0126] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like.

[0127] The fourth aspect of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the audio data processing method of the first aspect of the present application.

[0128] Another embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer can execute the corresponding content in the foregoing method embodiments.

[0129] In addition, an embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the steps of the foregoing method.

[0130] To sum up, the present application relates to the field of computer technology, and discloses an audio data processing method, system, device and medium, which comprises obtaining audio data to be processed and its service requirement, and adaptively adjusting operation parameters of a linear operation operator according to the audio data to be processed and the service requirement; the operation parameters comprise row scale, column scale and bit width, and the linear operation operator comprises a multiplier and a logic operator; the audio data to be processed is converted from time domain to frequency domain, and the gain of the converted audio data to be processed is calculated; the spectrum of the converted audio data to be processed is calculated through the linear operation operator after adjustment, and the gain is applied to the spectrum; the spectrum after the gain is applied is converted to time domain to obtain processed audio data; and the operation resource ratio is changed through dynamic configuration of operation parameters, so that the best match between audio signal processing and time section power consumption and delay is realized.

[0131] Each embodiment in the specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, since the system embodiment is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment. It should be noted that, each technical feature of the above-mentioned embodiments can be combined arbitrarily, in order to make the description simple, each technical feature of the above-mentioned embodiments is not described all possible combinations, however, as long as the combination of these technical features does not exist contradictory, it should be considered that it is within the scope of the present application.

[0132] The above-mentioned embodiments only express several preferred embodiments of the present application, the description is more specific and detailed, but it should not be understood as the limitation of the patent scope of the present application. It should be pointed out that, for ordinary skilled in the art, without departing from the technical principles of the present application, a number of improvements and replacements can be made, and these improvements and replacements should be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the protection scope of the claims.

Claims

1. An audio data processing method, characterized by, The method comprises the following steps: acquiring audio data to be processed and service requirements thereof, and adaptively adjusting operation parameters of a linear operation operator according to the audio data to be processed and the service requirements; the operation parameters comprise a row scale, a column scale and a bit width, and the linear operation operator comprises a multiplier and a logic operator; converting the audio data to be processed from a time domain to a frequency domain, and calculating a gain of the converted audio data to be processed; calculating a frequency spectrum of the converted audio data to be processed through the adjusted linear operation operator, and applying the gain to the frequency spectrum; converting the frequency spectrum to which the gain is applied to the time domain to obtain processed audio data; the step of adaptively adjusting the operation parameters of the linear operation operator according to the audio data to be processed and the service requirements thereof comprises the following steps: configuring a working voltage and a clock main frequency of the audio data to be processed according to the service requirements, and determining a maximum value of the column scale according to the clock main frequency; quantifying a delay of the audio data to be processed according to a callable resource scale; quantifying a power consumption of the audio data to be processed according to the working voltage and the clock main frequency, and adjusting the row scale and the column scale of the linear operation operator in a range of the column scale and the callable resource scale, so as to minimize a product of the row scale and the column scale, the power consumption and the delay; quantifying the power consumption of the audio data to be processed according to the working voltage and the clock main frequency; determining an operation precision of the audio data to be processed according to the service requirements, and adjusting a bit width of the linear operation operator in a range of the callable resource scale, so as to maximize the operation precision and minimize the power consumption.

2. The audio data processing method of claim 1, wherein, after the step of acquiring the audio data to be processed and the service requirements thereof, the method further comprises the following step: preprocessing the audio data to be processed, wherein the preprocessing comprises but is not limited to denoising and filtering.

3. The method of claim 1, wherein, the step of calculating the frequency spectrum of the converted audio data to be processed through the adjusted linear operation operator comprises the following steps: initializing a path address of the adjusted linear operation operator to obtain an application gain topology circuit; calculating a positive frequency expression and a negative frequency expression of the converted audio data to be processed through the application gain topology circuit respectively to obtain the frequency spectrum.

4. The audio data processing method of claim 3, wherein, the step of calculating the positive frequency expression of the converted audio data to be processed through the application gain topology circuit comprises the following steps: calculating, through a logic operator in a first column linear operation operator of the application gain topology circuit, a difference between a frequency spectrum array corresponding to the positive frequency and a preset linear threshold value to obtain a calculation result represented by a Boolean value; updating the frequency spectrum array by calling a base value element corresponding to the Boolean value from a base value array through a second column linear operation operator to a third from last column linear operation operator of the application gain topology circuit. The multiplier in the secondary tail column linear operation operator of the application gain topology circuit is used to calculate the gain of the corresponding frequency point of the updated spectrum array based on the gain difference and the Boolean value; the gain difference is the difference between the gain applied when the updated spectrum array exceeds the preset linear threshold and the gain applied when the updated spectrum array is lower than the preset linear threshold; The tail column linear operation operator of the application gain topology circuit is used to combine the updated spectrum array and the gain of the corresponding frequency point of the updated spectrum array to obtain the positive frequency expression.

5. The audio data processing method of claim 4, wherein, The multiplier in the secondary tail column linear operation operator of the application gain topology circuit is used to calculate the gain of the corresponding frequency point of the updated spectrum array based on the gain difference and the Boolean value, including: The Boolean value and the to-be-processed audio data are respectively converted into binary to obtain first input data and second input data, and the first input data and the second input data are input into the multiplier in the secondary tail column linear operation operator of the application gain topology circuit; the first input data is the converted Boolean value, and the second input data is the to-be-processed audio data with the bit width; The first input data and the second input data are sequentially subjected to logical AND operation to form a first shape data block with the bit width row, and the first shape data block is subjected to effective mapping to form a second shape data block with the bit width row; The data in the second shape data block is rearranged, and is sequentially subjected to addition operation to gradually reduce the number of rows of the calculation result, and the rearrangement and addition operation steps are repeated until the number of rows of the calculation result reaches a minimum value, so that a final calculation result output from high to low is obtained; The gain of the corresponding frequency point of the updated spectrum array is calculated based on the gain difference and according to the final calculation result.

6. An audio data processing system, characterized by Including: The parameter adjustment module is configured to obtain to-be-processed audio data and service requirements thereof, and to adaptively adjust operation parameters of a linear operation operator according to the to-be-processed audio data and the service requirements; the operation parameters include row scale, column scale and bit width, and the linear operation operator includes a multiplier and a logical operator; The gain calculation module is configured to convert the to-be-processed audio data from time domain to frequency domain, and to calculate the gain of the converted to-be-processed audio data; The gain application module is configured to calculate the spectrum of the converted to-be-processed audio data by using the adjusted linear operation operator, and to apply the gain to the spectrum; The spectrum conversion module is configured to convert the spectrum to which the gain is applied to time domain to obtain processed audio data; The adaptive adjustment of the operation parameters of the linear operation operator according to the to-be-processed audio data and the service requirements includes: The working voltage and clock main frequency of the to-be-processed audio data are configured according to the service requirements, and the maximum value of the column scale is determined according to the clock main frequency; The delay of the to-be-processed audio data is quantified according to the callable resource scale. quantify power consumption of the to-be-processed audio data according to the working voltage and clock frequency; quantify power consumption of the to-be-processed audio data according to the working voltage and clock frequency; determine operation precision of the to-be-processed audio data according to the service requirement, and adjust bit width of the linear operation operator within a range of the callable resource scale, so as to maximize the operation precision and minimize the power consumption.

7. An electronic device, comprising: The computer readable storage medium comprises a stored computer program, wherein a device in which the computer readable storage medium is located implements the audio data processing method as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein a device in which the computer readable storage medium is located implements the audio data processing method as claimed in any one of claims 1 to 5 when executing the computer program.

Images