Voiceprint processor and voiceprint verification execution method

Abstract

The present invention relates to a voiceprint processor and a voiceprint verification execution method. The voiceprint processor includes a configuration manager, a reconfigurable audio processing engine, and a main controller. The main controller is bidirectionally interconnected with the configuration manager via a configuration bus, the main controller is bidirectionally interconnected with the reconfigurable audio processing engine via an external interface, and the configuration manager and the reconfigurable audio processing engine are bidirectionally interconnected. The reconfigurable audio processing engine has vector dot product, tensor product, convolution, and fast Fourier transform functions. The voiceprint processor of the present invention has a small area and low power consumption, but is fully functional and can dynamically configure the layout and routing of different algorithms, thereby achieving high utilization of PE processing units.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a voiceprint processor and a voiceprint verification execution method. Background Art

[0002] Voiceprint recognition is a computer technology that uses audio data to identify a speaker. It primarily involves two tasks: speaker identification and speaker verification. The former verifies the speaker's identity based on the audio, while the latter confirms whether the speaker is a specific person.

[0003] Voiceprint recognition has a wide range of applications, including in numerous embedded systems, such as security, smart furniture, and autonomous driving. Currently, voiceprint recognition applications face two major challenges: the enormous computational workload and the real-time requirements of the application. Therefore, selecting the right hardware platform for embedded system design is a crucial issue.

[0004] To meet the performance requirements of applications, voiceprint recognition applications are currently mostly performed on the following embedded platforms:

[0005] 1. General Purpose Processors (GPP)

[0006] 2. Digital Signal Processors (DSP)

[0007] 3. Application Specific Integrated Circuits (ASIC)

[0008] 4. Field Programmable Gate Arrays (FPGA)

[0009] GPP and DSP offer great flexibility due to their programmability, but suffer from low computational efficiency. To accommodate a wide range of applications, GPP's architecture excels at processing diverse and large amounts of data, such as audio, images, and video. However, its lack of parallelism results in low computational efficiency.

[0010] Although DSP has a better computing speed than GPP, it still cannot meet the real-time requirements of voiceprint recognition. In addition, GPP and DSP have not completely solved the problems of energy consumption and area, which are very important indicators in the application of embedded systems.

[0011] ASICs and FPGAs both offer high computational efficiency in embedded voiceprint recognition applications, but they also have limitations. While ASICs offer advantages like high computing performance, strong electromagnetic impedance, low cost, and high integration, they are difficult to test once manufactured. Therefore, a detailed physical simulation process is required before ASIC production. Furthermore, the non-customizability of ASICs also hinders their application in the field of deep learning-based voiceprint recognition, where algorithms evolve rapidly.

[0012] FPGAs, with their fine-grained parallel computing capabilities, offer another high-performance solution for embedded voiceprint recognition. Compared to ASICs, they offer lower development costs. While FPGAs are an excellent choice for energy and area savings, the key focus for embedded voiceprint recognition lies in real-time performance.

[0013] In addition, although voiceprint recognition runs very efficiently on GPU, the high energy consumption of GPU makes it difficult to apply to embedded systems. Therefore, this patent does not discuss the performance comparison with GPU.

[0014] In summary, the current trend is that using reconfigurability to minimize production and development costs is an increasingly important requirement. CGRA (Coarse Grained Reconfigurable Architecture) is a coarse-grained reconfigurable architecture that balances performance and flexibility. A standard CGRA architecture consists of an array of processing units that execute static instructions and customizable interconnect converters. It features high data parallelism, reconfigurable processing units, low power consumption, and high memory bandwidth utilization, making it ideal as a processor for embedded systems in today's rapidly evolving algorithm landscape. Summary of the Invention

[0015] The technical problem to be solved by the present invention is to provide a voiceprint processor and a voiceprint verification execution method. The voiceprint processor has a small area and low power consumption, but is fully functional and can dynamically configure the layout and wiring of different algorithms, so the PE processing unit utilization rate is high.

[0016] The present invention solves the technical problem by adopting the following technical solution: providing a voiceprint processor, comprising a configuration manager, a reconfigurable audio processing engine, and a main controller, wherein the main controller is bidirectionally interconnected with the configuration manager via a configuration bus, the main controller is bidirectionally interconnected with the reconfigurable audio processing engine via an external interface, and the configuration manager and the reconfigurable audio processing engine are bidirectionally interconnected;

[0017] The reconfigurable audio processing engine has vector dot product, tensor product, convolution, and fast Fourier transform functions.

[0018] The reconfigurable audio processing engine includes a second storage unit, a PE array, a third storage unit, and a fourth storage unit connected in sequence, and the fourth storage unit is connected to the second storage unit.

[0019] The configuration manager includes a configuration decoder, a configuration flow controller and a first storage unit connected in sequence. The configuration decoder is used to decode the control instruction stream from the main controller, and the configuration flow controller is used to write the reconstruction method of the PE array into the first storage unit in a time sequence according to the decoded instruction stream.

[0020] The PE array includes m×n PE processing units arranged in an array, and two adjacent PE processing units are interconnected to form a grid structure; the PE processing units execute processing according to the tasks configured by the configuration manager.

[0021] The PE array connects two adjacent columns of PE processing units except the PE processing units parallel to the PE processing units in the array itself through a bridge unit to realize a fast Fourier transform function.

[0022] The first storage unit, the second storage unit, the third storage unit and the fourth storage unit are all static random access memories.

[0023] The technical solution adopted by the present invention to solve the technical problem is to provide a voiceprint verification execution method of the above-mentioned voiceprint processor, comprising the following steps:

[0024] (1) The configuration manager configures the PE processing units in the 1st column to the n-1th column in the PE array to perform multiplication and addition operations, and the PE processing units in the 1st column to the n-1th column only transmit data horizontally; configures the PE processing units in the nth column in the PE array to perform accumulation operations, and the PE processing units in the nth column only transmit data vertically; and uses the PE processing unit in the nth column and the mth row in the PE array as output;

[0025] (2) the configuration manager configures each PE processing unit in the PE array to perform a multiply-add function, and constructs two adjacent columns of PE processing units into a plurality of butterfly operation units; and uses the PE processing unit in the nth column of the PE array as an output;

[0026] (3) The configuration manager configures the PE processing units in the 1st to n-1th columns of the PE array for multiplication and addition operations, and the PE processing units in the 1st to n-1th columns only transmit data horizontally; configures the PE processing units in the nth column of the PE array for accumulation operations, and the PE processing units in the nth column only transmit data vertically; and uses the PE processing units in the nth column and mth row of the PE array as outputs; the configuration manager configures the PE processing units in the 1st column of the PE array for a comparison and larger value function, and configures the PE processing units in the 2nd to nth columns for data transmission functions, and the PE processing units in the 1st to nth columns only transmit data horizontally; and uses the PE processing units in the nth column of the PE array as outputs; the configuration manager configures each PE processing unit in the PE array for a data transmission function, and the PE processing units in the 1st to nth columns only transmit data horizontally; and uses the PE processing unit in the nth column of the PE array as outputs;

[0027] (4) The output result of step (3) is compared with the data to be verified in the database through Euclidean distance measurement to realize voiceprint verification.

[0028] Beneficial effects

[0029] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: the reconfigurable audio processing engine of the present invention is dynamically reconfigurable. The configuration manager is controlled by the main controller, and the configuration manager configures the layout and routing of the PE array of the reconfigurable audio processing engine, which is dynamically reconfigured during runtime to achieve efficient operation of the algorithm; the reconfigurable audio processing engine of the present invention is highly agile, and its PE array is very flexible in switching between different calculation methods. It can dynamically reconfigure the processing unit array at a speed of tens of nanoseconds to adapt to different calculations; the configuration manager of the present invention can configure different layouts and routings for the PE array, which can implement mathematical operations such as vector dot product, tensor product, convolution, and fast Fourier transform; the configuration flow controller in the configuration manager of the present invention can configure the layout and routing of the PE array in a chronological order, allowing the PE array to be dynamically configured into the layout and routing of different mathematical operations during operation, thereby achieving hardware implementation of different algorithms in the voiceprint recognition process; the voiceprint processor of the present invention has a small area and low power consumption, but is fully functional and can dynamically configure the layout and routing of different algorithms, thereby achieving high processing unit utilization, extremely low power consumption and a small area; the present invention has significant advantages over traditional ASIC, GPU, FPGA and DSP general-purpose processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the overall architecture of a voiceprint processor according to an embodiment of the present invention;

[0031] Figure 2This is the complete execution flow chart of the traditional voiceprint verification method;

[0032] Figure 3 This is a flow chart of the voiceprint verification method according to an embodiment of the present invention;

[0033] Figure 4 is a convolutional data flow graph of a reconfigurable audio processing engine according to an embodiment of the present invention;

[0034] Figure 5 2 is a schematic diagram of the configuration of parallel data streams in SRAM 2 during convolution operation according to an embodiment of the present invention;

[0035] Figure 6 is a data flow diagram of the reconfigurable audio processing engine implementing FFT according to an embodiment of the present invention;

[0036] Figure 7 is an FFT butterfly diagram according to an embodiment of the present invention;

[0037] Figure 8 is a maximum pooling data flow graph of a reconfigurable audio processing engine according to an embodiment of the present invention;

[0038] Figure 9 4 is a data flow graph of a nonlinear activation operation of a reconfigurable audio processing engine according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0040] The embodiment of the present invention relates to a voiceprint processor, see Figure 1 , including a main controller, a configuration manager, a reconfigurable audio processing engine, four static random access memories (SRAMs), an external interface, and a configuration bus. This embodiment obtains input data through the upper system bus and the external interface.

[0041] The configuration manager is responsible for laying out and routing the PE processing units of the reconfigurable audio processing engine, enabling it to be dynamically reconfigured while the algorithm is running.

[0042] The main controller is a general-purpose processor (including but not limited to ARM7, RISC-V core), which is responsible for receiving input data from the external interface and controlling the configuration manager.

[0043] The configuration decoder in the configuration manager is responsible for decoding the control instruction stream from the main controller.

[0044] The configuration flow controller in the configuration manager is responsible for writing the reconstruction mode of the PE array into SRAM 1 in a timely manner during operation.

[0045] The reconfigurable audio processing engine consists of a PE array and three SRAMs (SRAM2, SRAM3, and SRAM4). The PE array is m rows by n columns, where m and n are integers. The PE processing units in the PE array are interconnected in four directions: up, down, left, and right. The PE array reads data from SRAM2 and writes output data to SRAM3. SRAM4 is responsible for storing intermediate variables used in calculations.

[0046] The reconfigurable audio processing engine can realize functions such as vector dot multiplication, tensor product, convolution, and fast Fourier transform. The PE array connects two adjacent columns of PE processing units except the PE processing units parallel to itself through the bridge unit.

[0047] The main controller and the configuration manager are connected via a configuration bus.

[0048] The embodiment of the present invention relates to a method for executing voiceprint verification. Figure 2 , a generalized voiceprint recognition process includes the following Figure 2 For the ten steps shown, this embodiment can configure ten sets of data flow diagrams, which can be dynamically reconstructed during the voiceprint recognition process to achieve hardware implementation of all algorithms in these ten steps.

[0049] For the sake of convenience, the following briefly describes the characteristics of this embodiment, such as dynamic reconstruction, parallel processing, and high utilization of computing units. Figure 3 The voiceprint verification method shown in the figure is described, which mainly includes audio pre-enhancement, fast Fourier transform, feature extraction, and similarity calculation steps, which are described in detail below.

[0050] The first step is to pre-enhance the audio segment of 10-25ms in length. This is mainly done using FIR filters, and the main operation is one-dimensional convolution. Accelerated convolution calculation is the main feature of the PE array in the reconfigurable audio processing engine. Both multi-channel two-dimensional convolution of neural networks and one-dimensional convolution of audio can be achieved through Figure 4 Data flow diagram implementation.

[0051] Specifically, for an m×n reconfigurable audio processing engine PE array, when calculating convolution, except for the last column (i.e., the nth column), the functions of all PE processing units from the 1st column to the n-1th column are configured by the configuration manager as multiplication and addition, and the PE processing unit function of the last column is configured as accumulation; during layout and routing, all PE processing units except the last column can only transmit data horizontally, and the PE processing unit in the last column can only transmit data vertically. At the same time, the PE processing unit at the bottom of the last column transmits the output result to SRAM 3, see details. Figure 4 and Figure 5 , Figure 5 The parallel data stream in SRAM 2 moves one grid to the right in each clock cycle (to allow the PE processing unit in this row to read data), and SRAM 4 is used to store intermediate variables.

[0052] When configuring data, it is processed by the configuration manager and Figure 4 The function of each column of PE processing units is configured as a convolution kernel, that is, if the convolution kernel size is 3×3, then the 9 numbers in the convolution kernel are sequentially configured to the 9 PE processing units in the same column, and at the same time Figure 5 As shown, the convolved data sequence is properly configured in SRAM 2 so that the parallel data stream moves one grid to the right in each clock cycle (that is, the PE array of the reconstructed audio processing engine can read data once). Figure 5 As shown, not all PE processing units have to read data in every clock cycle.

[0053] from Figure 4 As can be seen, for an m×n reconfigurable audio processing engine, if m is 9, then n 3×3 convolution operations can be performed simultaneously, because each column of PE processing units represents the computational function of a convolution kernel. In the subsequent feature extraction process using Resnet50, some convolutional layers often perform 64, 256, 512, or 1024 convolution operations on the input tensor. In these cases, the parallel computing of this embodiment can greatly improve computational efficiency.

[0054] The above explains how the reconfigurable audio processing engine accelerates convolution operations. Accelerating convolution operations also speeds up the first step in voiceprint recognition: audio pre-enhancement.

[0055] In the second step, after obtaining the pre-emphasized audio signal, the next step is to perform a Fast Fourier Transform (FFT) on it. This embodiment can also accelerate the FFT operation, and its layout and wiring are as follows: Figure 6 As shown, Figure 6 The size of the PE array is 8×4 (m=8, n=4).

[0056] For a signal with eight components, the FFT hardware acceleration algorithm is configured as follows: Figure 6 As shown, the function of each PE processing unit is configured as multiplication and addition. The configuration manager will configure the data flow graph of the reconfigurable audio processing engine into Figure 6 In order to briefly explain the calculation process, this embodiment selects a smaller butterfly operation unit, see Figure 7 The butterfly unit takes A and B as input, and the data is multiplied by a weight when passing through each line, so its output is: C = 0.5 × (A + B × Wn) and D = 0.5 × (AB × Wn).

[0057] After performing an FFT on an audio signal, a spectrogram is generated. The horizontal axis of a spectrogram is time, and the vertical axis is frequency. The color of each pixel (the intensity value on the RGB channels) represents the energy intensity. A spectrogram comprehensively reflects the sound information, so obtaining a spectrogram means that voiceprint recognition can be performed using image processing methods.

[0058] In the third step, this implementation inputs the spectrogram into Resnet50 and uses a convolutional neural network to extract the audio feature information. The Resnet50 operation includes three convolutional layers (1×1, 3×3, 7×7), a maximum pooling layer, and a nonlinear activation layer. The layout of the convolutional layer has been Figure 4 Given, Figure 8 and Figure 9 The data flow diagrams of the maximum pooling operation and the nonlinear activation operation are given. The above data flow diagrams are all configured in sequence by the configuration manager.

[0059] The maximum pooling operation only requires that the PE processing unit function of the first column of this implementation be configured as "compare and take the larger value". At the same time, the PE processing unit of the first column continuously receives 4 inputs, and all subsequent PE processing units are configured as "data transmission" function (ie, "pass" function).

[0060] For nonlinear activation operations, Resnet50 uses the linear rectifier unit (ReLU) as the activation function, and its operation formula is as follows:

[0061] f(x)=max(0,x)

[0062] Where x is the input of the function, which in this embodiment represents the operand of the PE processing unit; max means taking the larger of the two numbers as the output. Since all data are normalized to the range of 0-1 during neural network operation, there is no case where the operand is less than 0. Therefore, if Figure 9As shown in the figure, for the data flow diagram of the nonlinear activation of Resnet50, the wiring only needs to be changed to horizontal, and all PE processing units are configured as the "data transmission" function (i.e., the "pass" function).

[0063] The fourth step is similarity calculation. After extracting a feature vector using Resnet50, you can use this vector to complete the downstream task. For the sake of simplicity, this implementation example selects the downstream task as speaker verification, assuming that the feature vector extracted in the previous step has a dimension of 256.

[0064] Given a 256-dimensional feature vector for voiceprint verification, we simply compare it to the feature vector of the speaker to be verified in the database and calculate the Euclidean distance between the two vectors. If the Euclidean distance is less than a threshold, the judgment is "yes", otherwise it is "no". Depending on the array size of the reconfigurable audio processing engine (the PE array size in this embodiment is m×n), operations on n vector components can be performed in parallel.

[0065] After calculating the similarity result, you only need to compare it with a given threshold (scalar ratio size) and then output the result "yes" or "no".

[0066] In summary, using this invention to complete voiceprint verification tasks requires dynamically switching data flow graphs during algorithm execution, fully demonstrating the dynamic reconfigurability of the voiceprint verification execution method of the present invention. Furthermore, each data flow graph also fully demonstrates the data parallel processing capability and high processing unit utilization (low power consumption) of the present invention.

[0067] The above is the whole process of completing the voiceprint verification task using the present invention.

[0068] The foregoing descriptions of specific exemplary embodiments of the present invention are for purposes of illustration and description. These descriptions are not intended to limit the invention to the precise forms disclosed, and it is apparent that many variations and modifications are possible in light of the foregoing teachings. The exemplary embodiments have been selected and described for the purpose of explaining the specific principles of the invention and their practical application, thereby enabling those skilled in the art to realize and utilize a variety of exemplary embodiments of the invention and various options and modifications. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for executing voiceprint verification, characterized in that: A voiceprint processor is used, which includes a configuration manager, a reconfigurable audio processing engine and a main controller. The main controller is bidirectionally interconnected with the configuration manager via a configuration bus, the main controller is bidirectionally interconnected with the reconfigurable audio processing engine via an external interface, and the configuration manager and the reconfigurable audio processing engine are bidirectionally interconnected. The reconfigurable audio processing engine has vector dot multiplication, tensor product, convolution, and fast Fourier transform functions. The reconfigurable audio processing engine includes a second storage unit, a PE array, a third storage unit, and a fourth storage unit connected in sequence, and the fourth storage unit is connected to the second storage unit. The configuration manager includes a configuration decoder, a configuration flow controller, and a first storage unit connected in sequence. The configuration decoder is used to decode the control instruction stream from the main controller, and the configuration flow controller is used to write the reconstruction mode of the PE array to the first storage unit in a time sequence according to the decoded instruction stream. Specifically, the following steps are included: (1) The configuration manager configures the PE processing units in the 1st column to the n-1th column in the PE array to perform multiplication and addition operations, and the PE processing units in the 1st column to the n-1th column only transmit data horizontally; configures the PE processing units in the nth column in the PE array to perform accumulation operations, and the PE processing units in the nth column only transmit data vertically; and uses the PE processing unit in the nth column and the mth row in the PE array as output; (2) the configuration manager configures each PE processing unit in the PE array to perform a multiplication-addition function, and constructs two adjacent columns of PE processing units into a plurality of butterfly operation units; and uses the PE processing unit in the nth column of the PE array as an output; (3) The configuration manager configures the PE processing units in the 1st to n-1th columns of the PE array for multiplication and addition operations, and the PE processing units in the 1st to n-1th columns only transmit data horizontally; configures the PE processing units in the nth column of the PE array for accumulation operations, and the PE processing units in the nth column only transmit data vertically; and uses the PE processing units in the nth column and mth row of the PE array as outputs; the configuration manager configures the PE processing units in the 1st column of the PE array for a comparison and larger value function, and configures the PE processing units in the 2nd to nth columns for data transmission functions, and the PE processing units in the 1st to nth columns only transmit data horizontally; and uses the PE processing units in the nth column of the PE array as outputs; the configuration manager configures each PE processing unit in the PE array for a data transmission function, and the PE processing units in the 1st to nth columns only transmit data horizontally; and uses the PE processing unit in the nth column of the PE array as outputs; (4) The output result of step (3) is compared with the data to be verified in the database through Euclidean distance measurement to realize voiceprint verification.

2. The voiceprint verification execution method according to claim 1, characterized in that: The PE array includes m×n PE processing units arranged in an array, and two adjacent PE processing units are interconnected to form a grid structure; the PE processing units execute processing according to the tasks configured by the configuration manager.

3. The voiceprint verification execution method according to claim 2, characterized in that: The PE array connects two adjacent columns of PE processing units except the PE processing units parallel to the PE processing units in the array itself through a bridge unit to realize a fast Fourier transform function.

4. The voiceprint verification execution method according to claim 1, characterized in that: The first storage unit, the second storage unit, the third storage unit and the fourth storage unit are all static random access memories.

Images