Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech recognition method and device, electronic equipment and storage medium, the method comprises the following steps: inputting a speech to be recognized into a speech recognition model to obtain a speech recognition result output by the speech recognition model; wherein, an attention layer of the speech recognition model is used to perform matrix multiplication on each sub-block attention matrix and a value matrix; each sub-block attention matrix is obtained by sequentially performing longitudinal division and horizontal truncation on a lower triangular attention matrix of the attention layer, or sequentially performing horizontal division and longitudinal truncation on the lower triangular attention matrix, and then removing zero matrices. Since there is no zero matrix in the sub-block attention matrix, the zero matrix will not participate in the matrix multiplication operation, thereby reducing unnecessary operation cost. Compared with the complete matrix multiplication operation of the entire lower triangular attention matrix and the value matrix in the traditional method, the multiplication operation of the zero element part is reduced, the overall operation speed is accelerated, and the training efficiency of the model is improved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and more particularly to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Automatic speech recognition (ASR) is the process of converting human speech into computer-readable input, typically intelligible text content, also known as speech-to-text.

[0003] Currently, speech recognition is mostly performed using end-to-end speech recognition models. These models typically include a front-end feature extractor, a CTC branch, and an Attention branch. During model training, to mask the association of each character with all future characters, a lower triangular matrix is ​​added to the attention matrix in the Attention branch to obtain a lower triangular attention matrix, which is then multiplied by the value matrix. Since the upper right diagonal portion of the lower triangular attention matrix involved in the calculation has zero values, this part of the calculation contributes nothing to the result, thus increasing computational cost and reducing the training efficiency of the speech recognition model. Summary of the Invention

[0004] This invention provides a speech recognition method, device, electronic device, and storage medium to address the shortcomings of existing speech recognition models, such as high computational load and low training efficiency.

[0005] This invention provides a speech recognition method, comprising:

[0006] Identify the speech to be recognized;

[0007] The speech to be recognized is input into the speech recognition model to obtain the speech recognition result output by the speech recognition model;

[0008] The speech recognition model is trained based on sample speech and its corresponding sample speech recognition results. The attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix. Each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix of the attention layer vertically and truncating it horizontally, or by sequentially dividing the lower triangular attention matrix horizontally and truncating it vertically and then removing the zero matrix.

[0009] According to a speech recognition method provided by the present invention, the attention matrix of each block is determined based on the following steps:

[0010] After vertically dividing the lower triangular attention matrix, multiple sectional matrices are obtained;

[0011] Starting from the first non-zero element of the first column of each sectional matrix, shift upwards by the first element interval, and then truncate each sectional matrix horizontally, removing the zero matrix to obtain the corresponding block attention matrix.

[0012] or,

[0013] After horizontally dividing the lower triangular attention matrix, multiple horizontally slicing matrices are obtained;

[0014] Starting from the last non-zero element of the bottom row of each cross-cutting matrix, shift to the right by the second element interval, truncate each cross-cutting matrix vertically, remove the zero matrix, and obtain the corresponding block attention matrix.

[0015] According to a speech recognition method provided by the present invention, the width of each vertical slicing matrix is ​​the same, and the height of each horizontal slicing matrix is ​​the same.

[0016] According to a speech recognition method provided by the present invention, the width of each vertical slicing matrix is ​​greater than or equal to 64, and the height of each horizontal slicing matrix is ​​greater than or equal to 64.

[0017] According to a speech recognition method provided by the present invention, the first element interval is zero, and the second element interval is zero.

[0018] The speech recognition method provided by the present invention further includes:

[0019] The attention matrix is ​​formed by sequentially dividing the lower triangular attention matrix of the attention layer vertically and truncating it horizontally. The attention layer is used to superimpose the matrix multiplication results of each block attention matrix and the value matrix.

[0020] The speech recognition method provided by the present invention further includes:

[0021] When each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix horizontally and vertically, the attention layer is used to stack the matrix multiplication results of each block attention matrix and the value matrix.

[0022] The present invention also provides a voice recognition device, comprising:

[0023] The determining unit is used to determine the speech to be recognized;

[0024] The recognition unit is used to input the speech to be recognized into the speech recognition model and obtain the speech recognition result output by the speech recognition model.

[0025] The speech recognition model is trained based on sample speech and its corresponding sample speech recognition results. The attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix. Each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix of the attention layer vertically and truncating it horizontally, or by sequentially dividing the lower triangular attention matrix horizontally and truncating it vertically and then removing the zero matrix.

[0026] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech recognition methods.

[0027] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0028] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0029] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention sequentially perform vertical and horizontal truncation on the lower triangular attention matrix of the attention layer, or sequentially perform horizontal and vertical truncation on the lower triangular attention matrix and remove zero matrices to obtain multiple block attention matrices. Since there are no zero matrices in the block attention matrices, these zero matrices do not participate in matrix multiplication operations, reducing unnecessary computational costs. Compared to traditional methods that perform a complete matrix multiplication operation between the entire lower triangular attention matrix and the value matrix, this reduces multiplication operations on zero-element parts, speeds up the overall computation, and improves the training efficiency of the model. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention;

[0032] Figure 2 This is one of the schematic diagrams of the lower triangular attention matrix segmentation method provided by the present invention;

[0033] Figure 3This is the second schematic diagram of the lower triangular attention matrix segmentation method provided by the present invention;

[0034] Figure 4 This is a schematic diagram of the structure of the speech recognition device provided by the present invention;

[0035] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] Existing end-to-end speech recognition models typically include a front-end feature extractor, a CTC branch, and an Attention branch. During model training, to mask the association of each character with all future characters, a lower triangular matrix is ​​added to the attention matrix in the Attention branch, resulting in a lower triangular attention matrix, which is then multiplied by the value matrix. Since the upper right diagonal portion of the lower triangular attention matrix involved in the calculation is zero, this part of the calculation contributes nothing to the result, thus increasing computational cost and reducing the training efficiency of the speech recognition model.

[0038] In response, this invention provides a speech recognition method. Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0039] Step 110: Determine the speech to be recognized.

[0040] Here, the speech to be recognized refers to the speech data that needs to be recognized. The speech to be recognized can be speech data recorded in real time by the user through an electronic device, such as a smartphone, tablet, or smart appliance like a speaker, television, or air conditioner. After receiving the speech, the electronic device can amplify and reduce noise. Alternatively, the speech to be recognized can also be stored or received speech data; this embodiment of the invention does not specifically limit this.

[0041] Step 120: Input the speech to be recognized into the speech recognition model and obtain the speech recognition result output by the speech recognition model;

[0042] The speech recognition model is trained based on sample speech and its corresponding sample speech recognition results. The attention layer of the speech recognition model is used to perform matrix multiplication between the attention matrix of each block and the value matrix. Each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix of the attention layer vertically and truncating it horizontally, or by sequentially dividing the lower triangular attention matrix horizontally and truncating it vertically and then removing the zero matrix.

[0043] Specifically, the attention layer of a speech recognition model can be understood as the attention branch of a traditional speech recognition model. During training, to ensure that each input character can only be associated with previously generated characters and not with future characters, thus maintaining consistency with the reasoning logic, the attention matrix of the attention layer needs to be masked. This is achieved by adding a lower triangular mask matrix after the attention matrix, resulting in a lower triangular attention matrix. The upper right diagonal portion of the lower triangular attention matrix contains zero values.

[0044] After obtaining the lower triangular attention matrix, it needs to be multiplied by the value matrix. During matrix multiplication, the upper right diagonal part of the lower triangular attention matrix is ​​also involved in the calculation, but this part contributes nothing to the result, thus wasting nearly half of the computing power in this step.

[0045] To address this, in this embodiment of the invention, the attention layer is obtained by first performing matrix multiplication, then sequentially dividing the lower triangular attention matrix vertically and truncating it horizontally, and finally removing the zero matrix; or by sequentially dividing the lower triangular attention matrix horizontally and truncating it vertically, and then removing the zero matrix. Specifically:

[0046] (a) After vertically segmenting and horizontally truncating the lower triangular attention matrix of the attention layer, and removing the zero matrix, the block attention matrix is ​​obtained:

[0047] After vertically dividing the lower triangular attention matrix of the attention layer, multiple sectional matrices can be obtained. The height of each sectional matrix is ​​the same as the height of the lower triangular attention matrix, while the width can be the same or different.

[0048] After obtaining multiple sectional matrices, each sectional matrix is ​​truncated laterally to obtain multiple cross-sectional matrices. Among these cross-sectional matrices, at least one is a zero matrix.

[0049] After obtaining multiple cross-sectional matrices, remove the zero matrices and use the remaining non-zero matrices as the block attention matrices.

[0050] (b) After sequentially dividing the lower triangular attention matrix horizontally and vertically, and removing the zero matrix, the block attention matrix is ​​obtained:

[0051] After horizontally dividing the lower triangular attention matrix of the attention layer, multiple cross-cut matrices can be obtained. The width of each cross-cut matrix is ​​the same as the width of the lower triangular attention matrix, while the height can be the same or different.

[0052] After obtaining multiple transverse matrices, each transverse matrix is ​​truncated vertically to obtain multiple longitudinal matrices. Among these longitudinal matrices, at least one is a zero matrix.

[0053] After obtaining multiple vertical matrices, remove the zero matrices and use the remaining non-zero matrices as the block attention matrices.

[0054] It should be noted that, compared with the horizontal segmentation method mentioned in (b), the vertical segmentation mentioned in (a) does not require any repeated loading of data in the value matrix, and has higher computational efficiency, while (b) requires repeated loading of data in the value matrix when performing matrix multiplication between each block attention matrix and the value matrix.

[0055] Therefore, the speech recognition method provided in this embodiment of the invention sequentially performs vertical and horizontal segmentation on the lower triangular attention matrix of the attention layer, or sequentially performs horizontal and vertical segmentation on the lower triangular attention matrix and removes the zero matrix to obtain multiple block attention matrices. Since there are no zero matrices in the block attention matrices, the zero matrices will not participate in matrix multiplication operations, reducing unnecessary computational costs. Compared with the traditional method of performing a complete matrix multiplication operation between the entire lower triangular attention matrix and the value matrix, this method reduces the multiplication operations of the zero element part, speeds up the overall computation speed, and improves the training efficiency of the model.

[0056] Based on the above embodiments, the attention matrix for each block is determined based on the following steps:

[0057] After vertically dividing the lower triangular attention matrix, multiple sectional matrices are obtained;

[0058] Starting from the first non-zero element of the first column of each sectional matrix, shift upwards by the first element interval, and then truncate each sectional matrix horizontally, removing the zero matrix to obtain the corresponding block attention matrix.

[0059] or,

[0060] After horizontally dividing the lower triangular attention matrix, multiple cross-cut matrices are obtained;

[0061] Starting from the last non-zero element of the bottom row of each cross-cutting matrix, shift to the right by the second element interval, truncate each cross-cutting matrix vertically, remove the zero matrix, and obtain the corresponding block attention matrix.

[0062] Specifically, as an optional embodiment, after vertically dividing the lower triangular attention matrix, multiple sectional matrices can be obtained. The height of each sectional matrix is ​​the same as the height of the lower triangular attention matrix, while the widths can be the same or different.

[0063] After obtaining multiple sectional matrices, the first non-zero element of the first column of each sectional matrix is ​​used as the starting point, and the matrix is ​​horizontally truncated by shifting upwards by the first element interval to obtain a cross-sectional matrix. Among the multiple cross-sectional matrices, at least one is a zero matrix.

[0064] After obtaining multiple cross-sectional matrices, remove the zero matrices and use the remaining non-zero matrices as the block attention matrices.

[0065] It should be noted that after truncating each ordinate matrix horizontally, two submatrices can be obtained. Since the upper right part of the diagonal of the lower triangular attention matrix is ​​zero, the first column of each ordinate matrix has the most non-zero elements. Therefore, by taking the first non-zero element of the first column of each ordinate matrix as the starting point and shifting upward by the first element interval, after truncating each ordinate matrix horizontally, it can be guaranteed that one of the two submatrices is a zero matrix and the other is a non-zero matrix.

[0066] Furthermore, since one of the two sub-matrices is a zero matrix and the other is a non-zero matrix, and in this embodiment of the invention, the zero matrix is ​​removed so that the zero matrix does not participate in the operation, that is, one of the two sub-matrices participates in the operation and the other matrix does not participate in the operation, thereby saving about half of the computing power and greatly improving the computing efficiency.

[0067] As another optional embodiment, after horizontally dividing the lower triangular attention matrix, multiple cross-cut matrices can be obtained. The height of each cross-cut matrix is ​​the same as the width of the lower triangular attention matrix; the heights can be the same or different.

[0068] After obtaining multiple transverse matrices, the last non-zero element of the bottom row of each transverse matrix is ​​used as the starting point, and the matrix is ​​shifted to the right by a second element interval to vertically truncate each transverse matrix, resulting in a longitudinal matrix. Among the multiple longitudinal matrices, at least one is a zero matrix.

[0069] After obtaining multiple longitudinal matrices, remove the zero matrices and use the remaining non-zero matrices as the block attention matrices.

[0070] It should be noted that after vertically truncating each transverse matrix, two submatrices can be obtained. Since the upper right part of the diagonal of the lower triangular attention matrix is ​​zero, the lowermost row of each transverse matrix has the most non-zero elements. Therefore, by taking the last non-zero element of the lowermost row of each transverse matrix as the starting point and shifting to the right by a second element interval, after vertically truncating each transverse matrix, it can be guaranteed that one of the two submatrices is a zero matrix and the other is a non-zero matrix.

[0071] Furthermore, since one of the two sub-matrices is a zero matrix and the other is a non-zero matrix, and in this embodiment of the invention, the zero matrix is ​​removed so that the zero matrix does not participate in the operation, that is, one of the two sub-matrices participates in the operation and the other matrix does not participate in the operation, thereby saving about half of the computing power and greatly improving the computing efficiency.

[0072] Based on any of the above embodiments, the width of each longitudinal sectional matrix is ​​the same, and the height of each transverse sectional matrix is ​​the same.

[0073] Specifically, all the sectional matrices have the same width, meaning that in this embodiment of the invention, the lower triangular attention matrix is ​​uniformly divided vertically, resulting in multiple sectional matrices with the same width. For example, if the size of the lower triangular attention matrix is ​​32×32, it can be divided into four sectional matrices, each with a width of 8, resulting in a sectional matrix size of 32×8.

[0074] Since all the cross-cutting matrices have the same height, this embodiment of the invention involves uniformly dividing the lower triangular attention matrix horizontally, resulting in multiple cross-cutting matrices with the same height. For example, if the size of the lower triangular attention matrix is ​​32×32, it can be divided into four cross-cutting matrices, each with a height of 8, resulting in cross-cutting matrices of size 8×32.

[0075] Based on any of the above embodiments, the width of each longitudinal slicing matrix is ​​greater than or equal to 64, and the height of each transverse slicing matrix is ​​greater than or equal to 64.

[0076] Specifically, in order to ensure the efficiency of matrix multiplication in the tensor computation kernel for each block attention matrix, the width of each vertical matrix and the height of each horizontal matrix should not be too small. Through experimental analysis, it is found that in the embodiment of the present invention, the width of each vertical matrix is ​​greater than or equal to 64 and the height of each horizontal matrix is ​​greater than or equal to 64.

[0077] Based on any of the above embodiments, the first element interval is zero, and the second element interval is zero.

[0078] Specifically, the zero-interval for the first element indicates that after vertically dividing the lower triangular attention matrix to obtain multiple sectional matrices, the first non-zero element of the first column of each sectional matrix is ​​used as the starting cutoff point to horizontally truncate each sectional matrix, resulting in two cross-sectional matrices corresponding to each sectional matrix. One cross-sectional matrix is ​​a zero matrix, and the other is a non-zero matrix, with all non-zero elements in the first column of the non-zero matrix.

[0079] The second element's interval is zero, indicating that after horizontally dividing the lower triangular attention matrix to obtain multiple cross-section matrices, the last non-zero element of the bottom row of each cross-section matrix is ​​used as the starting cutoff point to vertically truncate each cross-section matrix, resulting in two corresponding vertical cut matrices. One vertical cut matrix is ​​a zero matrix, and the other is a non-zero matrix, with all bottom rows of the non-zero matrix containing non-zero elements.

[0080] Based on any of the above embodiments, it further includes:

[0081] When each block attention matrix is ​​a vertically segmented and horizontally truncated lower triangular attention matrix of the attention layer, the attention layer is used to superimpose the matrix multiplication results of each block attention matrix and the value matrix.

[0082] Specifically, when each block attention matrix is ​​a vertically segmented and horizontally truncated lower triangular attention matrix of the attention layer, multiple matrix multiplication results are obtained when each block attention matrix is ​​multiplied by the value matrix. The current matrix multiplication result is then superimposed with the previous superposition result to obtain the current superposition result.

[0083] For example, according to the method of the above embodiment, the lower triangular attention matrix is ​​divided into four block attention matrices, namely block matrix 1, block matrix 2, block matrix 3 and block matrix 4. The results of matrix multiplication of these block matrices with the value matrix are matrix multiplication result 1, matrix multiplication result 2, matrix multiplication result 3 and matrix multiplication result 4, respectively. The superposition method is as follows: matrix multiplication result 1 and matrix multiplication result 2 are superimposed (i.e., matrix addition) to obtain superposition result 1; superposition result 1 and matrix multiplication result 3 are superimposed to obtain superposition result 2; superposition result 2 and matrix multiplication result 4 are superimposed to obtain the final superposition result.

[0084] Based on any of the above embodiments, it further includes:

[0085] When each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix horizontally and vertically, the attention layer is used to stack the matrix multiplication results of each block attention matrix and the value matrix.

[0086] Specifically, when each block attention matrix is ​​a lower triangular attention matrix of the attention layer that is horizontally divided and vertically truncated in sequence, the matrix multiplication of each block attention matrix with the value matrix will result in multiple matrix multiplication results. The matrix multiplication results are stacked in order (without matrix superposition operation) to obtain the stacked result.

[0087] For example, according to the method of the above embodiment, the lower triangular attention matrix is ​​divided into four block attention matrices, namely block matrix 1, block matrix 2, block matrix 3 and block matrix 4. The matrix multiplication results of these block matrices with the value matrix are matrix multiplication result 1, matrix multiplication result 2, matrix multiplication result 3 and matrix multiplication result 4 respectively. The stacking method is to stack matrix multiplication result 1, matrix multiplication result 2, matrix multiplication result 3 and matrix multiplication result 4 in a top-to-bottom order to obtain the stacking result.

[0088] Based on any of the above embodiments, the present invention also provides a method for segmenting the lower triangular attention matrix, such as... Figure 2 As shown, the following example illustrates the concept of a triangular attention matrix of size 16×16 and a value matrix of size 16×20:

[0089] First, the lower triangular attention matrix is ​​vertically and evenly divided into four parts, resulting in four 16×4 sectional matrices. Then, each sectional matrix is ​​horizontally divided from above the first non-zero element in the first column, and the lower left matrix after the division is retained as the block attention matrix, which are the four block attention matrices numbered 1, 2, 3, and 4 in the lower triangular attention matrix in the figure. The size of block attention matrix 1 is 16×4, the size of block attention matrix 2 is 12×4, the size of block attention matrix 3 is 8×4, and the size of block attention matrix 4 is 4×4.

[0090] Next, perform matrix multiplication between each block attention matrix and the value matrix. It should be noted that since the heights of the block attention matrices are inconsistent but their widths are consistent, it is only necessary to perform matrix multiplication between the portions of each block attention matrix and the value matrix whose heights are the same from top to bottom as the widths of the block attention matrices in the lower triangular attention matrix.

[0091] After performing matrix multiplication on each block attention matrix with a portion of the value matrix, resulting in separate result matrices, the height of these result matrices is the same as the height of the block attention matrix, while the width is the same as the width of the value matrix. Figure 2 There are 4 block attention matrices, so there are 4 corresponding result matrices (i.e., result matrix 1, result matrix 2, result matrix 3 and result matrix 4). Then, these result matrices are superimposed in a bottom-aligned manner to obtain the complete result matrix.

[0092] Assume the lower triangular attention matrix is ​​N×N and the value matrix is ​​N×D. Dividing the lower triangular attention matrix vertically into M parts as described above, the calculation process can be transformed into:

[0093]

[0094] Based on any of the above embodiments, the present invention also provides a method for segmenting the lower triangular attention matrix, such as... Figure 3 As shown, the following example illustrates the concept of a triangular attention matrix of size 16×16 and a value matrix of size 16×20:

[0095] First, the lower triangular attention matrix is ​​horizontally divided into four equal parts, resulting in four 4×16 horizontal matrices. Then, each horizontal matrix is ​​vertically divided from the right of the last non-zero element at the bottom, and the block attention matrices on the left side after the division are retained. These are the four block attention matrices numbered I, II, III, and IV in the lower triangular attention matrix shown in the figure. Block attention matrix I has a size of 4×4, block attention matrix II has a size of 4×8, block attention matrix III has a size of 4×12, and block attention matrix IV has a size of 4×16.

[0096] Next, perform matrix multiplication between each block attention matrix and the value matrix. It should be noted that since the widths of the block attention matrices are inconsistent, and except for the bottom block attention matrix IV, the widths of the other block attention matrices are all smaller than the height of the value matrix, for the remaining block attention matrices (except for the bottom block attention matrix IV), simply perform matrix multiplication between the corresponding portions of the block attention matrix and the value matrix whose height from top to bottom is equal to the width of the block attention matrix.

[0097] After performing matrix multiplication on the attention and value matrices of the segments with the same height in the previous step, we will obtain their respective result matrices (i.e., ... Figure 3 The result matrices (Result Matrix I, Result Matrix II, Result Matrix III, and Result Matrix IV) have the same height as the attention matrix of each block, and the same width as the value matrix. Figure 3 There are 4 block attention matrices, and therefore 4 corresponding result matrices (i.e., result matrix I, result matrix II, result matrix III and result matrix IV). These result matrices are then stacked from top to bottom to form the complete result matrix.

[0098] Assume the lower triangular attention matrix is ​​N×N and the value matrix is ​​N×D. Dividing the lower triangular attention matrix vertically into M parts as described above, the calculation process can be transformed into:

[0099]

[0100] Based on theoretical analysis, according to the method provided in the above embodiment, the lower triangular attention matrix with a size of 2048×2048 is first horizontally divided and then vertically cut into four parts with a height of 512. Theoretically, this can reduce the amount of computation by 3 / 8, and the actual test results are close to this ratio. In addition, if the lower triangular attention matrix is ​​first vertically divided and then horizontally cut, the computation time is even shorter.

[0101] The speech recognition device provided by the present invention is described below. The speech recognition device described below and the speech recognition method described above can be referred to in correspondence.

[0102] Based on any of the above embodiments, the present invention also provides a voice recognition device, such as... Figure 4 As shown, the device includes:

[0103] Determining unit 410 is used to determine the speech to be recognized;

[0104] The recognition unit 420 is used to input the speech to be recognized into the speech recognition model and obtain the speech recognition result output by the speech recognition model;

[0105] The speech recognition model is trained based on sample speech and its corresponding sample speech recognition results. The attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix. Each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix of the attention layer vertically and truncating it horizontally, or by sequentially dividing the lower triangular attention matrix horizontally and truncating it vertically and then removing the zero matrix.

[0106] Based on any of the above embodiments, the device further includes:

[0107] The longitudinal slicing unit is used to longitudinally slice the lower triangular attention matrix to obtain multiple longitudinal slicing matrices;

[0108] The transverse unit is used to truncate each sectional matrix horizontally by taking the first non-zero element of the first column of each sectional matrix as the starting point, shifting upward by the first element interval, and removing the zero matrix to obtain the corresponding block attention matrix.

[0109] or,

[0110] A transverse segmentation unit is used to transversely divide the lower triangular attention matrix to obtain multiple transverse segments;

[0111] The longitudinal segmentation unit is used to truncate each cross-section matrix longitudinally by shifting the last non-zero element of the bottom row of each cross-section matrix to the right by a second element interval, and then removing the zero matrix to obtain the corresponding block attention matrix.

[0112] Based on any of the above embodiments, the width of each longitudinal sectional matrix is ​​the same, and the height of each transverse sectional matrix is ​​the same.

[0113] Based on any of the above embodiments, the width of each longitudinal slicing matrix is ​​greater than or equal to 64, and the height of each transverse slicing matrix is ​​greater than or equal to 64.

[0114] Based on any of the above embodiments, the first element interval is zero, and the second element interval is zero.

[0115] Based on any of the above embodiments, the device further includes:

[0116] The superposition unit is used when each block attention matrix is ​​a lower triangular attention matrix of the attention layer that is vertically divided and horizontally truncated in sequence. The attention layer is used to superimpose the matrix multiplication results of each block attention matrix and the value matrix.

[0117] Based on any of the above embodiments, the device further includes:

[0118] The stacking unit is used to stack the matrix multiplication results of each block attention matrix and the value matrix when each block attention matrix is ​​obtained by sequentially dividing and truncating the lower triangular attention matrix.

[0119] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 510, a memory 520, a communication interface 530, and a communication bus 540, wherein the processor 510, memory 520, and communication interface 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 520 to execute a speech recognition method, which includes: determining the speech to be recognized; inputting the speech to be recognized into a speech recognition model to obtain the speech recognition result output by the speech recognition model; wherein the speech recognition model is trained based on sample speech and its corresponding sample speech recognition result; the attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix; each block attention matrix is ​​obtained by sequentially vertically dividing and horizontally truncating the lower triangular attention matrix of the attention layer, or by sequentially horizontally dividing and vertically truncating the lower triangular attention matrix and then removing the zero matrix.

[0120] Furthermore, the logical instructions in the aforementioned memory 520 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the speech recognition method provided by the above methods, the method comprising: determining the speech to be recognized; inputting the speech to be recognized into a speech recognition model to obtain the speech recognition result output by the speech recognition model; wherein the speech recognition model is trained based on sample speech and its corresponding sample speech recognition result; the attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix; each block attention matrix is ​​obtained by sequentially vertically dividing and horizontally truncating the lower triangular attention matrix of the attention layer, or by sequentially horizontally dividing and vertically truncating the lower triangular attention matrix and then removing the zero matrix.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned speech recognition methods. The method includes: determining a speech to be recognized; inputting the speech to be recognized into a speech recognition model to obtain a speech recognition result output by the speech recognition model; wherein the speech recognition model is trained based on sample speech and its corresponding sample speech recognition results; the attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix; each block attention matrix is ​​obtained by sequentially vertically dividing and horizontally truncating the lower triangular attention matrix of the attention layer, or by sequentially horizontally dividing and vertically truncating the lower triangular attention matrix and then removing the zero matrix.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Identify the speech to be recognized; The speech to be recognized is input into the speech recognition model to obtain the speech recognition result output by the speech recognition model; The speech recognition model is trained based on sample speech and its corresponding sample speech recognition results; the attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix; each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix of the attention layer vertically and truncating it horizontally, or by sequentially dividing the lower triangular attention matrix horizontally and truncating it vertically and then removing the zero matrix. The attention matrix for each block is determined based on the following steps: After vertically dividing the lower triangular attention matrix, multiple sectional matrices are obtained; Starting from the first non-zero element of the first column of each sectional matrix, shift the first element upwards by zero intervals, truncate each sectional matrix horizontally, remove the zero matrix, and obtain the corresponding block attention matrix. or, After horizontally dividing the lower triangular attention matrix, multiple horizontally slicing matrices are obtained; Starting from the last non-zero element at the bottom of each cross-cutting matrix, shift to the right by a second element with a zero interval, truncate each cross-cutting matrix vertically, remove the zero matrix, and obtain the corresponding block attention matrix.

2. The speech recognition method according to claim 1, characterized in that, All vertical cross matrices have the same width, and all horizontal cross matrices have the same height.

3. The speech recognition method according to claim 2, characterized in that, The width of each vertical matrix is ​​greater than or equal to 64, and the height of each horizontal matrix is ​​greater than or equal to 64.

4. The speech recognition method according to claim 1, characterized in that, Also includes: When each block attention matrix is ​​a vertically segmented and horizontally truncated lower triangular attention matrix of the attention layer, the attention layer is used to superimpose the matrix multiplication results of each block attention matrix and the value matrix.

5. The speech recognition method according to claim 1, characterized in that, Also includes: When each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix horizontally and vertically, the attention layer is used to stack the matrix multiplication results of each block attention matrix and the value matrix.

6. A voice recognition device, characterized in that, include: The determining unit is used to determine the speech to be recognized; The recognition unit is used to input the speech to be recognized into the speech recognition model and obtain the speech recognition result output by the speech recognition model. The speech recognition model is trained based on sample speech and its corresponding sample speech recognition results; the attention layer of the speech recognition model is used to perform matrix multiplication between each block attention matrix and the value matrix; each block attention matrix is ​​obtained by sequentially dividing the lower triangular attention matrix of the attention layer vertically and truncating it horizontally, or by sequentially dividing the lower triangular attention matrix horizontally and truncating it vertically and then removing the zero matrix. The attention matrix for each block is determined based on the following steps: After vertically dividing the lower triangular attention matrix, multiple sectional matrices are obtained; Starting from the first non-zero element of the first column of each sectional matrix, shift the first element upwards by zero intervals, truncate each sectional matrix horizontally, remove the zero matrix, and obtain the corresponding block attention matrix. or, After horizontally dividing the lower triangular attention matrix, multiple horizontally slicing matrices are obtained; Starting from the last non-zero element at the bottom of each cross-cutting matrix, shift to the right by a second element with a zero interval, truncate each cross-cutting matrix vertically, remove the zero matrix, and obtain the corresponding block attention matrix.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech recognition method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 5.

Images