Representation method, speech recognition method, device, electronic equipment
By using filters to generate query vectors, key vectors, and value vectors in a self-attention mechanism, the problem of excessive number of parameters is solved, the memory footprint of neural networks is reduced, and the application of neural networks in devices with limited memory is promoted.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA GROUP HOLDING LTD
- Filing Date
- 2020-07-23
- Publication Date
- 2026-05-12
AI Technical Summary
Existing self-attention mechanisms consume too much memory due to the large number of parameters, making them difficult to popularize in devices with limited memory or cache.
The feature vector is transformed using a filter to generate a query vector, a key vector, and a value vector, reducing the number of parameters. The filter only calculates a portion of the feature vector at a time.
The number of parameters in the self-attention mechanism has been reduced, the memory footprint of the neural network has been decreased, and the neural network has been promoted to be used in devices with limited memory.
Smart Images
Figure CN113971948B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and more particularly to a representation method, a speech recognition method, a device, and an electronic device. Background Technology
[0002] Self-attention is a network framework in deep learning, widely used in fields such as natural language processing and speech recognition. Self-attention enables neural network models to better connect with context, resulting in more accurate processing results.
[0003] In existing self-attention mechanisms, a linear transformation matrix is generally used to linearly transform the input feature vectors to obtain the Query, Key, and Value corresponding to the feature vectors. Then, based on the Query, Key, and Value corresponding to each feature vector, the network representation after applying the self-attention mechanism is obtained by querying all features.
[0004] However, this attention mechanism has a large number of parameters, which means that it requires a lot of memory when applying the self-attention mechanism, making it difficult to popularize, especially in devices with small memory or cache. Summary of the Invention
[0005] The purpose of this application is to provide a representation method, speech recognition method, device, or electronic device to at least solve or alleviate the above-mentioned problems.
[0006] According to a first aspect of the embodiments of this application, a representation method based on a self-attention mechanism is provided, comprising: acquiring data to be processed, wherein the data to be processed is one of the following: speech data to be processed, text data to be processed, and image data to be processed; acquiring a feature vector generated after feature extraction of a data vector corresponding to the data to be processed; performing a transformation process on the feature vector through a filter to obtain a triple for self-attention calculation, wherein the triple includes: a query vector, a key vector, and a value vector; and performing self-attention calculation based on the query vector, the key vector, and the value vector to obtain a corresponding network representation.
[0007] According to a second aspect of the embodiments of this application, a speech recognition method is provided, comprising: obtaining a speech feature vector generated by feature extraction on a data vector corresponding to the speech to be processed; performing a transformation process on the speech feature vector through a filter to obtain a triple for self-attention calculation, wherein the triple includes: a query vector, a key vector, and a value vector; performing self-attention calculation based on the query vector, the key vector, and the value vector to obtain a corresponding network representation; and recognizing the speech to be processed based on the network representation.
[0008] According to a third aspect of the embodiments of this application, a representation device based on a self-attention mechanism is provided, comprising: an acquisition module, configured to acquire data to be processed, wherein the data to be processed is one of the following: speech data to be processed, text data to be processed, and image data to be processed; and a feature vector generated after feature extraction of a data vector corresponding to the data to be processed; a vector generation module, configured to perform transformation processing on the feature vector through a filter to obtain a triple for self-attention calculation, wherein the triple includes: a query vector, a key vector, and a value vector; and a representation generation module, configured to perform self-attention calculation based on the query vector, the key vector, and the value vector to obtain a corresponding network representation.
[0009] According to a fourth aspect of the embodiments of this application, a speech recognition device is provided, comprising: a speech feature vector determination module, configured to obtain a speech feature vector generated after feature extraction of a data vector corresponding to the speech to be processed; a vector generation module, configured to perform transformation processing on the speech feature vector through a filter to obtain a triple for self-attention calculation, wherein the triple includes: a query vector, a key vector, and a value vector; a representation generation module, configured to perform self-attention calculation based on the query vector, the key vector, and the value vector to obtain a corresponding network representation; and a recognition module, configured to recognize the speech to be processed based on the network representation.
[0010] According to a fifth aspect of the present application, an electronic device is provided, comprising: one or more processors; and a computer-readable medium configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described representation method or speech recognition method.
[0011] According to a sixth aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described representation method or speech recognition method.
[0012] The technical solution provided in this application involves obtaining a feature vector generated after feature extraction from the data vector corresponding to the data to be processed. Then, a filter is used to transform the feature vector, obtaining a triplet for self-attention calculation. The triplet includes a query vector, a key vector, and a value vector. Compared to the commonly used linear transformation matrices Wq, Wk, and Wv, the filter can calculate only a portion of the feature vector at a time, rather than performing a linear transformation on the entire feature vector. Therefore, the number of parameters in the filter is less than the number of parameters in the linear transformation matrix, thus reducing the number of parameters in the process of generating the query vector, key vector, and value vector. After generating the query vector, key vector, and value vector, self-attention calculation can be performed based on them to obtain the corresponding network representation. Therefore, this implementation directly reduces the number of parameters corresponding to the self-attention mechanism, thereby reducing the number of parameters in the neural network using the self-attention mechanism, i.e., reducing the memory occupied by the neural network, which is beneficial for the widespread adoption of neural networks. Furthermore, when the above solution is applied to scenarios such as speech recognition, it can also reduce the number of parameters in the speech recognition model using the self-attention mechanism, reduce the memory occupied by the speech recognition model, and improve speech recognition efficiency. Attached Figure Description
[0013] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0014] Figure 1 This is a schematic diagram illustrating the principle of a self-attention mechanism.
[0015] Figure 2 This is a schematic diagram illustrating the application environment of the representation method in the embodiments of this application;
[0016] Figure 3A This is a flowchart illustrating the method of Embodiment 1 of this application;
[0017] Figure 3B This is a schematic diagram illustrating a usage scenario of the representation method in Embodiment 1 of this application;
[0018] Figure 4A This is a flowchart illustrating the method of embodiment two of this application;
[0019] Figure 4B This is a schematic diagram illustrating a usage scenario of the representation method in Embodiment 2 of this application;
[0020] Figure 5A This is a flowchart illustrating the speech recognition method of Embodiment 3 of this application;
[0021] Figure 5B This is a schematic diagram of the neural network model in Embodiment 3 of this application;
[0022] Figure 6 This is a schematic diagram of the structure of the display device according to Embodiment 4 of this application;
[0023] Figure 7 This is a schematic diagram of the speech recognition device according to Embodiment 5 of this application;
[0024] Figure 8 This is a schematic diagram of the electronic device in Embodiment Six of this application;
[0025] Figure 9 This is a schematic diagram of the hardware structure of the electronic device in Embodiment 7 of this application. Detailed Implementation
[0026] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0028] To explain this scheme in more detail, we will first introduce the commonly used self-attention mechanism.
[0029] Reference Figure 1 Typically, when applying the self-attention mechanism, the neural network model including the self-attention mechanism can be trained first to obtain three linear transformation matrices Wq, Wk, and Wv after training. The number of dimensions of the linear transformation matrices can be the same as the number of dimensions of the feature vectors input to the self-attention mechanism.
[0030] When applying the self-attention mechanism, for an input feature vector, input can be linearly transformed using three linear transformation matrices Wq, Wk, and Wv to obtain the corresponding query vector, key vector, and value vector. The specific calculation formula is as follows:
[0031] Query = input * Wq;
[0032] Key = input * Wk;
[0033] Value = input * Wv.
[0034] Taking a text as an example, the text contains n characters. Through word embedding transformation, the vector representation corresponding to each character can be determined. If the vector representation dimension of the character is m, then the n characters can be converted into an n*m sequence, with each character corresponding to an m-dimensional vector.
[0035] After introducing the self-attention mechanism, each element in the sequence can be used as input. By calculating using the above method, a set of (Query, Key, Value) corresponding to each element can be obtained. n characters can correspond to n sets of (Query, Key, Value); or, the above sequence can be directly used as input, and by calculating using the above method, three sequences [q1, q2, ..., qn], [k1, k2, ..., kn], and [v1, v2, ..., vn] corresponding to Query, Key, and Value can be obtained respectively, where n is the number of characters.
[0036] During computation, the query corresponding to a certain character can be used as the query object. The query is then queried among the keys of all characters to obtain the correlation or similarity between the query corresponding to the character and the keys of all characters. Then, the value vector of the element can be fused with the obtained correlation or similarity and the output can be output. Based on the output, a network representation corresponding to the character after applying the self-attention mechanism can be generated, such as generating a vector to represent the character.
[0037] This application of self-attention mechanism allows the neural network model to focus on certain characters, thereby improving the model's processing efficiency and performance.
[0038] However, this approach requires training three linear transformation matrices Wq, Wk, and Wv for the query vector, key vector, and value vector, resulting in a large number of parameters for the attention mechanism. In other words, the self-attention mechanism consumes a lot of resources when applied, making it difficult to popularize, especially for devices with limited memory or cache.
[0039] See Figure 2 The diagram illustrates the structural structure of an application system for the method provided in this embodiment. Figure 2 As shown, the system may include: terminal device 101 and computing device 102.
[0040] Terminal device 101 can be a desktop computer, mobile phone, tablet computer, server, etc. Terminal device 101 can also be an independent server or client, or a cluster of multiple servers, etc. This embodiment does not limit it in this way.
[0041] Terminal device 101 and computing device 102 can be connected via Bluetooth, network, etc. Terminal device 101 can send data to be processed to computing device 102. The data to be processed may include a piece of text content to be translated, or a piece of speech content to be recognized, etc.
[0042] Terminal device 101 can send data to be processed to computing device 102 in real time or at regular intervals.
[0043] The computing device 102 can also be a desktop computer, mobile phone, tablet computer, server, etc. The computing device 102 can also be used to provide cloud services, cloud storage, cloud computing, etc., but this embodiment does not limit it in this regard. The terminal device 101 and the computing device 102 can be the same or different. The terminal device 101 and the computing device 102 can belong to the same hardware or to different hardware, but this embodiment does not limit it in this regard.
[0044] A neural network can be deployed on the computing device 102. The computing device 102 can process the data to be processed, such as converting the data to be processed into a sequence that is input into the neural network. The neural network can process the input sequence and output an output.
[0045] The neural network deployed on computing device 102 can employ a self-attention mechanism. This embodiment does not limit the specific type of neural network; for example, it can be a neural network model for image recognition, a neural network model for translation, a neural network model for speech recognition, etc.
[0046] It should be noted that the above description is only an exemplary description of the application system of the method of this application. In other implementations, this is not limited. For example, the computing device 102 can generate the data to be processed and convert it into a sequence that is input to the neural network, and the neural network deployed on it can process the sequence and output it.
[0047] To facilitate understanding of the solutions provided in the embodiments of this application, the representation scheme based on the self-attention mechanism will be described first, and then the speech recognition scheme provided in the embodiments of this application will be described based on the representation scheme.
[0048] Reference Figure 3A , Figure 3B , Figure 3A A flowchart illustrating a representation method based on a self-attention mechanism according to an embodiment of this application is shown.
[0049] Specifically, the representation method provided in this embodiment includes the following steps:
[0050] S200, Obtain the data to be processed.
[0051] The data to be processed can be one of the following: voice data to be processed, text data to be processed, or image data to be processed.
[0052] S201. Obtain the feature vector generated after feature extraction of the data vector corresponding to the data to be processed.
[0053] In this embodiment, when the data to be processed is speech data, the feature vector can be a speech feature vector generated after extracting features from the data vector corresponding to the speech to be processed. For example, it can be a vector corresponding to the speech data of a frame of speech.
[0054] In translation scenarios, the speech feature vector can be the speech feature vector corresponding to the speech to be translated, and the speech feature vector can correspond one-to-one with the frame number in the speech to be translated. Of course, the above is only an example and is not intended to limit this application.
[0055] In this embodiment, if a self-attention mechanism is applied in a certain neural network, the above-mentioned speech feature vector can be the speech feature vector transformed by the neural network model, such as the speech feature vector transformed by the compiler. This embodiment does not limit this.
[0056] In this embodiment, when the data to be processed is text data, the feature vector can be the text feature vector obtained after feature extraction of the text data to be processed.
[0057] A character can be a Chinese character, an English word, a number symbol, etc., and this embodiment does not limit it.
[0058] In translation scenarios, a text feature vector can be the text feature vector corresponding to the text to be translated, and the text feature vector can correspond one-to-one with the characters in the text to be translated. Alternatively, the text to be translated can be segmented into words, and then the text feature vector can correspond one-to-one with the segmentation results.
[0059] In this embodiment, if a self-attention mechanism is applied in a neural network, the above-mentioned text feature vector can be the text feature vector transformed by the neural network model, such as the text feature vector transformed by a compiler. This embodiment does not limit this.
[0060] Similarly, when the data to be processed is image data, the feature vector can be an image feature vector from which features are extracted from the image data to be processed.
[0061] Of course, the above are merely illustrative examples, and the embodiments of this application are not limited thereto. Other data to be processed that can be processed using the self-attention mechanism are also within the scope of protection of this application.
[0062] S202. The feature vector is transformed by a filter to obtain triples for self-attention calculation, wherein the triples include: query vector, key vector and value vector.
[0063] The method provided in this embodiment is a representation method based on a self-attention mechanism. The self-attention mechanism is a mechanism that captures global context information by query vector, key vector, and value vector to improve the accuracy of the results.
[0064] In this embodiment, at least one of the query vector, key vector, and value vector is obtained by transforming the feature vector through a filter.
[0065] A filter, also known as a filter, can perform calculations on a portion of the input each time and produce a corresponding output. After calculating and outputting all portions, all outputs can be superimposed as the filter's actual output.
[0066] In this application, a filter can be used to calculate a portion of the feature vector each time and generate a corresponding output. After all parts of the feature vector have been calculated, the outputs can be superimposed as a query vector, a key vector, or a value vector.
[0067] For example, if the speech feature vector is [x1,x2,x3,…,xn], then through a filter, calculations can be performed on [xt-N1,…,xt,…,xt+N2] each time to obtain the corresponding output. t, N1, and N2 can be chosen according to requirements, as long as each x in the speech feature vector can be calculated. After calculating all parts of the speech feature vector, the outputs can be superimposed, and the superimposed output can be used as a query vector, key vector, or value vector.
[0068] Compared to the commonly used linear transformation matrices Wq, Wk, and Wv, filters can compute only a portion of the speech feature vector each time, rather than performing a linear transformation on the entire speech feature vector. Therefore, the number of parameters in a filter is smaller than that in a linear transformation matrix, which reduces the number of parameters corresponding to the self-attention mechanism, and thus reduces the number of parameters in a neural network that applies the self-attention mechanism.
[0069] For example, the feature vector can be 512-dimensional. In order to perform a linear transformation on the feature vector, the required linear transformation matrix is generally a 512*512 linear transformation matrix. The number of parameters corresponding to one linear transformation matrix is 512*512. In the self-attention mechanism, three vectors need to be generated: query vector, key vector, and value vector. Therefore, three linear transformation matrices are required, and the number of parameters is 512*512*3=786432b.
[0070] In this embodiment, the number of parameters of the filter can be 31. If one of the query vector, key vector, and value vector is generated by the filter, the corresponding number of parameters can be 512*512*2+31*512=540160b, which significantly reduces the number of model parameters.
[0071] S203. Perform self-attention calculation based on the query vector, key vector, and value vector to obtain the corresponding network representation.
[0072] In this embodiment, during computation, the similarity or relevance of the query vector Query to all other key vectors Key can be calculated to obtain the weight coefficient of the value vector Value corresponding to each key vector Key. Then, the value vectors Value are weighted and summed to obtain the attention level corresponding to the query vector Query. Subsequently, the network representation corresponding to the speech feature vector of the query vector Query can be generated based on the attention level. The specific method for generating the network representation can refer to related technologies, and this embodiment does not limit it.
[0073] See Figure 3B This illustrates a specific use case in this embodiment.
[0074] like Figure 3B As shown, after obtaining the feature vector generated by feature extraction of the data vector corresponding to the data to be processed, the feature vector can be transformed by a filter to obtain the query vector and key vector corresponding to the feature vector. The value vector corresponding to the feature vector can be obtained by linear transformation or by other means. This embodiment does not limit this.
[0075] After obtaining the query vector, key vector, and value vector corresponding to the feature vector, a dot product operation can be performed on the query vector and key vector, and the result can be normalized. Then, the normalized parameters can be fused with the value vector to generate the corresponding network representation.
[0076] The technical solution provided in this application involves obtaining a feature vector generated after feature extraction from the data vector corresponding to the data to be processed. Then, a filter is used to transform the feature vector to obtain a query vector, a key vector, and a value vector. Compared to the commonly used linear transformation matrices Wq, Wk, and Wv, the filter can calculate only a portion of the feature vector at a time, rather than performing a linear transformation on the entire feature vector. Therefore, the number of parameters in the filter is less than the number of parameters in the linear transformation matrix. This reduces the number of parameters involved in generating the query vector, key vector, and value vector. After generating the query vector, key vector, and value vector, self-attention calculation can be performed based on them to obtain the corresponding network representation. Therefore, this implementation directly reduces the number of parameters corresponding to the self-attention mechanism, thereby reducing the number of parameters in the neural network using the self-attention mechanism, i.e., reducing the memory occupied by the neural network, which is beneficial for the widespread adoption of neural networks.
[0077] The representation method of this embodiment can be executed by any suitable device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, in-vehicle devices, entertainment devices, advertising devices, personal digital assistants (PDAs), tablet computers, laptops, handheld game consoles, smart glasses, smartwatches, wearable devices, virtual display devices, or display enhancement devices, etc.
[0078] Reference Figure 4A , Figure 4B , Figure 4A A flowchart illustrating the steps of the representation method in Embodiment 2 of this application is shown. Figure 4B A usage scenario diagram of the representation method of Embodiment 2 of this application is shown. In this embodiment, the filter includes a first filter for generating a query vector and a second filter for generating a key vector; furthermore, this application uses speech to be processed as an example for illustration.
[0079] Specifically, the representation method provided in this embodiment includes the following steps:
[0080] S300: Acquire the voice data to be processed.
[0081] In practical applications, those skilled in the art can acquire the voice data to be processed in any appropriate manner, and the embodiments of this application do not impose any restrictions on this.
[0082] S301. Obtain the speech feature vector generated after feature extraction of the data vector corresponding to the speech to be processed.
[0083] In this embodiment, the obtained speech feature vector can be a single speech feature vector or a sequence of speech feature vectors composed of multiple speech feature vectors. This embodiment does not limit this.
[0084] If the self-attention mechanism is multi-head self-attention, after obtaining the speech feature vector, the speech feature vector can be segmented to obtain multiple segmented speech feature vectors. For each segmented speech feature vector, a query vector, key vector, and value vector are generated based on the speech feature vector, resulting in query vectors, key vectors, and value vectors corresponding to the multiple segmented speech feature vectors. In subsequent step S305, calculations can be performed based on the query vector, key vector, and value vector corresponding to each segment, and the calculation results are concatenated to generate the network representation corresponding to the speech feature vector. Alternatively, after obtaining the speech feature vector, query vectors, key vectors, and value vectors corresponding to the speech feature vector can be generated. The query vector, key vector, and value vector are segmented respectively, and calculations are performed based on the segmented query vector, key vector, and value vector. The calculation results are concatenated to generate the network representation corresponding to the speech feature vector.
[0085] By partitioning the computation, we can ensure that each computation results are small and that parallel computation can be performed, thereby improving the computation speed.
[0086] Of course, the above are merely illustrative examples and are not intended to limit this application. Other implementations of multi-head attention mechanisms are also within the scope of protection of this application.
[0087] S302. The speech feature vector is transformed by the first filter to generate the query vector.
[0088] S303. The speech feature vector is transformed by the second filter to generate the key vector.
[0089] In this embodiment, the first filter and the second filter in steps S302 and S303 can be the same or different, and this embodiment does not limit this.
[0090] In this embodiment, the feature vector is transformed using two filters to obtain two vectors in the triplet used for self-attention calculation. This embodiment illustrates obtaining a query vector and a key vector; however, those skilled in the art should understand that in practical applications, any two vectors in the triplet can be obtained, such as a key vector and a value vector, or a query vector and a value vector, etc. The other vector can be obtained by assigning values to the feature vector; this embodiment does not limit this.
[0091] Furthermore, the number of parameters in the filters (first filter and / or second filter) of this application can be at least one order of magnitude smaller than the number of dimensions of the query vector. This further reduces the number of parameters in the neural network applying the self-attention mechanism of this application, thus reducing the memory occupied by the neural network and facilitating its wider adoption.
[0092] Optionally, in this embodiment, the filter is a neural network model that models contextual information. By using a neural network model that models contextual information, contextual information can be incorporated into the generation of query vectors, key vectors, or value vectors, thereby improving the accuracy of the network representation corresponding to the generated speech feature vectors.
[0093] Furthermore, in this embodiment, the neural network model is a neural network model based on a feedforward sequence memory network, which is used to model contextual information. Through the feedforward sequence memory network, contextual information can be directly modeled, and the required number of parameters is relatively small.
[0094] A feed-forward sequential memory network (FSMN) includes a hidden layer, which contains a memory block. The memory block can be used to remember the information of the sequence of adjacent time steps of the input information at time t.
[0095] For example, in this application, if the speech feature vector is [x1,x2,x3,…,xn], then the output of the neural network model based on the feedforward sequence memory network can be:
[0096]
[0097] Where t can be 1, 2, 3, ..., n. N1 and N2 are the amount of information the neural network model looks at from the past and future, respectively. N1 and N2 are positive integers. The memory module can be used to remember information adjacent to xt, such as xt-1 and xt+1.
[0098] In this embodiment, the number of model parameters of the neural network model (filter) based on the feedforward sequence memory network is N1+N2+1, that is, the number of ai and cj.
[0099] By superimposing multiple FSMN(xt), a query vector or key vector can be generated. The total number of parameters required for the speech feature vector can be n*(N1+N2+1).
[0100] Of course, the above is only an illustrative example. In other implementations of this application, filters based on other neural networks may also be used, and this application does not limit this.
[0101] S304. Assign the speech feature vector as a value to the value vector.
[0102] Assignment is the process of assigning a value to a variable. In this application, it means setting the value vector Value to the speech feature vector.
[0103] Since the value vector is directly assigned from the speech feature vector, the assignment process does not require any model parameters to participate in the calculation. Therefore, the number of parameters corresponding to the self-attention mechanism can be further reduced, thereby reducing the number of parameters in the neural network that applies the self-attention mechanism.
[0104] For example, the dimension of a speech feature vector can be 512. In order to perform a linear transformation on the speech feature vector, the required linear transformation matrix is generally a 512*512 linear transformation matrix. The number of parameters corresponding to one linear transformation matrix is 512*512. In the self-attention mechanism, three vectors need to be generated: query vector, key vector, and value vector. Therefore, three linear transformation matrices are required, and the number of parameters is 512*512*3=786432b.
[0105] In this embodiment, the number of parameters for the first filter and the second filter can be 31. If the query vector is generated by the first filter, the key vector is generated by the second filter, and the value vector is directly assigned, the corresponding number of parameters can be 512*31*2=31744b. Compared with linear transformation, the number of model parameters is reduced by about half.
[0106] In an end-to-end speech recognition neural network, the speech encoder can include 10 self-attention layers, and the speech decoder can include three self-attention layers, thus reducing the number of model parameters:
[0107] (10+3)*(786432-31744)=9810944b≈9.36M
[0108] It should also be noted that the above steps S302-S305 are only illustrative examples and are not intended to limit this application. In other implementations of this application, the query vector can be generated only through a filter, and the key vector and value vector can be generated by linear transformation; or the query vector can be generated only through a filter, and the key vector can be generated by linear transformation, and the value vector can be generated by assignment.
[0109] Furthermore, this embodiment does not limit the execution process of the above steps S302-S305. The three steps can be executed in parallel or in sequence. Those skilled in the art can determine the appropriate method according to their needs.
[0110] S305. Perform self-attention calculation based on the query vector, key vector, and value vector to obtain the corresponding network representation.
[0111] In this embodiment, the query vector and the transpose of the key vector are multiplied by a dot product to obtain the attention coefficients. Then, these attention coefficients are divided by the square root of d_model, where d_model represents the dimension of the query vector. The attention coefficients after division are then normalized using the softmax function. The normalized attention coefficients are then multiplied by the value vector to obtain the output 'out' after applying the attention mechanism.
[0112] Specifically, the output can be represented as:
[0113] out = (softmax(query * key) T / √(d_model)))*value.
[0114] After determining the output out corresponding to the speech feature vector, the network representation corresponding to the speech feature vector can be further generated based on the output out.
[0115] The solution provided in this embodiment transforms the feature vectors using two filters to obtain two of the triplets used for self-attention calculation; then, it assigns values to the feature vectors to obtain the other vector in the triplets used for self-attention calculation; finally, it performs self-attention calculation based on the three vectors included in the triplets—the query vector, the key vector, and the value vector—to obtain the corresponding network representation, further reducing the number of parameters corresponding to the self-attention mechanism.
[0116] The representation method of this embodiment can be executed by any suitable device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, in-vehicle devices, entertainment devices, advertising devices, personal digital assistants (PDAs), tablet computers, laptops, handheld game consoles, smart glasses, smartwatches, wearable devices, virtual display devices, or display enhancement devices, etc.
[0117] Reference Figure 5A The flowchart of the speech recognition method according to Embodiment 3 of this application is shown as follows. Figure 5A As shown, the method includes:
[0118] S401. Obtain the speech feature vector generated after feature extraction of the data vector corresponding to the speech to be processed.
[0119] In this embodiment, one frame of the speech to be processed can correspond to one speech feature vector. Of course, one frame can correspond to multiple speech feature vectors, or multiple frames can correspond to one speech feature vector. This embodiment does not limit this.
[0120] S402. The speech feature vector is transformed by a filter to obtain triples for self-attention calculation, wherein the triples include: query vector, key vector and value vector.
[0121] For example, two filters can be used to transform the speech feature vectors to obtain two vectors in the triplet used for self-attention calculation; and an assignment operation can be performed on the speech feature vectors to obtain the other vector in the triplet used for self-attention calculation.
[0122] Specifically, in this embodiment, the query vector and key vector can be obtained by transforming the feature vector through a filter, and the speech feature vector is directly assigned to the value vector.
[0123] In this embodiment, the specific methods for generating query vectors, key vectors, and value vectors are the same as those in the above embodiments, and will not be repeated here.
[0124] S403. Perform self-attention calculation based on the query vector, key vector, and value vector to obtain the corresponding network representation.
[0125] In this embodiment, the method for generating network representations is the same as in the above embodiment, and will not be repeated here.
[0126] S404. Based on the network representation, recognize the speech to be processed.
[0127] In this embodiment, after recognition, the text content corresponding to the speech to be processed can be output according to the recognition result.
[0128] See Figure 5B The diagram shows a schematic of the neural network model that applies the self-attention mechanism described above. Here, an end-to-end speech recognition model is used as an example for illustration.
[0129] The speech recognition model includes a speech encoder and a speech decoder. Both the speech encoder and the speech decoder have self-attention layers. In practical applications, only one of the speech encoder and the speech decoder may use the solution provided in this application embodiment, or both may use the solution provided in this application embodiment.
[0130] The speech encoder receives the input sequence, which can be generated by processing the input speech at a low frame rate. The input sequence consists of speech feature vectors. For example, if the input to the speech recognition model is a speech consisting of 50 frames, the speech recognition model can obtain the data vector corresponding to the input speech, and then the feature extraction part of the speech recognition model can extract features from the data vector to generate speech feature vectors, which are then input to the speech encoder. The feature extraction part of the speech recognition model can be independent of the speech encoder, integrated into the speech encoder, or a combination of both. In this case, the input to the speech encoder can be a vector sequence consisting of 50 speech feature vectors, where each vector corresponds one-to-one with a word in the speech.
[0131] A speech encoder can output a corresponding encoded representation based on an input sequence. Specifically, the speech encoder may include a simplified self-attention mechanism layer (i.e., the self-attention mechanism using the method described above). The speech feature vector can be transformed using filters in the speech encoder to obtain triples for self-attention calculation: a query vector, a key vector, and a value vector. Then, the self-attention layer in the speech encoder performs self-attention calculation on the query vector, the key vector, and the value vector to obtain the corresponding network representation. The encoded representation output by the speech encoder can be a network representation generated from the self-attention processing corresponding to the speech feature vector, or it can be further processed to generate an encoded representation such as a speech encoded vector. This embodiment does not limit this approach.
[0132] Optionally, in this embodiment, the speech encoder may further include a normalization layer to normalize the output of the self-attention mechanism.
[0133] Additionally, the speech encoder may include a feedforward network to receive the output of the self-attention mechanism and transform it to obtain the output of the speech encoder. Similarly, the output of the feedforward network is also input to a normalization layer for normalization.
[0134] For the speech decoder in a speech recognition model, it can generate an output sequence of character vectors based on the encoded representation output by the speech encoder. If the input to the speech recognition neural network model is a speech consisting of 50 frames, the character vectors in the output sequence can identify the text corresponding to the speech.
[0135] In this embodiment, as Figure 5BAs shown, the speech decoder can include a simplified self-attention mechanism. Specifically, this simplified self-attention mechanism can be an encoder-decoder attention layer, etc., all of which can use the aforementioned self-attention mechanism. In this case, the speech decoder in the speech recognition model obtains the speech encoding vector output by the speech encoder and performs feature extraction on the speech encoding vector to generate a speech feature vector. Then, the speech feature vector is transformed through a filter in the speech decoder to obtain a triplet for self-attention calculation: a query vector, a key vector, and a value vector. Finally, the self-attention layer in the speech decoder performs self-attention calculation on the query vector, the key vector, and the value vector to obtain the corresponding network representation.
[0136] In a speech recognition model, the output sequence of the speech decoder can be the speech recognition result.
[0137] The above examples illustrate the application of the solution in a speech recognition model. However, those skilled in the art should understand that the speech recognition model is merely illustrative, and the solution in this application can be applied to any appropriate neural network model or data processing flow that employs a self-attention mechanism. For example, it can be applied to text detection models, text recognition models, image detection models, or image recognition models that use a self-attention mechanism, etc. In these scenarios, the application of this solution can refer to the processing implementation of the self-attention part in the above-described speech recognition model, and will not be elaborated further here.
[0138] In this embodiment, compared with the commonly used linear transformation matrices Wq, Wk, and Wv, the filter can calculate only a portion of the speech feature vector each time, rather than performing a linear transformation on the entire speech feature vector. Therefore, the number of parameters in the filter is less than the number of parameters in the linear transformation matrix. This reduces the number of parameters required to generate the query vector, key vector, and value vector. After generating the query vector, key vector, and value vector, self-attention calculation can be performed based on them to obtain the corresponding network representation. Thus, the speech to be processed can be recognized based on the network representation. Therefore, the query vector and key vector are obtained by transforming the feature vector through the filter. The speech feature vector is directly assigned to the value vector, which directly reduces the number of parameters corresponding to the self-attention mechanism, thereby reducing the number of parameters in the neural network that applies the self-attention mechanism, i.e., reducing the memory occupied by the neural network, which is conducive to the popularization of neural networks.
[0139] The representation method of this embodiment can be executed by any suitable device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, in-vehicle devices, entertainment devices, advertising devices, personal digital assistants (PDAs), tablet computers, laptops, handheld game consoles, smart glasses, smartwatches, wearable devices, virtual display devices, or display enhancement devices, etc.
[0140] Reference Figure 6 The diagram shows a schematic representation device based on a self-attention mechanism in Embodiment 4 of this application.
[0141] The representation device in this embodiment includes: an acquisition module 501, a vector generation module 502, and a representation generation module 503.
[0142] The acquisition module 501 is used to acquire data to be processed, wherein the data to be processed is one of the following: voice data to be processed, text data to be processed, image data to be processed; and to acquire a feature vector generated after feature extraction of the data vector corresponding to the data to be processed.
[0143] The vector generation module 502 is used to transform the feature vector through a filter to obtain triples for self-attention calculation, wherein the triples include: query vector, key vector and value vector;
[0144] The representation generation module 503 is used to perform self-attention calculation based on the query vector, key vector, and value vector to obtain the corresponding network representation.
[0145] Optionally, in any embodiment of this application, when the data to be processed is voice data, the corresponding feature vector is a voice feature vector; or, when the data to be processed is text data, the corresponding feature vector is a text feature vector; or, when the data to be processed is image data, the corresponding feature vector is an image feature vector.
[0146] Optionally, in any embodiment of this application, the vector generation module 502 includes: a conversion module, used to convert the feature vectors through two filters respectively to obtain two vectors in the triplet for self-attention calculation; and an assignment module to perform an assignment operation based on the feature vectors to obtain another vector in the triplet for self-attention calculation.
[0147] Optionally, in any embodiment of this application, the conversion module is specifically used to: convert the feature vector through two filters respectively to obtain the query vector and key vector in the triplet used for self-attention calculation; the assignment module is specifically used to: assign the feature vector as a value to the value vector in the triplet used for self-attention calculation.
[0148] Optionally, in any embodiment of this application, the number of parameters of the filter is at least one order of magnitude smaller than the number of dimensions of the query vector.
[0149] Optionally, in any embodiment of this application, the filter is a neural network model that models contextual information.
[0150] Optionally, in any embodiment of this application, the neural network model is a neural network model based on a feedforward sequence memory network, which is used to model contextual information.
[0151] The representation device based on the self-attention mechanism in this embodiment is used to implement the corresponding representation methods based on the self-attention mechanism in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0152] Reference Figure 7 The diagram shows a schematic representation of the speech recognition device in Embodiment 5 of this application.
[0153] The speech recognition device in this embodiment includes: a speech feature vector acquisition module 601, a vector generation module 602, a representation generation module 603, and a recognition module 604.
[0154] The speech feature vector acquisition module 601 is used to acquire the speech feature vector generated after feature extraction of the data vector corresponding to the speech to be processed.
[0155] The vector generation module 602 is used to transform the speech feature vector through a filter to obtain triples for self-attention calculation, wherein the triples include: query vector, key vector and value vector;
[0156] The representation generation module 603 is used to perform self-attention calculation based on the query vector, key vector, and value vector to obtain the corresponding network representation;
[0157] The recognition module 604 is used to recognize the speech to be processed based on the network representation.
[0158] Optionally, in any embodiment of this application, the vector generation module 602 is used to transform the speech feature vector through two filters respectively to obtain two vectors in the triplet for self-attention calculation; and to perform an assignment operation based on the speech feature vector to obtain the other vector in the triplet for self-attention calculation.
[0159] Optionally, in any embodiment of this application, the vector generation module 602 is used to transform the speech feature vector through two filters respectively to obtain the query vector and the key vector in the triplet used for self-attention calculation; and to assign the speech feature vector as a value to the value vector in the triplet used for self-attention calculation.
[0160] Optionally, in any embodiment of this application, the speech recognition device includes a speech recognition model, which includes a speech encoder. The speech recognition device obtains a data vector corresponding to the speech to be processed through the speech recognition model, and performs feature extraction on the data vector through the feature extraction part of the speech recognition model to generate a speech feature vector, which is then input into the speech encoder. The speech feature vector is transformed through a filter in the speech encoder to obtain a triplet for self-attention calculation. Self-attention calculation is performed on the query vector, the key vector, and the value vector through a self-attention layer in the speech encoder to obtain the corresponding network representation.
[0161] Optionally, in any embodiment of this application, the speech recognition device includes a speech recognition model, which includes a speech encoder and a speech decoder. The speech recognition device obtains the speech encoding vector output by the speech encoder through the speech decoder in the speech recognition model, and performs feature extraction on the speech encoding vector to generate a speech feature vector. The speech feature vector is transformed through a filter in the speech decoder to obtain triples for self-attention calculation. Self-attention calculation is performed on the query vector, the key vector, and the value vector through a self-attention layer in the speech decoder to obtain the corresponding network representation.
[0162] The speech recognition device in this embodiment is used to implement the corresponding speech recognition method in the aforementioned method embodiment and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0163] Figure 8 This is a schematic diagram of the structure of the electronic device in Embodiment Six of this application; the electronic device may include:
[0164] One or more processors 701;
[0165] Computer-readable medium 702 can be configured to store one or more programs.
[0166] When one or more programs are executed by one or more processors, the one or more processors implement the above representation method or speech recognition method.
[0167] Figure 9 This is a schematic diagram of the hardware structure of the electronic device in Embodiment 7 of this application; as shown Figure 9 As shown, the hardware structure of the electronic device may include: a processor 801, a communication interface 802, a computer-readable medium 803, and a communication bus 804.
[0168] The processor 801, communication interface 802, and computer-readable medium 803 communicate with each other through the communication bus 804.
[0169] Optionally, the communication interface 802 can be an interface of a communication module, such as the interface of a GSM module;
[0170] Specifically, the processor 801 can implement the aforementioned representation method or speech recognition method.
[0171] The processor 801 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.
[0172] The computer-readable medium 803 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.
[0173] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code configured to perform the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access storage media (RAM), read-only storage media (ROM), erasable programmable read-only storage media (EPROM or flash memory), optical fibers, portable compact disk read-only storage media (CD-ROM), optical storage media, magnetic storage media, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0174] Computer program code configured to perform the operations of this application can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions configured to perform a specified logical function. Specific sequences are present in the above specific embodiments, but these sequences are merely exemplary; in actual implementations, these steps may be fewer, more, or executed in a different order. That is, in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0176] In another aspect, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the representation method or text recognition method described in the above embodiments.
[0177] The terms "first," "second," "first," or "second" as used in the various embodiments of this disclosure may modify various components regardless of their order and / or importance, but these terms do not limit the corresponding components. The above terms are configured only for the purpose of distinguishing an element from other elements. For example, "first user equipment" and "second user equipment" refer to different user equipments, although both are user equipment. For example, without departing from the scope of this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.
[0178] When a component (e.g., a first component) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another component (e.g., a second component), it should be understood that the first component is directly connected to the second component or that the first component is indirectly connected to the second component via yet another component (e.g., a third component). Conversely, it can be understood that when a component (e.g., a first component) is referred to as being "directly connected" or "directly coupled" to another component (the second component), no component (e.g., a third component) is inserted between the two.
[0179] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A speech recognition method, comprising: Obtain the speech feature vector generated after extracting features from the data vector corresponding to the speech to be processed; The speech feature vector is transformed by a filter to obtain a triple for self-attention computation. The triple includes a query vector, a key vector, and a value vector. The filter includes a first filter and a second filter. The number of parameters in the first filter or the second filter is at least one order of magnitude smaller than the number of dimensions in the query vector. Any two vectors in the triple are determined based on the first filter and the second filter, respectively. The other vector in the triple is determined by assigning a value to the speech feature vector. Self-attention calculation is performed based on the query vector, the key vector, and the value vector to obtain a network representation corresponding to the speech feature vector. The network representation is determined based on the concatenation result, which is obtained by calculating the segmented query vector, the segmented key vector, and the segmented value vector, and then concatenating the calculation results. The speech to be processed is recognized according to the network representation.
2. The method according to claim 1, wherein, The step of transforming the speech feature vector through a filter to obtain triples for self-attention calculation includes: The speech feature vector is transformed by two filters to obtain two vectors in the triplet used for self-attention calculation. The speech feature vector is assigned a value to obtain another vector in the triplet used for self-attention calculation.
3. The method according to claim 2, wherein, The speech feature vector is transformed by two filters to obtain two vectors in the triplet used for self-attention calculation, including: transforming the speech feature vector by two filters to obtain the query vector and the key vector in the triplet used for self-attention calculation; The step of assigning a value to the speech feature vector to obtain another vector in the triplet used for self-attention calculation includes: assigning the speech feature vector as a value to the value vector in the triplet used for self-attention calculation.
4. The method according to claim 1, wherein, The method is applied to a speech recognition model, which includes a speech encoder. The step of obtaining the speech feature vector generated after extracting features from the data vector corresponding to the speech to be processed includes: obtaining the data vector corresponding to the speech to be processed through the speech recognition model, and extracting features from the data vector through the feature extraction part of the speech recognition model to generate a speech feature vector and inputting it into the speech encoder; The step of transforming the speech feature vector through a filter to obtain triples for self-attention calculation includes: transforming the speech feature vector through a filter in the speech encoder to obtain triples for self-attention calculation; The step of performing self-attention calculation based on the query vector, the key vector, and the value vector to obtain the corresponding network representation includes: performing self-attention calculation on the query vector, the key vector, and the value vector through the self-attention layer in the speech encoder to obtain the corresponding network representation.
5. The method according to claim 1, wherein, The method is applied to a speech recognition model, which includes a speech encoder and a speech decoder. The step of obtaining the speech feature vector generated after extracting features from the data vector corresponding to the speech to be processed includes: obtaining the speech encoding vector output by the speech encoder through the speech decoder in the speech recognition model, and extracting features from the speech encoding vector to generate the speech feature vector; The step of transforming the speech feature vector through a filter to obtain triples for self-attention calculation includes: transforming the speech feature vector through a filter in the speech decoder to obtain triples for self-attention calculation; The step of performing self-attention calculation based on the query vector, the key vector, and the value vector to obtain the corresponding network representation includes: performing self-attention calculation on the query vector, the key vector, and the value vector through the self-attention layer in the speech decoder to obtain the corresponding network representation.
6. A representation method based on a self-attention mechanism, comprising: Acquire data to be processed, wherein the data to be processed is one of the following: voice data to be processed, text data to be processed, or image data to be processed; Obtain the feature vector generated after feature extraction of the data vector corresponding to the data to be processed; The feature vector is transformed by a filter to obtain a triple for self-attention computation. The triple includes a query vector, a key vector, and a value vector. The filter includes a first filter and a second filter. The number of parameters in the first filter or the second filter is at least one order of magnitude smaller than the number of dimensions in the query vector. Any two vectors in the triple are determined based on the first filter and the second filter, respectively. The other vector in the triple is determined based on the feature vector through an assignment operation. Self-attention calculation is performed based on the query vector, the key vector, and the value vector to obtain a network representation corresponding to the feature vector. The network representation is determined based on the concatenation result, which is obtained by calculating the segmented query vector, the segmented key vector, and the segmented value vector, and then concatenating the calculation results.
7. The method according to claim 6, wherein, When the data to be processed is speech data to be processed, the corresponding feature vector is a speech feature vector; Alternatively, when the data to be processed is text data to be processed, the corresponding feature vector is a text feature vector; Alternatively, when the data to be processed is image data to be processed, the corresponding feature vector is an image feature vector.
8. The method according to claim 6, characterized in that, The step of transforming the feature vector through a filter to obtain triples for self-attention computation includes: The feature vectors are transformed by two filters to obtain two vectors in the triplet used for self-attention calculation. The feature vector is assigned a value to obtain another vector in the triplet used for self-attention calculation.
9. The method according to claim 8, wherein, The feature vector is transformed by two filters to obtain two vectors in the triplet used for self-attention calculation, including: transforming the feature vector by two filters to obtain the query vector and the key vector in the triplet used for self-attention calculation; The step of performing the assignment operation based on the feature vector to obtain another vector in the triplet used for self-attention calculation includes: assigning the feature vector as a value to the value vector in the triplet used for self-attention calculation.
10. The method according to any one of claims 6-9, wherein, The filter is a neural network model that models contextual information.
11. The method according to claim 10, wherein, The neural network model is a neural network model based on a feedforward sequence memory network, which is used to model contextual information.
12. A representation device based on a self-attention mechanism, comprising: An acquisition module is used to acquire data to be processed, wherein the data to be processed is one of the following: voice data to be processed, text data to be processed, and image data to be processed; and to acquire a feature vector generated after feature extraction of the data vector corresponding to the data to be processed; A vector generation module is used to transform the feature vector through a filter to obtain triples for self-attention computation. The triples include a query vector, a key vector, and a value vector. The filter includes a first filter and a second filter. The number of parameters in the first filter or the second filter is at least one order of magnitude smaller than the number of dimensions in the query vector. Any two vectors in the triples are determined based on the first filter and the second filter, respectively. The other vector in the triples is determined based on the feature vector through an assignment operation. The representation generation module is used to perform self-attention calculation based on the query vector, the key vector, and the value vector to obtain a network representation corresponding to the feature vector. The network representation is determined based on the concatenation result, which is obtained by calculating the segmented query vector, the segmented key vector, and the segmented value vector, and concatenating the calculation results.
13. A voice recognition device, comprising: The speech feature vector acquisition module is used to acquire the speech feature vector generated after feature extraction of the data vector corresponding to the speech to be processed. A vector generation module is used to transform the speech feature vector through filters to obtain triples for self-attention computation. The triples include a query vector, a key vector, and a value vector. The filters include a first filter and a second filter. The number of parameters in the first filter or the second filter is at least one order of magnitude smaller than the number of dimensions in the query vector. Any two vectors in the triples are determined based on the first filter and the second filter, respectively. The other vector in the triples is determined based on the speech feature vector through an assignment operation. The representation generation module is used to perform self-attention calculation based on the query vector, the key vector, and the value vector to obtain a network representation corresponding to the speech feature vector. The network representation is determined based on the concatenation result, which is obtained by calculating the segmented query vector, the segmented key vector, and the segmented value vector, and concatenating the calculation results. The recognition module is used to recognize the speech to be processed based on the network representation.
14. An electronic device comprising: One or more processors; Computer-readable media, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the speech recognition method as described in any one of claims 1-5, or implement the representation method as described in any one of claims 6-11.
15. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the speech recognition method as described in any one of claims 1-5, or the representation method as described in any one of claims 6-11.