Audio recognition method, apparatus, and storage medium

Abstract

The application discloses an audio recognition method and device and a storage medium, relates to the technical field of audio recognition, and comprises the following steps: inputting target audio data into an audio recognition model; dividing the target audio data into n chunks by the audio recognition model, wherein n is an integer greater than 1; for each chunk in the n chunks, obtaining a historical state of the current chunk by the audio recognition model, and calculating the current chunk according to the historical state and the current chunk; the historical state is a state calculated and saved before an attention operation when a previous chunk of the current chunk is calculated; and outputting the target audio data after recognition by the audio recognition model according to the calculation results of the chunks. The problem of low audio recognition efficiency in the prior art is solved, the historical state is recorded, the current chunk is calculated according to the historical state and the current chunk, the calculation based on all data of the previous chunk is not needed, and the effect of improving the audio recognition efficiency is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to an audio recognition method, device and storage medium, and belongs to the technical field of audio recognition. BACKGROUND

[0002] Time series models are dependent on the order of event occurrence. The same size value changes the order and the result of inputting the model is different. The most commonly used time series network model in deep learning includes RNN (Recurrent neural network) and LSTM (Long Short-Term Memor) network.

[0003] In the model streaming inference application scenario, such as in the process of audio call, the chunk attention has the minimum performance loss and is most widely used. The chunk attention divides the input into multiple chunks according to a fixed chunk size. Each chunk depends on itself and the previous chunk, and the information of the previous chunk is needed when calculating the next chunk. Therefore, the previous chunk and the current chunk need to be input at the same time for calculation, and the calculation amount is large. The audio recognition efficiency is low in the existing scheme. SUMMARY

[0004] The purpose of the present application is to provide an audio recognition method, device and storage medium to solve the problems in the prior art.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] According to the first aspect, the present application provides an audio recognition method, which comprises:

[0007] Inputting target audio data into an audio recognition model;

[0008] Dividing the target audio data into n chunks by the audio recognition model, n being an integer greater than 1;

[0009] For each chunk of the n chunks, obtaining a history state of the current chunk by the audio recognition model, and calculating the current chunk according to the history state and the current chunk; the history state is a state calculated and saved before the attention operation when calculating the previous chunk of the current chunk;

[0010] Outputting the target audio data after recognition by the audio recognition model according to the calculation results of the chunks.

[0011] Optionally, the obtaining, by the audio recognition model, a history state of a current chunk and performing calculation on the current chunk according to the history state and the current chunk comprises:

[0012] obtaining an input state of the target audio data;

[0013] obtaining, by the audio recognition model, a history state of a current chunk and performing calculation on the current chunk according to the input state, the history state and the current chunk.

[0014] Optionally, the obtaining an input state of the target audio data comprises:

[0015] recognizing, by a speech recognition module, a first and last state of the target audio data;

[0016] marking, by a state model, the input state according to the recognized first and last state, wherein the input state comprises a start time, an intermediate time and an end time of the target audio data.

[0017] Optionally, the obtaining, by the audio recognition model, a history state of a current chunk and performing calculation on the current chunk according to the input state, the history state and the current chunk comprises:

[0018] if the input state is the start time, initializing the history state and performing calculation on the current chunk according to the current chunk by the audio recognition model;

[0019] updating a calculation result before an attention operation as the history state.

[0020] Optionally, the obtaining, by the audio recognition model, a history state of a current chunk and performing calculation on the current chunk according to the input state, the history state and the current chunk comprises:

[0021] if the input state is the intermediate time, performing calculation on the current chunk according to the history state and the current chunk by the audio recognition model;

[0022] updating a calculation result before an attention operation as the history state.

[0023] Optionally, the obtaining, by the audio recognition model, a history state of a current chunk and performing calculation on the current chunk according to the input state, the history state and the current chunk comprises:

[0024] If the input state is an ending moment, the current chunk is calculated by the audio recognition model according to the history state and the current chunk, without updating the history state.

[0025] Optionally, the input state is marked by the state model according to the identified first / last state, comprising:

[0026] If the first / last state is a starting state, the input state is: start = 1, end = 0.

[0027] If the first / last state is an intermediate state, the input state is: start = 0, end = 0.

[0028] If the first / last state is an ending state, the input state is: start = 0, end = 1.

[0029] In a second aspect, an audio recognition device is provided, comprising:

[0030] An input module is configured to input target audio data to an audio recognition model.

[0031] A segmentation module is configured to segment the target audio data into n chunks by the audio recognition model, n being an integer greater than 1.

[0032] A calculation module is configured to, for each of the n chunks, obtain a history state of a current chunk by the audio recognition model, and calculate the current chunk according to the history state and the current chunk; the history state is a state calculated and saved before an attention operation when a previous chunk of the current chunk is calculated.

[0033] An output module is configured to output the target audio data after recognition by the audio recognition model according to the calculation results of the respective chunks.

[0034] In a third aspect, an audio recognition device is provided, comprising a memory and a processor, the memory storing at least one program instruction, and the processor loading and executing the at least one program instruction to implement the method of the first aspect.

[0035] In a fourth aspect, a computer storage medium is provided, the computer storage medium storing at least one program instruction, and the at least one program instruction being loaded and executed by a processor to implement the method of the first aspect.

[0036] The target audio data is input into an audio recognition model; the target audio data is segmented into n chunks by the audio recognition model, n being an integer greater than 1; for each of the n chunks, a history state of the current chunk is obtained by the audio recognition model, and the current chunk is calculated according to the history state and the current chunk; the history state is a state calculated and saved before an attention operation when a previous chunk of the current chunk is calculated; the target audio data after recognition is output by the audio recognition model according to the calculation results of the respective chunks. The problem of low audio recognition efficiency in the prior art is solved, and the effect of improving the audio recognition efficiency is achieved by recording the history state and then calculating the current chunk according to the history state and the current chunk without calculating based on all data of the previous chunk.

[0037] Meanwhile, the input state of the audio data is marked by start and end to replace the real-time transmission state of the client and the server in the prior art, thereby reducing the data transmission between the client and the server and reducing the inference delay.

[0038] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application and to implement the content of the description, the following will be described in detail with the preferred embodiments of the present application and with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A method flowchart of an audio recognition method provided by an embodiment of the present application;

[0040] Figure 2 A possible schematic diagram of audio recognition model input and output sequences provided by an embodiment of the present application;

[0041] Figure 3 A possible network model schematic diagram of an audio recognition model provided by an embodiment of the present application;

[0042] Figure 4 A possible flowchart of input state represented by start and end provided by an embodiment of the present application;

[0043] Figure 5 A possible calculation flowchart of a chunk calculated according to a history state provided by an embodiment of the present application;

[0044] Figure 6 A possible calculation flowchart of a chunk calculated according to a history state and an input state provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0046] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0047] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0048] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0049] Please refer to Figure 1 , which shows the method flowchart of the audio recognition method provided by an embodiment of the present application, as Figure 1 shown, the method comprises:

[0050] Step 101, input the target audio data into the audio recognition model;

[0051] There are various timing models in the field of speech recognition. At present, the more common one is an end-to-end model based on the Transformer structure. The end-to-end model includes an encoder and a decoder. The encoder is responsible for encoding, and the decoder is responsible for decoding. The input is a sequence, and the output is another sequence. Both the input sequence and the output sequence contain timing sequences. The above end-to-end model is the audio recognition model described in the present application.

[0052] An application example of the end-to-end model is as follows Figure 2 As shown, it describes the working process of the end-to-end translation model. The end-to-end speech recognition model is similar, both including two modules: encode and decode. The input of the translation model is the English sequence "it is a cat", and the output is the corresponding Chinese sequence "这是一只猫". The difference of the speech recognition model is that the input of the encode module is replaced by the fbank features extracted from the audio of "这是一只猫". The encode module outputs the encoded vector of this audio, and then it is decoded by the decode module into the text recognition result corresponding to the audio.

[0053] The above only takes the combination of encoding and decoding to form an audio recognition model as an example. In actual implementation, encoding and decoding can be two independent models respectively. The two models are on the same server and share a GPU (graphics processing unit) graphics card. Of course, in actual implementation, according to the actual architecture deployment, the two models can also be deployed on different servers, and this is not limited.

[0054] In addition, please refer to Figure 3 , which shows a schematic diagram of a possible network model of the audio recognition model. It can be seen from Figure 3 that in the audio recognition model of this application, the Multi-Head Attention operation depends on context information, while other network structures do not depend on context information.

[0055] Step 102, the target audio data is segmented into n chunks by the audio recognition model, where n is an integer greater than 1;

[0056] Chunk attention divides the input into multiple chunks according to a fixed chunk size. For each chunk, it contains the input [t+1, t+2, …, T+C], where C is the size of the chunk size. Each chunk depends on itself and the previous chunks. Therefore, the delay in the encode model depends on the size of the chunk size.

[0057] In actual implementation, the number of n obtained by segmentation can be determined according to the chunk size and the length of the target audio data, and the chunk size can be the default size or a custom size, and this is not limited.

[0058] Step 103, for each of the n chunks, obtaining a history state of the current chunk by the audio recognition model, and calculating the current chunk according to the history state and the current chunk; the history state is a state calculated and saved before the attention operation when calculating the previous chunk of the current chunk;

[0059] Optionally, the present step comprises:

[0060] Firstly, obtaining an input state of the target audio data;

[0061] (1) identifying a first and last state of the target audio data by a speech recognition module;

[0062] Optionally, in the audio recognition process, the client sends start, feed or stop to the server to indicate the beginning and end of an audio. These data are obtained by the vad module (Voice activity detection, also known as speech activity detection or speech detection) before the audio recognition model detects the first and last state of the speech signal. The first and last state includes: the initial state, the intermediate state and the end state.

[0063] Specifically, when the vad module detects that the client sends start, it can be identified that the target audio data is the initial state; when it detects that the client sends feed, it can be identified that the target audio data is the intermediate state; when it detects that the client sends stop, it can be identified that the target audio data is the end state.

[0064] (2) marking the input state according to the identified first and last state by a state model, wherein the input state includes the initial time, the intermediate time and the end time of the target audio data.

[0065] The state model can be a stateful model, which can identify the input state of the audio data by start and end. Specifically:

[0066] If the first and last state is the initial state, the input state is: start = 1, end = 0;

[0067] If the first and last state is the intermediate state, the input state is: start = 0, end = 0;

[0068] If the first and last state is the end state, the input state is: start = 0, end = 1.

[0069] For example, refer to Figure 4 which shows a possible schematic diagram of representing input states by state models. As shown in Figure 4 , at the start input moment start = 1 and end = 0, at the intermediate moment start = 0 and end = 0, and at the end moment start = 0 and end = 1.

[0070] The present application marks the input state of the audio data by start and end, thereby replacing the real-time transmission of the state of the client and the server in the prior art, reducing the data transmission time between the client and the server and the data copy time between the CPU and the GPU, that is, reducing the delay of the client requesting the audio recognition service.

[0071] The present application only takes marking the input state by the state model as an example, and in actual implementation, the input state can also be obtained by real-time interaction with the client, and the present embodiment does not limit this.

[0072] Secondly, the historical state of the current chunk is obtained by the audio recognition model, and the current chunk is calculated according to the input state, the historical state and the current chunk.

[0073] Optionally, the present step can include the following steps:

[0074] If the input state is the start moment, the historical state is initialized, the current chunk is calculated by the audio recognition model according to the current chunk, and the calculation result before the attention operation is updated as the historical state.

[0075] When the input state is the start moment, it is indicated that the current chunk is the first chunk, at this time the historical state can be initialized as empty and calculated directly according to the current chunk. And after calculation, in order to time series prediction of subsequent audio, the calculation result before the attention operation is updated as the historical state. Wherein, except for special description, the attention operation described in the present application refers to all operations in the audio recognition model which need to rely on the previous chunk for calculation, and in actual implementation, different network models can be named differently, and the present application does not limit this.

[0076] If the input state is the intermediate moment, the current chunk is calculated by the audio recognition model according to the historical state and the current chunk, and the calculation result before the attention operation is updated as the historical state.

[0077] If the input state is the end time, the current chunk is calculated by the audio recognition model according to the history state and the current chunk, and the history state does not need to be updated.

[0078] If the input state is the end time, it means that the current chunk is the last chunk, the speech recognition is completed after the calculation, and therefore, the history state does not need to be updated after the calculation of the current chunk.

[0079] In a possible embodiment, the history state is illustrated by a state representation, please refer to Figure 5 , which shows a possible schematic diagram of the present application, as shown in Figure 5 , when each chunk is calculated, the output is calculated according to the current input chunk and the history state.

[0080] In another possible embodiment, each input state is represented by a state model based on Figure 5 , please refer to Figure 6 , at input1, that is, the start time, start = 1 and end = 0, at input2 and input3, that is, the middle time, start = 0 and end = 0, and at inputn, that is, the end time, start = 0 and end = 1, for each chunk, the output is calculated according to the state in which the current chunk is located and the history state.

[0081] In a possible embodiment, the encode in stream recognition is segmented inference, for example, when chunksize = 80, the input length of the audio recognition model is 80 frames of audio data, and the speech recognition extracts features every 10 ms, so 80 frames are 800 ms, and therefore the audio is input to the audio recognition model every 800 ms for calculation. After the 80 frames of audio features are input to the encode, they are first passed through a vgg network, and the two layers of maxpoll in the vgg reduce the feature length to 20 frames. When the output of the vgg enters the attention operation in the encode, attention calculation is performed on the current chunk and the previous 10 frames of data, and therefore the last 10 frames of results need to be retained for the next chunk calculation. Because the second time the next 80 frames of data are received, the last 80 frames of data calculation is completed, and these data need to be used as historical information together with the second input to be input into the encode for calculation, so the encode model needs to record the state of the last time and update the state after the calculation is completed. In the present application, the encode needs to rely on the last 10 frames of results of the previous chunk when calculating, and the network can also be fed with repeated data to see the previous information. In the existing scheme, the first time 1-80 frames of data are fed, and the second time 40-160 (so that in addition to the current 80-160, the last half of the data that needs to be relied on are also input, so that the historical information before the current input can be seen), but this will cause the model to increase the calculation amount, in addition to the attention calculation, the calculation of 80 frames of data will change to the calculation of 120 frames each time. In the present application, by using the state to record the data required by the attention part, the calculation amount can be reduced by one third, and the efficiency of speech recognition is improved.

[0082] In addition, in the present application, the decode model is the encoding of a segment of audio output by each chunk, and in addition to the encoding result of the encode, a ctc header is also output. According to the output result of the ctc, it can be predicted that there are several characters in this segment of audio. Because the decode network contains an lstm structure, and the decoding process needs to be calculated multiple times, a character is output every time the decode is performed, and when a new character is obtained, the current sequence is also used to predict the next sequence. Therefore, the decode model is also a stateful network model.

[0083] In addition, in the present application, the decode model is the encoding of a segment of audio output by each chunk, and in addition to the encoding result of the encode, a ctc header is also output. According to the output result of the ctc, it can be predicted that there are several characters in this segment of audio. Because the decode network contains an lstm structure, and the decoding process needs to be calculated multiple times, a character is output every time the decode is performed, and when a new character is obtained, the current sequence is also used to predict the next sequence. Therefore, the decode model is also a stateful network model. Figure 3The decoding end in the application adopts two layers of LSTM and one layer of attention (other structures are also possible, and multi-layer attention without using LSTM is also possible). The difference between the multi-layer attention structure is that the state of each layer is recorded separately and then spliced into a large state. For example, if a layer of attention needs to record 10 frames of state and the feature dimension is 512, then 10*512 = 5120 data need to be recorded for one layer. If there are 10 such structures, 5120*10 = 51200 data are recorded as hidden layer states. When used, 5120 is allocated to each layer, and the results of each layer are summarized and returned after calculation.

[0084] The above only takes the audio recognition model as an example of a model based on the Transformer structure. Alternatively, the model based on LSTM (Long Short-Term Memory), GRU, DFSMN, etc. can also be used for the scene of streaming calculation through a request service, and the specific application scenario of the application is not limited.

[0085] Step 104: outputting the target audio data after recognition according to the calculation results of the chunks by the audio recognition model.

[0086] In summary, by inputting the target audio data into the audio recognition model; the target audio data is divided into n chunks by the audio recognition model, n being an integer greater than 1; for each chunk of the n chunks, the historical state of the current chunk is obtained by the audio recognition model, and the current chunk is calculated according to the historical state and the current chunk; the historical state is the state calculated and saved before the attention operation when calculating the previous chunk of the current chunk; the target audio data after recognition is output according to the calculation results of the chunks by the audio recognition model. The problem of low audio recognition efficiency in the prior art is solved, and the effect of improving the audio recognition efficiency by recording the historical state and then calculating the current chunk based on the historical state and the current chunk without calculating based on all the data of the previous chunk is achieved.

[0087] At the same time, the application marks the input state of the audio data by start and end, thereby replacing the real-time transmission of the state of the client and the server in the prior art, reducing the data transmission between the client and the server and reducing the inference delay.

[0088] The application also provides an audio recognition device, which comprises:

[0089] The input module is configured to input target audio data into the audio recognition model.

[0090] The segmentation module is configured to segment the target audio data into n chunks by using the audio recognition model, where n is an integer greater than 1.

[0091] The calculation module is configured to, for each of the n chunks, obtain a history state of a current chunk by using the audio recognition model, and perform calculation on the current chunk according to the history state and the current chunk; the history state is a state calculated and saved before an attention operation when a previous chunk of the current chunk is calculated.

[0092] The output module is configured to output the target audio data after recognition by using the audio recognition model according to the calculation results of the respective chunks.

[0093] The present application also provides an audio recognition device, which comprises a memory and a processor, the memory stores at least one program instruction, and the processor loads and executes the at least one program instruction to realize the method described above.

[0094] The present application also provides a computer storage medium, which stores at least one program instruction, and the at least one program instruction is loaded and executed by a processor to realize the method described above.

[0095] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present disclosure.

[0096] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of variations and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An audio recognition method, characterized by, The method comprises: inputting target audio data into an audio recognition model; segmenting the target audio data into n chunks by the audio recognition model, n being an integer greater than 1; for each of the n chunks, obtaining a history state of the current chunk by the audio recognition model, and calculating the current chunk according to the history state and the current chunk; the history state is a state calculated and saved before an attention operation when calculating a previous chunk of the current chunk, and the audio recognition model is a model based on a Transformer structure; outputting the target audio data after recognition by the audio recognition model according to the calculation results of the respective chunks.

2. The method of claim 1, wherein, The method comprises: obtaining an input state of the target audio data; obtaining a history state of the current chunk by the audio recognition model, and calculating the current chunk according to the input state, the history state and the current chunk.

3. The method of claim 2, wherein, The method comprises: recognizing a first and last state of the target audio data by a speech recognition module; marking the input state according to the recognized first and last state by a state model, wherein the input state comprises a start time, an intermediate time and an end time of the target audio data.

4. The method of claim 3, wherein, The method comprises: if the input state is the start time, initializing the history state, and calculating the current chunk according to the current chunk by the audio recognition model; updating the calculation result before the attention operation as the history state.

5. The method of claim 3, wherein, The method comprises: if the input state is the intermediate time, calculating the current chunk according to the history state and the current chunk by the audio recognition model; updating the calculation result before the attention operation as the history state.

6. The method of claim 3, wherein, The method comprises: if the input state is the end time, calculating the current chunk according to the history state and the current chunk by the audio recognition model, and not updating the history state.

7. The method of claim 3, wherein, The method comprises: If the first and last states are start states, the input states are: start = 1, end = 0; If the first and last states are intermediate states, the input states are: start = 0, end = 0; If the first and last states are end states, the input states are: start = 0, end = 1.

8. An audio recognition apparatus, characterized by comprising: The device comprises: an input module configured to input target audio data to an audio recognition model; a segmentation module configured to segment the target audio data into n chunks by using the audio recognition model, n being an integer greater than 1; a calculation module configured to, for each of the n chunks, obtain a historical state of the current chunk by using the audio recognition model, and calculate the current chunk according to the historical state and the current chunk; the historical state being a state calculated and saved before an attention operation when a previous chunk of the current chunk is calculated, and the audio recognition model being a model based on a Transformer structure; an output module configured to output the target audio data after recognition according to calculation results of the chunks by using the audio recognition model.

9. An audio recognition apparatus, characterized by comprising: The device comprises a memory and a processor, the memory storing at least one program instruction, and the processor loads and executes the at least one program instruction to implement the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores at least one program instruction, and the at least one program instruction is loaded and executed by a processor to implement the method according to any one of claims 1 to 7.

Images