Audio recognition method, system and related device

Abstract

The present application discloses an audio recognition method, system and related device, the method comprising: obtaining audio to be recognized; inputting the audio to be recognized into a target recognition model to obtain a target recognition text that matches the audio to be recognized; wherein the target recognition model is obtained based on a trained initial recognition model, the initial recognition model comprises a parameter-sharing audio parsing network and a text parsing network, the audio parsing network is trained using a plurality of training audios, the text parsing network is trained using a plurality of training texts, the training audios are matched with text labels, and the training texts are matched with category labels. In the above manner, the present application can improve the accuracy of audio recognition.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio recognition method, system and related devices. Background Art

[0002] Audio recognition technology is an important branch of modern computer science. It enables computers to receive, process and understand human spoken commands and has been widely used in many fields, including intelligent assistants, automatic customer service, voice input, etc. Existing audio recognition technology mainly trains neural network models through a large number of training samples, so that the trained neural network models have the ability to recognize audio. However, since the neural network model often only learns information related to speech during the training process and ignores text information in different fields, there are certain limitations in recognition accuracy. For example, existing audio recognition technology easily misidentifies "lingering sound lingering in the beam" as "voice lingering in the beam".

[0003] In view of this, how to propose an audio recognition method with higher recognition accuracy has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an audio recognition method, system and related devices, which can improve the accuracy of audio recognition.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide an audio recognition method, comprising: obtaining audio to be recognized; inputting the audio to be recognized into a target recognition model to obtain a target recognition text matching the audio to be recognized; wherein the target recognition model is obtained based on a trained initial recognition model, the initial recognition model includes a parameter-sharing audio parsing network and a text parsing network, the audio parsing network is trained using multiple training audios, the text parsing network is trained using multiple training texts, the training audios are matched with text labels, the training texts are matched with category labels, and the audio parsing network in the trained initial recognition model serves as the target recognition model.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an audio recognition system, comprising: an acquisition module, used to acquire audio to be recognized; a recognition module, used to input the audio to be recognized into a target recognition model, and obtain a target recognition text matching the audio to be recognized; wherein the target recognition model is obtained based on a trained initial recognition model, the initial recognition model includes an audio parsing network and a text parsing network that share at least some parameters, the audio parsing network is trained using multiple training audios, the text parsing network is trained using multiple training texts, the training audios are matched with text labels, the training texts are matched with category labels, and the audio parsing network in the trained initial recognition model serves as the target recognition model.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio recognition method mentioned in the above technical solution.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which a program quality is stored, and when the program quality is executed by a processor, the audio recognition method mentioned in the above technical solution is implemented.

[0009] The beneficial effect of the present application is as follows: Different from the prior art, in the audio recognition method proposed in the present application, the initial recognition model includes an audio parsing network and a text parsing network, and at least part of the parameters in the audio parsing network and the text parsing network are shared. In the process of training the initial recognition model, multiple training audios adjust the parameters in the audio parsing network and share at least part of the adjusted parameters to the text parsing network, and multiple training texts adjust the parameters in the text parsing network and share at least part of the adjusted parameters to the audio parsing network. In response to the training text matching, there are corresponding category labels, so that the text parsing network and the audio parsing network learn text corpora under different field categories during the training process. Therefore, the target recognition model obtained according to the trained initial recognition model has a high accuracy when recognizing audio to be recognized in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0011] Figure 1 It is a flowchart of an implementation method of the audio recognition method of the present application;

[0012] Figure 2 It is a flowchart of an implementation method of the initial recognition model training process of the present application;

[0013] Figure 3 yes Figure 2 Step S202 corresponds to a flow chart of another implementation method;

[0014] Figure 4 It is a structural schematic diagram of an implementation method of the initial recognition model of the present application;

[0015] Figure 5 yes Figure 2 Step S202 corresponds to a flowchart of another implementation method;

[0016] Figure 6 It is a structural diagram of an implementation method of a category prediction model of the present application;

[0017] Figure 7 It is a structural diagram of an implementation method of an audio recognition system of the present application;

[0018] Figure 8 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0019] Fig. 9 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an audio recognition method of the present application, the method comprising:

[0022] S101: Acquire audio to be recognized.

[0023] In one implementation, audio to be recognized that needs to be recognized into text is obtained, and the audio to be recognized may be acquired in real time using an audio acquisition device.

[0024] In another embodiment, the audio to be recognized may also be obtained in other ways, for example, audio downloaded from the network is used as the audio to be recognized; or audio received through the network or other means is used as the audio to be recognized.

[0025] S102: Input the audio to be recognized into the target recognition model to obtain the target recognition text matching the audio to be recognized; wherein, the target recognition model is obtained based on the trained initial recognition model, the initial recognition model includes an audio parsing network and a text parsing network that share at least some parameters, the audio parsing network is trained using multiple training audios, the text parsing network is trained using multiple training texts, the training audios are matched with text labels, the training texts are matched with category labels, and the audio parsing network in the trained initial recognition model serves as the target recognition model.

[0026] In one embodiment, the obtained audio to be recognized is input into a target recognition model, so that the target recognition model analyzes and recognizes the audio to be recognized and outputs corresponding target recognition text. The target recognition model is obtained based on the trained initial recognition model.

[0027] In one implementation scenario, in response to the trained initial recognition model including an audio parsing network and a text parsing network, and at least some parameters of the audio parsing network and the text parsing network are shared, the audio parsing network is used as the target recognition model. In the process of training the initial recognition model, the audio parsing network is trained using a plurality of training audios, the training audios are matched with text labels, and the text parsing network is trained using a plurality of training texts, the training texts are matched with category labels.

[0028] It should be noted that the training text can also be a general corpus, that is, the training text can have no clear domain category, and the matching category label is replaced by a default value, such as "0". In addition, the same training text can match one or more category labels. For example, when the training text is "In this class, we will learn the calculation formula of L'Hôpital's rule", its matching category labels can include "education" and "mathematics".

[0029] In another embodiment, after obtaining the trained initial recognition model, the text parsing network in the initial recognition model is shielded to obtain a target recognition model, that is, the initial recognition model after shielding only supports the recognition of the input audio to be recognized through the audio parsing network to obtain the corresponding target recognition text.

[0030] In the audio recognition method proposed in the present application, the initial recognition model includes an audio parsing network and a text parsing network, and at least part of the parameters in the audio parsing network and the text parsing network are shared. In the process of training the initial recognition model, multiple training audios adjust the parameters in the audio parsing network and share at least part of the adjusted parameters to the text parsing network, and multiple training texts adjust the parameters in the text parsing network and share at least part of the adjusted parameters to the audio parsing network. In response to the training text matching, there are corresponding category labels so that the text parsing network and the audio parsing network learn text corpora under different field categories during the training process. Therefore, the target recognition model obtained according to the trained initial recognition model has a high accuracy when recognizing audio to be recognized in different fields.

[0031] See also Figure 2 , Figure 2 1 is a flow chart of an implementation method of the initial recognition model training process of the present application. Specifically, the training process of the initial recognition model includes:

[0032] S201: Acquire a first training set and a second training set for a current training cycle; wherein the first training set includes a plurality of training audios, and the second training set includes a plurality of training texts.

[0033] In one embodiment, a plurality of training audios and a plurality of training texts are obtained.

[0034] In one implementation scenario, each training audio is an audio sequence composed of initial audio features corresponding to a plurality of audio frames, and a text label matched by the training audio is a text sequence composed of a plurality of characters.

[0035] Specifically, the Fbank feature extraction is performed on the audio modality data used for training to obtain the initial audio features corresponding to each frame of audio modality data, and the audio sequence composed of the initial audio features is used as the training audio. For example, the training audio is represented as ,in represents the initial audio features corresponding to the audio modality data of the mth frame; the text label matched by the training audio is expressed as ,in represents the tth character, and t is the total number of characters in the text corresponding to the training audio. The training text is a text sequence consisting of multiple characters obtained after word segmentation, and a start symbol is set at the starting position of the text sequence and a terminator is set at the ending position.

[0036] Optionally, in other implementation scenarios, for the audio modality data used for training, MFCC (Mel-scale Frequency Cepstral Coefficients) or LPC (Linear Predictive Coding) features may be extracted frame by frame, and the obtained audio sequence may be used as training data.

[0037] Further, the plurality of training audios are divided into a plurality of first training sets, and the plurality of training texts are divided into a plurality of second training sets, each of the first training sets includes a plurality of training audios, and each of the second training sets includes a plurality of training texts. The first training set and the second training set for the current training cycle are selected from the plurality of first training sets and the plurality of second training sets.

[0038] In one implementation scenario, in response to the fact that the cost of acquiring training text is much higher than the cost of acquiring training audio, and the training resources consumed by the training text during the training process are much smaller than the training resources consumed by the training audio, in order to balance the training resources, the second training set also includes multiple training subsets, each training subset includes a preset number of training texts.

[0039] In a specific application scenario, the second training set can be expressed as , .in, represents the number of training subsets in each second training set, Represents the training subset The cth training text in, c represents the training subset The total number of training texts in .

[0040] In another embodiment, in the above embodiment, the training text in the second training set is divided into multiple training subsets to facilitate the subsequent calculation of the model loss. In other embodiments, it is also necessary not to first construct multiple training subsets and then obtain the second training set based on the training subsets, that is, the number of training texts included in the second training set is greater than the number of training audios in the first training set.

[0041] S202: Alternately train the audio parsing network and the text parsing network using the first training set and the second training set until a preset stop condition is met, thereby obtaining a trained initial recognition model.

[0042] In one embodiment, the first training set is used to adjust the parameters in the audio parsing network, and at least part of the adjusted parameters are shared with the text parsing network to obtain an updated text parsing network. The second training set is used to adjust the parameters in the updated text parsing network, and at least part of the adjusted parameters are shared with the audio parsing network to obtain an updated audio parsing network. The audio parsing network is provided with an audio mask, and the text parsing network is provided with a text mask, and the text mask is determined based on the text length of the training text in the second training set.

[0043] Specifically, in response to the audio parsing network and the text parsing network including parts with the same structure, during the alternating training process, the parameters corresponding to the parts with the same structure in the audio parsing network and the text parsing network are shared, so that the audio parsing network and the text parsing network can learn from each other. The specific process of parameter sharing is parameter replacement, and the specific implementation process is described in the subsequent implementation method.

[0044] Further, after completing the training of the current training cycle, continue to obtain the first training set and the second training set for the next training cycle, and perform training for the next training cycle. In response to the initial recognition model satisfying the preset convergence condition, or the number of training cycles reaches a preset value, stop training, and obtain the trained initial recognition model. Among them, the specific training process of the next training cycle can refer to the training process corresponding to the current training cycle, and will not be elaborated here.

[0045] In another embodiment, when training in the current training cycle is performed, the second training set may be used to adjust at least some of the parameters in the text parsing network, and the adjusted at least some of the parameters are shared with the audio parsing network to obtain an updated audio parsing network. Then, the first training set is used to adjust the parameters in the audio parsing network, and the adjusted at least some of the parameters are shared with the text parsing network to obtain an updated text parsing network.

[0046] See also Figure 3 and Figure 4 , Figure 3 yes Figure 2 Step S202 in the figure corresponds to a flow chart of another embodiment. Figure 4 1 is a schematic diagram of the structure of an implementation of the initial recognition model of the present application. Specifically, in response to the audio parsing network 10 including an audio encoding subnetwork and an audio decoding subnetwork, in step S202, the parameters in the audio parsing network are adjusted using the first training set, and at least part of the adjusted parameters are shared with the text parsing network. The process of obtaining the updated text parsing network includes:

[0047] S301: Input the training audio in the first training set into the audio coding subnetwork to obtain an audio coding vector corresponding to the training audio.

[0048] In one embodiment, the training audio in the first training set is input into the audio encoding sub-network in the audio parsing network 10, so that the audio encoding sub-network outputs an audio encoding vector corresponding to the training audio.

[0049] Specifically, Figure 4 As shown, the training audio is input into the convolution layer in the audio coding subnetwork, so that the convolution layer performs feature extraction and downsampling processing on the training audio to obtain reference features corresponding to the training audio. After the reference features are layer-normalized, they are sequentially input into the multi-head attention layer and the forward feedback layer, and after repeated iterations for a preset number of times, a deeper audio coding vector is obtained. Among them, the multi-head attention layer is the Multi-Head Attention layer, the forward feedback layer is the Feed Forward layer, and residual connections and layer normalization operations are also provided between the Multi-Head Attention layer and the Feed Forward layer, and a layer normalization operation is also provided after the Feed Forward layer; that is, the specific structure and implementation principle of the audio coding subnetwork can refer to the specific structure and implementation principle of the encoder in the transformer structure, and will not be elaborated here.

[0050] S302: Input the audio encoding vector into the audio decoding subnetwork to obtain the corresponding first recognition text.

[0051] In one embodiment, the obtained audio encoding vector is input into the audio decoding subnetwork, so that the audio decoding subnetwork sequentially outputs the decoded characters corresponding to each decoding round until all the decoded characters corresponding to the training audio are obtained, and all the corresponding decoded characters are concatenated in the decoding order to obtain the first recognized text.

[0052] Specifically, the audio decoding subnetwork includes a word embedding layer, a self-attention subnetwork, a cross-attention subnetwork, a feedforward subnetwork, and a text classification subnetwork. The self-attention subnetwork includes layer normalization and a masked multi-head attention layer, namely, a layer Norm layer and a Masked Multi-Head Attention layer; the cross-attention subnetwork includes layer normalization and a masked multi-head cross-attention layer, namely, a layer Norm layer and a Masked Multi-Head Cross-Attention layer; the feedforward subnetwork includes layer normalization and a forward feedback layer, namely, a layer Norm layer and a Feed Forward layer; and the text classification subnetwork includes layer normalization and a linear layer, namely, a layer Norm layer and a Linear layer. The decoded characters that have been recognized are input into the word embedding layer for semantic encoding and position encoding, and then input into the self-attention subnetwork. The audio encoding vector corresponding to the training audio output by the audio encoding subnetwork is input into the cross-attention subnetwork as the key and value, and finally the decoded characters corresponding to each decoding round output by the text classification subnetwork are obtained. Among them, the specific structure and implementation principle of the audio decoding subnetwork in the audio analysis network 10 can refer to the structure and implementation principle of the decoder in the existing Transformer model, and will not be elaborated in detail here.

[0053] In addition, audio masks are applied in the Multi-Head Attention layer in the audio encoding subnetwork, the self-attention subnetwork in the audio decoding subnetwork, and the cross-attention subnetwork in the decoding subnetwork. The audio mask is used to make the network focus on the information before the current decoding round and shield the information after the current decoding round, so as to optimize the real-time recognition effect of the audio parsing network 10.

[0054] S303: Obtaining audio training loss based on the text label matched between the first recognized text and the training audio.

[0055] In one embodiment, the corresponding audio training loss is calculated based on the first recognized text and text label corresponding to the training audio. Specifically, the specific calculation formula of the audio training loss is as follows:

[0056]

[0057] in, represents the audio training loss, represents the number of training audios in the first training set, and represents the trainable parameters of the text classification subnetwork in the audio decoding subnetwork, represents the bias term of the text classification subnetwork in the audio decoding subnetwork, Indicates Training audios, Indicates training audio The matching text label.

[0058] S304: Using the audio training loss to adjust at least part of the parameters in the audio parsing network, and sharing at least part of the parameters in the adjusted audio decoding subnetwork to the text parsing network to obtain an updated text parsing network.

[0059] In one embodiment, the obtained audio training loss is used to adjust at least part of the parameters in the audio parsing network 10.

[0060] Further, after adjusting the parameters in the audio parsing network 10, the parameters in the self-attention sub-network in the audio parsing network 10 are shared with the self-attention sub-network in the text parsing network, that is, the parameters in the self-attention sub-network in the text parsing network are replaced by the parameters in the self-attention sub-network in the audio parsing network 10; the parameters in the feedforward sub-network in the audio parsing network 10 are shared with the feedforward sub-network in the text parsing network, that is, the parameters in the feedforward sub-network in the text parsing network are replaced by the parameters in the feedforward sub-network in the audio parsing network 10; and the parameters in the text classification sub-network in the audio parsing network 10 are shared with the text classification sub-network in the text parsing network, that is, the parameters in the text classification sub-network in the text parsing network are replaced by the parameters in the text classification sub-network in the text parsing network. After completing parameter sharing, an updated text parsing network is obtained.

[0061] Please combine Figure 4 See also Figure 5 , Figure 5 yes Figure 2 Step S202 in the figure corresponds to a flowchart of another embodiment. Specifically, the text parsing network 20 includes a text encoding subnetwork, a text classification subnetwork, and a pre-trained category classification subnetwork. In step S202, the parameters in the updated text parsing network 20 are adjusted using the second training set, and at least part of the adjusted parameters are shared with the audio parsing network. The process of obtaining the updated audio parsing network includes:

[0062] S401: Input the training text in the second training set into the text parsing network after parameter sharing, and obtain the second recognition text corresponding to the training text output by the text classification subnetwork, and the predicted category corresponding to the training text output by the category classification subnetwork.

[0063] In one embodiment, the training text in the second training set is input into the text parsing network 20 after parameter sharing, and the text encoding subnetwork in the text parsing network 20 first performs semantic encoding and position encoding on the training text, and the obtained encoding vector is sequentially passed through the self-attention subnetwork and the feedforward subnetwork structure in the text parsing network 20, and finally the second recognition text output by the text classification subnetwork in the text parsing network 20 and the predicted category output by the category classification subnetwork in the text parsing network 20 are obtained.

[0064] Specifically, the self-attention subnetwork in the text parsing network 20 includes layer normalization and masked multi-head attention layers, namely, layer Norm layers and Masked Multi-Head Attention layers; the feedforward subnetwork in the text parsing network 20 includes layer normalization and forward feedback layers, namely, layer Norm layers and Feed Forward layers. In order to facilitate the subsequent sharing of parameters, the self-attention subnetwork structure in the text parsing network 20 is the same as the self-attention subnetwork structure in the audio decoding subnetwork, and the feedforward subnetwork structure in the text parsing network 20 is the same as the feedforward subnetwork structure in the audio decoding subnetwork. Since the text parsing network 20 does not need to analyze the data of the audio modality, compared with the audio decoding subnetwork, the cross-attention subnetwork is not set in the text parsing network 20. In addition, the text classification subnetwork structure in the text parsing network 20 is the same as the text classification subnetwork structure in the audio decoding subnetwork, and the category classification subnetwork in the text parsing network 20 includes layer normalization and linear layers, namely, layer Norm layers and Linear layers.

[0065] In another embodiment, in order to improve the classification effect of the category classification subnetwork in the text parsing network 20, the category classification subnetwork includes a plurality of stacked network structures, each of which includes a layer Norm layer and a Linear layer.

[0066] S402: Obtain text training loss based on the second recognized text, the predicted category, and the category label matched by the training text.

[0067] In one embodiment, in response to the second training set including multiple training subsets, each training subset including multiple training texts, a first loss is calculated based on the training texts and the corresponding second recognition texts in each training subset; and a second loss is calculated based on the predicted category and category label corresponding to the training text. Based on the first loss and the second loss, a reference loss of the corresponding training subset is obtained. Based on the reference loss corresponding to each training subset in the second training set, a text training loss is obtained.

[0068] Specifically, the calculation formula for the first loss corresponding to each training subset is as follows:

[0069]

[0070] in, Represents the number of training texts in the corresponding training subset, and represents the trainable parameters of the text classification subnetwork in the text parsing network 20, represents the bias term of the text classification subnetwork in the text parsing network 20, Indicates training text, Represents training text The actual text that was matched.

[0071] Furthermore, in response to the fact that each training text in the training subset may match multiple category labels, a multi-label loss function is used to calculate the second loss corresponding to each training subset. The specific calculation formula is as follows:

[0072]

[0073] in, Represents the number of training texts in the corresponding training subset, Indicates The number of class labels that match the training texts, Indicates The training text corresponds to The class labels of the categories, Indicates The training text corresponds to The predicted category of each category.

[0074] And, after determining the first loss and the second loss corresponding to each training subset, the first loss and the second loss are weightedly summed to obtain the text training loss of the second training set corresponding to the current training cycle. The specific calculation formula is as follows:

[0075]

[0076]

[0077] Among them, by calculating the training subset The corresponding second loss With hyperparameters and multiply the product with the training subset The corresponding first loss The sum is used as the training subset The corresponding reference loss ; represents the number of training subsets in the second training set, represents the text training loss of the second training set.

[0078] S403: Based on the text training loss, at least part of the parameters in the text parsing network are adjusted, and at least part of the adjusted parameters are shared with the audio parsing network to obtain an updated audio parsing network.

[0079] In one embodiment, in response to the fact that the category classification subnetwork is pre-trained, in order to reduce the difficulty of training the initial recognition model, the parameters of the category classification subnetwork in the text parsing network 20 are first fixed during the training process until the number of training cycles reaches a preset cycle threshold.

[0080] In one implementation scenario, in response to the current training cycle failing to reach a preset cycle threshold, at least some parameters in the text encoding subnetwork and the text classification subnetwork in the text parsing network 20 are adjusted using the text training loss, and at least some of the adjusted parameters are shared with the audio parsing network to obtain an updated audio parsing network.

[0081] Specifically, the obtained text training loss is used to adjust at least part of the parameters in the text parsing network 20. After adjusting the parameters in the text parsing network 20, the parameters in the self-attention subnetwork in the text parsing network 20 are shared with the self-attention subnetwork in the audio parsing network, that is, the parameters in the self-attention subnetwork in the audio parsing network are replaced with the parameters in the self-attention subnetwork in the text parsing network 20; the parameters in the feedforward subnetwork in the text parsing network 20 are shared with the feedforward subnetwork in the audio parsing network, that is, the parameters in the feedforward subnetwork in the audio parsing network are replaced with the parameters in the feedforward subnetwork in the text parsing network 20; and, the parameters in the text classification subnetwork in the text parsing network 20 are shared with the text classification subnetwork in the audio parsing network, that is, the parameters in the text classification subnetwork in the audio parsing network are replaced with the parameters in the text classification subnetwork in the text parsing network 20.

[0082] Alternatively, in response to the current training cycle reaching a preset cycle threshold, the category classification subnetwork in the text parsing network 20 is added to the training, that is, the text training loss is used to adjust at least part of the parameters in the text encoding subnetwork, text classification subnetwork and category classification subnetwork in the text parsing network 20, and at least part of the adjusted parameters are shared to the audio parsing network to obtain an updated audio parsing network.

[0083] Specifically, after adjusting the parameters in the text parsing network 20, the parameters in the self-attention subnetwork, feedforward subnetwork and text classification subnetwork in the text parsing network 20 are shared to the corresponding subnetworks in the audio parsing network. The specific sharing process can refer to the corresponding implementation method mentioned above and will not be elaborated here.

[0084] Also, see Figure 6 , Figure 6 Schematic diagram of the structure of an implementation of the category prediction model of the present application. The process of pre-training the category classification subnetwork includes: pre-constructing Figure 6 The category prediction model shown in the figure includes a word embedding layer, a self-attention module, a feedforward module and a category classification module. The self-attention module includes a layer normalization and a masked multi-head attention layer, namely, a layer Norm layer and a Masked Multi-Head Attention layer; the feedforward module includes a layer normalization and a forward feedback layer, namely, a layer Norm layer and a Feed Forward layer. By obtaining multiple training samples, the loss value of the category prediction model is calculated using the multiple training samples, so that the parameters in the category prediction model are adjusted using the loss value. In response to the number of training times reaching the number threshold or reaching the preset convergence condition, the training is stopped and the trained category prediction model is obtained. The category classification module in the trained category prediction model is used as the pre-trained category classification subnetwork in the above-mentioned text analysis model. By obtaining the pre-trained category classification subnetwork, it is avoided to take into account multi-task training at the same time when training the initial recognition model, resulting in greater training difficulty. Among them, during the training process of the category prediction model, the loss value is calculated by a multi-label loss function, and the specific details can be referred to the above-mentioned corresponding implementation method.

[0085] In the above scheme, in the process of training the text parsing network 20 with the training text, the text parsing network 20 learns the training texts in different fields. Furthermore, by sharing the relevant parameters in the text parsing network 20 with the audio parsing network, the updated audio parsing network has the function of performing text recognition on the audio modality data, and further improves the recognition accuracy.

[0086] In another embodiment, in response to the second training set including multiple training subsets, each training subset including multiple training texts, step S201, after acquiring the first training set and the second training set for the current training cycle, further includes: determining the maximum length from the text lengths of each training text in the training subset, and determining a text field threshold matching the training subset based on the maximum length. Based on the text field threshold, acquiring a text mask matching the corresponding training subset.

[0087] Specifically, the corresponding text length is determined according to the number of characters contained in each training text in the training subset, and the maximum length is determined from the text lengths corresponding to all training texts in the training subset. The maximum length is compared with a preset length threshold. In response to the maximum length being greater than the preset length threshold, the maximum length is used as the text field of view threshold. Alternatively, in response to the maximum length being less than or equal to the preset length threshold, the text field of view threshold is determined based on the maximum length, and the text field of view threshold increases with the increase of the maximum length. Then, a text mask is generated according to the text field of view threshold. By generating the adaptively adjustable text mask, it is possible to avoid the text information that the text parsing network 20 pays attention to being too rich during the training process of the text parsing network 20, thereby increasing the difficulty of training. Among them, the specific calculation formula for the above-mentioned text field of view threshold is as follows:

[0088]

[0089] in, Indicates the text field of view threshold, Indicates the maximum length. Indicates the preset length threshold.

[0090] In another embodiment, after the text mask is generated, the parameters in the text parsing network 20 are adjusted in the above corresponding embodiments, and at least part of the adjusted parameters are shared with the audio parsing network, and the process of obtaining the updated audio parsing network includes: for each training subset in the second training set, the training text is input into the text parsing network 20 after parameter sharing, and based on the training text and the matching text mask, the second recognition text and predicted category corresponding to the training text are obtained.

[0091] Specifically, for the training subset in the second training set, the matching text mask is added to the Masked Multi-Head Attention layer in the text parsing network 20, and the training text in the training subset is input into the text parsing network 20, so as to finally obtain the second recognition text and predicted category corresponding to the training text. The specific acquisition process of the second recognition text and the predicted category can be combined with the above corresponding implementation methods, and will not be elaborated in detail here.

[0092] Furthermore, based on the second recognition text and the predicted category corresponding to the training text, at least some parameters in the text parsing network 20 are adjusted, and at least some parameters in the adjusted text parsing network 20 are shared with the audio parsing network to obtain an updated audio parsing network.

[0093] Specifically, after obtaining the second recognition text and prediction category corresponding to each training text in the training subset, the reference loss corresponding to the training subset is calculated, and the text training loss is obtained according to the reference loss corresponding to each training subset in the second training set. The text training loss is used to adjust at least part of the parameters in the text parsing network 20, and at least part of the parameters in the adjusted text parsing network 20 are shared with the audio parsing network to obtain an updated audio parsing network. The specific implementation process can refer to the corresponding implementation method above.

[0094] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of an audio recognition system of the present application corresponding to an embodiment. Specifically, the audio recognition system includes an acquisition module 30 and a recognition module 40 coupled to each other.

[0095] Specifically, the acquisition module 30 is used to acquire the audio to be recognized.

[0096] The recognition module 40 is used to input the audio to be recognized into the target recognition model to obtain the target recognition text matching the audio to be recognized; wherein, the target recognition model is obtained based on the trained initial recognition model, the initial recognition model includes an audio parsing network and a text parsing network that share at least some parameters, the audio parsing network is trained using multiple training audios, the text parsing network is trained using multiple training texts, the training audios are matched with text labels, the training texts are matched with category labels, and the audio parsing network in the trained initial recognition model serves as the target recognition model.

[0097] In one embodiment, please continue to refer to Figure 7 The audio recognition system proposed in the present application also includes a training module 50 coupled to the recognition module 40. The training process of the initial recognition model by the training module 50 includes: obtaining a first training set and a second training set for the current training cycle; wherein the first training set includes a plurality of training audios, and the second training set includes a plurality of training texts; and alternately training the audio parsing network and the text parsing network using the first training set and the second training set until a preset stop condition is met, thereby obtaining a trained initial recognition model.

[0098] In one embodiment, the training module 50 uses the first training set and the second training set to alternately train the audio parsing network and the text parsing network, including: using the first training set to adjust the parameters in the audio parsing network, and sharing at least part of the adjusted parameters to the text parsing network to obtain an updated text parsing network; using the second training set to adjust the parameters in the updated text parsing network, and sharing at least part of the adjusted parameters to the audio parsing network to obtain an updated audio parsing network.

[0099] In one embodiment, the audio parsing network includes an audio encoding subnetwork and an audio decoding subnetwork. The training module 50 uses the first training set to adjust the parameters in the audio parsing network, and shares at least part of the adjusted parameters to the text parsing network to obtain an updated text parsing network, including: inputting the training audio in the first training set into the audio encoding subnetwork to obtain an audio encoding vector corresponding to the training audio; inputting the audio encoding vector into the audio decoding subnetwork to obtain the corresponding first recognition text; obtaining the audio training loss based on the text label matched between the first recognition text and the training audio; using the audio training loss to adjust at least part of the parameters in the audio parsing network, and sharing at least part of the adjusted parameters in the audio decoding subnetwork to the text parsing network to obtain an updated text parsing network.

[0100] In one embodiment, the text parsing network includes a text encoding subnetwork, a text classification subnetwork and a pre-trained category classification subnetwork, and the training module 50 uses the second training set to adjust the parameters in the updated text parsing network, and shares at least part of the adjusted parameters with the audio parsing network to obtain an updated audio parsing network, including: inputting the training text in the second training set into the text parsing network after parameter sharing, obtaining a second recognition text corresponding to the training text output by the text classification subnetwork, and a predicted category corresponding to the training text output by the category classification subnetwork; obtaining a text training loss based on the second recognition text, the predicted category and the category label matched by the training text; in response to the current training cycle not reaching a preset cycle threshold, adjusting at least part of the parameters in the text encoding subnetwork and the text classification subnetwork in the text parsing network using the text training loss, and sharing at least part of the adjusted parameters with the audio parsing network to obtain an updated audio parsing network; in response to the current training cycle reaching the preset cycle threshold, adjusting at least part of the parameters in the text encoding subnetwork, the text classification subnetwork and the category classification subnetwork in the text parsing network using the text training loss, and sharing at least part of the adjusted parameters with the audio parsing network to obtain an updated audio parsing network.

[0101] In one embodiment, please continue to refer to Figure 7 The audio recognition system proposed in the present application also includes a processing module 60 coupled to the training module 50. In response to the second training set including multiple training subsets, each training subset including multiple training texts, after obtaining the first training set and the second training set for the current training cycle, the processing module 60 is used to determine the maximum length from the text lengths of each training text in the training subset, and based on the maximum length, determine the text field threshold that matches the training subset; based on the text field threshold, obtain the text mask that matches the training subset.

[0102] In one embodiment, the training module 50 uses the second training set and the audio training loss to adjust at least part of the parameters in the audio parsing network, and shares at least part of the parameters in the adjusted audio decoding subnetwork to the text parsing network to obtain an updated text parsing network, including: for each training subset in the second training set, inputting the training text into the text parsing network after parameter sharing, and obtaining a second recognition text and a predicted category corresponding to the training text based on the training text and the matching text mask; based on the second recognition text and the predicted category corresponding to the training text, adjusting at least part of the parameters in the text parsing network, and sharing at least part of the parameters in the adjusted text parsing network to the audio parsing network to obtain an updated audio parsing network.

[0103] See also Figure 8 , Figure 8 : is a structural diagram of an embodiment of an electronic device of the present application. The electronic device includes: a memory 70 and a processor 80 coupled to each other. The memory 70 stores program instructions, and the processor 80 is used to execute the program instructions to implement the method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 80 can also be called a CPU (Center Processing Unit). The processor 80 may be an integrated circuit chip with signal processing capabilities. The processor 80 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 80 can be implemented by an integrated circuit chip.

[0104] See also Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 90 stores program instructions 100 that can be executed by a processor. When the program instructions 100 are executed by the processor, the method mentioned in any of the above embodiments is implemented.

[0105] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0109] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An audio recognition method, characterized in that: include: Get the audio to be recognized; The audio to be recognized is input into a target recognition model to obtain a target recognition text that matches the audio to be recognized; wherein the target recognition model is obtained based on a trained initial recognition model, the initial recognition model includes an audio parsing network and a text parsing network that share at least some parameters, the audio parsing network is trained using a plurality of training audios, the text parsing network is trained using a plurality of training texts, the training audios are matched with text labels, the training texts are matched with category labels, and the audio parsing network in the trained initial recognition model serves as the target recognition model; The training process of the initial recognition model includes: obtaining a first training set and a second training set for the current training cycle; wherein the first training set includes a plurality of the training audios, and the second training set includes a plurality of the training texts; using the first training set and the second training set to alternately train the audio parsing network and the text parsing network until a preset stop condition is met, thereby obtaining the trained initial recognition model; Among them, the alternate training of the audio parsing network and the text parsing network using the first training set and the second training set includes: using the first training set to adjust the parameters in the audio parsing network, sharing at least part of the adjusted parameters to the text parsing network, and obtaining the updated text parsing network; using the second training set to adjust the parameters in the updated text parsing network, sharing at least part of the adjusted parameters to the audio parsing network, and obtaining the updated audio parsing network.

2. The audio recognition method according to claim 1, characterized in that: The audio parsing network includes an audio encoding subnetwork and an audio decoding subnetwork. The first training set is used to adjust the parameters in the audio parsing network, and at least part of the adjusted parameters are shared with the text parsing network to obtain the updated text parsing network, including: Inputting the training audio in the first training set into the audio coding subnetwork to obtain an audio coding vector corresponding to the training audio; Inputting the audio encoding vector into the audio decoding subnetwork to obtain a corresponding first recognition text; Obtaining an audio training loss based on the text label matched between the first recognition text and the training audio; The audio training loss is used to adjust at least part of the parameters in the audio parsing network, and at least part of the parameters in the adjusted audio decoding subnetwork are shared with the text parsing network to obtain an updated text parsing network.

3. The audio recognition method according to claim 1, characterized in that: The text parsing network includes a text encoding subnetwork, a text classification subnetwork and a pre-trained category classification subnetwork. The second training set is used to adjust the parameters in the updated text parsing network, and at least part of the adjusted parameters are shared with the audio parsing network to obtain the updated audio parsing network, including: Input the training text in the second training set into the text parsing network after parameter sharing, and obtain the second recognition text corresponding to the training text output by the text classification subnetwork, and the predicted category corresponding to the training text output by the category classification subnetwork; Obtaining text training loss based on the second recognized text, the predicted category, and the category label matched by the training text; In response to the current training cycle not reaching a preset cycle threshold, adjusting at least part of the parameters in the text encoding subnetwork and the text classification subnetwork in the text parsing network using the text training loss, and sharing at least part of the adjusted parameters with the audio parsing network to obtain an updated audio parsing network; In response to the current training cycle reaching a preset cycle threshold, the text training loss is used to adjust at least part of the parameters in the text encoding subnetwork, the text classification subnetwork and the category classification subnetwork in the text parsing network, and at least part of the adjusted parameters are shared with the audio parsing network to obtain an updated audio parsing network.

4. The audio recognition method according to claim 1, characterized in that: In response to the second training set including a plurality of training subsets, each training subset including a plurality of the training texts, after acquiring the first training set and the second training set for the current training cycle, the method further includes: Determine a maximum length from the text lengths of each of the training texts in the training subset, and determine a text field threshold matching the training subset based on the maximum length; Based on the text field threshold, a text mask matching the training subset is obtained.

5. The audio recognition method according to claim 4, characterized in that: The method of using the second training set to adjust the parameters in the updated text parsing network and sharing at least part of the adjusted parameters with the audio parsing network to obtain the updated audio parsing network includes: For each of the training subsets in the second training set, the training text is input into the text parsing network after parameter sharing, and based on the training text and the matched text mask, a second recognition text and a predicted category corresponding to the training text are obtained; Based on the second recognition text and the predicted category corresponding to the training text, at least some parameters in the text parsing network are adjusted, and at least some parameters in the adjusted text parsing network are shared with the audio parsing network to obtain an updated audio parsing network.

6. An audio recognition system, characterized in that: include: An acquisition module, used to acquire the audio to be recognized; A recognition module, used for inputting the audio to be recognized into a target recognition model to obtain a target recognition text matching the audio to be recognized; wherein the target recognition model is obtained based on a trained initial recognition model, the initial recognition model includes an audio parsing network and a text parsing network that share at least some parameters, the audio parsing network is trained using a plurality of training audios, the text parsing network is trained using a plurality of training texts, the training audios are matched with text labels, the training texts are matched with category labels, and the audio parsing network in the trained initial recognition model serves as the target recognition model; The training process of the initial recognition model includes: obtaining a first training set and a second training set for the current training cycle; wherein the first training set includes a plurality of the training audios, and the second training set includes a plurality of the training texts; using the first training set and the second training set to alternately train the audio parsing network and the text parsing network until a preset stop condition is met, thereby obtaining the trained initial recognition model; Among them, the alternate training of the audio parsing network and the text parsing network using the first training set and the second training set includes: using the first training set to adjust the parameters in the audio parsing network, sharing at least part of the adjusted parameters to the text parsing network, and obtaining the updated text parsing network; using the second training set to adjust the parameters in the updated text parsing network, sharing at least part of the adjusted parameters to the audio parsing network, and obtaining the updated audio parsing network.

7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the audio recognition method according to any one of claims 1 to 5 is implemented.

Images