A keyword recognition method and system

Abstract

The application provides a keyword recognition method and system, the method comprising: constructing a speech recognition model, a keyword encoding model and a speaker information encoding model based on a self-attention mechanism respectively; calculating a first loss function value of the speech recognition model based on general audio data; calculating a second loss function value of the keyword encoding model and the speaker information encoding model based on keyword text data and speaker audio data; training the speech recognition model by using the first loss function value; and jointly training the speech recognition model, the keyword encoding model and the speaker information encoding model by using the second loss function value; until the model converges, obtaining a trained joint recognition model; and performing keyword recognition by using the joint recognition model to obtain a keyword recognition result based on a specific speaker. By the application, the keywords called by a specific speaker in a speech stream can be effectively captured, the false wake-up rate is greatly reduced, and the number of keywords is not limited.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a keyword recognition method and system. Background Technology

[0002] With the development of neural networks, speech recognition performance has improved significantly, and keyword-based smart home systems are gradually entering the lives of ordinary people. However, single keyword models often require massive amounts of data for fitting, while the general public has diverse keyword needs, with each person having different requirements. Existing speech recognition methods include large-scale continuous speech recognition (LSCR) and keyword recognition methods. LSCR uses massive amounts of data to train a general speech recognition model by transcribing audio data completely and then extracting keywords from the transcribed text. The advantage of this method is that keywords can be customized arbitrarily. Its disadvantages include the need for massive amounts of audio data, and the speech recognition model is often quite large, requiring significant resources for decoding, making it unsuitable for smart home applications. Keyword recognition methods, on the other hand, use massive amounts of data to fit a single keyword, such as "Xiaodu Xiaodu," "Xiao Ai Tongxue," or "Hisiri." This method has good robustness to single keywords and high recognition accuracy. Its disadvantage is that it often lacks massive amounts of single command word data, making it unsuitable for customized keywords. Summary of the Invention

[0003] To address this issue, the present invention provides a keyword recognition method and system, aiming to solve the problem that existing technologies cannot meet the diverse keyword needs of the public, resulting in low accuracy and easy false wake-up of smart home voice recognition.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] According to a first aspect of the present invention, the present invention provides a keyword identification method, the method comprising:

[0006] A speech recognition model, a keyword encoding model, and a speaker information encoding model based on the self-attention mechanism were constructed respectively.

[0007] The first loss function value of the speech recognition model is calculated based on general audio data; and the second loss function value of the keyword encoding model and the speaker information encoding model is calculated based on keyword text data and speaker audio data.

[0008] The speech recognition model is trained using the first loss function value; and the speech recognition model, keyword encoding model, and speaker information encoding model are jointly trained using the second loss function value; until the model converges, a trained joint recognition model is obtained.

[0009] The joint recognition model is used to identify keywords, and keyword recognition results based on specific speakers are obtained.

[0010] Optionally, the construction of the speech recognition model, keyword encoding model, and speaker information encoding model based on the self-attention mechanism includes:

[0011] A speech recognition model with a 6-layer attention structure is constructed; the speech recognition model is used to recognize keywords in general audio data; and,

[0012] A keyword encoding model with a 4-layer attention structure is constructed; the keyword encoding model is used to encode features of keyword text data; and,

[0013] A speaker information encoding model with a 4-layer attention structure is constructed; the speaker information encoding model is used to encode features of speaker audio data.

[0014] Optionally, calculating the first loss function value of the speech recognition model based on general audio data includes:

[0015] The general audio data is input into the speech recognition model, and the self-attention module of the speech recognition model outputs the first feature vector corresponding to the general audio data.

[0016] The first feature vector is input into the first loss function ASR loss of the speech recognition model, and the first loss function value is calculated based on the first label data corresponding to the general audio data;

[0017] The first tag data is the target keyword corresponding to the general audio data.

[0018] Optionally, calculating the second loss function value of the keyword encoding model and the speaker information encoding model based on keyword text data and speaker audio data includes:

[0019] The keyword text data is input into the keyword encoding model to obtain the second feature vector corresponding to the keyword text data; and,

[0020] The speaker audio data is input into the speaker information encoding model to obtain the third feature vector corresponding to the speaker audio data;

[0021] The first correlation parameter between the first feature vector and the second feature vector and the second correlation parameter between the first feature vector and the third feature vector are calculated using the Cross Attention mechanism, respectively.

[0022] The second loss function value corresponding to the second loss function BCE loss is calculated using the first correlation parameter and the second correlation parameter.

[0023] Optionally, the step of inputting the keyword text data into the keyword encoding model to obtain the second feature vector corresponding to the keyword text data includes:

[0024] The keyword text data is encoded using a preset first encoding rule, and the encoded data is input into the self-attention module of the keyword encoding model to obtain the second feature vector corresponding to the keyword text data; and / or,

[0025] The step of inputting the speaker audio data into the speaker information encoding model to obtain the third feature vector corresponding to the speaker audio data includes:

[0026] The speaker's audio data is encoded using a preset second encoding rule, and the encoded data is input into the self-attention module of the speaker information encoding model to obtain the third feature vector corresponding to the speaker's audio data.

[0027] Optionally, the first encoding rule includes at least one of BERT and word2vec;

[0028] The process of encoding the speaker's audio data using a preset second encoding rule includes:

[0029] MFCC features are extracted from the speaker's audio data, and Iverctor features are extracted using the UBM background model.

[0030] Optionally, the step of calculating the first correlation parameter between the first feature vector and the second feature vector and the second correlation parameter between the first feature vector and the third feature vector using the Cross Attention mechanism includes:

[0031] The first correlation parameter and the second correlation parameter are calculated using the following formulas:

[0032]

[0033]

[0034] in, Z b This is the first associated parameter; Z c This is the first associated parameter; K a This is the first eigenvector; Q b This is the second feature vector; Q c It is the third eigenvector; V a This is a copy of the first eigenvector; d k For feature dimensions.

[0035] Optionally, calculating the second loss function value corresponding to the second loss function BCE loss using the first association parameter and the second association parameter includes:

[0036] Concatenate the first association parameter and the second association parameter; or, weight the first association parameter and the second association parameter.

[0037] The first association parameter and the second association parameter after splicing or weighting are input into the second loss function BCE loss, and the second loss function value is calculated based on the second tag data corresponding to the general audio data;

[0038] Wherein, the second tag data is the target keyword and target speaker corresponding to the general audio data; the second loss function value is 0 or 1.

[0039] Optionally, when the second loss function value is 1, the general audio data, the keyword text data, and the speaker audio data correspond completely; when the second loss function value is 0, the general audio data, the keyword text data, and the speaker audio data do not correspond completely; the step of jointly training the speech recognition model, the keyword encoding model, and the speaker information encoding model using the second loss function value includes:

[0040] The speech recognition model is jointly trained based on the general audio data, the keyword text data, and the speaker audio data corresponding to the second loss function value of 1.

[0041] According to a first aspect of the present invention, the present invention provides a keyword recognition system, the system comprising:

[0042] The model building module is used to build a speech recognition model, a keyword encoding model, and a speaker information encoding model based on the self-attention mechanism, respectively.

[0043] The parameter calculation module is used to calculate the first loss function value of the speech recognition model based on general audio data; and to calculate the second loss function value of the keyword encoding model and the speaker information encoding model based on keyword text data and speaker audio data.

[0044] The model training module is used to train the speech recognition model using the first loss function value; and to jointly train the speech recognition model, the keyword encoding model, and the speaker information encoding model using the second loss function value; until the model converges, a trained joint recognition model is obtained.

[0045] The model recognition module is used to perform keyword recognition using the joint recognition model to obtain keyword recognition results.

[0046] The present invention, by adopting the above technical solution, has at least the following beneficial effects:

[0047] This invention constructs a speech recognition model, a keyword encoding model, and a speaker information encoding model based on a self-attention mechanism. A first loss function value is calculated for the speech recognition model based on general audio data. A second loss function value is calculated for the keyword encoding model and the speaker information encoding model based on keyword text data and speaker audio data. The speech recognition model is trained using the first loss function value. The speech recognition model, keyword encoding model, and speaker information encoding model are then jointly trained using the second loss function value until the model converges, resulting in a trained joint recognition model. The joint recognition model is then used for keyword recognition to obtain keyword recognition results based on a specific speaker. Therefore, a keyword recognition method based on a bias between text information and speaker information is proposed. In the training process of a general speech recognition system, keyword text information and speaker information are incorporated, and joint training using three modalities—keyword text information, speech information, and speaker information—effectively captures keywords from the speech stream while effectively controlling false wake-ups. Furthermore, the number of keywords is unlimited, allowing for simultaneous training of multiple keywords.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating a keyword recognition method provided in an embodiment of the present invention is shown;

[0051] Figure 2 A schematic diagram of the keyword recognition system provided in an embodiment of the present invention is shown. Detailed Implementation

[0052] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0053] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0054] This invention provides a keyword recognition method, such as... Figure 1 As shown, it may include at least the following steps S101~S104:

[0055] Step S101: Construct a speech recognition model, a keyword encoding model, and a speaker information encoding model based on the self-attention mechanism, respectively.

[0056] The speech recognition model, keyword encoding model, and speaker information encoding model in the embodiments of the present invention are all neural network models containing multi-layer attention structures.

[0057] Specifically, the speech recognition model comprises a 6-layer attention structure for recognizing keywords in general audio data. General audio data refers to audio data from an unknown speaker. The keyword encoding model comprises a 4-layer attention structure for feature encoding of keyword text data. Keyword text data does not contain pre-defined keyword text information. The speaker information encoding model also comprises a 4-layer attention structure for feature encoding of speaker audio data. Speaker audio data refers to audio data from a specific speaker.

[0058] Step S102: Calculate the first loss function value of the speech recognition model based on general audio data; and calculate the second loss function values ​​of the keyword encoding model and the speaker information encoding model based on keyword text data and speaker audio data.

[0059] It is understandable that by inputting general audio data into a speech recognition model to obtain the output result, and comparing it with label data representing the correct output result, a first loss function value is calculated. This first loss function value can then be used to train the speech recognition model, improving its recognition performance. In this embodiment of the invention, when training the speech recognition model, keyword text data and speaker audio data are introduced to improve the correlation between keyword recognition and specific speakers.

[0060] Specifically, the first loss function value of the speech recognition model can be calculated based on general audio data by inputting general audio data into the speech recognition model, outputting the first feature vector corresponding to the general audio data in the self-attention module of the speech recognition model, inputting the first feature vector into the first loss function ASR loss of the speech recognition model, and calculating the first loss function value based on the first label data corresponding to the general audio data.

[0061] It should be noted that the input to the speech recognition system can be audio data, FBank features, or MFCC features; this invention does not impose any restrictions on this. Generally, ordinary audio data can be input into the speech recognition model, and the self-attention module outputs the first feature vector corresponding to the ordinary audio data, denoted as... K a Furthermore, K a The first loss function, ASR loss, input into the speech recognition model is compared with the first label data corresponding to the general audio data to calculate the value of the first loss function. Here, the first label data is the target keyword corresponding to the general audio data.

[0062] Furthermore, based on keyword text data and speaker audio data, the second loss function values ​​of the keyword encoding model and speaker information encoding model are calculated. This can be achieved by inputting the keyword text data into the keyword encoding model to obtain the second feature vector corresponding to the keyword text data; and by inputting the speaker audio data into the speaker information encoding model to obtain the third feature vector corresponding to the speaker audio data. The first correlation parameter between the first feature vector and the second feature vector and the second correlation parameter between the first feature vector and the third feature vector are calculated using the Cross Attention mechanism, respectively. The second loss function value corresponding to the second loss function BCE loss is then calculated using the first correlation parameter and the second correlation parameter.

[0063] Specifically, keyword text data is input into a keyword encoding model to obtain the second feature vector corresponding to the keyword text data. This can be achieved by encoding the keyword text data using a pre-defined first encoding rule, and then inputting the encoded data into a four-layer self-attention structure of the keyword encoding model to obtain the second feature vector corresponding to the keyword text data. Q b The first encoding rule can be a method such as BERT or word2vec, and this invention does not limit this method.

[0064] The speaker's audio data is input into the speaker information encoding model to obtain the third feature vector corresponding to the speaker's audio data. Specifically, the speaker's audio data can be encoded using a preset second encoding rule. The encoded data is then input into the four-layer self-attention structure of the speaker information encoding model to obtain the third feature vector corresponding to the speaker's audio data. Q c Specifically, when encoding speaker audio data using the preset second encoding rule, MFCC features can be extracted from the speaker audio data, and Iverctor features can be extracted using the UBM background model to obtain the third feature vector. Q c .

[0065] Furthermore, the Cross Attention mechanism is used to concatenate the audio model modal information with the keyword text modal information and the corresponding speaker modal information, and then input them into the attention model. This yields the association parameters between the audio model modal information and the text modal information (i.e., the first association parameter between the first feature vector and the second feature vector) and the association parameters between the audio model modal information and the speaker modal information (i.e., the second association parameter between the first feature vector and the third feature vector).

[0066] Specifically, the first association parameter can be expressed as:

[0067]

[0068] The second correlation parameter can be expressed as:

[0069]

[0070] in, K a This is the first eigenvector; Q b This is the second feature vector; Q c It is the third eigenvector; V a This is a copy of the first eigenvector; d k For feature dimensions.

[0071] Understandably, if the general audio data input to the speech recognition model contains keywords, then the attention model can capture the keywords in the audio stream, and the first feature vector... K a With the second eigenvector Q b First associated parameter Z b The value in the first feature vector will be relatively large, otherwise it will be small. Similarly, if the general audio data input to the speech recognition model is audio data of a specific speaker, then the first feature vector will be relatively large. K a With the second eigenvector Q b Second correlation parameter Z c The value in the middle will be relatively large, otherwise it will be small.

[0072] Furthermore, the second loss function value corresponding to the second loss function BCE loss is calculated using the first association parameter and the second association parameter. Specifically, the first association parameter and the second association parameter can be concatenated; or the first association parameter and the second association parameter can be weighted. The concatenated or weighted first association parameter and the second association parameter are input into the second loss function BCE loss, and the second loss function value is calculated based on the second tag data corresponding to the general audio data.

[0073] In other words, this embodiment of the invention can concatenate the two feature vectors, the first association parameter and the second association parameter, to obtain a target feature vector whose dimension is the sum of the dimensions of the first association parameter and the second association parameter. The target feature vector is then input into a second loss function, and the BCE loss value is calculated based on the second label data. The second label data consists of the target keywords and the target speaker corresponding to the general audio data. It should be noted that the BCE loss is a binary classification vector, with an output of either 0 or 1. When the second loss function value is 1, the general audio data, keyword text data, and speaker audio data correspond completely; when the second loss function value is 0, the general audio data, keyword text data, and speaker audio data do not correspond completely. That is, 1 indicates that the general audio data contains keywords and the audio data is from a specific speaker. If the general audio data does not contain keywords or is not from a specific speaker, the BCE loss value is 0.

[0074] Step S103: Train the speech recognition model using the first loss function value; and train the speech recognition model, keyword encoding model, and speaker information encoding model jointly using the second loss function value; until the model converges, a trained joint recognition model is obtained.

[0075] Based on the above description, it can be understood that in this embodiment of the invention, the speech recognition model is trained using the first loss function value, ASR loss, to enable the speech recognition model to learn the feature information of keyword text corresponding to general audio data; the speech recognition model is trained using the second loss function value, BCE loss, to enable the speech recognition model to learn the feature information of the three modalities fused together: a specific speaker, audio data, and keyword text. Through joint training with the two losses until the model converges, a joint recognition model based on specific speaker correlation can be obtained.

[0076] Step S104: Use the joint recognition model to perform keyword recognition and obtain keyword recognition results based on a specific speaker.

[0077] Based on the above, the joint recognition model trained in this invention can detect keywords when audio is detected, and can also detect audio keywords from specific speakers. Applied to the smart home field, it can accurately identify keyword calls from specific speakers. That is, when two people speak simultaneously, their audio overlaps, and ordinary keyword recognition systems cannot accurately capture this aliased audio. However, the joint recognition model proposed in this invention can accurately capture effective keywords from the audio stream for specific speakers, thus waking up the smart home system. Even if a non-specific key person calls the keyword, the smart home system will not be woken up, effectively controlling the false wake-up rate.

[0078] This invention provides a keyword recognition method, which involves constructing a speech recognition model, a keyword encoding model, and a speaker information encoding model based on a self-attention mechanism; calculating a first loss function value for the speech recognition model based on general audio data; and calculating second loss function values ​​for the keyword encoding model and the speaker information encoding model based on keyword text data and speaker audio data; training the speech recognition model using the first loss function value; and jointly training the speech recognition model, keyword encoding model, and speaker information encoding model using the second loss function value until the model converges, obtaining a trained joint recognition model; and using the joint recognition model to perform keyword recognition, obtaining keyword recognition results based on a specific speaker. Therefore, a keyword recognition method based on a bias of text information and speaker information is proposed. In the training process of a general speech recognition system, keyword text information and speaker information are incorporated, and joint training using three modalities—keyword text information, speech information, and speaker information—effectively captures keywords in the speech stream, while also effectively controlling false wake-ups. Furthermore, the number of keywords is unlimited, allowing for the simultaneous training of multiple keywords.

[0079] Furthermore, as Figure 1 In specific implementation, embodiments of the present invention provide a keyword recognition system, such as... Figure 2 As shown, the system may include: a model building module 210, a parameter calculation module 220, a model training module 230, and a model recognition module 240.

[0080] The model building module 210 can be used to build a speech recognition model, a keyword encoding model, and a speaker information encoding model based on the self-attention mechanism, respectively.

[0081] The parameter calculation module 220 can be used to calculate the first loss function value of the speech recognition model based on general audio data; and to calculate the second loss function value of the keyword encoding model and the speaker information encoding model based on keyword text data and speaker audio data.

[0082] The model training module 230 can be used to train a speech recognition model using the first loss function value; and to jointly train a speech recognition model, a keyword encoding model, and a speaker information encoding model using the second loss function value; until the model converges, a well-trained joint recognition model is obtained.

[0083] The model recognition module 240 can be used to perform keyword recognition using a joint recognition model to obtain keyword recognition results.

[0084] Optionally, the model building module 210 can also be used to build a speech recognition model containing a 6-layer attention structure; the speech recognition model is used to recognize keywords in general audio data; and,

[0085] Construct a keyword encoding model with a 4-layer attention structure; the keyword encoding model is used to encode features of keyword text data; and,

[0086] A speaker information encoding model with a 4-layer attention structure is constructed; the speaker information encoding model is used to encode features of speaker audio data.

[0087] The parameter calculation module 220 can also be used to input general audio data into the speech recognition model and output the first feature vector corresponding to the general audio data in the self-attention module of the speech recognition model.

[0088] The first feature vector is input into the first loss function ASR loss of the speech recognition model, and the value of the first loss function is calculated based on the first label data corresponding to the general audio data.

[0089] The first tag data consists of the target keywords corresponding to the general audio data.

[0090] The parameter calculation module 220 can also be used to input keyword text data into the keyword encoding model to obtain the second feature vector corresponding to the keyword text data; and,

[0091] Input the speaker's audio data into the speaker information encoding model to obtain the third feature vector corresponding to the speaker's audio data;

[0092] The first correlation parameter between the first feature vector and the second feature vector, and the second correlation parameter between the first feature vector and the third feature vector are calculated using the Cross Attention mechanism respectively.

[0093] The second loss function value corresponding to the BCE loss is calculated using the first and second correlation parameters.

[0094] The parameter calculation module 220 can also be used to encode the keyword text data using a preset first encoding rule, input the encoded data into the self-attention module of the keyword encoding model, and obtain the second feature vector corresponding to the keyword text data; and / or,

[0095] The speaker's audio data is encoded using a pre-defined second encoding rule. The encoded data is then input into the self-attention module of the speaker information encoding model to obtain the third feature vector corresponding to the speaker's audio data.

[0096] The parameter calculation module 220 can also be used to extract MFCC features from speaker audio data and extract ivector features using the UBM background model; the first encoding rule includes at least one of BERT and word2vec.

[0097] The parameter calculation module 220 can also be used to calculate the first correlation parameter and the second correlation parameter using the following formula:

[0098]

[0099]

[0100] in, Z b This is the first associated parameter; Z c This is the first associated parameter; K a This is the first eigenvector; Q b This is the second feature vector; Q c It is the third eigenvector; V a This is a copy of the first eigenvector; d k For feature dimensions.

[0101] The parameter calculation module 220 can also be used to concatenate the first associated parameter and the second associated parameter; or to weight the first associated parameter and the second associated parameter.

[0102] The first and second association parameters, after being spliced ​​or weighted, are input into the second loss function BCE loss, and the value of the second loss function is calculated based on the second tag data corresponding to the general audio data.

[0103] The second label data consists of the target keywords and target speaker corresponding to the general audio data; the second loss function value is 0 or 1.

[0104] The model training module 230 can also be used to jointly train the speech recognition model based on the general audio data, keyword text data and speaker audio data corresponding to the second loss function value of 1.

[0105] It should be noted that other corresponding descriptions of the functional modules involved in the keyword recognition system provided in this embodiment of the invention can be found in [reference needed]. Figure 1 The corresponding description of the method shown will not be repeated here.

[0106] Those skilled in the art will clearly understand that the specific working process of the systems, devices, modules and units described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0107] Furthermore, the functional units in the various embodiments of the present invention can be physically independent of each other, or two or more functional units can be integrated together, or all functional units can be integrated into one processing unit. The integrated functional units described above can be implemented in hardware, or in software or firmware.

[0108] Those skilled in the art will understand that if the integrated functional unit is implemented in software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of the present invention when running the instructions. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as a computing device, personal computer, server, or network device) related to program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the computing device, the computing device executes all or part of the steps of the methods described in the various embodiments of the present invention.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to depart from the protection scope of the present invention.

Claims

1. A keyword recognition method, characterized in that, The method includes: A speech recognition model, a keyword encoding model, and a speaker information encoding model based on the self-attention mechanism were constructed respectively. The first loss function value of the speech recognition model is calculated based on general audio data, including: The general audio data is input into the speech recognition model, and the self-attention module of the speech recognition model outputs the first feature vector corresponding to the general audio data. The first feature vector is input into the first loss function ASR loss of the speech recognition model, and the first loss function value is calculated based on the first label data corresponding to the general audio data; Wherein, the first tag data is the target keyword corresponding to the general audio data; the general audio data is audio data of an unknown speaker; And, based on the keyword text data and speaker audio data, calculate the second loss function value of the keyword encoding model and the speaker information encoding model, including: The keyword text data is input into the keyword encoding model to obtain the second feature vector corresponding to the keyword text data; and, The speaker audio data is input into the speaker information encoding model to obtain the third feature vector corresponding to the speaker audio data; The first correlation parameter between the first feature vector and the second feature vector and the second correlation parameter between the first feature vector and the third feature vector are calculated using the Cross Attention mechanism, respectively. The calculation of the second loss function value corresponding to the second loss function BCE loss using the first association parameter and the second association parameter includes: Concatenate the first association parameter and the second association parameter; or, weight the first association parameter and the second association parameter. The first association parameter and the second association parameter after splicing or weighting are input into the second loss function BCE loss, and the second loss function value is calculated based on the second tag data corresponding to the general audio data; Wherein, the second tag data consists of the target keywords and target speaker corresponding to the general audio data; the second loss function value is 0 or 1; The speech recognition model is trained using the first loss function value; and the speech recognition model, keyword encoding model, and speaker information encoding model are jointly trained using the second loss function value; until the model converges, a trained joint recognition model is obtained. The joint recognition model is used to identify keywords, and keyword recognition results based on specific speakers are obtained.

2. The method according to claim 1, characterized in that, The construction of a speech recognition model, a keyword encoding model, and a speaker information encoding model based on the self-attention mechanism includes: A speech recognition model with a 6-layer attention structure is constructed; the speech recognition model is used to recognize keywords in general audio data; and, A keyword encoding model with a 4-layer attention structure is constructed; the keyword encoding model is used to encode features of keyword text data; and, A speaker information encoding model with a 4-layer attention structure is constructed; the speaker information encoding model is used to encode features of speaker audio data.

3. The method according to claim 1, characterized in that, The step of inputting the keyword text data into the keyword encoding model to obtain the second feature vector corresponding to the keyword text data includes: The keyword text data is encoded using a preset first encoding rule, and the encoded data is input into the self-attention module of the keyword encoding model to obtain the second feature vector corresponding to the keyword text data; and / or, The step of inputting the speaker audio data into the speaker information encoding model to obtain the third feature vector corresponding to the speaker audio data includes: The speaker's audio data is encoded using a preset second encoding rule, and the encoded data is input into the self-attention module of the speaker information encoding model to obtain the third feature vector corresponding to the speaker's audio data.

4. The method according to claim 3, characterized in that, The first encoding rule includes at least one of BERT and word2vec; The process of encoding the speaker's audio data using a preset second encoding rule includes: MFCC features are extracted from the speaker's audio data, and Iverctor features are extracted using the UBM background model.

5. The method according to claim 1, characterized in that, The step of calculating the first correlation parameter between the first feature vector and the second feature vector and the second correlation parameter between the first feature vector and the third feature vector using the Cross Attention mechanism includes: The first correlation parameter and the second correlation parameter are calculated using the following formulas: in, Z b This is the first associated parameter; Z c This is the second associated parameter; K a This is the first eigenvector; Q b This is the second feature vector; Q c It is the third eigenvector; V a This is a copy of the first eigenvector; d k For feature dimensions.

6. The method according to claim 1, characterized in that, When the second loss function value is 1, the general audio data, the keyword text data, and the speaker audio data correspond completely; when the second loss function value is 0, the general audio data, the keyword text data, and the speaker audio data do not correspond completely; the step of jointly training the speech recognition model, the keyword encoding model, and the speaker information encoding model using the second loss function value includes: The speech recognition model is jointly trained based on the general audio data, the keyword text data, and the speaker audio data corresponding to the second loss function value of 1.

7. A keyword recognition system, characterized in that, The system includes: The model building module is used to build a speech recognition model, a keyword encoding model, and a speaker information encoding model based on the self-attention mechanism, respectively. The parameter calculation module is used to calculate the first loss function value of the speech recognition model based on general audio data, including: The general audio data is input into the speech recognition model, and the self-attention module of the speech recognition model outputs the first feature vector corresponding to the general audio data. The first feature vector is input into the first loss function ASR loss of the speech recognition model, and the first loss function value is calculated based on the first label data corresponding to the general audio data; Wherein, the first tag data is the target keyword corresponding to the general audio data; the general audio data is audio data of an unknown speaker; And, based on the keyword text data and speaker audio data, calculate the second loss function value of the keyword encoding model and the speaker information encoding model, including: The keyword text data is input into the keyword encoding model to obtain the second feature vector corresponding to the keyword text data; and, The speaker audio data is input into the speaker information encoding model to obtain the third feature vector corresponding to the speaker audio data; The first correlation parameter between the first feature vector and the second feature vector and the second correlation parameter between the first feature vector and the third feature vector are calculated using the Cross Attention mechanism, respectively; wherein, the first feature vector is generated by the self-attention module of the speech recognition model by inputting the general audio data; The calculation of the second loss function value corresponding to the second loss function BCE loss using the first association parameter and the second association parameter includes: Concatenate the first association parameter and the second association parameter; or, weight the first association parameter and the second association parameter. The first association parameter and the second association parameter after splicing or weighting are input into the second loss function BCE loss, and the second loss function value is calculated based on the second tag data corresponding to the general audio data; Wherein, the second tag data consists of the target keywords and target speaker corresponding to the general audio data; the second loss function value is 0 or 1; The model training module is used to train the speech recognition model using the first loss function value; and to jointly train the speech recognition model, the keyword encoding model, and the speaker information encoding model using the second loss function value; until the model converges, a trained joint recognition model is obtained. The model recognition module is used to perform keyword recognition using the joint recognition model to obtain keyword recognition results.

Images