Speech recognition method and model training method, device, electronic equipment and medium

Abstract

The present disclosure provides a speech recognition method, a model training method, an apparatus, an electronic device and a medium, relates to the technical field of artificial intelligence, and in particular to the field of large models, speech recognition and natural language processing. The specific implementation scheme is: performing semantic fusion on audio features of to-be-processed audio, text features of a target text and semantic features of the to-be-processed audio to obtain target fusion features; wherein the target text and the to-be-processed audio have semantic association; the target fusion features fuse sound information and semantic information of the to-be-processed audio and context information of the target text; the semantic features are obtained by decoding the audio features; decoding the target fusion features to generate a corrected text; wherein the corrected text includes a position identifier for identifying a correction position and a correction field; and according to the position identifier, the initial text is corrected by using the correction field to obtain a target recognition text; wherein the initial text is obtained by decoding the semantic features.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of large models, speech recognition, and natural language processing. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI-based speech recognition methods have been widely used. However, traditional speech recognition methods typically only process the audio data being processed, resulting in low accuracy of the recognition results. Summary of the Invention

[0003] This disclosure provides a speech recognition method, a model training method, an apparatus, an electronic device, and a medium.

[0004] According to one aspect of this disclosure, a speech recognition method is provided, comprising: semantically fusing audio features of an audio to be processed, text features of a target text, and semantic features of the audio to be processed to obtain target fusion features; wherein, there is a semantic relationship between the target text and the audio to be processed; the target fusion features fuse sound information and semantic information of the audio to be processed, as well as contextual information of the target text; the semantic features are obtained by decoding the audio features; decoding the target fusion features to generate corrected text; wherein the corrected text includes a position identifier for identifying the correction position and a correction field; and correcting the initial text according to the position identifier and using the correction field to obtain target recognition text; wherein the initial text is obtained by decoding the semantic features.

[0005] According to another aspect of this disclosure, a model training method is provided, comprising: using a correction module to semantically fuse sample audio features of sample audio, sample text features of sample text, and sample semantic features of sample audio to obtain sample fusion features; wherein, there is a semantic relationship between sample text and sample audio; the sample fusion features fuse sound information and semantic information of sample audio, as well as contextual information of sample text; the sample semantic features are obtained by decoding sample audio features using a speech recognition module; the correction module decodes the sample fusion features to generate sample corrected text; wherein the sample corrected text includes a sample position identifier for identifying the correction position and a sample correction field; and based on a first loss function, fixing the parameters of the speech recognition module, training the correction module using the sample corrected text to obtain a trained target model.

[0006] According to another aspect of this disclosure, a speech recognition device is provided, comprising: a semantic fusion module, configured to semantically fuse audio features of an audio to be processed, text features of a target text, and semantic features of the audio to be processed to obtain target fusion features; wherein, there is a semantic relationship between the target text and the audio to be processed; the target fusion features fuse sound information and semantic information of the audio to be processed, as well as contextual information of the target text; the semantic features are obtained by decoding the audio features; a fusion decoding module, configured to decode the target fusion features to generate corrected text; wherein, the corrected text includes a position identifier for identifying the correction position and a correction field; and a text correction module, configured to correct the initial text according to the position identifier and using the correction field to obtain target recognition text; wherein, the initial text is obtained by decoding the semantic features.

[0007] According to another aspect of this disclosure, a model training apparatus is provided, comprising: a sample fusion module, configured to semantically fuse sample audio features of sample audio, sample text features of sample text, and sample semantic features of sample audio using a correction module to obtain sample fusion features; wherein there is a semantic relationship between sample text and sample audio; the sample fusion features fuse sound information and semantic information of sample audio, as well as contextual information of sample text; the sample semantic features are obtained by decoding sample audio features using a speech recognition module; a sample decoding module, configured to decode the sample fusion features using the correction module to generate sample corrected text; wherein the sample corrected text includes a sample position identifier for identifying the correction position and a sample correction field; and a training module, configured to train the correction module using the sample corrected text based on a first loss function, fixing the parameters of the speech recognition module, to obtain a trained target model.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This illustration schematically shows an exemplary system architecture of a speech recognition method and apparatus according to embodiments of the present disclosure;

[0014] Figure 2 A flowchart illustrating a speech recognition method according to an embodiment of the present disclosure is shown schematically.

[0015] Figure 3 A schematic diagram illustrating the generation of target-identified text according to an embodiment of the present disclosure is shown.

[0016] Figure 4A The illustration shows a schematic diagram of generating modified text according to an embodiment of the present disclosure;

[0017] Figure 4B A schematic diagram illustrating the generation of modified text according to another embodiment of this disclosure is shown;

[0018] Figure 5 The illustration shows a schematic diagram of modifying initial text according to an embodiment of the present disclosure;

[0019] Figure 6 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 7 The illustration shows a schematic diagram of the generation of sample labels and sample text according to embodiments of the present disclosure;

[0021] Figure 8 A schematic diagram of the structure of a target model according to an embodiment of the present disclosure is shown.

[0022] Figure 9 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 10 A block diagram of a model training apparatus according to an embodiment of the present disclosure is schematically shown; and

[0024] Figure 11 A block diagram of an electronic device suitable for implementing a speech recognition method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0027] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0028] With the rapid development of artificial intelligence technology, AI-based speech recognition methods have been widely applied. However, traditional speech recognition methods typically only process the audio data being processed, resulting in low accuracy. For example, traditional speech recognition methods often use general datasets to train speech recognition models, but these models cannot guarantee performance when faced with specific words, such as trending terms and domain-specific terms.

[0029] To improve the accuracy of speech recognition models in identifying timely trending words, related technologies typically use labeled training data containing the latest trending words and information to iterate the speech recognition model. However, additional labeled training data significantly increases costs, and periodic iteration is lagging and cannot meet recognition needs in a timely manner.

[0030] Furthermore, in order to improve the accuracy of speech recognition models in recognizing proper nouns in specific domains, related technologies usually combine speech recognition models with slotting techniques for recognition. However, slotting techniques often rely heavily on semantic rule matching, the processing is relatively mechanical, and the cross-domain generalization ability is insufficient. Retraining is required in different domains, which increases deployment costs.

[0031] Embodiments of this disclosure provide a speech recognition method, comprising: semantically fusing audio features of an audio to be processed, text features of a target text, and semantic features of the audio to be processed to obtain target fusion features; wherein, there is a semantic relationship between the target text and the audio to be processed; the target fusion features fuse sound information and semantic information of the audio to be processed, as well as contextual information of the target text; the semantic features are obtained by decoding the audio features; decoding the target fusion features to generate corrected text; wherein, the corrected text includes a position identifier for identifying the correction position and a correction field; and correcting the initial text according to the position identifier and using the correction field to obtain target recognition text; wherein, the initial text is obtained by decoding the semantic features.

[0032] According to embodiments of this disclosure, by introducing target text that has a semantic relationship with the audio to be processed during the speech recognition process, rich contextual information can be provided to the audio. This contextual information helps to more accurately parse the audio, especially when dealing with difficulties caused by factors such as trending words and domain-specific proper nouns. The contextual information of the target text can serve as important auxiliary information to more accurately identify the speech content. Furthermore, by semantically fusing the audio features of the audio to be processed, the text features of the target text, and the semantic features of the audio to be processed, the sound information, semantic information, and contextual information of the target text can be fully utilized, making the target fused features contain more comprehensive and accurate semantically related content. Based on this, the target fused features are decoded to generate corrected text. The positional identifiers and correction fields in the corrected text are used to correct the initial text to obtain the target recognition text, effectively improving the accuracy and quality of speech recognition and better meeting the accuracy requirements of speech recognition in different practical application scenarios.

[0033] In this embodiment of the disclosure, the quality of semantic recognition includes, but is not limited to, the speed, accuracy, and robustness of speech recognition.

[0034] Figure 1 The illustration schematically shows an exemplary system architecture to which speech recognition methods and apparatus can be applied according to embodiments of the present disclosure.

[0035] It is important to note that Figure 1 The examples shown are merely examples of system architectures applicable to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. They do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For instance, in another embodiment, an exemplary system architecture to which the speech recognition method and apparatus can be applied may include a terminal device. However, the terminal device can implement the speech recognition method and apparatus provided in the embodiments of this disclosure without interacting with a server.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the speech recognition method provided in this embodiment can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the speech recognition device provided in this embodiment can also be disposed in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0041] Alternatively, the speech recognition method provided in this embodiment can generally be executed by server 105. Correspondingly, the speech recognition device provided in this embodiment can generally be located in server 105. The speech recognition method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the speech recognition device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0042] For example, when a user inputs audio to be processed into a first terminal device 101, a second terminal device 102, and a third terminal device 103, the first terminal device 101, the second terminal device 102, and the third terminal device 103 can semantically fuse the audio features of the audio to be processed, the text features of the target text, and the semantic features of the audio to be processed to obtain target fusion features; wherein, there is a semantic relationship between the target text and the audio to be processed; the target fusion features fuse the sound information and semantic information of the audio to be processed, as well as the context information of the target text; the semantic features are obtained by decoding the audio features; the target fusion features are decoded to generate corrected text; wherein, the corrected text includes a position identifier and a correction field for identifying the correction position; and the initial text is corrected according to the position identifier using the correction field to obtain target recognition text; wherein, the initial text is obtained by decoding the semantic features.

[0043] It should be understood that Figure 1 The number of first terminal devices, second terminal devices, third terminal devices, networks, and servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of first terminal devices, second terminal devices, third terminal devices, networks, and servers can be included.

[0044] Figure 2 A flowchart illustrating a speech recognition method according to an embodiment of the present disclosure is shown schematically.

[0045] like Figure 2 As shown, the method includes operations S210~S230.

[0046] In operation S210, the audio features of the audio to be processed, the text features of the target text, and the semantic features of the audio to be processed are semantically fused to obtain the target fused features.

[0047] In operation S220, the target fusion features are decoded to generate corrected text.

[0048] In operation S230, the initial text is corrected according to the location identifier using the correction field to obtain the target recognition text.

[0049] According to embodiments of this disclosure, there is a semantic relationship between the target text and the audio to be processed, and the textual features of the target text can be obtained by encoding the target text. For example, the target text can be hot words that have a semantic relationship with the audio to be processed, enabling the speech recognition process to adapt to the latest trending words and information. The target text can also be proper nouns in a specific field, enabling the speech recognition process to adapt to the requirements of a specific field. In addition, the target text can also be historical interaction content and user-defined information that have a semantic relationship with the audio to be processed, enabling the speech recognition process to meet the personalized needs of users.

[0050] In some embodiments, the audio to be processed can be input into a large language model to obtain target text that is semantically related to the audio to be processed. Alternatively, it can be matched in a text library to obtain pre-stored text related to the topic of the audio to be processed as target text. As long as the target text can provide valuable contextual information for the audio to be processed and help improve the accuracy of speech recognition, there are no restrictions on the method of obtaining the target text.

[0051] According to embodiments of this disclosure, the audio features of the audio to be processed can be obtained by encoding the audio to be processed, the semantic features are obtained by decoding the audio features, and further decoding of the audio features can obtain the initial text.

[0052] In a specific example, a large speech recognition model can be used to process the audio to be processed, obtaining the frame-level audio features output by the coding layer, the semantic features output by the decoding layer, and the final output of the initial text. By introducing frame-level speech features, the acoustic features in the audio to be processed can be preserved as much as possible, reducing the dependence on the large speech recognition model, and introducing richer acoustic information to improve the reliability of the recognition process.

[0053] The target fusion feature integrates the audio and semantic information of the audio to be processed, as well as the contextual information of the target text. In some embodiments, the audio features, text features, and speech features can be semantically fused based on an attention mechanism, automatically learning and assigning weights between different features to more effectively capture key information. This allows the target fusion feature to more comprehensively reflect the semantic relationship between the audio to be processed and the target text, thereby improving the accuracy of speech recognition.

[0054] After obtaining the target fusion features, they need to be decoded to generate corrected text. The corrected text includes a location identifier to identify the correction location and a correction field. The location identifier is used to accurately point out the specific location where the error occurred in the audio to be processed during speech recognition, while the correction field is the correct text content obtained after accurately correcting the content at the error location based on the semantic information contained in the target fusion features.

[0055] When correcting the initial text based on the corrected text, the initial text can be corrected using the correction field according to the location identifier, ultimately yielding the target recognition text. In this process, the precise matching of the correction field and the location identifier ensures the accuracy and effectiveness of the correction. As the final output of speech recognition, the target recognition text not only retains the valid information in the initial text but also corrects errors or ambiguous parts through the correction field, thereby improving the overall quality of speech recognition.

[0056] According to embodiments of this disclosure, by introducing target text that has a semantic relationship with the audio to be processed during the speech recognition process, rich contextual information can be provided to the audio. This contextual information helps to more accurately parse the audio, especially when dealing with difficulties caused by factors such as trending words and domain-specific proper nouns. The contextual information of the target text can serve as important auxiliary information to more accurately identify the speech content. Furthermore, by semantically fusing the audio features of the audio to be processed, the text features of the target text, and the semantic features of the audio to be processed, the sound information, semantic information, and contextual information of the target text can be fully utilized, making the target fused features contain more comprehensive and accurate semantically related content. Based on this, the target fused features are decoded to generate corrected text. The positional identifiers and correction fields in the corrected text are used to correct the initial text to obtain the target recognition text, effectively improving the accuracy and quality of speech recognition and better meeting the accuracy requirements of speech recognition in different practical application scenarios.

[0057] The following is for reference. Figures 3-5 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0058] Figure 3 A schematic diagram illustrating the generation of target recognition text according to an embodiment of the present disclosure is shown.

[0059] like Figure 3 As shown, the target text 10 is encoded to obtain text feature 20. The audio to be processed 30 is encoded to obtain audio feature 40. The audio feature 40 is decoded to obtain semantic feature 50. The semantic feature 50 is then decoded to obtain the initial text 60.

[0060] like Figure 3 As shown, semantic fusion is performed on text feature 20, audio feature 40, and semantic feature 50 to obtain target fusion feature 70. Decoding target fusion feature 70 yields corrected text 80. The corrected text 80 is then used to correct the initial text 60, resulting in target recognition text 90.

[0061] According to embodiments of this disclosure, the target fusion features include: audio fusion features and semantic fusion features. For Figure 2 The operation S210 shown performs semantic fusion on the audio features of the audio to be processed, the text features of the target text, and the semantic features of the audio to be processed to obtain target fusion features. This includes: fusing audio features and text features based on an interactive attention mechanism to obtain audio fusion features; and fusing semantic features and text features based on an interactive attention mechanism to obtain semantic fusion features.

[0062] According to embodiments of this disclosure, audio fusion features characterize the features of the fused contextual information of the target text and the acoustic information of the audio to be processed, while semantic fusion features characterize the features of the fused contextual information of the target text and the semantic information of the audio to be processed.

[0063] In the embodiments of this disclosure, the feature fusion process is implemented based on an interactive attention mechanism. The interactive attention mechanism can automatically learn the correlation between different features and dynamically allocate weights according to these correlations, thereby ensuring that key information can be fully captured during the fusion process, thus making the fusion between audio features, text features, and semantic features more accurate.

[0064] For the fusion of audio and text features, the interactive attention mechanism analyzes the correlation between the acoustic information contained in the audio features and the contextual information contained in the text features. For example, since certain acoustic features may be correlated with the text features of specific words or phrases, the interactive attention mechanism can generate a weight matrix through the information interaction between these two different modalities. This weight matrix indicates the degree of correlation between each element in the audio feature and each element in the text feature. Based on this weight matrix, the audio and text features are fused into an audio fusion feature, which simultaneously contains the acoustic information of the audio to be processed and the contextual information of the target text.

[0065] Similarly, for the fusion of semantic and textual features, the interactive attention mechanism analyzes the correlation between the semantic information contained in the speech features and the contextual information contained in the text features. For example, some semantic features may be closely related to specific paragraphs or sentences in the target text. Through the analysis of the interactive attention mechanism, another weight matrix can be generated to indicate the fusion process of semantic and textual features. The final semantic fusion feature not only contains the semantic information of the audio to be processed, but also incorporates the contextual information of the target text, thus providing a richer semantic background for the parsing of the audio.

[0066] According to embodiments of this disclosure, by fusing text features with semantic features and audio features respectively based on an interactive attention mechanism, rich acoustic information can be introduced, enhancing contextual semantic interaction. This allows the target fusion feature to be constructed as a composite feature containing audio fusion features and semantic fusion features, which not only has a more comprehensive information representation capability but also more accurately reflects the semantic relationship between the audio to be processed and the target text. In turn, the target fusion feature can be decoded into accurate corrected text, improving the accuracy of speech recognition.

[0067] According to embodiments of this disclosure, when the target fusion features include audio fusion features and semantic fusion features, for Figure 2 The operation S220 shown decodes the target fusion features to generate corrected text, including: fusing audio fusion features based on a self-attention mechanism to obtain a first fusion feature; fusing the first fusion feature and semantic fusion features based on an interactive attention mechanism to generate a second fusion feature; and recognizing the second fusion feature to obtain corrected text.

[0068] When the target fusion features include both audio fusion features and semantic fusion features, self-attention interaction can be applied to the audio fusion features first. This allows the decoding process to focus more on the semantic relevance between contextual and acoustic information. Understandably, the self-attention mechanism can identify the correlation between different elements within the audio fusion feature. By calculating the attention weights between each element and all other elements, a first fusion feature is generated that enhances key sound and contextual information. This helps to more accurately extract and utilize key information in the audio during subsequent fusion and recognition steps.

[0069] After obtaining the first fusion feature, an interactive attention mechanism can be used to fuse the first fusion feature with the semantic fusion feature to generate a second fusion feature. It is understood that the interactive attention mechanism can analyze the correlation between the key acoustic and contextual information in the first fusion feature and the semantic information in the semantic fusion feature. By dynamically allocating weights, it ensures that the correlation between these information is fully captured during the fusion process, thereby generating a second fusion feature containing more comprehensive and accurate semantic information.

[0070] After obtaining the second fusion feature, it can be identified to obtain the corrected text. In one embodiment, the second fusion feature can be decoded, and the semantic information in the second fusion feature can be converted into corrected text in natural language form using a decoder. The corrected text includes a location identifier for identifying the correction location and a correction field for the correction content.

[0071] According to embodiments of this disclosure, when the target fusion features include audio fusion features and semantic fusion features, by first fusing the audio fusion features based on a self-attention mechanism to enhance key sound information and key context information, and then fusing the first fusion feature with the semantic fusion feature based on an interactive attention mechanism to generate the second fusion feature, the correlation between key acoustic information, key context information and semantic information is further integrated. This makes the second fusion feature incorporate richer and more accurate sound information, speech information and context information. As a result, after recognizing and decoding the second fusion feature, accurate corrected text can be generated, thereby effectively improving the accuracy and quality of speech recognition and meeting the requirements for speech recognition accuracy in different scenarios.

[0072] Figure 4A The illustration shows a schematic diagram of generating modified text according to an embodiment of the present disclosure.

[0073] like Figure 4A As shown, based on the interactive attention mechanism, audio feature 40 and text feature 20 are fused to obtain audio fusion feature 410. Based on the interactive attention mechanism, semantic feature 50 and text feature 20 are fused to obtain semantic fusion feature 420.

[0074] like Figure 4A As shown, based on the self-attention mechanism, the audio fusion feature 410 is fused to obtain the first fusion feature 430. Based on the interactive attention mechanism, the first fusion feature 430 and the semantic fusion feature 420 are fused to obtain the second fusion feature 440. The second fusion feature 440 is then recognized to obtain the corrected text 450.

[0075] According to embodiments of this disclosure, the target fusion features include audio enhancement features and semantic fusion features. For Figure 2 The operation S210 shown above performs semantic fusion of the audio features of the audio to be processed, the text features of the target text, and the semantic features of the audio to be processed to obtain the target fusion features. It also includes fusing the audio fusion features with the audio features to obtain the audio enhancement features.

[0076] In the embodiments of this disclosure, after fusing audio features and text features based on an interactive attention mechanism to obtain audio fusion features, the audio fusion features can be further fused with the audio features to obtain audio enhancement features. It is understood that fusing audio features with audio fusion features can reduce the loss of the original audio due to the interactive attention mechanism and preserve more details and features from the original audio.

[0077] For example, in certain complex sound environments, audio features may contain subtle acoustic changes that are difficult to capture through the interactive attention mechanism fusion process, while audio enhancement features can better preserve these changes, thereby improving the accuracy and robustness of speech recognition.

[0078] Meanwhile, the processing of semantic fusion features remains unchanged, still based on the interactive attention mechanism to fuse semantic features with text features, resulting in features representing the contextual information of the fused target text and the semantic information of the audio to be processed. In this way, the target fusion features are composed of audio enhancement features and semantic fusion features, providing more comprehensive and accurate information for subsequent decoding and generation of corrected text.

[0079] According to embodiments of this disclosure, by fusing audio fusion features with audio features to obtain audio enhancement features, the audio information can be further enriched while preserving the key details and features of the original audio. Furthermore, by constructing the target fusion feature as a composite feature containing audio enhancement features and semantic fusion features, the generation of the target fusion feature considers not only the acoustic characteristics of the audio to be processed but also its interaction with the contextual information of the target text, as well as the integrity of the audio to be processed. Therefore, in the subsequent decoding process, the audio enhancement features can provide more accurate and detailed sound information, helping to generate more precise corrected text and improving the accuracy and quality of speech recognition.

[0080] According to embodiments of this disclosure, when the target fusion features include audio enhancement features and semantic fusion features, for Figure 2 The operation S220 shown decodes the target fusion features to generate corrected text, including: fusing audio enhancement features based on a self-attention mechanism to obtain a third fusion feature; fusing the third fusion feature and semantic fusion features based on an interactive attention mechanism to generate a fourth fusion feature; and recognizing the fourth fusion feature to obtain corrected text.

[0081] When the target fusion features include both audio enhancement features and semantic fusion features, the process of decoding the target fusion features to generate corrected text is similar to the case where the target fusion features include both audio and semantic fusion features. Similarly, a self-attention mechanism is first used to process the audio enhancement features. Understandably, the self-attention mechanism analyzes the inherent relationships between the elements within the audio enhancement features. By calculating the attention weights between each element and all other elements, it generates a third fusion feature that highlights key sound details and key contextual information. This ensures that in subsequent operations, key information in the audio can be captured and utilized more accurately.

[0082] After obtaining the third fusion feature, an interactive attention mechanism can be used to fuse the third fusion feature with the semantic fusion feature to generate a fourth fusion feature. Understandably, the interactive attention mechanism analyzes the correlation between key acoustic details and key contextual information in the third fusion feature and the rich semantic information in the semantic fusion feature. By dynamically adjusting the weights, it ensures that the correlation between these information can be obtained during the fusion process, thereby generating a fourth fusion feature containing more comprehensive and accurate semantic information.

[0083] After obtaining the fourth fusion feature, it can be identified to obtain the corrected text. In one embodiment, the semantic information in the fourth fusion feature can be converted into corrected text in natural language form by a decoder. The corrected text includes a location identifier for identifying the correction location and a correction field for the correction content.

[0084] According to embodiments of this disclosure, when the target fusion feature includes audio enhancement features and semantic fusion features, by first fusing the audio enhancement features based on a self-attention mechanism to enhance key sound detail information and key context information, and then fusing the third fusion feature with the semantic fusion feature based on an interactive attention mechanism to generate the fourth fusion feature, the correlation between key sound detail information, key context information and semantic information is further integrated. This makes the fourth fusion feature incorporate richer and more accurate sound detail information, speech information and context information. As a result, after recognizing and decoding the fourth fusion feature, more accurate corrected text can be generated, thereby effectively improving the accuracy and quality of speech recognition and meeting the requirements for speech recognition accuracy in different scenarios.

[0085] Figure 4B A schematic diagram illustrating the generation of modified text according to another embodiment of this disclosure is shown.

[0086] like Figure 4BAs shown, based on the interactive attention mechanism, audio feature 40 and text feature 20 are fused to obtain audio fusion feature 410. Audio fusion feature 410 is then fused with audio feature 30 to obtain audio enhancement feature 460. Based on the interactive attention mechanism, semantic feature 40 and text feature 20 are fused to obtain semantic fusion feature 420.

[0087] like Figure 4B As shown, based on the self-attention mechanism, the audio enhancement feature 460 is fused to obtain the third fused feature 470. Based on the interactive attention mechanism, the third fused feature 470 and the semantic fused feature 420 are fused to obtain the fourth fused feature 480. The fourth fused feature 480 is then recognized to obtain the corrected text 450.

[0088] According to embodiments of this disclosure, the correction field includes at least one correction word, and the initial text includes an initial word sequence. For Figure 2 The operation S230 shown involves correcting the initial text using a correction field according to the position identifier to obtain the target recognition text. This includes: determining at least one initial word located at the correction position from the initial word sequence according to the position identifier; obtaining the confidence of at least one correction word and the confidence of at least one initial word; and in response to determining that the confidence of at least one correction word is greater than the confidence of at least one initial word, replacing at least one initial word with at least one correction word to obtain the target recognition text.

[0089] When correcting the initial text using the corrected text, in order to further improve the accuracy of the speech recognition process, the confidence of the corrected word can be compared with the confidence of the initial word. Only when the confidence of the corrected word is greater than that of the initial word will the corrected word replace the initial word.

[0090] Understandably, confidence level reflects the reliability of the revised or initial word. By comparing the confidence levels of the revised and initial words, it is possible to avoid introducing erroneous information during the revision process and ensure that a replacement operation is only performed when the revised word is more reliable than the initial word.

[0091] For example, when using a large speech recognition model to obtain the initial text from the audio to be processed, the initial text may be misrecognized because the large speech recognition model cannot identify timely hot words. The corrected words in the corrected text may be timely hot words with higher confidence. In this case, replacing the initial words with the corrected words can effectively improve the accuracy of speech recognition.

[0092] According to embodiments of this disclosure, the initial word is replaced only when the confidence level of the modified word is greater than that of the initial word. This avoids errors that may be introduced by blind replacement and ensures that replacement is only performed when the modified word is indeed more reliable than the initial word, thereby generating the final target recognition text and improving the accuracy of the target recognition text.

[0093] Figure 5 The illustration shows a schematic diagram of the modification of initial text according to an embodiment of the present disclosure.

[0094] like Figure 5 As shown, the length of the corrected text can be the same as the length of the initial text, and the position identifier can be the placeholder "#". Taking the corrected text as "##Weather in Location A####", the corrected word as "Weather in Location A", and the initial text as "What will the weather quality be like in Location B in the future" as an example, the initial word at the corrected position is "Weather in Location B". When the confidence level of the corrected word "Weather in Location A" is 0.7 and the confidence level of the initial word "Weather in Location B" at the corrected position is 0.65, the initial word "Weather in Location B" can be replaced with the corrected word "Weather in Location A" to obtain the target recognition text "What will the weather quality be like in Location A in the future".

[0095] Figure 6 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.

[0096] like Figure 6 As shown, the model training method includes operations S610 to S630.

[0097] In operation S610, the correction module is used to perform semantic fusion of the sample audio features, sample text features, and sample semantic features of the sample audio to obtain sample fusion features.

[0098] When operating the S620, the correction module is used to decode the sample fusion features and generate sample correction text.

[0099] When operating the S630, based on the first loss function, the parameters of the speech recognition module are fixed, and the correction module is trained using sample corrected text to obtain the trained target model.

[0100] In embodiments of this disclosure, the target model for implementing the speech recognition method includes a correction module and a speech recognition module. Before using the target model for speech recognition, the target model can be trained to improve the accuracy of speech recognition, resulting in a trained target model, which is then used for speech recognition.

[0101] The correction module in the target model can fuse sample audio features, sample text features, and sample semantic features to obtain sample fused features, and then decode the sample fused features. Meanwhile, the speech recognition module in the target model can encode sample audio to obtain sample audio features, and decode the sample audio features to obtain sample semantic features.

[0102] According to embodiments of this disclosure, there is a semantic relationship between sample text and sample audio; the sample fusion feature integrates the sound information and semantic information of the sample audio, as well as the contextual information of the sample text; the sample semantic feature is obtained by decoding the sample audio feature using a speech recognition module.

[0103] The corrected sample text includes a sample location identifier and a sample correction field for identifying the correction location. In some embodiments, the correction module performs semantic fusion on the sample audio features, sample text features, and sample semantic features of the sample audio to obtain sample fusion features. The decoding process of the sample fusion features is similar to the process of obtaining and decoding the target fusion features in speech recognition methods, and will not be described in detail here.

[0104] Understandably, the target model has a dual-decoder structure. The correction module decodes the fused features of the samples, while the speech recognition module decodes the audio features of the samples. This decouples the correction module and the speech recognition module, allowing the target model to be trained by training only the correction module.

[0105] The speech recognition module can be a pre-trained large-scale speech recognition model. During training, the parameters of the speech recognition module can be fixed, and the module can be trained using only sample corrected text. Specifically, a first loss function can be used to determine the loss value of the sample corrected text, and the parameters of the correction module can be adjusted based on this loss value.

[0106] Once the correction module is trained, it can be combined with the speech recognition module to obtain a trained target model. The trained target model can use the correction module to correct the output of the speech recognition module, thereby effectively improving the accuracy and quality of speech recognition.

[0107] According to embodiments of this disclosure, by setting the target model as a dual-decoding structure, the speech recognition module and the correction module are decoupled. When training the target model, only the correction module needs to be trained, eliminating the need for joint debugging and optimization, resulting in faster iteration speed and reduced training difficulty. Furthermore, the dual-decoder structure of the target model also offers high flexibility and scalability. The correction module and speech recognition module can be disassembled and iteratively optimized, facilitating maintenance. Specifically, the correction module can be trained specifically according to different speech recognition scenarios and needs without requiring large-scale adjustments to the entire target model. Simultaneously, the speech recognition module can be replaced and upgraded according to different audio encoding and decoding requirements to adapt to different types and qualities of audio input. Therefore, this dual-decoding structure design enables the target model to better adapt to diverse speech recognition application scenarios and meet different types of speech recognition needs.

[0108] According to embodiments of this disclosure, the model training method further includes: using a large language model to generate sample text based on associated text related to the sample audio.

[0109] In the embodiments of this disclosure, the powerful text generation capabilities of a large language model can be utilized to generate sample text. Furthermore, to improve the accuracy of sample generation, associated text related to the sample audio can be provided to the large language model for sample text generation, rather than directly providing the sample audio to the large language model. This avoids errors in the large language model's speech recognition process, which could lead to inaccurate sample text. The associated text can be the actual text content represented by the sample audio. For example, the associated text could be "What will the weather quality be like in location A in the future?", and the sample audio could be a reading of that text.

[0110] In one example, the database can store sample audio and associated text together. When it is necessary to generate sample text corresponding to the sample audio, the corresponding associated text can be determined from the database based on the relationship between the sample audio and the associated text.

[0111] After determining the associated text, sample text that has a contextual semantic relationship with the associated text can be generated based on the associated text. In some embodiments, the associated text can be input into a prompt template used to guide the large language model to generate sample text, prompt information can be obtained, and the prompt information can be input into the large language model to obtain the sample text output by the large language model.

[0112] For example, a prompt containing the related text "What will the weather quality be like in location A in the future?" can be input into a large language model to guide it to output sample text that has a contextual semantic relationship with the related text, such as "The weather in location A is pretty good this weekend, suitable for going to the park...".

[0113] According to embodiments of this disclosure, by utilizing a large language model to generate sample text based on associated text, the accuracy and reliability of sample text generation are improved. Furthermore, the content and style of the associated text can be adjusted according to different speech recognition scenarios and needs, thereby generating more diverse sample text and further enhancing the generalization ability and recognition accuracy of the target model.

[0114] According to embodiments of this disclosure, a large language model is used to generate sample text based on associated text related to sample audio, including: segmenting the associated text related to sample audio into multiple text segments; and using the large language model to generate sample text that has a semantic relationship with any one of the multiple text segments.

[0115] In the process of generating sample text, the related text can first be segmented into multiple text segments. In one example, word segmentation tools such as the bag-of-words model can be used to segment the related text, and there are no restrictions on the specific word segmentation method.

[0116] After segmenting the associated text into multiple text segments, you can select the text segment of interest from the multiple text segments according to actual needs, and use the text segment to generate sample text, thereby achieving "customization" of the text segment to be corrected and the correction position.

[0117] For example, for the related text "What will the weather quality be like in location A in the future", multiple text segments can be obtained, such as "future", "location A in the future", and "weather in location A". Based on the most recent trending words, the text segment "weather in location A" can be selected to generate sample text. This allows the correction module to be trained using the sample text, thereby improving the module's ability to identify outdated trending words.

[0118] For example, for the related text "The recognition model significantly improves the long-distance feature extraction capability in natural language processing tasks through the attention mechanism", multiple text segments such as "recognition model", "through" and "attention mechanism" can be obtained. Based on computer science terminology, the text segment "attention mechanism" can be selected for generating sample text. This allows the correction module to be trained using the sample text, thereby improving the module's ability to recognize computer science vocabulary.

[0119] After determining the text segment to be used to generate sample text, the text segment can be input into the prompt template to obtain prompt information, which can then be used to guide the large language model to generate sample text based on the text segment.

[0120] In embodiments of this disclosure, the sample text may include a text segment used to generate the sample text. For example, if the selected text segment is "weather in location A", the generated sample text could be "The weather in location A was pretty good this weekend, ...". As another example, if the selected text segment is "attention mechanism", the generated sample text could be "Attention mechanism allows the model to focus on key information".

[0121] According to embodiments of this disclosure, by segmenting associated text and selecting specific text segments to generate sample text, it is possible to more accurately "customize" the specific text segments that need to be corrected and the corresponding correction positions. In the process of generating sample text using specific text segments, the correction module's ability to identify and correct specific types of words can be improved more effectively, thereby improving the accuracy of the speech recognition process.

[0122] According to embodiments of this disclosure, the model training method further includes: replacing characters in the sample text other than any text segment with position identifiers to obtain sample labels.

[0123] Accordingly, after determining the specific text segment used to generate the sample text, the characters in the sample text other than that text segment can be replaced with position identifiers to obtain sample labels, so that when training with the sample labels, the correction module can learn more contextual knowledge through the training process.

[0124] In some embodiments, the location identifier can be a placeholder "#". Taking the associated text as "What will the weather quality be like in location A in the future", and the selected text segment used to generate the sample text as "Weather in location A" as an example, both "future" and "what will the quality be like" need to be replaced with placeholders. The constructed sample label can be "##Weather in location A####".

[0125] According to embodiments of this disclosure, sample labels are obtained by replacing characters in the sample text other than specific text segments with location identifiers. The sample labels contain location information of specific text segments, so that when the correction module is trained using the sample labels, the correction module can learn more contextual information based on this location information, thereby improving the accuracy and quality of speech recognition.

[0126] Figure 7 The illustration shows a schematic diagram of the generation of sample labels and sample text according to an embodiment of the present disclosure.

[0127] like Figure 7As shown, the associated text is "What will the weather quality be like in location A in the future?". The associated text is divided into multiple segments such as "future", "location A in the future", and "weather in location A", with "weather in location A" selected for sample text generation. After processing the text segment "weather in location A" using a large language model, the sample text "The weather in location A is pretty good this weekend..." is obtained. Then, characters other than "weather in location A" in the associated text "What will the weather quality be like in location A in the future?" are replaced with placeholders, resulting in the sample label "##weather in location A####".

[0128] According to embodiments of this disclosure, based on a first target loss function, the parameters of the speech recognition module are fixed, and the correction module is trained using sample corrected text to obtain a trained target model, including: processing the sample corrected text and sample labels based on the first loss function to obtain a first loss value; and based on the first loss value, fixing the parameters of the speech recognition module, adjusting the parameters of the correction module to obtain a trained target model.

[0129] When training the correction module, the first loss function can be used to determine the first loss value between the sample corrected text output by the correction module and the sample label. Based on the first loss value, the parameters of the correction module can be adjusted in a targeted manner while keeping the parameters of the speech recognition module unchanged.

[0130] Understandably, the first loss value can be used to define the error between the corrected sample text and the sample label. The smaller the first loss value, the closer the corrected sample text is to the sample label, meaning the more accurate the output of the correction module is. By continuously adjusting the parameters of the correction module, the first loss value is gradually reduced, thereby improving the performance of the correction module.

[0131] It is understandable that the performance of the correction module includes, but is not limited to, correction accuracy and correction speed.

[0132] According to embodiments of this disclosure, by adjusting the parameters of the correction module based on the first loss value between the sample corrected text and the sample label, the correction module can gradually learn how to generate corrected text more accurately based on the sample audio features, sample text features and sample semantic features, thereby improving its ability to correct speech recognition results.

[0133] According to embodiments of this disclosure, the model training method further includes: using a decision module to correct the initial sample text according to the sample location identifier using a sample correction field to obtain the target sample text; wherein the initial sample text is obtained by decoding semantic features using a speech recognition module; and training the target model using the target sample text based on a second loss function to obtain the trained target model.

[0134] In embodiments of this disclosure, the target model further includes a decision module. The decision module can be used to correct the initial sample text using the corrected sample text. In some embodiments, the method by which the decision module corrects the initial sample text using the corrected sample text is the same as the method in speech recognition methods for correcting the initial text using corrected text, and will not be described again here.

[0135] The target text can be the final recognition result output by the target model when performing speech recognition on the sample audio. When training the target model using the target text, a second loss function can be used to determine the loss value between the target text and the associated text, and this loss value can be used to adjust the parameters of the target model.

[0136] For example, the parameters of the decision module, correction module, and speech recognition module can be adjusted simultaneously using the loss value between the target text and associated text. Alternatively, to improve training efficiency, the parameters of the speech recognition module can be fixed, and only the parameters of the decision module and correction module can be adjusted.

[0137] According to embodiments of this disclosure, by further training the target model using a second loss function, the overall performance of the correction module, decision module, and speech recognition module can be comprehensively considered, making the sample target text output by the target model closer to the actual associated text, thereby improving the accuracy and reliability of the target model in speech recognition tasks.

[0138] Figure 8 A schematic diagram of the structure of a target model according to an embodiment of the present disclosure is shown.

[0139] like Figure 8 As shown, the target model includes a correction module 810, a speech recognition module 820, and a decision module 830.

[0140] like Figure 8 As shown, the correction module 810 includes a bias encoder 811 and a bias decoder 812. The bias encoder 811 may include a bias summing layer 811_1 and a context coding layer 811_2. The bias decoder 812 may include a speech-semantic interaction layer 812_1 and a decoding layer 812_2. Figure 8 As shown, the speech recognition module 820 includes an encoder 821, a decoder 822, and a classifier 823.

[0141] like Figure 8As shown, the target text is input into the bias encoder 811 to obtain text features. The audio to be processed is input into the encoder 821 to obtain audio features. The audio features are input into the decoder 822 to obtain semantic features. The semantic features are input into the classifier 823 to obtain the initial text. The text features, audio features, and semantic features are input into the speech-semantic interaction layer 812_1 to obtain the target fusion features. The target fusion features are input into the decoding layer 812_2 to obtain the corrected text. The initial text and the corrected text are input into the decision module 830 to obtain the target recognition text.

[0142] According to an embodiment of this disclosure, a target model is trained using sample target text based on a second loss function to obtain a trained target model, including: fixing the parameters of the speech recognition module and the correction module based on the second loss function, and training the decision module using sample target text to obtain a trained target model.

[0143] In the embodiments of this disclosure, since the parameters of the correction module have been adjusted using the first loss value, the parameters of the speech recognition module and the correction module can be fixed, and only the decision module can be trained to improve the overall training efficiency.

[0144] When training the decision module, a second loss value can be determined based on the sample target text and the second loss function. The parameters of the decision module can then be adjusted based on the second loss value to obtain a trained decision module. The speech recognition module, the correction module, and the trained decision module can then be combined to obtain a trained target model.

[0145] According to embodiments of this disclosure, by fixing the parameters of the speech recognition module and the correction module, and training only the decision module, the decision module can be optimized centrally while ensuring the stable performance of the speech recognition and correction modules. This not only reduces training complexity but also more accurately improves the decision module's ability to correct the initial text based on the corrected text, further enhancing the speech recognition performance of the target model.

[0146] According to embodiments of this disclosure, based on a second loss function, the parameters of the speech recognition module and the correction module are fixed, and the decision module is trained using sample target text to obtain a trained target model. This includes: processing the sample target text and associated text related to the sample audio based on the second loss function to obtain a second loss value; and adjusting the parameters of the decision module based on the second loss value, while fixing the parameters of the speech recognition module and the correction module, to obtain the trained target model.

[0147] When training the decision module using sample target text, the associated text related to the sample audio can be used as a label. The second loss function is used to determine the second loss value between the sample target text and the associated text, and the parameters of the decision module are adjusted based on the second loss value.

[0148] According to embodiments of this disclosure, by using associated text as labels to determine the second loss value, the gap between the sample target text and the true semantics can be measured more directly, thereby providing a more accurate basis for adjusting the parameters of the decision module. This training method helps the decision module learn more accurately how to use corrected text to correct the initial text in order to obtain sample target text that is closer to the true semantics, thereby improving the accuracy of the target model in speech recognition.

[0149] According to embodiments of this disclosure, a target model is trained using sample target text based on a second loss function to obtain a trained target model, including: processing the sample target text and associated text related to the sample audio based on the second loss function to obtain a second loss value; and fixing the parameters of the speech recognition module, adjusting the parameters of the decision module and the correction module based on the second loss function to obtain the trained target model.

[0150] In embodiments of this disclosure, in order to further improve the accuracy of the target model in speech recognition, the decision module and the correction module can also be trained simultaneously using a second loss value.

[0151] Similar to the process of training only the decision module, when training both the decision module and the correction module simultaneously, the associated text related to the sample audio can be used as a label. The second loss function is used to determine the second loss value between the target sample text and the associated text, and the parameters of the decision module and the correction module are adjusted based on the second loss value.

[0152] According to embodiments of this disclosure, by using associated text as labels and simultaneously adjusting the parameters of the decision-making module and the correction module based on a second loss value, the synergistic effect of the two modules in the speech recognition process can be comprehensively considered. This training method helps the decision-making module and the correction module cooperate better. The decision-making module can more effectively use the corrected text generated by the correction module to correct the initial text, while the correction module can further optimize its ability to generate corrected text based on feedback from the decision-making module, thereby improving the overall accuracy and reliability of the target model in speech recognition.

[0153] Figure 9 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown schematically.

[0154] like Figure 9As shown, the speech recognition device 900 includes a semantic fusion module 910, a fusion decoding module 920, and a text correction module 930.

[0155] The semantic fusion module 910 is used to semantically fuse the audio features of the audio to be processed, the text features of the target text, and the semantic features of the audio to be processed to obtain the target fusion features; wherein, there is a semantic relationship between the target text and the audio to be processed; the target fusion features fuse the sound information and semantic information of the audio to be processed, as well as the context information of the target text; the semantic features are obtained by decoding the audio features.

[0156] The fusion decoding module 920 is used to decode the target fusion features and generate corrected text; wherein, the corrected text includes a location identifier and a correction field for identifying the correction position.

[0157] The text correction module 930 is used to correct the initial text according to the location identifier using the correction field to obtain the target recognition text; wherein, the initial text is obtained by decoding semantic features.

[0158] According to embodiments of this disclosure, the target fusion features include: audio fusion features and semantic fusion features. The semantic fusion module 910 includes an audio fusion submodule and a semantic fusion submodule.

[0159] The audio fusion submodule is used to fuse audio features with text features based on an interactive attention mechanism to obtain audio fusion features; the audio fusion features represent the contextual information of the fused target text and the acoustic information of the audio to be processed.

[0160] The semantic fusion submodule is used to fuse semantic features with text features based on the interactive attention mechanism to obtain semantic fusion features; the semantic fusion features represent the contextual information of the fused target text and the semantic information of the audio to be processed.

[0161] According to embodiments of this disclosure, the target fusion features include audio enhancement features and semantic fusion features. The semantic fusion module 910 further includes an enhancement fusion submodule.

[0162] The enhanced fusion submodule is used to fuse audio fusion features with audio features to obtain audio enhancement features.

[0163] According to embodiments of this disclosure, the target fusion features include: audio fusion features and semantic fusion features. The fusion decoding module 920 includes a first fusion submodule, a second fusion submodule, and a first recognition submodule.

[0164] The first fusion submodule is used to fuse audio fusion features based on a self-attention mechanism to obtain the first fusion feature.

[0165] The second fusion submodule is used to fuse the first fusion feature and the semantic fusion feature based on the interactive attention mechanism to generate the second fusion feature.

[0166] The first recognition submodule is used to recognize the second fused feature to obtain the corrected text.

[0167] According to embodiments of this disclosure, the target fusion features include audio enhancement features and semantic fusion features. The fusion decoding module 920 includes a third fusion submodule, a fourth fusion submodule, and a second recognition submodule.

[0168] The third fusion submodule is used to fuse audio enhancement features based on a self-attention mechanism to obtain the third fused feature.

[0169] The fourth fusion submodule is used to fuse the third fusion feature and the semantic fusion feature based on the interactive attention mechanism to generate the fourth fusion feature.

[0170] The second recognition submodule is used to recognize the fourth fusion feature to obtain the corrected text.

[0171] According to embodiments of this disclosure, the correction field includes at least one correction word, and the initial text includes an initial word sequence. The text correction module 930 includes a correction determination submodule, a confidence level acquisition submodule, and a replacement submodule.

[0172] The correction determination submodule is used to determine at least one initial word located at a correction position from the initial word sequence according to the position identifier.

[0173] The confidence acquisition submodule is used to acquire the confidence of at least one corrected word and the confidence of at least one initial word.

[0174] The replacement submodule is used to replace at least one sample initial word with at least one correction word in response to determining that the confidence of at least one correction word is greater than the confidence of at least one initial word, thereby obtaining the target recognition text.

[0175] Figure 10 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically.

[0176] like Figure 10 As shown, the model training device 1000 includes a sample fusion module 1010, a sample decoding module 1020, and a correction training module 1030.

[0177] The sample fusion module 1010 is used to perform semantic fusion of the sample audio features, sample text features, and sample semantic features of the sample audio using the correction module to obtain sample fusion features; wherein, there is a semantic relationship between the sample text and the sample audio; the sample fusion features fuse the sound information and semantic information of the sample audio, as well as the context information of the sample text; the sample semantic features are obtained by decoding the sample audio features using the speech recognition module.

[0178] The sample decoding module 1020 is used to decode the sample fusion features using the correction module to generate sample correction text; wherein, the sample correction text includes a sample location identifier and a sample correction field for identifying the correction position.

[0179] The training module 1030 is used to train the correction module based on the first loss function, with the parameters of the speech recognition module fixed, and the correction module is trained using sample correction text to obtain the trained target model.

[0180] According to embodiments of this disclosure, the model training apparatus 1000 further includes a sample generation module.

[0181] The sample generation module is used to generate sample text based on associated text related to the sample audio using a large language model; where the sample text and the associated text have a contextual semantic relationship.

[0182] According to embodiments of this disclosure, the sample generation module includes a word segmentation submodule and a generation submodule.

[0183] The word segmentation submodule is used to segment the associated text related to the sample audio into multiple text segments.

[0184] The generation submodule is used to generate sample text that has a semantic relationship with any of the multiple text segments based on a large language model.

[0185] According to embodiments of this disclosure, the sample generation module further includes a label generation submodule.

[0186] The tag generation submodule is used to replace characters in the sample text except for any text segment with location identifiers to obtain sample tags.

[0187] According to embodiments of this disclosure, the corrected training module 1030 includes a first loss determination submodule and a first parameter adjustment submodule.

[0188] The first loss determination submodule is used to process the sample corrected text and sample labels based on the first loss function to obtain the first loss value.

[0189] The first parameter adjustment submodule is used to fix the parameters of the speech recognition module and adjust the parameters of the correction module based on the first loss value to obtain the trained target model.

[0190] According to embodiments of this disclosure, the model training apparatus 1000 further includes a sample correction module and a target training module.

[0191] The sample correction module is used by the decision module to correct the initial sample text according to the sample location identifier using the sample correction field to obtain the target sample text; wherein, the initial sample text is obtained by decoding the semantic features using the speech recognition module.

[0192] The target training module is used to train the target model using sample target text based on the second loss function, so as to obtain the trained target model.

[0193] According to embodiments of this disclosure, the target training module includes a target training submodule.

[0194] The target training submodule is used to train the decision module using sample target text based on the second loss function, fixing the parameters of the speech recognition module and the correction module, to obtain the trained target model.

[0195] According to embodiments of this disclosure, the target training submodule includes a loss determination unit and a parameter adjustment unit.

[0196] The loss determination unit is used to process the sample target text and the associated text related to the sample audio based on the second loss function to obtain the second loss value.

[0197] The parameter adjustment unit is used to adjust the parameters of the decision module based on the second loss value, fixing the parameters of the speech recognition module and the correction module, to obtain the trained target model.

[0198] According to embodiments of this disclosure, the target training module further includes a second loss determination submodule and a second parameter adjustment submodule.

[0199] The second loss determination submodule is used to process the sample target text and the associated text related to the sample audio based on the second loss function to obtain the second loss value.

[0200] The second parameter adjustment submodule is used to fix the parameters of the speech recognition module, adjust the parameters of the decision module and the correction module based on the second loss function, and obtain the trained target model.

[0201] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0202] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0203] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0204] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0205] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0206] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded into random access memory (RAM) 1103 from storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0207] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0208] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as speech recognition methods. For example, in some embodiments, the speech recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the speech recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the speech recognition method by any other suitable means (e.g., by means of firmware).

[0209] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0210] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0211] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0212] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0213] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0214] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0215] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0216] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An audio recognition method, comprising: Based on the interactive attention mechanism, the audio features of the audio to be processed and the text features of the target text are fused to obtain the audio fusion features in the target fusion features; Based on the interactive attention mechanism, the semantic features of the audio to be processed and the text features are semantically fused to obtain the semantic fusion features in the target fusion features; wherein, there is a semantic relationship between the target text and the audio to be processed; the audio fusion features represent the features of the fused context information of the target text and the acoustic information of the audio to be processed; the semantic fusion features represent the features of the fused context information and the semantic information of the audio to be processed; the semantic features are obtained by decoding the audio features; the target text is obtained by inputting the audio to be processed into a large language model, so that the speech recognition process can adapt to the latest hot words and information or the requirements of specific fields; The target fusion features are decoded to generate corrected text; wherein, the corrected text includes a location identifier for identifying the correction location and a correction field; the correction field is the correct text content obtained after accurately correcting the content at the error location based on the semantic information contained in the target fusion features; and According to the location identifier, the initial text is corrected using the correction field to obtain the target recognition text; wherein, the initial text is obtained by decoding the semantic features.

2. The method according to claim 1, wherein, The method further includes: The audio fusion feature is fused with the audio feature to obtain the audio enhancement feature.

3. The method according to claim 1, wherein, Decoding the target fusion features to generate corrected text includes: The audio fusion features are fused based on a self-attention mechanism to obtain a first fusion feature; The first fusion feature and the semantic fusion feature are fused based on the interactive attention mechanism to generate a second fusion feature; and The second fusion feature is identified to obtain the corrected text.

4. The method according to claim 2, wherein, The target fusion features include audio enhancement features and semantic fusion features; Decoding the target fusion features to generate corrected text includes: The audio enhancement features are fused based on a self-attention mechanism to obtain a third fused feature; The third fusion feature and the semantic fusion feature are fused based on the interactive attention mechanism to generate a fourth fusion feature; and The fourth fusion feature is identified to obtain the corrected text.

5. The method according to any one of claims 1-4, wherein, The correction field includes at least one correction word, and the initial text includes an initial word sequence; The step of correcting the initial text according to the location identifier and using the correction field to obtain the target recognition text includes: Based on the location identifier, determine at least one initial word located at the corrected position from the initial word sequence; Obtain the confidence scores of the at least one modified word and the at least one initial word; and In response to determining that the confidence level of the at least one modified word is greater than the confidence level of the at least one initial word, the at least one modified word is used to replace the at least one initial word to obtain the target recognition text.

6. A model training method, wherein the target model includes a correction module and a speech recognition module, comprising: Using the aforementioned correction module, the sample audio features of the sample audio and the sample text features of the sample text are fused based on the interactive attention mechanism to obtain the sample audio fusion features in the sample fusion features. Using the correction module, the sample semantic features of the sample audio and the sample text features are semantically fused based on the interactive attention mechanism to obtain the sample semantic fusion features in the sample fusion features; wherein, there is a semantic relationship between the sample text and the sample audio; the sample audio fusion features represent the features of the sample context information and the acoustic information of the sample audio that have been fused; the sample semantic fusion features represent the features of the sample context information and the semantic information of the sample audio that have been fused; the sample semantic features are obtained by decoding the sample audio features using the speech recognition module; the sample text is obtained by inputting the sample audio into a large language model, so that the speech recognition process can adapt to the latest hot words and information or the requirements of specific fields; The correction module decodes the sample fusion features to generate corrected sample text; wherein, the corrected sample text includes a sample position identifier for identifying the correction location and a sample correction field; the sample correction field is the correct sample text content obtained after accurately correcting the content at the sample error location based on the semantic information contained in the sample fusion features; and Based on the first loss function, the parameters of the speech recognition module are fixed, and the correction module is trained using sample corrected text to obtain the trained target model.

7. The method according to claim 6, further comprising: Using a large language model, sample text is generated based on associated text related to the sample audio; wherein, the sample text and the associated text have a contextual semantic relationship.

8. The method according to claim 7, wherein, The step of generating sample text based on associated text related to the sample audio using a large language model includes: The associated text related to the sample audio is segmented into multiple text segments; and Using a large language model, a sample text with semantic association to any one of the multiple text segments is generated.

9. The method according to claim 8, further comprising: Replace all characters in the sample text except for any of the text segments with location identifiers to obtain sample labels.

10. The method according to claim 9, wherein, The step of fixing the parameters of the speech recognition module based on the first loss function and training the correction module using sample corrected text to obtain the trained target model includes: The sample corrected text and the sample label are processed based on the first loss function to obtain a first loss value; and Based on the first loss value, the parameters of the speech recognition module are fixed, and the parameters of the correction module are adjusted to obtain the trained target model.

11. The method according to any one of claims 6-10, wherein the target model further comprises a decision module, and the method further comprises: Using the decision module, the initial sample text is corrected according to the sample location identifier using the sample correction field to obtain the target sample text; wherein, the initial sample text is obtained by decoding the semantic features using the speech recognition module; and Based on the second loss function, the target model is trained using the sample target text to obtain the trained target model.

12. The method according to claim 11, wherein, The step of training the target model using the sample target text based on the second loss function to obtain the trained target model includes: Based on the second loss function, the parameters of the speech recognition module and the correction module are fixed, and the decision module is trained using sample target text to obtain the trained target model.

13. The method according to claim 12, wherein, The step of fixing the parameters of the speech recognition module and the correction module based on the second loss function, and training the decision module using sample target text to obtain the trained target model includes: Based on the second loss function, the sample target text and associated text related to the sample audio are processed to obtain a second loss value; and Based on the second loss value, the parameters of the speech recognition module and the correction module are fixed, and the parameters of the decision module are adjusted to obtain the trained target model.

14. The method according to claim 11, wherein, The step of training the target model using the sample target text based on the second loss function to obtain the trained target model includes: Based on the second loss function, the sample target text and associated text related to the sample audio are processed to obtain a second loss value; and Based on the second loss function, the parameters of the speech recognition module are fixed, and the parameters of the decision module and the correction module are adjusted to obtain the trained target model.

15. A voice recognition device, comprising: The semantic fusion module is used to fuse the audio features of the audio to be processed and the text features of the target text based on the interactive attention mechanism to obtain the audio fusion features in the target fusion features; Based on the interactive attention mechanism, the text features and the semantic features of the audio to be processed are semantically fused to obtain the implied fusion features in the target fusion features; wherein, there is a semantic relationship between the target text and the audio to be processed; the audio fusion features represent the features of the fused contextual information of the target text and the acoustic information of the audio to be processed; the semantic fusion features represent the features of the fused contextual information and the semantic information of the audio to be processed; the semantic features are obtained by decoding the audio features; the target text is obtained by inputting the audio to be processed into a large language model, so that the speech recognition process can adapt to the latest hot words and information or the requirements of specific fields; A fusion decoding module is used to decode the target fusion features and generate corrected text; wherein, the corrected text includes a location identifier for identifying the correction position and a correction field; the correction field is the correct text content obtained after accurately correcting the content at the error position based on the semantic information contained in the target fusion features; and The text correction module is used to correct the initial text according to the location identifier and using the correction field to obtain the target recognition text; wherein the initial text is obtained by decoding the semantic features.

16. A model training apparatus, wherein the target model includes a correction module and a speech recognition module, comprising: The sample fusion module is used to fuse the sample audio features of the sample audio and the sample text features of the sample text based on the interactive attention mechanism of the correction module to obtain the sample audio fusion features in the sample fusion features. Based on the interactive attention mechanism, the sample text features and sample audio semantic features are semantically fused to obtain the sample semantic fusion features in the sample fusion features; wherein, there is a semantic relationship between the sample text and the sample audio; the sample audio fusion features represent the features of the sample context information of the fused sample text and the acoustic information of the sample audio; the sample semantic fusion features represent the features of the fused sample context information and the semantic information of the sample audio; the sample semantic features are obtained by decoding the sample audio features using the speech recognition module; the sample text is obtained by inputting the sample audio into a large language model, so that the speech recognition process can adapt to the latest hot words and information or the requirements of specific fields; A sample decoding module is used to decode the sample fusion features using the correction module to generate corrected sample text; wherein, the corrected sample text includes a sample position identifier for identifying the correction position and a sample correction field; the sample correction field is the correct sample text content obtained after accurately correcting the content at the sample error position based on the semantic information contained in the sample fusion features; and The correction training module is used to fix the parameters of the speech recognition module based on the first loss function, and train the correction module using sample correction text to obtain the trained target model.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.

Images