Speech recognition method and device, equipment and medium

Through the generative large language model, the output of the speech recognition model is predicted and integrated, which solves the problem of low accuracy of the existing speech recognition model and improves the accuracy of speech recognition and context modeling capabilities.

CN118942462APending Publication Date: 2024-11-12LENOVO (BEIJING) LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411215280.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing speech recognition model has poor recognition capabilities and low accuracy of text recognition, especially due to insufficient reliance on manual design rules and context modeling capabilities, resulting in poor accuracy.

Method used

The generative large language model is used to predict the intermediate output results of the speech recognition model, and the final text information is comprehensively determined based on the prediction results and the output results of the speech recognition model. The speech recognition ability is enhanced through the large language model and the recognition accuracy is improved.

Benefits of technology

Through the introduction of large language models, the accuracy and output performance of the speech recognition model are improved, the contextual relationship modeling ability of speech recognition is enhanced, and the accuracy of the recognition results are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942462A_ABST
    Figure CN118942462A_ABST
Patent Text Reader

Abstract

The invention provides a voice recognition method, which comprises the following steps: recognizing a to-be-recognized voice segment in to-be-recognized voice information by using a voice recognition model to obtain at least one initial recognition result corresponding to the to-be-recognized voice segment; the to-be-recognized voice information comprises a plurality of voice segments; determining a target recognition result of the to-be-recognized voice segment according to the at least one initial recognition result based on target recognition results of other voice segments before the to-be-recognized voice segment in the to-be-recognized voice information by using the target large model; and determining text information corresponding to the to-be-recognized voice information according to the target recognition result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of large language model technology. More specifically, the present disclosure provides a speech recognition method, apparatus, device and medium. Background Art

[0002] Automatic Speech Recognition (ASR) technology can convert input audio into text. However, the recognition ability of current speech recognition models is not ideal, and the accuracy of text recognition is low. Summary of the invention

[0003] In view of this, the present disclosure provides a speech recognition method, apparatus, electronic device, medium, and program product.

[0004] One aspect of the present disclosure provides a speech recognition method, including: using a speech recognition model to recognize a speech segment to be recognized in speech information to be recognized, and obtaining at least one initial recognition result corresponding to the speech segment to be recognized; the speech information to be recognized includes multiple speech segments; using a target large model, based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized, determining the target recognition result of the speech segment to be recognized according to at least one initial recognition result; and determining the text information corresponding to the speech information to be recognized according to the target recognition result.

[0005] According to an embodiment of the present disclosure, a target large model is used to determine a target recognition result of a speech segment to be recognized based on target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized according to at least one initial recognition result, including: using a target large model to generate a prediction result based on target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized; determining a first probability of at least one initial recognition result based on the prediction result to determine the target recognition result of the speech segment to be recognized.

[0006] According to an embodiment of the present disclosure, a speech segment to be recognized in speech information to be recognized is recognized using a speech recognition model to obtain at least one initial recognition result corresponding to the speech segment to be recognized, including: a speech segment to be recognized in speech information to be recognized is recognized using a speech recognition model to obtain at least one initial recognition result corresponding to the speech segment to be recognized and a second probability of each initial recognition result; a first probability of at least one initial recognition result is determined according to the prediction result to determine a target recognition result of the speech segment to be recognized, including: according to the first probability and the second probability of at least one initial recognition result, the target recognition result of the speech segment to be recognized is determined from at least one initial recognition result.

[0007] According to an embodiment of the present disclosure, the method further includes: using a target large model to determine a target recognition result of the speech segment to be recognized based on at least one initial recognition result.

[0008] According to an embodiment of the present disclosure, the voice segment to be recognized includes a time period identifier, and the text information corresponding to the voice information to be recognized is determined according to the target recognition result, including: obtaining the text information corresponding to the voice information to be recognized according to the target recognition results corresponding to multiple voice segments to be recognized and the time period identifiers of the multiple voice segments to be recognized.

[0009] According to an embodiment of the present disclosure, the target large model is obtained by the following method: obtaining a sample corpus, the sample corpus includes annotated text, initial recognition results of speech fragment samples and / or processed corpus, the initial recognition results of the speech fragment samples are obtained by recognizing the speech fragment samples using a speech recognition model, and the processed corpus is obtained by intercepting and processing the initial recognition results of the speech fragments; adjusting the parameters of the initial large model according to the sample corpus and prompt information to obtain the target large model, and the target large model is suitable for the target task scenario corresponding to the sample corpus.

[0010] According to an embodiment of the present disclosure, parameters of an initial large model are adjusted according to sample corpus and prompt information to obtain a target large model, including: inputting the sample corpus and prompt information into the initial large model; the initial large model obtains an initial recognition result for the speech fragment sample and / or a generation result of the processed corpus based on the prompt information and the initial recognition result of the speech fragment sample and / or the processed corpus; based on the generated result and the annotated text, parameters of the initial large model are adjusted to obtain a target large model.

[0011] According to an embodiment of the present disclosure, the prompt information is used to instruct the initial large model to generate content after the initial recognition result of the speech segment sample and / or the specified position of the processing corpus.

[0012] According to an embodiment of the present disclosure, the annotated text includes a first time period identifier, and the initial recognition result of the speech segment sample includes a second time period identifier, and the first time period identifier corresponds to the second time period identifier.

[0013] Another aspect of the present disclosure also provides a speech recognition device, including: a recognition module, used to use a speech recognition model to recognize a speech segment to be recognized in the speech information to be recognized, and obtain at least one initial recognition result corresponding to the speech segment to be recognized; the speech information to be recognized includes multiple speech segments; a recognition result determination module, used to use a target large model, based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized, and determine the target recognition result of the speech segment to be recognized according to at least one initial recognition result; a text information determination module, used to determine the text information corresponding to the speech information to be recognized according to the target recognition result.

[0014] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0015] Another aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the above method.

[0016] Another aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 The application scenario diagram of the speech recognition method, device, electronic device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0020] Figure 2 The flowchart of the speech recognition method according to the embodiment of the present disclosure is schematically shown;

[0021] Figure 3 A schematic diagram schematically shows a speech recognition method according to another embodiment of the present disclosure;

[0022] Figure 4 A flowchart of a speech recognition method according to another embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A schematic diagram of target large model training according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 6 A structural block diagram of a speech recognition device according to an embodiment of the present disclosure is schematically shown; and

[0025] Figure 7 A schematic block diagram of an electronic device that can be used to implement the speech recognition method of an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0028] When the speech recognition model converts the input audio into text, there will be multiple candidate texts. The acoustic model, language model, pronunciation dictionary, etc. are used to select the text with the highest probability of being correct from the multiple candidate texts as the output result.

[0029] In some embodiments, an N-gram language model or a language model based on a recurrent neural network can be used to assist the output of the speech recognition model. The N-gram language model breaks down the text into multiple words and obtains the final result by determining the probability of each word appearing in turn, where N represents the probability of the current word appearing relative to the previous N words. This speech model relies on manually designed rules, is not accurate enough, and the context modeling capabilities of both language models are not ideal.

[0030] In another embodiment, a large language model can be used to score multiple text results output by the speech recognition model, and the final result can be selected after re-ranking. The accuracy of this solution is easily affected by the output performance of the speech recognition model. If there is no correct result in the text results output by the speech recognition model, the large language model cannot correct the result.

[0031] The disclosed embodiments provide a speech recognition method. The method predicts the intermediate output results of the speech recognition model by using a generative large language model, and comprehensively determines the final text information based on the prediction results and the intermediate output results of the speech recognition model, thereby enhancing the speech recognition capability through the large language model and improving the accuracy of the speech recognition model.

[0032] Figure 1 The application scenario diagram of the speech recognition method, device, electronic device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0033] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a network 103, and a server 104. The network 103 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, and the server 104. The network 103 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0034] The user may use at least one of the first terminal device 101 and the second terminal device 102 to interact with the server 104 through the network 103 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101 and the second terminal device 102, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0035] The first terminal device 101 and the second terminal device 102 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 104 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101 and the second terminal device 102. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0037] The server 104 may be deployed with a speech recognition model and a generative large language model, obtain the speech information to be recognized through the interface of the application on the first terminal device 101 or the second terminal device 102, and generate the text corresponding to the speech information through the speech recognition model and the generative large language model. It should be noted that in other embodiments, the speech recognition model and the generative large language model may also be deployed locally on the first terminal device 101 or the second terminal device 102 to generate the text corresponding to the speech information.

[0038] It should be noted that the speech recognition method provided in the embodiment of the present disclosure can generally be executed by the server 104. Accordingly, the speech recognition device provided in the embodiment of the present disclosure can generally be set in the server 104. The speech recognition method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 104 and can communicate with the first terminal device 101, the second terminal device 102 and / or the server 104. Correspondingly, the speech recognition device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 104 and can communicate with the first terminal device 101, the second terminal device 102 and / or the server 104.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Figure 2 The flowchart of the speech recognition method according to the embodiment of the present disclosure is schematically shown.

[0041] like Figure 2 As shown, the speech recognition method of this embodiment includes operations S210-230.

[0042] In operation S210, a speech segment to be recognized in the speech information to be recognized is recognized using a speech recognition model to obtain at least one initial recognition result corresponding to the speech segment to be recognized; the speech information to be recognized includes a plurality of speech segments.

[0043] In the embodiments of the present disclosure, the speech recognition model may be a language model, an acoustic model, etc. that can convert speech into text. In the process of recognizing the speech information to be recognized, the speech recognition model recognizes multiple speech segments in the speech information to be recognized in sequence based on the time sequence and in segments.

[0044] The speech information to be recognized may be complete speech information that needs to be recognized by the speech recognition model. The speech segment to be recognized refers to the speech segment recognized by the speech recognition model at the current moment. The speech recognition model extracts features of the speech segment to be recognized and can obtain at least one initial recognition result corresponding to the speech segment to be recognized. The pronunciation of at least one initial recognition result is similar, but the semantics expressed may be different.

[0045] For example, the speech information to be recognized is "The Forbidden City is a landmark building in Beijing". The speech segment to be recognized is "Beijing" in the speech information to be recognized. The initial recognition result obtained by the speech recognition model based on the speech segment to be recognized may be "Beijing", "background", and "North Pole".

[0046] In operation S220, a target recognition result of the speech segment to be recognized is determined according to at least one initial recognition result by using a target large model based on target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized.

[0047] In the embodiments of the present disclosure, the target large model refers to a large model (Adapt Language Model, ADP-LM) adapted to the target domain obtained by training the generative large language model based on the corresponding domain of the speech information to be recognized. The target recognition result refers to the recognition result determined in at least one initial recognition result corresponding to other speech segments. The target large model can determine the target recognition result of the speech segment to be recognized from at least one initial recognition result based on the target recognition results of other speech segments before the speech segment to be recognized.

[0048] For example, the voice information to be recognized is "The Forbidden City is a landmark building in Beijing." The voice segment to be recognized is "Beijing" in the voice information to be recognized. The initial recognition results obtained by the voice recognition model based on the voice segment to be recognized are "Beijing", "background", and "North Pole". The other voice segments refer to the voice segments before "Beijing", and the corresponding target recognition results are "The Forbidden City is". Based on the target recognition result of "The Forbidden City is", the target large model can determine that the target recognition result of the voice segment to be recognized is "Beijing" from "Beijing", "background", and "North Pole".

[0049] In operation S230, text information corresponding to the to-be-recognized voice information is determined according to the target recognition result.

[0050] In the embodiments of the present disclosure, the target recognition result refers to the recognition result corresponding to each voice segment in the voice information to be recognized. The text information refers to the complete recognition result of the voice information to be recognized. The target recognition results of each voice segment are spliced ​​together to obtain the text information of the voice information to be recognized.

[0051] For example, the target recognition results of each voice segment to be recognized in the voice information to be recognized are "Forbidden City", "is", "Beijing's", "landmark", and "building" respectively. According to the target recognition results, the text information of the voice information to be recognized can be obtained as "The Forbidden City is a landmark building in Beijing."

[0052] According to the embodiments of the present disclosure, LM is integrated into the process of speech recognition. By using the trained large language model, based on the recognition results of the previous text in the speech information to be recognized, the correct answer is determined from multiple recognition results of the speech segment to be recognized at the current moment. By using the large language model to determine the recognition results based on the construction of contextual relationships, the accuracy and output performance of the speech recognition model are improved.

[0053] According to an embodiment of the present disclosure, it also includes using a target large model to determine a target recognition result of the speech segment to be recognized based on at least one initial recognition result.

[0054] In the embodiments of the present disclosure, the target large model can determine the target recognition result of the speech segment to be recognized based on the recognition result of the previous text in the speech information to be recognized. When recognizing the first speech segment of the speech information to be recognized, there is no previous target recognition result. In this case, the target large model can directly reorder the initial recognition results based on the language model rescore technology (Language Model Rescore, LM rescore), and determine the target recognition result with a higher accuracy from multiple initial recognition results.

[0055] According to an embodiment of the present disclosure, the voice segment to be recognized includes a time period identifier, and the text information corresponding to the voice information to be recognized is determined according to the target recognition result, including: obtaining the text information corresponding to the voice information to be recognized according to the target recognition results corresponding to multiple voice segments to be recognized and the time period identifiers of the multiple voice segments to be recognized.

[0056] In the embodiment of the present disclosure, the time period identifier is used to characterize the time period information of the speech segment to be recognized. Since the target large model determines the target recognition result of each speech segment to be recognized in segments, the final text information of the speech information to be recognized can be obtained based on the time period information and the target recognition results of multiple speech segments to be recognized.

[0057] For example, the target recognition results of the target large model for multiple speech segments to be recognized are "today", "snowing", and "have". Among them, the time period corresponding to "today" is marked as 0-2s, the time period corresponding to "snowing" is marked as 2-3s, and the time period corresponding to "have" is marked as 3-4s. According to the time period mark corresponding to each target recognition result, the text information of the speech information to be recognized can be obtained as "it snowed today".

[0058] According to an embodiment of the present disclosure, by utilizing time period identifiers to align the target recognition results of speech segments with the time axis of each speech segment to be recognized, it is possible to help the target large model build a longer context and improve the accuracy of speech recognition.

[0059] According to an embodiment of the present disclosure, a target large model is used to determine a target recognition result of a speech segment to be recognized based on target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized according to at least one initial recognition result, including: using a target large model to generate a prediction result based on target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized; determining a first probability of at least one initial recognition result based on the prediction result to determine the target recognition result of the speech segment to be recognized.

[0060] In the embodiments of the present disclosure, the prediction result is obtained by generating subsequent text based on the target recognition results of other speech segments by the target large model. The prediction result includes the result of the same text as the initial recognition result. The first probability refers to the probability value obtained by using the target large model, which is used to characterize the accuracy of the initial recognition result. The target large model determines the first probability of at least one initial recognition result by scoring the prediction results, and determines the target recognition result of the speech segment to be recognized based on the first probability.

[0061] For example, the voice information to be recognized is "Lenovo's God Machine Factory". The voice segment to be recognized is "machine" in the voice information to be recognized. The initial recognition results obtained by the voice recognition model based on the voice segment to be recognized are "machine", "plan", and "level". The target large model generates multiple prediction results based on the target recognition result "Lenovo's God" of other voice segments, and generates a probability value corresponding to each prediction result. For example, multiple prediction results such as "Lenovo's Myth 0.3", "Lenovo's God Level 0.4", "Lenovo's God Machine 0.75", "Lenovo's God Plan 0.1", "Lenovo's Review 0.6" and corresponding probability values ​​can be obtained. Among them, the prediction results "Lenovo's God Level", "Lenovo's God Machine", and "Lenovo's God Plan" correspond to multiple initial recognition results of the voice recognition model, so it can be determined that the first probability of "machine" in the initial recognition result is 0.75, the first probability of "plan" is 0.1, and the first probability of "level" is 0.4. According to the first probability of each initial recognition result, the target large model can determine that the target recognition result of the voice segment to be recognized is "machine".

[0062] According to an embodiment of the present disclosure, a speech segment to be recognized in speech information to be recognized is recognized using a speech recognition model to obtain at least one initial recognition result corresponding to the speech segment to be recognized, including: a speech segment to be recognized in speech information to be recognized is recognized using a speech recognition model to obtain at least one initial recognition result corresponding to the speech segment to be recognized and a second probability of each initial recognition result; a first probability of at least one initial recognition result is determined according to the prediction result to determine a target recognition result of the speech segment to be recognized, including: according to the first probability and the second probability of at least one initial recognition result, the target recognition result of the speech segment to be recognized is determined from at least one initial recognition result.

[0063] In an embodiment of the present disclosure, when a speech recognition model recognizes a speech segment to be recognized, it can not only obtain at least one initial recognition result, but also determine a second probability corresponding to each initial recognition result. The second probability is a probability value obtained by using the speech recognition model and used to characterize the accuracy rate of the initial recognition result. When determining the target recognition result of the speech segment to be recognized, it is necessary to jointly determine based on the first probability and the second probability. For example, the average value, sum, weighted average, etc. of the first probability and the second probability can be calculated.

[0064] For example, in the initial recognition results determined by the target large model, the first probability of "machine" is 0.75, the first probability of "calculate" is 0.1, and the first probability of "level" is 0.4. In the initial recognition results determined by the speech recognition model, the second probability of "machine" is 0.5, the second probability of "calculate" is 0.3, and the second probability of "level" is 0.6. The weight value of the first probability is set to 0.7, and the weight value of the second probability is set to 0.3. The weighted average is calculated according to the weight values to obtain the score of each initial recognition result, and the target recognition result of the speech segment to be recognized is determined as "machine".

[0065] According to an embodiment of the present disclosure, the speech recognition model performs segmented recognition on speech information, and the generative large language model generates a prediction result for the initial recognition result based on the recognition results of the previous text. The speech recognition model and the generative large language model respectively give the probability values of the initial recognition results, and determine the target recognition result in the form of shallow fusion, so that the speech recognition model can perform path selection according to the guidance of the large language model during the recognition process, improving the recognition performance and increasing the accuracy rate of the recognition result at the same time.

[0066] Next, in combination with Figure 3 The speech recognition method of the embodiment of the present disclosure will be further described.

[0067] Figure 3 A schematic diagram showing a speech recognition method according to another embodiment of the present disclosure is schematically illustrated.

[0068] As Figure 3 shown, the speech information 301 to be recognized includes multiple speech segments, and the speech segments 302 to be recognized are sequentially sent into the speech recognition model 310 for recognition. The speech recognition model 310 obtains at least one initial recognition result 303 corresponding to the speech segment 302 to be recognized through recognition, and sends the at least one initial recognition result 303 to the target large model 320. The target large model generates multiple prediction results based on the target recognition results 304 of other language segments, and determines the target recognition result 305 of the speech segment 302 to be recognized from the at least one initial recognition result 303 based on the prediction results.

[0069] In response to the speech information to be recognized not being finished, the speech recognition model 310 can continue to recognize the subsequent speech segment 302 to be recognized based on the determined target recognition result 305. In response to the speech information to be recognized being finished, the target large model 320 determines the final text information 306 of the speech information to be recognized based on the target recognition results of the multiple speech segments.

[0070] Figure 4 The flowchart of a speech recognition method according to another embodiment of the present disclosure is schematically shown.

[0071] like Figure 4 As shown, the speech recognition method of this embodiment includes operations S401-S404.

[0072] In operation S401, a speech segment to be recognized in speech information to be recognized is recognized using a speech recognition model to obtain at least one initial recognition result corresponding to the speech segment to be recognized and a second probability of each initial recognition result.

[0073] In operation S402, a prediction result is generated based on target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized by using the target large model.

[0074] In operation S403, a target recognition result of the speech segment to be recognized is determined from the at least one initial recognition result according to the first probability and the second probability of the at least one initial recognition result.

[0075] In operation S404, text information corresponding to the voice information to be recognized is obtained according to the target recognition results corresponding to the multiple voice segments to be recognized and the time period identifiers of the multiple voice segments to be recognized.

[0076] According to an embodiment of the present disclosure, the target large model is obtained by the following method: obtaining a sample corpus, the sample corpus includes annotated text, initial recognition results of speech fragment samples and / or processed corpus, the initial recognition results of the speech fragment samples are obtained by recognizing the speech fragment samples using a speech recognition model, and the processed corpus is obtained by intercepting and processing the initial recognition results of the speech fragments; adjusting the parameters of the initial large model according to the sample corpus and prompt information to obtain the target large model, and the target large model is suitable for the target task scenario corresponding to the sample corpus.

[0077] Next, combine Figure 5 The training process of the target large model of the embodiment of the present disclosure is further explained.

[0078] Figure 5 A schematic diagram of target large model training according to an embodiment of the present disclosure is schematically shown.

[0079] like Figure 5 As shown, the initial recognition result 501 of the speech segment sample is obtained via the speech recognition model. The processed corpus 502 may refer to the corpus after the initial recognition result 501 of the speech segment sample is manually segmented, and is used to generate the text after segmentation according to the content before segmentation through the target large model. The annotated text 503 refers to the correct text of the speech segment sample. The sample corpus 504 may include at least one of the initial recognition result 501 of the speech segment sample and the processed corpus 502, as well as the corresponding annotated text 503.

[0080] In the embodiments of the present disclosure, the sample corpus 504 may be corpus in some specific fields such as finance, education, and culture. By training the initial large model 511 using the sample corpus 504 in these fields and the prompt information 505 for indicating the operation of the sample corpus 504, a target large model 512 suitable for the corresponding field can be obtained. The initial large model 511 may use a pre-trained language model such as the Llama2 large model, or a language model based on ASR N-gram and a language model based on a recurrent neural network. The prompt information 505 is used to indicate the processing method of the initial large model for the sample corpus 504.

[0081] For example, the initial large model is trained using sample corpus in the financial field, which contains terminology information related to economy and market. The target large model obtained after training can be used in relevant scenarios in the financial field.

[0082] For example, the annotated sample in the sample corpus can be "Mobile Internet Internal Testing Platform", and the initial recognition result of the corresponding voice fragment sample can be the correct recognition result "Mobile Internet Internal Testing Platform", or it can be an incorrect recognition result, such as "Mobile Internet Inside Platform", "Mobile Internet Internal Testing Platform", etc. In order to enrich the sample corpus, there can also be processed corpus such as "Mobile Internet Inside*", "Mobile Internet*", etc., where * indicates that the subsequent characters are any characters. The prompt information set for the initial recognition result of the voice fragment sample can represent the segmentation from a certain place in the initial recognition result and the generation of subsequent content. The prompt information set for the processed corpus can represent the generation of subsequent text based on the previous content.

[0083] According to an embodiment of the present disclosure, parameters of an initial large model are adjusted according to sample corpus and prompt information to obtain a target large model, including: inputting the sample corpus and prompt information into the initial large model; the initial large model obtains an initial recognition result for the speech fragment sample and / or a generation result of the processed corpus based on the prompt information and the initial recognition result of the speech fragment sample and / or the processed corpus; based on the generated result and the annotated text, parameters of the initial large model are adjusted to obtain a target large model.

[0084] In an embodiment of the present disclosure, an initial large model is trained according to sample corpus and prompt information by an instruction learning method, and the initial large model determines the corresponding generation result based on the initial recognition result of the speech segment sample and at least one of the processed corpus and the correspondingly set prompt information. Based on the initial recognition result of the speech segment sample and the annotated text corresponding to the processed corpus, the accuracy of the initial large model generation result is determined, so as to adjust the parameters of the initial large model to obtain a target large model suitable for the field corresponding to the sample corpus.

[0085] According to an embodiment of the present disclosure, by using predefined control prompt information, the initial pre-trained model is guided to process the sample corpus, and the model parameters are adjusted according to the generated results and annotated text to obtain a target large model that can adapt to the speech recognition task.

[0086] According to an embodiment of the present disclosure, the prompt information is used to instruct the initial large model to generate content after the initial recognition result of the speech segment sample and / or the specified position of the processing corpus.

[0087] For example, the initial recognition result of a voice segment sample may be a “mobile Internet internal testing platform” or a “mobile Internet internal platform”. By setting prompt information, the initial large model may be instructed to generate subsequent content based on “mobile Internet internal”.

[0088] According to an embodiment of the present disclosure, the annotated text includes a first time period identifier, and the initial recognition result of the speech segment sample includes a second time period identifier, and the first time period identifier corresponds to the second time period identifier.

[0089] In an embodiment of the present disclosure, the annotated text is the correct answer corresponding to the speech segment sample, and the first time period identifier is used to characterize the time period information of the annotated text. The second time period identifier is used to characterize the time period information of the initial recognition result of the speech segment sample. After the initial large model determines the generation result based on the initial recognition result of the speech segment sample, the corresponding first time period identifier and the annotated text are determined according to the second time period identifier of the initial recognition result of the speech segment sample, thereby determining the accuracy of the result generated by the initial large model.

[0090] Figure 6 The structural block diagram of the speech recognition device according to the embodiment of the present disclosure is schematically shown.

[0091] like Figure 6 As shown, the speech recognition device 600 of this embodiment includes a recognition module 610 , a recognition result determination module 620 and a text information determination module 630 .

[0092] The recognition module 610 is used to recognize the speech segment to be recognized in the speech information to be recognized by using the speech recognition model, and obtain at least one initial recognition result corresponding to the speech segment to be recognized; the speech information to be recognized includes multiple speech segments. In one embodiment, the recognition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0093] The recognition result determination module 620 is used to determine the target recognition result of the speech segment to be recognized based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized and based on at least one initial recognition result using the target large model. In one embodiment, the recognition result determination module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0094] The text information determination module 630 is used to determine the text information corresponding to the to-be-recognized voice information according to the target recognition result. In one embodiment, the text information determination module 630 can be used to perform the operation S230 described above, which will not be described in detail here.

[0095] According to an embodiment of the present disclosure, the recognition result determination module 620 includes a generation submodule and a determination submodule.

[0096] The generation submodule is used to generate prediction results based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized by using the target large model.

[0097] The determination submodule is used to determine a first probability of at least one initial recognition result according to the prediction result to determine a target recognition result of the speech segment to be recognized.

[0098] According to an embodiment of the present disclosure, the recognition module 610 includes a recognition submodule, which is used to recognize the speech segment to be recognized in the speech information to be recognized by using the speech recognition model, and obtain at least one initial recognition result corresponding to the speech segment to be recognized and the second probability of each initial recognition result. The determination submodule includes a result determination unit, which is used to determine the target recognition result of the speech segment to be recognized from at least one initial recognition result according to the first probability and the second probability of at least one initial recognition result.

[0099] According to an embodiment of the present disclosure, the speech recognition device 600 further includes an initial determination module for determining a target recognition result of a speech segment to be recognized based on at least one initial recognition result by using a target large model.

[0100] According to an embodiment of the present disclosure, the voice segment to be recognized includes a time period identifier, and the text information determination module 630 includes an identification module for obtaining text information corresponding to the voice information to be recognized based on target recognition results corresponding to multiple voice segments to be recognized and time period identifiers of multiple voice segments to be recognized.

[0101] According to the embodiments of the present disclosure, the target large model is obtained through the following submodules: an acquisition submodule, which is used to acquire sample corpus, the sample corpus includes annotated text, initial recognition results of speech fragment samples and / or processed corpus, the initial recognition results of speech fragment samples are obtained by recognizing speech fragment samples using a speech recognition model, and the processed corpus is obtained by intercepting and processing the initial recognition results of speech fragments. An adjustment submodule is used to adjust the parameters of the initial large model according to the sample corpus and prompt information to obtain the target large model, which is suitable for the target task scenario corresponding to the sample corpus.

[0102] According to an embodiment of the present disclosure, the adjustment submodule includes an input unit, a generation unit and an adjustment unit.

[0103] The input unit is used to input sample corpus and prompt information into the initial large model.

[0104] The generating unit is used for obtaining the initial recognition result and / or the generated result of the processed corpus for the speech segment sample based on the prompt information and the initial recognition result and / or the processed corpus of the speech segment sample by the initial large model.

[0105] The adjustment unit is used to adjust the parameters of the initial large model based on the generated results and the annotated text to obtain the target large model.

[0106] According to an embodiment of the present disclosure, the prompt information is used to instruct the initial large model to generate content after the initial recognition result of the speech segment sample and / or the specified position of the processing corpus.

[0107] According to an embodiment of the present disclosure, the annotated text includes a first time period identifier, and the initial recognition result of the speech segment sample includes a second time period identifier, and the first time period identifier corresponds to the second time period identifier.

[0108] Figure 7 A schematic block diagram of an electronic device that can be used to implement the speech recognition method of an embodiment of the present disclosure is schematically shown.

[0109] like Figure 7As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 707 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0110] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also implement the method of the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0111] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0112] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0113] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0114] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0115] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0116] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0117] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0118] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0119] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] In addition, unless the steps are specifically described or must occur in sequence, the order of the above steps is not limited to the above list, and can be changed or rearranged according to the required design. And the above embodiments can be mixed and matched with each other or with other embodiments based on design and reliability considerations, that is, the technical features in different embodiments can be freely combined to form more embodiments.

[0122] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations shown in the flowchart in the embodiments of the present disclosure, or there is a sequence of execution between different operations in technical implementation, otherwise, the execution order of multiple operations may not be prioritized, and multiple operations may also be executed simultaneously.

[0123] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems may also be used together with the enlightenment based thereon. According to the above description, it is apparent that the structure required for constructing such systems is. In addition, the present disclosure is not directed to any particular programming language either. It should be understood that various programming languages ​​may be utilized to implement the disclosure described herein, and the description of the specific languages ​​above is for the purpose of disclosing the best mode of implementation of the present disclosure.

[0124] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in a variety of ways, even if such combinations and / or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in a variety of ways without departing from the spirit and teachings of the present invention. All of these combinations and / or couplings fall within the scope of the present invention.

[0125] The embodiments of the present invention have been described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment has been described above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present invention is limited by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A speech recognition method, comprising: Using a speech recognition model to recognize a speech segment to be recognized in the speech information to be recognized, and obtaining at least one initial recognition result corresponding to the speech segment to be recognized; the speech information to be recognized includes a plurality of speech segments; Determine the target recognition result of the speech segment to be recognized according to the at least one initial recognition result by using the target large model based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized; The text information corresponding to the voice information to be recognized is determined according to the target recognition result.

2. The method according to claim 1, wherein the step of using the target large model to determine the target recognition result of the speech segment to be recognized based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized and based on the at least one initial recognition result comprises: Generate a prediction result based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized by using the target large model; A first probability of the at least one initial recognition result is determined according to the prediction result to determine a target recognition result of the speech segment to be recognized.

3. The method according to claim 2, wherein the step of using a speech recognition model to recognize a speech segment to be recognized in the speech information to be recognized and obtaining at least one initial recognition result corresponding to the speech segment to be recognized comprises: Recognize the speech segment to be recognized in the speech information to be recognized by using the speech recognition model, and obtain at least one initial recognition result corresponding to the speech segment to be recognized and a second probability of each initial recognition result; The step of determining a first probability of the at least one initial recognition result according to the prediction result to determine a target recognition result of the speech segment to be recognized includes: According to the first probability and the second probability of the at least one initial recognition result, a target recognition result of the speech segment to be recognized is determined from the at least one initial recognition result.

4. The method according to claim 1, further comprising: The target large model is used to determine a target recognition result of the speech segment to be recognized based on the at least one initial recognition result.

5. The method according to claim 1, wherein the speech segment to be recognized includes a time period identifier, and the determining of text information corresponding to the speech information to be recognized according to the target recognition result comprises: According to the target recognition results corresponding to the multiple voice segments to be recognized and the time period identifiers of the multiple voice segments to be recognized, the text information corresponding to the voice information to be recognized is obtained.

6. The method according to claim 1, wherein: The target macro model is obtained by the following method: Acquire sample corpus, wherein the sample corpus includes annotated text, an initial recognition result of a speech segment sample and / or processed corpus, wherein the initial recognition result of the speech segment sample is obtained by recognizing the speech segment sample using a speech recognition model, and the processed corpus is obtained by intercepting and processing the initial recognition result of the speech segment; The parameters of the initial large model are adjusted according to the sample corpus and the prompt information to obtain the target large model, and the target large model is suitable for the target task scenario corresponding to the sample corpus.

7. The method according to claim 6, wherein: The step of adjusting parameters of the initial large model according to the sample corpus and the prompt information to obtain the target large model includes: Inputting the sample corpus and the prompt information into the initial large model; The initial large model obtains an initial recognition result and / or a generation result of the processed corpus for the speech segment sample based on the prompt information and the initial recognition result and / or the processed corpus of the speech segment sample; Based on the generation result and the annotated text, the parameters of the initial large model are adjusted to obtain the target large model.

8. The method according to claim 6 or 7, wherein: The prompt information is used to instruct the initial large model to generate content after the initial recognition result of the speech segment sample and / or the specified position of the processed corpus.

9. The method according to claim 6, wherein: The annotated text includes a first time period identifier, and the initial recognition result of the speech segment sample includes a second time period identifier, and the first time period identifier corresponds to the second time period identifier.

10. A speech recognition device, comprising: A recognition module, configured to recognize a speech segment to be recognized in the speech information to be recognized by using a speech recognition model, and obtain at least one initial recognition result corresponding to the speech segment to be recognized; the speech information to be recognized includes a plurality of speech segments; A recognition result determination module, configured to determine a target recognition result of the speech segment to be recognized based on the target recognition results of other speech segments before the speech segment to be recognized in the speech information to be recognized and based on the at least one initial recognition result using a target large model; The text information determination module is used to determine the text information corresponding to the voice information to be recognized according to the target recognition result.

Citation Information

Cited By

  • Long speech recognition method based on large language model post-processing and electronic equipment

    CN119993136A

  • Speech recognition method and electronic equipment

    CN120452445A