A speech recognition method, system, device, and medium based on a large language model

Abstract

This application discloses a speech recognition method, system, device, and medium based on a large language model, relating to the field of speech recognition technology. The method includes: inputting the speech to be recognized into a speech recognition model to obtain several recognition results; constructing an input sequence containing the recognition results and their contextual information; a speech recognition model score reflecting the degree of matching between the speech to be recognized and the output text; scoring the input sequence using a large language model trained based on a domain knowledge graph to obtain a large language model score; calculating the indices of the speech recognition model score and the large language model score respectively to obtain corresponding first and second index values; determining the weights corresponding to the speech recognition model score and the large language model score; determining a comprehensive score for each recognition result based on the weights, the first index value, and the second index value; and selecting a target recognition result based on the comprehensive score. Therefore, this application improves the accuracy of speech recognition.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition method, system, device and medium based on a large language model. Background Technology

[0002] In recent years, speech recognition technology has been widely used in human-computer interaction, office automation, and other fields. However, domain-specific speech recognition has consistently faced challenges such as low accuracy in recognizing specialized terminology and high model adaptation costs. Existing technologies suffer from the following main shortcomings: First, traditional speech recognition systems often require a large amount of speech-text pair training data for processing domain-specific speech. Acquiring this type of data is extremely costly, especially in professional fields such as medicine and law, where high-quality labeled speech data is particularly scarce. Second, existing domain adaptation methods primarily rely on retraining or fine-tuning the complete speech recognition model, which not only requires substantial computational resources but also faces the problem of catastrophic forgetting. Even with incremental learning schemes, model performance remains difficult to improve significantly. Finally, the n-gram language models or simple neural network language models used in traditional speech recognition systems struggle to accurately understand and process complex domain-specific expressions, particularly new specialized terminology or unconventional expressions, often performing poorly. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a speech recognition method, system, device, and medium based on a large language model, which can significantly reduce the adaptation cost of speech recognition systems in specific domains, improve recognition accuracy, and enhance the ability to recognize specialized terminology. The specific solution is as follows:

[0004] Firstly, this application discloses a speech recognition method based on a large language model, comprising:

[0005] The speech to be recognized is input into a speech recognition model to obtain several recognition results, and an input sequence containing the recognition results and contextual information of the recognition results is constructed; the recognition results include output text and a speech recognition model score; the speech recognition model score is a score that reflects the degree of matching between the speech to be recognized and the output text;

[0006] The input sequence is scored using a large language model trained on a domain knowledge graph to obtain a large language model score.

[0007] Calculate the indices of the speech recognition model score and the large language model score respectively to obtain the corresponding first index value and second index value, and determine the weights corresponding to the speech recognition model score and the large language model score respectively.

[0008] A comprehensive score is determined for each recognition result based on the weight, the first index value, and the second index value, and a target recognition result is selected based on the comprehensive score.

[0009] Optionally, before inputting the speech to be recognized into the speech recognition model, the method further includes:

[0010] Collect the domain text data corresponding to the speech to be recognized; the domain text data includes professional literature and glossary.

[0011] Technical terms and usage patterns are extracted from the domain text data, and a domain knowledge graph is constructed based on the hierarchical relationship between the technical terms, the frequency of use of the technical terms, the contextual information of the technical terms, and the usage patterns.

[0012] The initial large language model is trained using the domain knowledge graph and LoRA technology to obtain the large language model.

[0013] Optionally, the step of scoring the input sequence using a large language model trained on a domain knowledge graph to obtain a large language model score includes:

[0014] The fluency of the input sequence is determined by calculating the perplexity score of the input sequence.

[0015] The domain relevance of the input sequence is determined based on the amount of domain data contained in the identification results;

[0016] The professional standardization of the input sequence is determined by judging whether the language of the input sequence conforms to the expression habits and norms of the professional field;

[0017] The input sequence is scored using a large language model trained on a domain knowledge graph, the language fluency, the domain relevance, and the professional standardization to obtain a large language model score.

[0018] Optionally, determining the weights corresponding to the speech recognition model score and the large language model score respectively includes:

[0019] The weights corresponding to the speech recognition model score are determined based on the signal-to-noise ratio and speech clarity of the speech to be recognized.

[0020] The weights corresponding to the large language model score are determined based on the term density and professional standardization of the recognition results; the term density represents the number of terms in the speech to be recognized.

[0021] Optionally, determining the weights corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized includes:

[0022] Calculate the signal-to-noise ratio (SNR) of the speech to be recognized, and determine the SNR score based on the relationship between the SNR and a preset SNR threshold;

[0023] Calculate the speech clarity index of the speech to be recognized, and determine the speech clarity score based on the speech clarity index;

[0024] The weights corresponding to the speech recognition model score are determined based on the pre-set first basic weights, the signal-to-noise ratio score, the speech clarity score, and the weight ratio between the signal-to-noise ratio score and the speech clarity score.

[0025] Optionally, determining the weights corresponding to the large language model score based on the term density and professional standardization of the recognition results includes:

[0026] The identification results are matched with the domain knowledge graph by maximum positive matching to determine the number of terms in the identification results;

[0027] The term density is determined based on the number of terms and a pre-set term number threshold.

[0028] A term density score is determined based on the term density.

[0029] The recognition result and the preceding information of the recognition result are input into the prompt template of the large language model to obtain the professional standardization score of the recognition result;

[0030] The weights corresponding to the large language model score are determined based on the pre-set second basic weights, the terminology density score, the professional standardization score, and the weight ratio between the terminology density score and the professional standardization score.

[0031] Optionally, determining the comprehensive score for each recognition result based on the weight, the first index value, and the second index value includes:

[0032] Determine the first product result between the first index value and the weight corresponding to the speech recognition model score;

[0033] Determine the second product result between the second index value and the weight corresponding to the large language model score;

[0034] A comprehensive score for each recognition result is determined based on the sum of the first product result and the second product result.

[0035] Secondly, this application discloses a speech recognition system based on a large language model, comprising:

[0036] An input sequence construction module is used to input the speech to be recognized into a speech recognition model to obtain several recognition results and construct an input sequence containing the recognition results and contextual information of the recognition results; the recognition results include output text and a speech recognition model score; the speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text;

[0037] The scoring acquisition module is used to score the input sequence using a large language model trained based on a domain knowledge graph, so as to obtain a large language model score.

[0038] The weight determination module is used to calculate the indices of the speech recognition model score and the large language model score respectively, to obtain the corresponding first index value and second index value, and to determine the weights corresponding to the speech recognition model score and the large language model score respectively.

[0039] The target recognition result selection module is used to determine a comprehensive score for each recognition result based on the weight, the first index value, and the second index value, and to select a target recognition result based on the comprehensive score.

[0040] Thirdly, this application discloses an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor is used to execute the computer program to implement the speech recognition method based on a large language model as described above.

[0043] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned speech recognition method based on a large language model.

[0044] In this application, when performing speech recognition, the speech to be recognized is first input into a speech recognition model to obtain several recognition results, and an input sequence containing the recognition results and their contextual information is constructed. The recognition results include output text and a speech recognition model score. The speech recognition model score reflects the degree of matching between the speech to be recognized and the output text. The input sequence is scored using a large language model trained based on a domain knowledge graph to obtain a large language model score. The indices of the speech recognition model score and the large language model score are calculated respectively to obtain corresponding first and second index values, and the weights corresponding to the speech recognition model score and the large language model score are determined. A comprehensive score for each recognition result is determined based on the weights, the first index value, and the second index value, and a target recognition result is selected based on the comprehensive score. Therefore, this application first outputs multiple most likely candidate recognition results from the speech recognition model, then scores the candidate results using a fine-tuned large language model, then calculates the model weights, and finally fuses the scores based on the model weights to select the final recognition result. In this way, this application can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve the recognition accuracy of complex language scenarios and the ability to recognize professional terms. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a flowchart of a speech recognition method based on a large language model disclosed in this application;

[0047] Figure 2 This application discloses a flowchart of knowledge injection and model optimization.

[0048] Figure 3 This is a flowchart of a specific speech recognition method based on a large language model disclosed in this application;

[0049] Figure 4 This application discloses a dynamic scoring fusion flowchart;

[0050] Figure 5 This is a schematic diagram of the structure of a speech recognition system based on a large language model disclosed in this application;

[0051] Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Currently, traditional speech recognition systems often require a large amount of speech-text pair training data for specific domains when processing speech in those domains. Acquiring such data is extremely costly, especially in professional fields such as medicine and law, where high-quality speech annotation data is particularly scarce. Secondly, existing domain adaptation methods mainly rely on retraining or fine-tuning the complete speech recognition model, which not only requires significant computational resources but also faces the problem of catastrophic forgetting. Even with incremental learning schemes, model performance is difficult to improve as desired. Finally, the n-gram language models or simple neural network language models used in traditional speech recognition systems struggle to accurately understand and process complex domain-specific expressions, particularly new technical terms or unconventional expressions, often performing poorly. To address these technical problems, this application discloses a speech recognition method, system, device, and medium based on a large language model, which can significantly reduce the adaptation cost of domain-specific speech recognition systems and improve recognition accuracy and the ability to recognize technical terms.

[0054] See Figure 1 As shown, this embodiment of the invention discloses a speech recognition method based on a large language model, including:

[0055] Step S11: Input the speech to be recognized into the speech recognition model to obtain several recognition results, and construct an input sequence containing the recognition results and the context information of the recognition results; the recognition results include output text and speech recognition model score; the speech recognition model score is a score that reflects the degree of matching between the speech to be recognized and the output text.

[0056] In this embodiment, before inputting the speech to be recognized into the speech recognition model, a large language model is first obtained, such as... Figure 2As shown, domain text data corresponding to the speech to be recognized is collected. This domain text data includes professional literature and glossaries. Professional terms and usage patterns are extracted from the domain text data. A domain knowledge graph is constructed based on the hierarchical relationships between professional terms, their usage frequency, contextual information, and usage patterns. The initial large language model is trained using the domain knowledge graph and LoRA (Low-Rank Adaptation) technology to obtain a larger language model. Specifically, firstly, professional terms and expression patterns are extracted from the domain text to construct a domain knowledge graph. The knowledge graph contains not only hierarchical relationships between terms but also their usage frequency and contextual information. For example, in the medical field, the term "pleural effusion" is synonymous with terms such as "pleural fluid" and "pleural cavity effusion," and is also associated with symptom terms such as "chest tightness" and "difficulty breathing." Then, during the fine-tuning of the large language model, LoRA (Low-Rank Adaptation) technology is used for efficient parameter fine-tuning. Specifically, for the weight matrix W in the model, two low-rank matrices A and B are introduced, making the fine-tuned weight matrix W + AB, where the ranks of A and B are much smaller than the rank of the original weight matrix. This approach maintains the model's basic language understanding capabilities while quickly adapting to the language characteristics of new domains. Practice shows that setting the rank of the low-rank matrices to 8 achieves good domain adaptation while maintaining a relatively small number of parameters. In this way, a low-cost domain adaptation mechanism is achieved through text-based domain knowledge injection.

[0057] After the large language model is trained, it is applied to the decoding process of speech recognition. In traditional speech recognition, the decoder typically uses an n-gram language model or a simple neural network language model. This invention proposes using a domain-adapted large language model to evaluate and optimize the decoding results. Specifically, in implementation, as follows... Figure 3 As shown, the speech recognition model first outputs N most likely candidate recognition results. The recognition result includes the output text and a speech recognition model score, which reflects the degree of matching between the speech to be recognized and the output text. For each candidate result, an input sequence containing complete contextual information is constructed and fed into a large language model for scoring.

[0058] Step S12: The input sequence is scored using a large language model trained based on a domain knowledge graph to obtain a large language model score.

[0059] In this embodiment, the fluency of the input sequence is determined by calculating its perplexity score; the domain relevance of the input sequence is determined based on the amount of domain data contained in the recognition results; the professional standardization of the input sequence is determined by judging whether the language of the input sequence conforms to the expression habits and norms of the professional domain; and the input sequence is scored using a large language model trained on a domain knowledge graph, considering fluency, domain relevance, and professional standardization to obtain a large language model score. Specifically, the process of feeding an input sequence containing complete contextual information into the large language model for scoring considers the following three aspects: fluency, domain relevance, and professional standardization. Fluency is obtained by calculating the perplexity score; domain relevance is evaluated based on the number of domain terms contained in the candidate results; and professional standardization is evaluated by the large model based on whether the language conforms to the expression habits and norms of the professional domain. Thus, the method of optimizing the decoding process using a large language model improves the recognition accuracy of complex language scenarios.

[0060] Step S13: Calculate the indices of the speech recognition model score and the large language model score respectively to obtain the corresponding first index value and second index value, and determine the weights corresponding to the speech recognition model score and the large language model score respectively.

[0061] In this embodiment, to comprehensively utilize the advantages of speech recognition models and large language models, this application discloses a dynamic scoring multi-model fusion strategy. Unlike traditional fixed-weight fusion methods, this strategy dynamically adjusts the weights of each model based on the reliability of the input features. Thus, by employing a dynamic scoring multi-model fusion strategy, adaptive fusion of speech recognition models and large language models is achieved. Figure 4 As shown, the fusion process first converts the acoustic model score and language model score into probability values, specifically by calculating the exponent exp(x) of the score x. The speech recognition model score reflects the degree of matching between the speech signal and the text, while the large language model score includes the comprehensive evaluation of the candidate results by the large language model. Therefore, the exponents of the speech recognition model score and the large language model score are calculated first and second respectively to obtain the corresponding first and second exponent values, and then the weights corresponding to the speech recognition model score and the large language model score are determined. The weights corresponding to the speech recognition model score are determined based on the signal-to-noise ratio and speech clarity of the speech to be recognized; the weights corresponding to the large language model score are determined based on the term density and professional standardization of the recognition results; term density represents the number of terms in the speech to be recognized.

[0062] In determining the weights corresponding to the speech recognition model score, the signal-to-noise ratio (SNR) of the speech to be recognized is first calculated, and the SNR score is determined based on the relationship between the SNR and a pre-set SNR threshold. The speech clarity index of the speech to be recognized is then calculated, and the speech clarity score is determined based on the speech clarity index. Finally, the weights corresponding to the speech recognition model score are determined based on the pre-set first basic weights, the SNR score, the speech clarity score, and the weight ratio between the SNR score and the speech clarity score.

[0063] When determining the weights corresponding to the large language model score, the recognition results are first matched with the domain knowledge graph to determine the number of terms in the recognition results; the term density is determined based on the number of terms and a pre-set term number threshold; the term density score is determined based on the term density; the recognition results and their context information are input into the prompt template of the large language model to obtain the professional standardization score of the recognition results; the weights corresponding to the large language model score are determined based on the pre-set second basic weights, the term density score, the professional standardization score, and the weight ratio between the term density score and the professional standardization score.

[0064] Step S14: Determine the comprehensive score of each recognition result based on the weight, the first index value, and the second index value, and select the target recognition result based on the comprehensive score.

[0065] In this embodiment, after determining the weights, first index value, and second index value corresponding to the speech recognition model score and the large language model score, the system determines the first product result between the first index value and the weight corresponding to the speech recognition model score; the system also determines the second product result between the second index value and the weight corresponding to the large language model score; finally, the system determines the comprehensive score of each recognition result based on the sum of the first and second product results. In other words, when determining the final score, the system dynamically adjusts the weights of the two scores based on factors such as the signal-to-noise ratio of the input speech and the speaker's pronunciation clarity. For example, when the input speech quality is good, the weight of the acoustic model score will increase accordingly; when the input contains a large number of technical terms, the weight of the language model score will increase accordingly. The specific weight calculation formula is as follows:

[0066] Score= (SNR,CLR)*exp(ASR_Score)+ (TERM,COH)* exp(LM_Score);

[0067] in, It is the weight function of the speech recognition model, which is related to the signal-to-noise ratio (SNR) and speech intelligibility (CLR); It is the weight function of the large language model, which is related to term density (TERM) and semantic coherence (COH); Score is the comprehensive score of the recognition results.

[0068] After obtaining the comprehensive score of the recognition results, the target recognition result with the highest score is selected from several recognition results based on the comprehensive score.

[0069] In summary, this application, when performing speech recognition, first inputs the speech to be recognized into a speech recognition model to obtain several recognition results, constructing an input sequence containing the recognition results and their contextual information; the recognition results include output text and a speech recognition model score; the speech recognition model score reflects the degree of matching between the speech to be recognized and the output text; the input sequence is scored using a large language model trained based on a domain knowledge graph to obtain a large language model score; the indices of the speech recognition model score and the large language model score are calculated respectively to obtain corresponding first and second index values, determining the weights corresponding to the speech recognition model score and the large language model score; a comprehensive score for each recognition result is determined based on the weights, the first index value, and the second index value, and a target recognition result is selected based on the comprehensive score. It can be seen that this application first outputs multiple most likely candidate recognition results from the speech recognition model, then scores the candidate results using a fine-tuned large language model, then calculates the model weights, and finally fuses the scores based on the model weights to select the final recognition result. In this way, this application can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve the recognition accuracy of complex language scenarios and the ability to recognize professional terms.

[0070] As described in the previous embodiment, this application determines the weights corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized, and determines the weights corresponding to the large language model score based on the terminology density and professional standardization of the recognition results. The specific weight determination process will be explained in detail below.

[0071] Firstly, when determining the weights corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized, it can be known that the acoustic model weights... It is related to signal-to-noise ratio (SNR) and speech intelligibility (CLR). The calculation of the SNR score is as follows:

[0072] Signal-to-noise ratio (SNR) calculation formula: SNR = 10 * log10 (signal energy / noise energy);

[0073] Normalization range:

[0074] SNR_min = 5dB (speech quality is poor below this value);

[0075] SNR_max = 20dB (the improvement is not significant above this value, and there is no need to increase the acoustic weight further);

[0076] These two thresholds are set based on practical experience in speech recognition. If SNR < SNR_min, then SNR_score (signal-to-noise ratio score) = 0 (this speech segment is poor and cannot be used as a reference); if SNR > SNR_max, then SNR_score (signal-to-noise ratio score) = 1; if SNR_min < SNR < SNR_max, then SNR_score (signal-to-noise ratio score) = (SNR - SNR_min) / (SNR_max - SNR_min).

[0077] The speech intelligibility score is calculated by calculating the Speech Intelligibility Index (SII) based on the ANSI S3.5-1997 standard; CLR_score (speech intelligibility score) = SII (in the range of 0-1).

[0078] Final acoustic model weights Calculation:

[0079] = 0.2 + 0.6 * (0.6 * SNR_score + 0.4 * CLR_score);

[0080] Here, 0.2 is the base weight, and 0.6 is the dynamic adjustment range. The weight ratio of SNR to CLR is 6:4, reflecting the main impact of signal-to-noise ratio on recognition accuracy. These parameters are all derived empirically and can be adjusted according to the actual performance in real-world scenarios.

[0081] When determining the weights corresponding to the scores of large language models based on the term density and professional standardization of the recognition results, the language model weights... This is related to Terminology Density (TERM) and Professional Normativity (PROF). For Terminology Density score calculation:

[0082] Using the domain terminology knowledge graph constructed in the previous embodiment, the maximum positive matching is calculated as follows:

[0083] Term density = number of detected terms / 3 (set upper limit to 3 terms);

[0084] Final term density score: If no term is detected, then TERM_score (term density score) = 0; if term density > 1, then TERM_score (term density score) = 1; otherwise, TERM_score (term density score) = term density.

[0085] For calculating the professional standardization score: An evaluation using a large language model is conducted, and the currently being recognized sentence and its preceding sentence are entered into the prompt template. The template needs to be adjusted according to the application domain; for example, a template for the medical field is shown below:

[0086] Please assess the professionalism and standardization of the following medical description:

[0087] Previous sentence: [Previous sentence];

[0088] Current sentence: [Current recognition result];

[0089] Please rate the professionalism of this sentence on a scale of 0 to 1, where:

[0090] - 0 points: Completely inconsistent with the expression habits in the medical field;

[0091] - 0.5 points: Generally in line with medical terminology, but may contain non-standard language;

[0092] - 1 point: Fully compliant with professional medical expression standards;

[0093] Only the score needs to be output; no explanation is required.

[0094] PROF_score (Professional Standard Score) = The score returned by the large language model (within the range of 0-1).

[0095] Final language model weights Calculation:

[0096] = 0.3 + 0.5 * (0.7 * TERM_score + 0.3 * PROF_score);

[0097] Here, 0.3 is the base weight, and 0.5 is the dynamic adjustment range. The weight ratio of TERM to PROF is 7:3, reflecting the importance of terminology recognition in specialized fields. These parameters can be adjusted according to specific application scenarios.

[0098] In this way, the application adopts a multi-model fusion strategy with dynamic scoring to achieve adaptive fusion of acoustic and language models, which further improves the accuracy of speech recognition and the ability to recognize technical terms.

[0099] See Figure 5As shown, this embodiment of the invention discloses a speech recognition system based on a large language model, comprising:

[0100] The input sequence construction module 11 is used to input the speech to be recognized into the speech recognition model to obtain several recognition results and construct an input sequence containing the recognition results and contextual information of the recognition results; the recognition results include output text and speech recognition model score; the speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text;

[0101] The scoring acquisition module 12 is used to score the input sequence using a large language model trained based on a domain knowledge graph, so as to obtain a large language model score.

[0102] The weight determination module 13 is used to calculate the indices of the speech recognition model score and the large language model score respectively, to obtain the corresponding first index value and second index value, and to determine the weights corresponding to the speech recognition model score and the large language model score respectively.

[0103] The target recognition result selection module 14 is used to determine a comprehensive score for each recognition result based on the weight, the first index value, and the second index value, and to select a target recognition result based on the comprehensive score.

[0104] In summary, this application, when performing speech recognition, first inputs the speech to be recognized into a speech recognition model to obtain several recognition results, constructing an input sequence containing the recognition results and their contextual information; the recognition results include output text and a speech recognition model score; the speech recognition model score reflects the degree of matching between the speech to be recognized and the output text; the input sequence is scored using a large language model trained based on a domain knowledge graph to obtain a large language model score; the indices of the speech recognition model score and the large language model score are calculated respectively to obtain corresponding first and second index values, determining the weights corresponding to the speech recognition model score and the large language model score; a comprehensive score for each recognition result is determined based on the weights, the first index value, and the second index value, and a target recognition result is selected based on the comprehensive score. It can be seen that this application first outputs multiple most likely candidate recognition results from the speech recognition model, then scores the candidate results using a fine-tuned large language model, then calculates the model weights, and finally fuses the scores based on the model weights to select the final recognition result. In this way, this application can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve the recognition accuracy of complex language scenarios and the ability to recognize professional terms.

[0105] In some specific embodiments, the system can also be used to collect domain text data corresponding to the speech to be recognized; the domain text data includes professional literature and glossary; extract professional terms and usage patterns from the domain text data, construct the domain knowledge graph based on the hierarchical relationship between the professional terms, the usage frequency of the professional terms, the contextual information of the professional terms and the usage patterns; and train the initial large language model using the domain knowledge graph and LoRA technology to obtain the large language model.

[0106] In some specific embodiments, the input sequence construction module 11 can be used to determine the language fluency of the input sequence by calculating the perplexity score of the input sequence; determine the domain relevance of the input sequence based on the amount of domain data contained in the recognition result; determine the professional standardization of the input sequence by judging whether the language of the input sequence conforms to the expression habits and norms of the professional domain; and score the input sequence based on a large language model trained on a domain knowledge graph, the language fluency, the domain relevance, and the professional standardization to obtain a large language model score.

[0107] In some specific embodiments, the weight determination module 13 can be used to determine the weight corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized; and to determine the weight corresponding to the large language model score based on the term density and professional standardization of the recognition result; the term density represents the number of terms in the speech to be recognized.

[0108] In some specific embodiments, the weight determination module 13 can be used to calculate the signal-to-noise ratio (SNR) of the speech to be recognized, determine an SNR score based on the relationship between the SNR and a pre-set SNR threshold, calculate the speech clarity index of the speech to be recognized, determine a speech clarity score based on the speech clarity index, and determine the weight corresponding to the speech recognition model score based on a pre-set first basic weight, the SNR score, the speech clarity score, and the weight ratio between the SNR score and the speech clarity score.

[0109] In some specific embodiments, the weight determination module 13 can be used to perform maximum positive matching between the recognition result and the domain knowledge graph to determine the number of terms in the recognition result; determine the term density based on the number of terms and a pre-set term number threshold; determine the term density score based on the term density; input the recognition result and the context information of the recognition result into the prompt template of the large language model to obtain the professional standardization score of the recognition result; and determine the weight corresponding to the large language model score based on the pre-set second basic weight, the term density score, the professional standardization score, and the weight ratio between the term density score and the professional standardization score.

[0110] In some specific embodiments, the target recognition result selection module 14 can be used to determine the first product result between the first index value and the weight corresponding to the speech recognition model score; determine the second product result between the second index value and the weight corresponding to the large language model score; and determine the comprehensive score of each recognition result based on the sum of the first product result and the second product result.

[0111] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0112] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the speech recognition method based on a large language model disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0113] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0114] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0115] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the speech recognition method based on a large language model executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0116] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned speech recognition method based on a large language model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0117] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0118] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0120] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0121] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech recognition method based on a large language model, characterized in that, include: The speech to be recognized is input into a speech recognition model to obtain several recognition results, and an input sequence containing the recognition results and contextual information of the recognition results is constructed; the recognition results include output text and speech recognition model scores; The speech recognition model score is a score that reflects the degree of matching between the speech to be recognized and the output text; The input sequence is scored using a large language model trained on a domain knowledge graph to obtain a large language model score. Calculate the indices of the speech recognition model score and the large language model score respectively to obtain the corresponding first index value and second index value, and determine the weights corresponding to the speech recognition model score and the large language model score respectively. A comprehensive score is determined for each recognition result based on the weight, the first index value, and the second index value, and a target recognition result is selected based on the comprehensive score. The determination of the weights corresponding to the speech recognition model score and the large language model score includes: The weights corresponding to the speech recognition model score are determined based on the signal-to-noise ratio and speech clarity of the speech to be recognized. The weights corresponding to the large language model score are determined based on the term density and professional standardization of the recognition results; the term density represents the number of terms in the speech to be recognized.

2. The speech recognition method based on a large language model according to claim 1, characterized in that, Before inputting the speech to be recognized into the speech recognition model, the process also includes: Collect the domain text data corresponding to the speech to be recognized; the domain text data includes professional literature and glossary. Technical terms and usage patterns are extracted from the domain text data, and a domain knowledge graph is constructed based on the hierarchical relationship between the technical terms, the frequency of use of the technical terms, the contextual information of the technical terms, and the usage patterns. The initial large language model is trained using the domain knowledge graph and LoRA technology to obtain the large language model.

3. The speech recognition method based on a large language model according to claim 1, characterized in that, The step of scoring the input sequence using a large language model trained on a domain knowledge graph to obtain a large language model score includes: The fluency of the input sequence is determined by calculating the perplexity score of the input sequence. The domain relevance of the input sequence is determined based on the amount of domain data contained in the identification results; The professional standardization of the input sequence is determined by judging whether the language of the input sequence conforms to the expression habits and norms of the professional field; The input sequence is scored using a large language model trained on a domain knowledge graph, the language fluency, the domain relevance, and the professional standardization to obtain a large language model score.

4. The speech recognition method based on a large language model according to claim 1, characterized in that, The step of determining the weights corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized includes: Calculate the signal-to-noise ratio (SNR) of the speech to be recognized, and determine the SNR score based on the relationship between the SNR and a preset SNR threshold; Calculate the speech clarity index of the speech to be recognized, and determine the speech clarity score based on the speech clarity index; The weights corresponding to the speech recognition model score are determined based on the pre-set first basic weights, the signal-to-noise ratio score, the speech clarity score, and the weight ratio between the signal-to-noise ratio score and the speech clarity score.

5. The speech recognition method based on a large language model according to claim 1, characterized in that, The determination of the weights corresponding to the large language model score based on the term density and professional standardization of the recognition results includes: The identification results are matched with the domain knowledge graph by maximum positive matching to determine the number of terms in the identification results; The term density is determined based on the number of terms and a pre-set term number threshold. A term density score is determined based on the term density. The recognition result and the preceding information of the recognition result are input into the prompt template of the large language model to obtain the professional standardization score of the recognition result; The weights corresponding to the large language model score are determined based on the pre-set second basic weights, the terminology density score, the professional standardization score, and the weight ratio between the terminology density score and the professional standardization score.

6. The speech recognition method based on a large language model according to any one of claims 1 to 5, characterized in that, The step of determining the comprehensive score for each recognition result based on the weight, the first index value, and the second index value includes: Determine the first product result between the first index value and the weight corresponding to the speech recognition model score; Determine the second product result between the second index value and the weight corresponding to the large language model score; A comprehensive score for each recognition result is determined based on the sum of the first product result and the second product result.

7. A speech recognition system based on a large language model, characterized in that, include: An input sequence construction module is used to input the speech to be recognized into a speech recognition model to obtain several recognition results and construct an input sequence containing the recognition results and contextual information of the recognition results; the recognition results include output text and speech recognition model scores; The speech recognition model score is a score that reflects the degree of matching between the speech to be recognized and the output text; The scoring acquisition module is used to score the input sequence using a large language model trained based on a domain knowledge graph, so as to obtain a large language model score. The weight determination module is used to calculate the indices of the speech recognition model score and the large language model score respectively, to obtain the corresponding first index value and second index value, and to determine the weights corresponding to the speech recognition model score and the large language model score respectively. The target recognition result selection module is used to determine a comprehensive score for each recognition result based on the weight, the first index value, and the second index value, and to select a target recognition result based on the comprehensive score. The weight determination module is used to determine the weights corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized; and to determine the weights corresponding to the large language model score based on the term density and professional standardization of the recognition results; the term density represents the number of terms in the speech to be recognized.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the speech recognition method based on a large language model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the speech recognition method based on a large language model as described in any one of claims 1 to 6.

Images