Speech recognition method, system and device based on large language model and medium

Abstract

The invention discloses a voice recognition method, system and device based on a large language model, and a medium, and relates to the technical field of voice recognition, and the method comprises the steps: inputting a to-be-recognized voice into a voice recognition model, so as to obtain a plurality of recognition results, and constructing an input sequence containing the recognition results and the context information of the recognition results; the speech recognition model score is a score reflecting the matching degree of the speech to be recognized and the output text; scoring the input sequence through a big language model trained based on a domain knowledge graph to obtain a big language model score; respectively calculating indexes of the speech recognition model score and the big language model score to obtain a corresponding first index value and a corresponding second index value, and determining weights corresponding to the speech recognition model score and the big language model score; and determining a comprehensive score of each identification result according to the weight, the first index value and the second index value, and selecting a target identification result according to the comprehensive score. Therefore, the accuracy of speech recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method, system, device and medium based on a large language model. Background Art

[0002] In recent years, speech recognition technology has been widely used in the fields of human-computer interaction and automated office. Speech recognition in specific fields has always faced problems such as low accuracy in professional terminology recognition and high model adaptation cost. The existing technology mainly has the following shortcomings: First, when processing speech in a specific field, traditional speech recognition systems often require a large amount of speech-text pair training data in the field. The cost of obtaining such data is extremely high, especially in professional fields such as medicine and law, where high-quality speech annotation data is particularly scarce. Secondly, existing domain adaptation methods mainly rely on retraining or fine-tuning the complete speech recognition model, which not only requires a lot of computing resources, but also faces the problem of catastrophic forgetting. Even with the use of incremental learning schemes, it is still difficult to achieve ideal improvement in model performance. Finally, the n-gram language model or simple neural network language model used by traditional speech recognition systems is difficult to accurately understand and process complex domain-specific expressions, especially for newly emerging professional terms or unconventional expressions, and often performs poorly. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a speech recognition method, system, device and medium based on a large language model, which can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve recognition accuracy and the ability to recognize professional terms. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a speech recognition method based on a large language model, comprising:

[0005] Inputting the speech to be recognized into the speech recognition model to obtain a plurality of recognition results, and constructing an input sequence including the recognition results and context information of the recognition results; the recognition results include output text and a speech recognition model score; the speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text;

[0006] Scoring the input sequence using a large language model trained based on a domain knowledge graph to obtain a large language model score;

[0007] Calculating the indexes of the speech recognition model score and the large language model score respectively to obtain corresponding first index values ​​and second index values, and determining the weights corresponding to the speech recognition model score and the large language model score respectively;

[0008] A comprehensive score of each of the recognition results is determined according to the weight, the first index value, and the second index value, and a target recognition result is selected according to the comprehensive score.

[0009] Optionally, before inputting the speech to be recognized into the speech recognition model, the method further includes:

[0010] Collecting domain text data corresponding to the speech to be recognized; the domain text data includes professional literature and a glossary;

[0011] Extracting professional terms and usage patterns from the domain text data, and constructing the domain knowledge graph according to the hierarchical relationship between the professional terms, the usage frequency of the professional terms, the context information of the professional terms, and the usage patterns;

[0012] The domain knowledge graph and LoRA technology are used to train the initial large language model to obtain the large language model.

[0013] Optionally, scoring the input sequence by using a large language model trained based on a domain knowledge graph to obtain a large language model score includes:

[0014] Determining the language fluency of the input sequence by calculating the perplexity score of the input sequence;

[0015] Determining the domain relevance of the input sequence according to the amount of domain data included in the recognition result;

[0016] Determining the professional standardization of the input sequence by judging whether the language of the input sequence conforms to the expression habits and expression standards of the professional field;

[0017] The input sequence is scored by using a large language model trained based on a domain knowledge graph, the language fluency, the domain relevance, and the professional norms to obtain a large language model score.

[0018] Optionally, determining the weights corresponding to the speech recognition model score and the large language model score respectively includes:

[0019] Determining a weight corresponding to the speech recognition model score according to the signal-to-noise ratio and speech clarity of the speech to be recognized;

[0020] The weight corresponding to the large language model score is determined based on the term density and professional norms of the recognition result; the term density represents the number of terms in the speech to be recognized.

[0021] Optionally, determining the weight corresponding to the speech recognition model score according to the signal-to-noise ratio and speech clarity of the speech to be recognized includes:

[0022] Calculating the signal-to-noise ratio of the speech to be recognized, and determining a signal-to-noise ratio score according to a magnitude relationship between the signal-to-noise ratio and a preset signal-to-noise ratio threshold;

[0023] Calculating a speech clarity index of the speech to be recognized, and determining a speech clarity score according to the speech clarity index;

[0024] The weight corresponding to the speech recognition model score is determined based on a preset first basic weight, the signal-to-noise ratio score, the speech intelligibility score, and a weight ratio between the signal-to-noise ratio score and the speech intelligibility score.

[0025] Optionally, determining the weight corresponding to the large language model score based on the term density and professional norms of the recognition result includes:

[0026] Performing maximum positive matching between the recognition result and the domain knowledge graph to determine the number of terms in the recognition result;

[0027] Determining the term density according to the term quantity and a preset term quantity threshold;

[0028] determining a term density score based on the term density;

[0029] Inputting the recognition result and the context information of the recognition result into the prompt template of the large language model to obtain a professional normative score of the recognition result;

[0030] The weight corresponding to the large language model score is determined based on a preset second basic weight, the term density score, the professional normative score, and a weight ratio between the term density score and the professional normative score.

[0031] Optionally, determining a comprehensive score of each of the recognition results according to the weight, the first index value, and the second index value includes:

[0032] Determine a first product result between the first index value and the weight corresponding to the speech recognition model score;

[0033] Determine a second product result between the second index value and the weight corresponding to the large language model score;

[0034] A comprehensive score of each of the recognition results is determined based on the sum of the first multiplication result and the second multiplication result.

[0035] In a second aspect, the present application discloses a speech recognition system based on a large language model, comprising:

[0036] An input sequence construction module is used to input the speech to be recognized into the speech recognition model to obtain a number of recognition results, and construct an input sequence including the recognition results and context information of the recognition results; the recognition results include output text and speech recognition model scores; the speech recognition model scores are scores reflecting the degree of matching between the speech to be recognized and the output text;

[0037] A scoring acquisition module, used to score the input sequence using a large language model trained based on a domain knowledge graph to obtain a large language model score;

[0038] A weight determination module, used to calculate the indexes of the speech recognition model score and the large language model score respectively to obtain corresponding first index values ​​and second index values, and determine the weights corresponding to the speech recognition model score and the large language model score respectively;

[0039] The target recognition result selection module is used to determine the comprehensive score of each recognition result according to the weight, the first index value and the second index value, and select the target recognition result according to the comprehensive score.

[0040] In a third aspect, the present application discloses an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor is used to execute the computer program to implement the speech recognition method based on the large language model as described above.

[0043] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned large language model-based speech recognition method.

[0044] When performing speech recognition, the present application first inputs the speech to be recognized into the speech recognition model to obtain several recognition results, and constructs an input sequence containing the recognition results and the context information of the recognition results; the recognition results include output text and speech recognition model scores; the speech recognition model scores are scores reflecting the degree of matching between the speech to be recognized and the output text; the input sequence is scored by a large language model trained based on a domain knowledge graph to obtain a large language model score; the indexes of the speech recognition model score and the large language model score are calculated respectively to obtain the corresponding first index value and second index value, and the weights corresponding to the speech recognition model score and the large language model score are determined; the comprehensive score of each of the recognition results is determined according to the weight, the first index value and the second index value, and the target recognition result is selected according to the comprehensive score. It can be seen that the present application first outputs a plurality of most likely candidate recognition results by the speech recognition model, then scores the candidate results using the fine-tuned large language model, and then calculates the model weights, and finally fuses the scores according to the model weights and selects the final recognition result. In this way, the present application can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve the recognition accuracy of complex language scenarios and the ability to recognize professional terms. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flow chart of a speech recognition method based on a large language model disclosed in this application;

[0047] Figure 2 A knowledge injection and model optimization flow chart disclosed in this application;

[0048] Figure 3 A flowchart of a specific speech recognition method based on a large language model disclosed in this application;

[0049] Figure 4 A dynamic scoring fusion flow chart disclosed in this application;

[0050] Figure 5 A schematic diagram of the structure of a speech recognition system based on a large language model disclosed in this application;

[0051] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] At present, when processing speech in a specific field, traditional speech recognition systems often require a large amount of speech-text pair training data in that field. The cost of obtaining such data is extremely high, especially in professional fields such as medicine and law, where high-quality speech annotation data is particularly scarce. Secondly, existing domain adaptation methods mainly rely on retraining or fine-tuning of the complete speech recognition model, which not only requires a lot of computing resources, but also faces the problem of catastrophic forgetting. Even if incremental learning schemes are adopted, it is still difficult to achieve ideal improvement in model performance. Finally, the n-gram language model or simple neural network language model used by traditional speech recognition systems is difficult to accurately understand and process complex domain-specific expressions, especially for newly emerging professional terms or unconventional expressions, and often performs poorly. In order to solve the above technical problems, the present application discloses a speech recognition method, system, device and medium based on a large language model, which can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve recognition accuracy and recognition ability of professional terms.

[0054] See also Figure 1 As shown, the embodiment of the present invention discloses a speech recognition method based on a large language model, comprising:

[0055] Step S11, input the speech to be recognized into the speech recognition model to obtain a number of recognition results, and construct an input sequence including the recognition results and the context information of the recognition results; the recognition results include output text and speech recognition model scores; the speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text.

[0056] In this embodiment, before the speech to be recognized is input into the speech recognition model, a large language model is first obtained, such as Figure 2As shown in the figure, the domain text data corresponding to the speech to be recognized is collected; the domain text data includes professional literature and terminology; professional terms and usage patterns are extracted from the domain text data, and a domain knowledge graph is constructed according to the hierarchical relationship between professional terms, the frequency of use of professional terms, the contextual information of professional terms, and the usage pattern; the initial large language model is trained using the domain knowledge graph and LoRA technology to obtain a large language model. Specifically, professional terms and expression patterns are first extracted from the domain text to construct a domain knowledge graph. The knowledge graph contains not only the hierarchical relationship between terms, but also their frequency of use and contextual information. For example, in the medical field, the term "pleural effusion" has a synonymous relationship with terms such as "pleural effusion" and "pleural effusion", and has an associated relationship with symptom terms such as "chest tightness" and "dyspnea". Then, in the fine-tuning process of the large language model, the LoRA (Low-Rank Adaptation) technology is used to efficiently fine-tune the parameters. Specifically, for the weight matrix W in the model, two low-rank matrices A and B are introduced, so that the fine-tuned weight matrix becomes W + AB, where the ranks of A and B are much smaller than the rank of the original weight matrix. This approach not only maintains the basic language understanding ability of the model, but also can quickly adapt to the language characteristics of the new domain. Practice has shown that when the rank of the low-rank matrix is ​​set to 8, good domain adaptation effects can be achieved while maintaining a small number of parameters. In this way, low-cost domain adaptation is achieved through a text-based domain knowledge injection mechanism.

[0057] After the large language model is trained, it is applied to the decoding process of speech recognition. In traditional speech recognition, the decoder usually adopts n-gram language model or simple neural network language model. The present invention proposes to use a domain-adapted large language model to evaluate and optimize the decoding results. In specific implementation, Figure 3 As shown, the speech recognition model first outputs N most likely candidate recognition results. The recognition result includes the output text and the speech recognition model score, which is a score reflecting the matching degree between the speech to be recognized and the output text. For each candidate result, an input sequence containing complete context information is constructed and sent to the large language model for scoring.

[0058] Step S12: Score the input sequence using a large language model trained based on the domain knowledge graph to obtain a large language model score.

[0059] In this embodiment, the language fluency of the input sequence is determined by calculating the perplexity score of the input sequence; the domain relevance of the input sequence is determined according to the number of domain data contained in the recognition result; the professional standardization of the input sequence is determined by judging whether the language of the input sequence conforms to the expression habits and expression norms of the professional field; the input sequence is scored by a large language model trained based on the domain knowledge graph, language fluency, domain relevance and professional standardization to obtain a large language model score. Specifically, the input sequence containing complete context information is sent to the large language model for scoring, and the following three aspects are considered: language fluency, domain relevance and professional standardization. Language fluency is obtained by calculating the perplexity score; domain relevance is evaluated according to the number of domain terms contained in the candidate results; professional standardization is evaluated by the large model based on whether the language conforms to the expression habits and expression norms of the professional field. Therefore, the method of optimizing the decoding process using a large language model improves the recognition accuracy of complex language scenes.

[0060] Step S13: Calculate the indexes of the speech recognition model score and the large language model score respectively to obtain corresponding first index values ​​and second index values, and determine the weights corresponding to the speech recognition model score and the large language model score respectively.

[0061] In this embodiment, in order to comprehensively utilize the advantages of the speech recognition model and the large language model, the present application discloses a dynamic scoring multi-model fusion strategy, which is different from the traditional fixed weight fusion method. The weights of each model are dynamically adjusted according to the reliability of the input features. In this way, the dynamic scoring multi-model fusion strategy is adopted to achieve the adaptive fusion of the speech recognition model and the large language model. Figure 4 As shown, the fusion process first converts the acoustic model score and the language model score into probability values, specifically by calculating the exponent exp(x) of the score x. The speech recognition model score reflects the degree of match between the speech signal and the text, and the large language model score includes the comprehensive evaluation of the candidate results by the large language model. Therefore, firstly, the indices of the speech recognition model score and the large language model score are calculated respectively to obtain the corresponding first index value and second index value, and then the weights corresponding to the speech recognition model score and the large language model score are determined. The weight corresponding to the speech recognition model score is determined according to the signal-to-noise ratio and speech clarity of the speech to be recognized; the weight corresponding to the large language model score is determined based on the term density and professional norms of the recognition result; the term density represents the number of terms in the speech to be recognized.

[0062] In the process of determining the weight corresponding to the speech recognition model score, the signal-to-noise ratio of the speech to be recognized is first calculated, and the signal-to-noise ratio score is determined based on the relationship between the signal-to-noise ratio and a preset signal-to-noise ratio threshold; the speech clarity index of the speech to be recognized is calculated, and the speech clarity score is determined based on the speech clarity index; finally, the weight corresponding to the speech recognition model score is determined based on the preset first basic weight, the signal-to-noise ratio score, the speech clarity score, and the weight ratio between the signal-to-noise ratio score and the speech clarity score.

[0063] When determining the weight corresponding to the large language model score, first perform maximum positive matching on the recognition result and the domain knowledge graph to determine the number of terms in the recognition result; determine the term density based on the number of terms and a preset term number threshold; determine the term density score based on the term density; input the recognition result and the above information of the recognition result into the prompt template of the large language model to obtain the professional normative score of the recognition result; determine the weight corresponding to the large language model score based on the preset second basic weight, the term density score, the professional normative score, and the weight ratio between the term density score and the professional normative score.

[0064] Step S14: determine a comprehensive score for each of the recognition results according to the weight, the first index value, and the second index value, and select a target recognition result according to the comprehensive score.

[0065] In this embodiment, after determining the weights, first index values, and second index values ​​corresponding to the speech recognition model score and the large language model score, the first product result between the first index value and the weight corresponding to the speech recognition model score is determined; the second product result between the second index value and the weight corresponding to the large language model score is determined; and finally, the comprehensive score of each recognition result is determined based on the sum of the first product result and the second product result. In other words, when determining the final score, the system will dynamically adjust the weights of the two scores based on factors such as the signal-to-noise ratio of the input speech and the clarity of the speaker's pronunciation. For example, when the input speech quality is good, the weight of the acoustic model score will increase accordingly; when the input contains a large number of professional terms, the weight of the language model score will increase accordingly. The specific weight calculation formula is:

[0066] Score= (SNR,CLR)*exp(ASR_Score)+ (TERM,COH)*exp(LM_Score);

[0067] in, It is the weight function of the speech recognition model, which is related to the signal-to-noise ratio (SNR) and the clarity of pronunciation (CLR); It is the weight function of the large language model, which is related to the term density (TERM) and semantic coherence (COH); Score is the comprehensive score of the recognition result.

[0068] After finally obtaining the comprehensive score of the recognition results, select the target recognition results with high scores from several recognition results according to the comprehensive score.

[0069] In summary, when performing speech recognition, the present application first inputs the speech to be recognized into the speech recognition model to obtain several recognition results, and constructs an input sequence containing the recognition results and the context information of the recognition results; the recognition results include output text and speech recognition model scores; the speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text; the input sequence is scored by a large language model trained based on a domain knowledge graph to obtain a large language model score; the indexes of the speech recognition model score and the large language model score are calculated respectively to obtain the corresponding first index value and second index value, and the weights corresponding to the speech recognition model score and the large language model score are determined; the comprehensive score of each of the recognition results is determined according to the weight, the first index value and the second index value, and the target recognition result is selected according to the comprehensive score. It can be seen that the present application first outputs multiple most likely candidate recognition results from the speech recognition model, then scores the candidate results using the fine-tuned large language model, and then calculates the model weight, and finally fuses the score according to the model weight and selects the final recognition result. In this way, the present application can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve the recognition accuracy of complex language scenarios and the ability to recognize professional terms.

[0070] Based on the previous embodiment, it can be seen that the present application determines the weight corresponding to the speech recognition model score according to the signal-to-noise ratio and speech clarity of the speech to be recognized, and determines the weight corresponding to the large language model score based on the term density and professional standardization of the recognition result. Next, the specific weight determination process will be described in detail.

[0071] First, when determining the weight corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized, it can be known that the acoustic model weight , which is related to the signal-to-noise ratio (SNR) and the clarity of pronunciation (CLR). For the calculation of the signal-to-noise ratio score:

[0072] Signal-to-noise ratio calculation formula: SNR = 10 * log10 (signal energy / noise energy);

[0073] Normalized range:

[0074] SNR_min = 5dB (below which the speech quality is poor);

[0075] SNR_max = 20dB (the improvement above this value is not obvious, and there is no need to increase the acoustic weight);

[0076] These two thresholds are set based on practical experience in speech recognition. If SNR< SNR_min, then SNR_score (signal-to-noise ratio score) = 0 (this speech is poor and cannot be used as a reference); if SNR> SNR_max, then SNR_score (signal-to-noise ratio score) = 1; if SNR_min < SNR< SNR_max, then SNR_score (signal-to-noise ratio score) = (SNR - SNR_min) / (SNR_max - SNR_min).

[0077] To calculate the speech intelligibility score, you need to calculate the Speech Intelligibility Index (SII) based on the ANSI S3.5-1997 standard; CLR_score (speech intelligibility score) = SII (already in the range of 0-1).

[0078] Final acoustic model weights Calculation:

[0079] = 0.2 + 0.6 * (0.6 * SNR_score + 0.4 * CLR_score);

[0080] Among them, 0.2 is the basic weight and 0.6 is the dynamic adjustment range. The weight ratio of SNR and CLR is 6:4, reflecting the main impact of signal-to-noise ratio on recognition accuracy. These parameters are derived from experience and can be adjusted according to the actual scene effect.

[0081] When determining the weight corresponding to the large language model score based on the term density and professional norms of the recognition results, the language model weight , which is related to term density (TERM) and professional normativeness (PROF). For term density score calculation:

[0082] Use the domain term knowledge graph constructed in the previous embodiment to perform the maximum positive match and calculate:

[0083] Term density = number of detected terms / 3 (set upper limit to 3 terms);

[0084] Final term density score: if the term is not detected, TERM_score = 0; if term density > 1, TERM_score = 1; otherwise, TERM_score = term density.

[0085] For the calculation of professional normativeness score: through the large language model evaluation, the sentence currently being recognized and its previous sentence are filled into the prompt template. The template needs to be adjusted according to the application field. For example, the template example in the medical field is as follows:

[0086] Please rate the professionalism and standardization of the following medical descriptions:

[0087] Above: [previous sentence];

[0088] Current sentence: [current recognition result];

[0089] Please rate the professionalism of this statement from 0 to 1, where:

[0090] - 0 points: completely inconsistent with the expression habits in the medical field;

[0091] - 0.5 points: basically in line with medical expressions, but there may be non-standard terms;

[0092] - 1 point: fully complies with professional medical expression standards;

[0093] Just output the score, no explanation required.

[0094] PROF_score = the score returned by the large language model (already in the range 0-1).

[0095] Final language model weights Calculation:

[0096] = 0.3 + 0.5 * (0.7 * TERM_score + 0.3 * PROF_score);

[0097] Among them, 0.3 is the basic weight and 0.5 is the dynamic adjustment range. The weight ratio of TERM and PROF is 7:3, reflecting the importance of terminology recognition in professional fields. These parameters can be adjusted according to specific application scenarios.

[0098] In this way, this application adopts a multi-model fusion strategy with dynamic scoring, realizes the adaptive fusion of acoustic model and language model, and further improves the accuracy of speech recognition and the ability to recognize professional terms.

[0099] See also Figure 5As shown, the embodiment of the present invention discloses a speech recognition system based on a large language model, including:

[0100] An input sequence construction module 11 is used to input the speech to be recognized into the speech recognition model to obtain a plurality of recognition results, and to construct an input sequence including the recognition results and context information of the recognition results; the recognition results include output text and speech recognition model scores; the speech recognition model scores are scores reflecting the degree of matching between the speech to be recognized and the output text;

[0101] A score acquisition module 12 is used to score the input sequence using a large language model trained based on a domain knowledge graph to obtain a large language model score;

[0102] A weight determination module 13, used to calculate the indexes of the speech recognition model score and the large language model score respectively to obtain corresponding first index values ​​and second index values, and determine the weights corresponding to the speech recognition model score and the large language model score respectively;

[0103] The target recognition result selection module 14 is used to determine the comprehensive score of each of the recognition results according to the weight, the first index value and the second index value, and select the target recognition result according to the comprehensive score.

[0104] In summary, when performing speech recognition, the present application first inputs the speech to be recognized into the speech recognition model to obtain several recognition results, and constructs an input sequence containing the recognition results and the context information of the recognition results; the recognition results include output text and speech recognition model scores; the speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text; the input sequence is scored by a large language model trained based on a domain knowledge graph to obtain a large language model score; the indexes of the speech recognition model score and the large language model score are calculated respectively to obtain the corresponding first index value and second index value, and the weights corresponding to the speech recognition model score and the large language model score are determined; the comprehensive score of each of the recognition results is determined according to the weight, the first index value and the second index value, and the target recognition result is selected according to the comprehensive score. It can be seen that the present application first outputs multiple most likely candidate recognition results from the speech recognition model, then scores the candidate results using the fine-tuned large language model, and then calculates the model weight, and finally fuses the score according to the model weight and selects the final recognition result. In this way, the present application can significantly reduce the adaptation cost of speech recognition systems in specific fields, improve the recognition accuracy of complex language scenarios and the ability to recognize professional terms.

[0105] In some specific embodiments, the system can also be used to collect domain text data corresponding to the speech to be recognized; the domain text data includes professional literature and a glossary; professional terms and usage patterns are extracted from the domain text data, and the domain knowledge graph is constructed according to the hierarchical relationship between the professional terms, the frequency of use of the professional terms, the contextual information of the professional terms and the usage patterns; the initial large language model is trained using the domain knowledge graph and LoRA technology to obtain the large language model.

[0106] In some specific embodiments, the input sequence construction module 11 can be specifically used to determine the language fluency of the input sequence by calculating the perplexity score of the input sequence; determine the domain relevance of the input sequence according to the amount of domain data contained in the recognition result; determine the professional standardization of the input sequence by judging whether the language of the input sequence conforms to the expression habits and expression norms of the professional field; score the input sequence based on a large language model trained based on the domain knowledge graph, the language fluency, the domain relevance and the professional standardization to obtain a large language model score.

[0107] In some specific embodiments, the weight determination module 13 can be specifically used to determine the weight corresponding to the speech recognition model score based on the signal-to-noise ratio and speech clarity of the speech to be recognized; determine the weight corresponding to the large language model score based on the term density and professional standardization of the recognition result; the term density characterizes the number of terms in the speech to be recognized.

[0108] In some specific embodiments, the weight determination module 13 can be specifically used to calculate the signal-to-noise ratio of the speech to be recognized, and determine the signal-to-noise ratio score according to the relationship between the signal-to-noise ratio and a preset signal-to-noise ratio threshold; calculate the speech clarity index of the speech to be recognized, and determine the speech clarity score according to the speech clarity index; determine the weight corresponding to the speech recognition model score based on a preset first basic weight, the signal-to-noise ratio score, the speech clarity score, and the weight ratio between the signal-to-noise ratio score and the speech clarity score.

[0109] In some specific embodiments, the weight determination module 13 can be specifically used to perform maximum positive matching between the recognition result and the domain knowledge graph to determine the number of terms in the recognition result; determine the term density based on the number of terms and a preset term number threshold; determine the term density score based on the term density; input the recognition result and the above information of the recognition result into the prompt template of the large language model to obtain the professional normative score of the recognition result; determine the weight corresponding to the large language model score based on a preset second basic weight, the term density score, the professional normative score, and the weight ratio between the term density score and the professional normative score.

[0110] In some specific embodiments, the target recognition result selection module 14 can be specifically used to determine a first product result between the first index value and the weight corresponding to the speech recognition model score; determine a second product result between the second index value and the weight corresponding to the large language model score; and determine a comprehensive score for each of the recognition results based on the sum of the first product result and the second product result.

[0111] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0112] Figure 6 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the speech recognition method based on a large language model disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0113] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0114] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0115] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the speech recognition method based on the large language model performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0116] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned large language model-based speech recognition method is implemented. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0117] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0118] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0120] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0121] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech recognition method based on a large language model, characterized in that: include: Inputting the speech to be recognized into the speech recognition model to obtain a number of recognition results, and constructing an input sequence including the recognition results and context information of the recognition results; the recognition results include output text and speech recognition model scores; The speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text; Scoring the input sequence using a large language model trained based on a domain knowledge graph to obtain a large language model score; Calculating the indexes of the speech recognition model score and the large language model score respectively to obtain corresponding first index values ​​and second index values, and determining the weights corresponding to the speech recognition model score and the large language model score respectively; A comprehensive score of each of the recognition results is determined according to the weight, the first index value, and the second index value, and a target recognition result is selected according to the comprehensive score.

2. The speech recognition method based on a large language model according to claim 1, characterized in that: Before inputting the speech to be recognized into the speech recognition model, the method further includes: Collecting domain text data corresponding to the speech to be recognized; the domain text data includes professional literature and a glossary; Extracting professional terms and usage patterns from the domain text data, and constructing the domain knowledge graph according to the hierarchical relationship between the professional terms, the usage frequency of the professional terms, the context information of the professional terms, and the usage patterns; The domain knowledge graph and LoRA technology are used to train the initial large language model to obtain the large language model.

3. The speech recognition method based on a large language model according to claim 1, characterized in that: Scoring the input sequence by using a large language model trained based on a domain knowledge graph to obtain a large language model score includes: Determining the language fluency of the input sequence by calculating the perplexity score of the input sequence; Determining the domain relevance of the input sequence according to the amount of domain data included in the recognition result; Determining the professional standardization of the input sequence by judging whether the language of the input sequence conforms to the expression habits and expression standards of the professional field; The input sequence is scored by using a large language model trained based on a domain knowledge graph, the language fluency, the domain relevance, and the professional norms to obtain a large language model score.

4. The speech recognition method based on a large language model according to claim 1, characterized in that: The determining of the weights corresponding to the speech recognition model score and the large language model score respectively includes: Determining a weight corresponding to the speech recognition model score according to the signal-to-noise ratio and speech clarity of the speech to be recognized; The weight corresponding to the large language model score is determined based on the term density and professional norms of the recognition result; the term density represents the number of terms in the speech to be recognized.

5. The speech recognition method based on a large language model according to claim 4, characterized in that: The step of determining the weight corresponding to the speech recognition model score according to the signal-to-noise ratio and speech clarity of the speech to be recognized includes: Calculating the signal-to-noise ratio of the speech to be recognized, and determining a signal-to-noise ratio score according to a magnitude relationship between the signal-to-noise ratio and a preset signal-to-noise ratio threshold; Calculating a speech clarity index of the speech to be recognized, and determining a speech clarity score according to the speech clarity index; The weight corresponding to the speech recognition model score is determined based on a preset first basic weight, the signal-to-noise ratio score, the speech intelligibility score, and a weight ratio between the signal-to-noise ratio score and the speech intelligibility score.

6. The speech recognition method based on a large language model according to claim 4, characterized in that: The determining of the weight corresponding to the large language model score based on the term density and professional norms of the recognition result includes: Performing maximum positive matching between the recognition result and the domain knowledge graph to determine the number of terms in the recognition result; Determining the term density according to the term quantity and a preset term quantity threshold; determining a term density score based on the term density; Inputting the recognition result and the context information of the recognition result into the prompt template of the large language model to obtain a professional normative score of the recognition result; The weight corresponding to the large language model score is determined based on a preset second basic weight, the term density score, the professional normative score, and a weight ratio between the term density score and the professional normative score.

7. The speech recognition method based on a large language model according to any one of claims 1 to 6, characterized in that: Determining a comprehensive score of each of the recognition results according to the weight, the first index value, and the second index value includes: Determine a first product result between the first index value and the weight corresponding to the speech recognition model score; Determine a second product result between the second index value and the weight corresponding to the large language model score; A comprehensive score of each of the recognition results is determined based on the sum of the first multiplication result and the second multiplication result.

8. A speech recognition system based on a large language model, characterized in that: include: An input sequence construction module is used to input the speech to be recognized into the speech recognition model to obtain a number of recognition results, and construct an input sequence including the recognition results and context information of the recognition results; the recognition results include output text and speech recognition model scores; The speech recognition model score is a score reflecting the degree of matching between the speech to be recognized and the output text; A scoring acquisition module, used to score the input sequence using a large language model trained based on a domain knowledge graph to obtain a large language model score; A weight determination module, used to calculate the indexes of the speech recognition model score and the large language model score respectively to obtain corresponding first index values ​​and second index values, and determine the weights corresponding to the speech recognition model score and the large language model score respectively; The target recognition result selection module is used to determine the comprehensive score of each recognition result according to the weight, the first index value and the second index value, and select the target recognition result according to the comprehensive score.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the speech recognition method based on a large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the speech recognition method based on a large language model as described in any one of claims 1 to 7 is implemented.

Images