Scoring method and device of recognition engine, storage medium and electronic device

Abstract

The invention discloses a scoring method and device of a recognition engine, a storage medium and an electronic device, and relates to the technical field of smart home, and the method comprises the steps: inputting a target audio into a trained voice large model to obtain a first recognition text, and obtaining a second recognition text output by processing the target audio through a third-party text recognition engine; calculating a plurality of index parameters between a first phoneme sequence corresponding to the first recognition text and a second phoneme sequence corresponding to the second recognition text, the plurality of index parameters comprise at least one of the following parameters: similarity between different phoneme sequences, common phoneme proportion between different phoneme sequences, phoneme error rate between different phoneme sequences, and phoneme length matching degree between different phoneme sequences; summarizing the index parameters according to preset parameter weights to obtain a target score of the third-party text recognition engine for recognizing the target audio, the target score being used for indicating the quality of the recognition text output by the third-party text recognition engine.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a scoring method, apparatus, storage medium, and electronic device for a recognition engine. Background Technology

[0002] With an increasing number of devices supporting voice input, voice recognition has become a crucial component of voice services. Device manufacturers often rely on third-party recognition services due to technological and cost constraints. Different devices operate in varying noise environments, resulting in inconsistent audio quality. While the recognition engine provides the text, it cannot guarantee that the recognized text is the speaker's exact words; in fact, the engine may output entirely different text. Since subsequent skill-based services depend on the recognized text, this can lead to irrelevant responses. For example, unclear narration might be misinterpreted as "20 degrees." Therefore, recognition engines sometimes misinterpret unclear audio as meaningful control or skill commands, resulting in misrecognition / overrecognition in voice recognition. This can easily lead to interaction failures during subsequent device voice interactions.

[0003] Therefore, in related technologies, there is no effective solution to the problem of not being able to effectively assess the reliability of the recognition results of third-party recognition engines. Summary of the Invention

[0004] This application provides a scoring method, apparatus, storage medium, and electronic device for a recognition engine, to at least solve the problem in the related art that the reliability of the recognition results of third-party recognition engines cannot be effectively evaluated.

[0005] According to one embodiment of this application, a scoring method for a recognition engine is provided, comprising: inputting target audio into a pre-trained large speech model to obtain a first recognized text, and obtaining a second recognized text output by a third-party text recognition engine after processing the target audio; calculating multiple indicator parameters between a first phoneme sequence corresponding to the first recognized text and a second phoneme sequence corresponding to the second recognized text, wherein the multiple indicator parameters include at least one of the following: similarity between different phoneme sequences, proportion of common phonemes between different phoneme sequences, phoneme error rate between different phoneme sequences, and phoneme length matching degree between different phoneme sequences; summarizing the multiple indicator parameters according to preset parameter weights to obtain a target score for the third-party text recognition engine in recognizing the target audio, wherein the target score is used to indicate the quality of the recognized text output by the third-party text recognition engine.

[0006] In an exemplary embodiment, calculating multiple index parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text includes: when the currently calculated index parameter is the similarity between different phoneme sequences, determining the first length corresponding to the longest common subsequence between the first phoneme sequence and the second phoneme sequence, and the second length corresponding to the total sequence after connecting the first phoneme sequence and the second phoneme sequence; substituting the first length and the second length into a preset first formula to obtain the first index parameter corresponding to the similarity, wherein the preset first formula is: similarity = 2.0 × first length / second length.

[0007] In an exemplary embodiment, calculating multiple index parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text includes: when the currently calculated index parameter is the proportion of common phonemes between different phoneme sequences, determining a third length corresponding to the common phonemes between the first phoneme sequence and the second phoneme sequence, and a fourth length corresponding to the first phoneme sequence; substituting the third length and the fourth length into a preset second formula to obtain a second index parameter corresponding to the proportion of common phonemes, wherein the preset second formula is: proportion of common phonemes = third length / fourth length.

[0008] In an exemplary embodiment, calculating multiple index parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text includes: when the currently calculated index parameter is the phoneme error rate between different phoneme sequences, determining a fifth length corresponding to the edit distance between the first phoneme sequence and the second phoneme sequence involved in sequence conversion, and a sixth length corresponding to the longest phoneme sequence among the first phoneme sequence and the second phoneme sequence; substituting the fifth length and the sixth length into a preset third formula to obtain a third index parameter corresponding to the phoneme error rate, wherein the preset third formula is: phoneme error rate = fourth length / fifth length.

[0009] In an exemplary embodiment, calculating multiple index parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text includes: when the currently calculated index parameter is the phoneme length matching degree between different phoneme sequences, dividing the seventh length corresponding to the second phoneme sequence by the fourth length corresponding to the first phoneme sequence to obtain the first length ratio; substituting the first length ratio into a preset fourth formula to obtain the fourth index parameter corresponding to the phoneme length matching degree, wherein the preset fourth formula is: phoneme length matching degree = 1.0 - min(abs(1.0 - first length ratio), 1.0).

[0010] In one exemplary embodiment, the third-party text recognition engine summarizes and processes the multiple indicator parameters according to preset parameter weights to obtain a target score for recognizing the target audio. This includes: obtaining multiple target weight values ​​corresponding to the multiple indicator parameters, wherein the multiple target weight values ​​are dynamic values ​​determined based on the historical audio recognition accuracy of the third-party text recognition engine; and performing a weighted summation based on the multiple target weight values ​​and the multiple target indicator parameters to obtain the target score.

[0011] In an exemplary embodiment, after summarizing the multiple indicator parameters according to preset parameter weights to obtain the target score for the third-party text recognition engine to recognize the target audio, the method further includes: if the target score is less than a preset score threshold, determining that the second recognized text output by the third-party text recognition engine has a recognition anomaly, and adjusting the configuration of the third-party text recognition engine; if the target score is greater than or equal to the preset score threshold, determining that the second recognized text output by the third-party text recognition engine does not have a recognition anomaly, and allowing the recognized text output by the third-party text recognition engine to be used as reference text for audio interaction.

[0012] According to another aspect of the embodiments of this application, a scoring device for a recognition engine is also provided, comprising: a recognition module, configured to input target audio into a pre-trained large speech model to obtain a first recognized text, and to obtain a second recognized text output by a third-party text recognition engine after processing the target audio; a calculation module, configured to calculate a plurality of index parameters between a first phoneme sequence corresponding to the first recognized text and a second phoneme sequence corresponding to the second recognized text, wherein the plurality of index parameters include at least one of the following: similarity between different phoneme sequences, proportion of common phonemes between different phoneme sequences, phoneme error rate between different phoneme sequences, and phoneme length matching degree between different phoneme sequences; and a processing module, configured to summarize and process the plurality of index parameters according to preset parameter weights to obtain a target score for the recognition of the target audio by the third-party text recognition engine, wherein the target score is used to indicate the quality of the recognized text output by the third-party text recognition engine.

[0013] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0014] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0016] In this embodiment, the audio data to be evaluated is input into a high-performance open-source speech recognition model. This model, after thorough training, is capable of generating a first recognized text with high accuracy. Subsequently, a second recognized text is obtained from the same target audio, output by a third-party recognition engine. Next, based on the comparison of phoneme sequences of initials and finals, multiple evaluation metrics are calculated between the phoneme sequences converted from the first and second recognized texts, including phoneme sequence similarity, common phoneme proportion, phoneme error rate, and phoneme length matching. These metrics comprehensively examine the accuracy of the recognition results from different dimensions. For example, phoneme sequence similarity measures the degree of matching between the two texts at the phoneme level, the common phoneme proportion reflects the proportion of shared phonemes between the two texts, the phoneme error rate assesses the differences at the phoneme level, and the phoneme length matching examines the consistency of phoneme lengths in the recognized texts. Finally, these metrics are combined with preset weights, and a target score is generated through weighted summation to directly reflect the recognition quality of the third-party recognition engine when processing specific audio. The above technical solution solves the problem of the inability to effectively evaluate the reliability of the recognition results of third-party recognition engines in related technologies. By introducing an independent, open-source speech recognition model as a notary, it fills the gap of lack of objective third-party evaluation in existing technologies, effectively improves the credibility and accuracy of speech recognition results, and realizes effective judgment of third-party recognition engines in handling non-standardized pronunciation, dialects and noisy environments. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment for a scoring method of a recognition engine according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of the scoring method of the recognition engine according to the embodiments of this application;

[0021] Figure 3 This is a schematic diagram of the system architecture for scoring the recognition results of a third-party recognition engine according to an embodiment of this application;

[0022] Figure 4 This is a structural block diagram of a scoring device for a recognition engine according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, apparatus, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, apparatus, or devices.

[0025] According to one aspect of the embodiments of this application, a scoring method for a recognition engine is provided. This scoring method for the recognition engine is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned scoring method for the recognition engine can be applied to, for example... Figure 1 The hardware environment shown consists of voice device 102 and server 104. Figure 1 This is a schematic diagram of the hardware environment for a scoring method of a recognition engine according to an embodiment of this application, such as... Figure 1As shown, server 104 is connected to voice device 102 via a network and can be used to provide services (such as application services) to terminals or clients installed on terminals. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0026] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless FiDelity), Bluetooth. The voice device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0027] This embodiment provides a scoring method for a recognition engine, applied to the aforementioned voice device. Figure 2 This is a flowchart of a scoring method for a recognition engine according to an embodiment of this application, which includes the following steps:

[0028] Step S202: Input the target audio into the trained speech model to obtain the first recognized text, and obtain the second recognized text output by the third-party text recognition engine after processing the target audio;

[0029] Optionally, the target audio file requiring scoring can be submitted to a pre-trained open-source speech recognition model. This model, using advanced technologies such as deep learning, can extract clear speech signals from the audio and convert them into the first recognized text. Simultaneously, the second recognized text is obtained from the same audio file after processing by a third-party speech recognition service. These two independent recognition results will be used for subsequent phoneme sequence comparison and scoring.

[0030] Step S204: Calculate multiple indicator parameters between the first phoneme sequence corresponding to the first recognized text and the second phoneme sequence corresponding to the second recognized text. The multiple indicator parameters include at least one of the following: similarity between different phoneme sequences, proportion of common phonemes between different phoneme sequences, phoneme error rate between different phoneme sequences, and phoneme length matching degree between different phoneme sequences.

[0031] Optionally, the first and second identified texts are converted into a first phoneme sequence and a second phoneme sequence, respectively, consisting of initials and finals. This conversion process is based on Chinese pronunciation rules, mapping characters to their most basic pronunciation units. By comparing the phoneme sequences, several index parameters are calculated, including the similarity of the phoneme sequences, the proportion of common phonemes, the phoneme error rate, and the phoneme length matching degree.

[0032] Step S206: Summarize and process multiple indicator parameters according to preset parameter weights to obtain the target score of the target audio recognized by the third-party text recognition engine. The target score is used to indicate the quality of the recognized text output by the third-party text recognition engine.

[0033] Optionally, multiple calculated indicator parameters can be weighted and summed according to preset parameter weights. The choice of weights depends on the specific application requirements and the importance of the indicators. By adjusting the weights of different indicators, the evaluation results can be optimized to more accurately reflect the recognition quality. After summarization, a comprehensive score, or target score, is generated to intuitively represent the accuracy and reliability of the third-party recognition engine in recognizing the target audio. A high score means that the recognition result closely matches the content of the original audio, while a low score indicates that there may be recognition errors or uncertainties.

[0034] Through the steps described above, the audio data to be evaluated is input into a high-performance open-source speech recognition model. This model, after thorough training, can generate highly accurate first-recognition text. Subsequently, the same target audio is used to obtain second-recognition text output by a third-party recognition engine. Next, based on the comparison of phoneme sequences of initials and finals, multiple evaluation metrics are calculated between the phoneme sequences converted from the first and second-recognition texts, including phoneme sequence similarity, common phoneme percentage, phoneme error rate, and phoneme length matching. These metrics comprehensively examine the accuracy of the recognition results from different dimensions. For example, phoneme sequence similarity measures the degree of matching between the two texts at the phoneme level, the common phoneme percentage reflects the proportion of shared phonemes between the two texts, the phoneme error rate assesses differences at the phoneme level, and the phoneme length matching examines the consistency of phoneme lengths in the recognized texts. Finally, these metrics are combined with preset weights and a target score is generated through weighted summation to directly reflect the recognition quality of the third-party recognition engine when processing specific audio. The above technical solution solves the problem of the inability to effectively evaluate the reliability of the recognition results of third-party recognition engines in related technologies. By introducing an independent, open-source speech recognition model as a notary, it fills the gap of lack of objective third-party evaluation in existing technologies, effectively improves the credibility and accuracy of speech recognition results, and realizes effective judgment of third-party recognition engines in handling non-standardized pronunciation, dialects and noisy environments.

[0035] In an exemplary embodiment, calculating multiple index parameters between a first phoneme sequence corresponding to a first identified text and a second phoneme sequence corresponding to a second identified text includes: when the currently calculated index parameter is the similarity between different phoneme sequences, determining a first length corresponding to the longest common subsequence between the first and second phoneme sequences, and a second length corresponding to the total sequence after connecting the first and second phoneme sequences; substituting the first length and the second length into a preset first formula to obtain the first index parameter corresponding to the similarity, wherein the preset first formula is: similarity = 2.0 × first length / second length.

[0036] Understandably, when calculating the similarity between different phoneme sequences, the algorithm first identifies the phonemes of the first and second identified texts, generating their respective phoneme sequences: the first phoneme sequence and the second phoneme sequence. Next, the algorithm finds the longest common subsequence between these two sets of phoneme sequences—the longest continuous string of phonemes shared by both sequences, regardless of phoneme order. After determining this longest common subsequence, the number of phonemes it contains is calculated, which is the first length. Then, the first and second phoneme sequences are merged, and the total number of phonemes in the merged sequence is calculated, yielding the second length. These two length values ​​are used to calculate the similarity index. Finally, a pre-defined first formula is used to calculate the similarity of the phoneme sequences: Similarity = 2.0 × First Length / Second Length. By utilizing the ratio between the length of the common subsequence and the total sequence length, the similarity between the two sets of phoneme sequences is effectively evaluated. Multiplying the ratio by 2.0 is to better balance the contributions of the two sequences in the calculation, ensuring a more fair and reasonable result. The first indicator parameter obtained, namely similarity, is that the higher the value, the higher the degree of matching between the two phoneme sequences, thus indirectly reflecting the consistency level between the second recognized text output by the third-party recognition engine and the recognition result of the large model at the phoneme level.

[0037] In an exemplary embodiment, calculating multiple index parameters between a first phoneme sequence corresponding to a first identified text and a second phoneme sequence corresponding to a second identified text includes: when the currently calculated index parameter is the proportion of common phonemes between different phoneme sequences, determining a third length corresponding to the common phonemes between the first phoneme sequence and the second phoneme sequence, and a fourth length corresponding to the first phoneme sequence; substituting the third length and the fourth length into a preset second formula to obtain a second index parameter corresponding to the proportion of common phonemes, wherein the preset second formula is: proportion of common phonemes = third length / fourth length.

[0038] In simple terms, the evaluation assesses the degree of overlap at the phoneme level between the phoneme sequences converted from the first and second recognized texts, i.e., the proportion of common phonemes between them. This evaluation is achieved by calculating the proportion of common phonemes, which directly reflects the degree of phoneme matching between the two recognition results, thus revealing the reliability of the third-party recognition engine's output. Specifically, the process first requires determining the number of common phonemes in the first and second phoneme sequences, i.e., the third length. This step essentially involves finding and counting completely identical phonemes in the two phoneme sequences, regardless of their position within the sequences. Next, the total number of phonemes in the first phoneme sequence is counted, i.e., the fourth length, which provides a base for subsequent calculations. The formula for calculating the proportion of common phonemes is: Proportion of common phonemes = Third length / Fourth length. This formula expresses the proportion of common phonemes to the total number of phonemes in the first recognized text. The higher this ratio, the higher the degree of phoneme matching between the third-party recognition engine and the open-source large model when recognizing the same audio, thus indirectly verifying the accuracy of the recognition results. Conversely, a low ratio suggests a greater number of phoneme differences, which may indicate misidentification or overidentification during the recognition process.

[0039] In an exemplary embodiment, calculating multiple index parameters between a first phoneme sequence corresponding to a first identified text and a second phoneme sequence corresponding to a second identified text includes: when the currently calculated index parameter is the phoneme error rate between different phoneme sequences, determining a fifth length corresponding to the edit distance between the first and second phoneme sequences involved in sequence conversion, and a sixth length corresponding to the longest phoneme sequence among the first and second phoneme sequences; substituting the fifth length and the sixth length into a preset third formula to obtain a third index parameter corresponding to the phoneme error rate, wherein the preset third formula is: phoneme error rate = fourth length / fifth length.

[0040] It should be noted that the phoneme error rate is used as a metric to evaluate the difference in phoneme sequences after conversion between the first and second recognized texts. The phoneme error rate is calculated based on the concept of edit distance, a general algorithm for measuring the similarity between two sequences. It effectively reflects the minimum number of editing operations required to convert one sequence to another, including the insertion, deletion, and replacement of phonemes.

[0041] The specific calculation process includes: determining the minimum distance required for the first and second phoneme sequences to achieve complete consistency through editing operations, i.e., the fifth length, which is called the edit distance in sequence alignment algorithms. Then, the longer of the two phoneme sequences is identified, i.e., the sixth length, as a benchmark for error rate calculation. The choice of the sixth length is based on the logic that the longer phoneme sequence in the recognition result usually contains more information, thus making it more representative for error rate calculation. The phoneme error rate calculation formula follows the preset third formula: Phoneme Error Rate = Fifth Length / Sixth Length. The preset third formula expresses the proportional relationship between the edit distance and the length of the longer phoneme sequence. The lower the phoneme error rate, the smaller the difference between the two sequences, meaning that the output of the third-party recognition engine is closer to the recognition result of the open-source large model at the phoneme level, and the higher the recognition quality. Conversely, a high phoneme error rate indicates that there may be many misidentifications or overidentifications during the recognition process, affecting the accuracy of recognition.

[0042] In an exemplary embodiment, calculating multiple index parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text includes: when the currently calculated index parameter is the phoneme length matching degree between different phoneme sequences, dividing the seventh length corresponding to the second phoneme sequence by the fourth length corresponding to the first phoneme sequence to obtain the first length ratio; substituting the first length ratio into a preset fourth formula to obtain the fourth index parameter corresponding to the phoneme length matching degree, wherein the preset fourth formula is: phoneme length matching degree = 1.0 - min(abs(1.0 - first length ratio), 1.0).

[0043] Optionally, phoneme length matching is used to determine the compatibility of the first and second recognized texts in terms of phoneme sequence length, thereby evaluating whether the recognition result of the third-party recognition engine matches the expected audio content length. Phoneme length matching reflects the degree of consistency between the number of syllables in the recognition result and the actual number of syllables in the audio content, and can be used to determine whether the recognition result is affected by additional factors such as background noise. The calculation process of phoneme length matching includes: obtaining the phoneme sequence length corresponding to the second recognized text, denoted as the seventh length, and correspondingly the phoneme sequence length of the first recognized text, denoted as the fourth length. Then, the ratio between the two lengths is calculated, that is, the result of the seventh length divided by the fourth length, which is called the first length ratio. This ratio intuitively shows the relative difference between the lengths of the two phoneme sequences. The preset fourth formula is used to convert the first length ratio into a phoneme length matching index parameter, and the formula is defined as: phoneme length matching = 1.0 - min(abs(1.0 - first length ratio), 1.0). Through this formula, the length ratio can be standardized to the interval [0,1], where 1 indicates that the lengths of the two sequences are completely consistent, and 0 indicates that the length difference is the largest. Here, a combination of absolute value function and minimum value function is used to ensure that the calculation result of phoneme length matching degree can reflect the positive matching case, such as when the proportion of the first length is close to 1, the matching degree approaches 1, and can also properly handle the reverse case, such as when the proportion of the first length is much less than or greater than 1, the matching degree is limited to a lower range, thereby avoiding the deviation that may be introduced by directly using the length difference.

[0044] In one exemplary embodiment, the target score for the target audio recognized by the third-party text recognition engine is obtained by summarizing multiple indicator parameters according to preset parameter weights. This includes: obtaining multiple target weight values ​​corresponding to multiple indicator parameters, wherein the multiple target weight values ​​are dynamic values ​​determined based on the historical audio recognition accuracy of the third-party text recognition engine; and performing a weighted summation based on the multiple target weight values ​​and multiple target indicator parameters to obtain the target score.

[0045] Understandably, each indicator parameter is assigned a target weight value. These weights are not fixed but dynamically adjusted based on the historical audio recognition accuracy of the third-party recognition engine. Analyzing historical accuracy reveals which indicators are more decisive in specific recognition environments. For example, in high-noise environments, phoneme error rate may be given higher weight because it is more sensitive to capturing potential phoneme misjudgments during recognition; conversely, when text length varies significantly, phoneme length matching becomes more important, and the weight allocation is adjusted accordingly. Then, these indicator parameters are combined with their corresponding target weight values ​​and summed using a weighted average. This process is achieved through a pre-defined summation formula, which allows the scores of different indicator parameters to be linearly superimposed according to their weights, resulting in a comprehensive score, or target score. The target score directly reflects the overall recognition quality and reliability of the third-party recognition engine when processing target audio. Furthermore, by dynamically adjusting the weight values ​​of each indicator parameter, the scoring results are ensured to flexibly adapt to the recognition quality assessment needs of different scenarios.

[0046] In an exemplary embodiment, after summarizing multiple indicator parameters according to preset parameter weights to obtain the target score for the target audio recognized by the third-party text recognition engine, the method further includes: if the target score is less than a preset score threshold, determining that the second recognized text output by the third-party text recognition engine has a recognition anomaly, and adjusting the configuration of the third-party text recognition engine accordingly; if the target score is greater than or equal to the preset score threshold, determining that the second recognized text output by the third-party text recognition engine does not have a recognition anomaly, and allowing the recognized text output by the third-party text recognition engine to be used as the reference text for audio interaction.

[0047] That is to say, the calculated target score comprehensively reflects the evaluation results in multiple dimensions such as the phoneme sequence similarity, the proportion of common phonemes, the phoneme error rate, and the phoneme length matching degree. And this target score is obtained by weighted summation of each index parameter under the action of preset parameter weights. The weight value is dynamically adjusted according to the historical audio recognition accuracy of the third-party recognition engine, so as to ensure that the scoring result can more accurately reflect the real-time recognition performance of the engine. Further, based on the comparison between the target score and the preset score threshold, it is decided whether to use the recognition result of the third-party recognition engine as the basis for audio interaction. Specifically, when the target score is lower than the preset score threshold, the recognition result of the third-party recognition engine is marked as having a recognition anomaly, prompting the user or system administrator that the recognition result may contain misrecognition or over-recognition errors, and the third-party recognition engine needs to be configured and adjusted to optimize its recognition accuracy. On the contrary, if the target score reaches or exceeds the preset threshold, then it can be determined that the output result of the third-party recognition engine is accurate and can be safely used as the reference text for the subsequent audio interaction process without additional review or adjustment.

[0048] To better understand the process of the above-mentioned scoring method of the recognition engine, the following further describes the scoring method process of the above-mentioned recognition engine in combination with optional embodiments, but it is not used to limit the technical solutions of the embodiments of the present application.

[0049] As an optional implementation solution, an embodiment of the present application proposes a method for scoring speech recognition results. The main process includes: sending the audio to an open-source speech recognition large model for secondary recognition to obtain the text recognized by the large model. Converting the text recognized by the large model into a first initial-final phoneme sequence; converting the third-party recognized text into a second initial-final phoneme sequence; calculating the phoneme sequence similarity, the proportion of common phonemes, the phoneme error rate, and the phoneme length matching degree between the two initial-final phoneme sequences, and performing weighted summation on the phoneme sequence similarity, the proportion of common phonemes, the phoneme error rate, and the phoneme length matching degree to obtain the score corresponding to the third-party recognized text. Furthermore, the open-source speech recognition large model is used to recognize and verify the output result of the third-party speech recognition service. If the score is high, it means that the third-party recognition is accurate. If the score is low, it means that the third-party recognition is problematic and is a misrecognition. For this situation, it needs to be rejected during subsequent voice interaction.

[0050] It should be noted that using the initial-final phoneme sequence for comparison can ensure that the phoneme sequence focuses on the sound. A high phoneme matching degree indicates that it is a human voice rather than noise, and the speaker is indeed saying this text. At the same time, it can also compensate for the phenomenon that the open-source large model cannot recognize proper nouns. For example, "using troops" and "using ice". A low score indicates that the third-party recognition engine has a large difference in the recognition of common audio, and it may not be normal audio. The text recognized by the third-party engine may be a misrecognition.

[0051] Optionally, the aforementioned large model is a large model disclosed in related technologies that can be used for audio recognition, and this application does not impose any further limitations on it.

[0052] Optional, Figure 3 This is a schematic diagram of a system architecture for scoring the recognition results of a third-party recognition engine according to an embodiment of this application. The system mainly consists of a third-party recognition engine, an open-source speech recognition model, and a scoring module. The specific system operation process is as follows:

[0053] The target audio is input into a third-party recognition engine to obtain the second recognized text. Then, the same target audio is input into an open-source speech recognition model for recognition, resulting in the first recognized text, which corresponds to the model's recognition result. Next, a scoring process for the third-party recognition engine is initiated through a scoring module. For example, the first and second recognized texts are converted into initial and final phoneme sequences, respectively. The similarity of the two converted phoneme sequences, the proportion of common phonemes, the phoneme error rate, and the phoneme length matching degree are compared. Finally, based on a weighted sum of these factors, a score is obtained for the third-party recognition engine's recognition of the target audio. Then, based on the score and the second recognized text, it is determined whether to use the third-party recognition engine with the current parameter settings and the direction for adjusting the third-party recognition engine.

[0054] In summary, upon receiving the recognition results from a third-party recognition engine, the audio and the third-party recognition results can be used for secondary recognition using an open-source large model. Furthermore, relevant phoneme algorithms can be applied to score the third-party recognition results. This determines whether the third-party recognition results can be used as an interaction reference in subsequent device voice interactions. By scoring the third-party recognition engine, the accuracy of its subsequent recognition is ensured, and misrecognition by the third-party recognition engine is avoided.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software device. This computer software device is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a voice device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0056] Figure 4 This is a structural block diagram of a scoring device for a recognition engine according to an embodiment of this application; as shown below. Figure 4As shown, it includes:

[0057] The recognition module 42 is used to input the target audio into a pre-trained speech model to obtain the first recognized text, and to obtain the second recognized text output by a third-party text recognition engine after processing the target audio.

[0058] The calculation module 44 is used to calculate multiple index parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text, wherein the multiple index parameters include at least one of the following: similarity between different phoneme sequences, proportion of common phonemes between different phoneme sequences, phoneme error rate between different phoneme sequences, and phoneme length matching degree between different phoneme sequences.

[0059] The processing module 46 is used to summarize and process the multiple indicator parameters according to preset parameter weights to obtain the target score of the third-party text recognition engine in recognizing the target audio, wherein the target score is used to indicate the quality of the recognized text output by the third-party text recognition engine.

[0060] Using the aforementioned device, the audio data to be evaluated is input into a high-performance open-source speech recognition model. This model, after thorough training, is capable of generating highly accurate first-recognition text. Subsequently, a second-recognition text is obtained from the same target audio, output by a third-party recognition engine. Next, based on the comparison of phoneme sequences of initials and finals, multiple evaluation metrics are calculated between the phoneme sequences converted from the first and second-recognition texts, including phoneme sequence similarity, common phoneme proportion, phoneme error rate, and phoneme length matching degree. These metrics comprehensively examine the accuracy of the recognition results from different dimensions. For example, phoneme sequence similarity measures the degree of matching between the two texts at the phoneme level, the common phoneme proportion reflects the proportion of phonemes shared by the two texts, the phoneme error rate assesses the differences at the phoneme level, and the phoneme length matching degree examines the consistency of phoneme lengths in the recognized texts. Finally, these metric parameters are combined with preset weights, and a target score is generated through weighted summation to directly reflect the recognition quality of the third-party recognition engine when processing specific audio. The above technical solution solves the problem of the inability to effectively evaluate the reliability of the recognition results of third-party recognition engines in related technologies. By introducing an independent, open-source speech recognition model as a notary, it fills the gap of lack of objective third-party evaluation in existing technologies, effectively improves the credibility and accuracy of speech recognition results, and realizes effective judgment of third-party recognition engines in handling non-standardized pronunciation, dialects and noisy environments.

[0061] In an exemplary embodiment, the above-mentioned calculation module is further configured to determine, when the currently calculated index parameter is the similarity between different phoneme sequences, the first length corresponding to the longest common subsequence between the first phoneme sequence and the second phoneme sequence, and the second length corresponding to the total sequence after connecting the first phoneme sequence and the second phoneme sequence; and substitute the first length and the second length into a preset first formula to obtain the first index parameter corresponding to the similarity, wherein the preset first formula is: similarity = 2.0 × first length / second length.

[0062] In an exemplary embodiment, the above-mentioned calculation module is further configured to determine, when the currently calculated index parameter is the proportion of common phonemes among different phoneme sequences, a third length corresponding to the common phonemes between the first phoneme sequence and the second phoneme sequence, and a fourth length corresponding to the first phoneme sequence; and to substitute the third length and the fourth length into a preset second formula to obtain a second index parameter corresponding to the proportion of common phonemes, wherein the preset second formula is: proportion of common phonemes = third length / fourth length.

[0063] In an exemplary embodiment, the above-mentioned calculation module is further configured to determine, when the currently calculated index parameter is the phoneme error rate between different phoneme sequences, the fifth length corresponding to the edit distance between the first phoneme sequence and the second phoneme sequence participating in sequence conversion, and the sixth length corresponding to the longest phoneme sequence among the first phoneme sequence and the second phoneme sequence; and to substitute the fifth length and the sixth length into a preset third formula to obtain the third index parameter corresponding to the phoneme error rate, wherein the preset third formula is: phoneme error rate = fourth length / fifth length.

[0064] In an exemplary embodiment, the above-mentioned calculation module is further configured to, when the currently calculated index parameter is the phoneme length matching degree between different phoneme sequences, divide the seventh length corresponding to the second phoneme sequence by the fourth length corresponding to the first phoneme sequence to obtain the first length ratio; substitute the first length ratio into a preset fourth formula to obtain the fourth index parameter corresponding to the phoneme length matching degree, wherein the preset fourth formula is: phoneme length matching degree = 1.0 - min(abs(1.0 - first length ratio), 1.0).

[0065] In an exemplary embodiment, the above-described processing module is further configured to obtain multiple target weight values ​​corresponding to the multiple indicator parameters, wherein the multiple target weight values ​​are dynamic values ​​determined based on the historical audio recognition accuracy of the third-party text recognition engine; and to obtain the target score by performing a weighted summation based on the multiple target weight values ​​and the multiple target indicator parameters.

[0066] In an exemplary embodiment, the above-described apparatus further includes: a determining module, configured to, after summarizing and processing the plurality of indicator parameters according to preset parameter weights to obtain a target score for the target audio recognized by the third-party text recognition engine, determine that the second recognized text output by the third-party text recognition engine has a recognition anomaly if the target score is less than a preset score threshold, and adjust the configuration of the third-party text recognition engine accordingly; and determine that the second recognized text output by the third-party text recognition engine does not have a recognition anomaly if the target score is greater than or equal to the preset score threshold, and allow the recognized text output by the third-party text recognition engine to be used as reference text for audio interaction.

[0067] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same target processor; or, the above modules are located in different target processors in any combination.

[0068] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0069] Embodiments of this application also provide an electronic device, including a target memory and a target processor, wherein the target memory stores a computer program and the target processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0070] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0071] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0072] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0073] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0074] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0075] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of N computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or N modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0076] The scoring method, apparatus, storage medium, and electronic device of the recognition engine provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A scoring method for a recognition engine, characterized in that, include: The target audio is input into a pre-trained speech model to obtain the first recognized text, and the second recognized text is obtained from the output of a third-party text recognition engine after processing the target audio. Calculate multiple index parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text, wherein the multiple index parameters include at least one of the following: similarity between different phoneme sequences, proportion of common phonemes between different phoneme sequences, phoneme error rate between different phoneme sequences, and phoneme length matching degree between different phoneme sequences. The multiple indicator parameters are aggregated and processed according to preset parameter weights to obtain the target score for the third-party text recognition engine to recognize the target audio. The target score is used to indicate the quality of the recognized text output by the third-party text recognition engine.

2. The scoring method of the recognition engine according to claim 1, characterized in that, Calculate multiple index parameters between the first phoneme sequence corresponding to the first recognized text and the second phoneme sequence corresponding to the second recognized text, including: Given that the current metric parameter is the similarity between different phoneme sequences, determine the first length corresponding to the longest common subsequence between the first phoneme sequence and the second phoneme sequence, and the second length corresponding to the total sequence after connecting the first phoneme sequence and the second phoneme sequence. Substituting the first length and the second length into a preset first formula, the first index parameter corresponding to the similarity is obtained, wherein the preset first formula is: similarity = 2.0 × first length / second length.

3. The scoring method of the recognition engine according to claim 1, characterized in that, Calculate multiple index parameters between the first phoneme sequence corresponding to the first recognized text and the second phoneme sequence corresponding to the second recognized text, including: Given that the current calculated index parameter is the proportion of common phonemes among different phoneme sequences, determine the third length corresponding to the common phonemes between the first phoneme sequence and the second phoneme sequence, and the fourth length corresponding to the first phoneme sequence. Substituting the third length and the fourth length into the preset second formula, the second index parameter corresponding to the common phoneme ratio is obtained, wherein the preset second formula is: common phoneme ratio = third length / fourth length.

4. The scoring method of the recognition engine according to claim 1, characterized in that, Calculate multiple index parameters between the first phoneme sequence corresponding to the first recognized text and the second phoneme sequence corresponding to the second recognized text, including: Given that the current calculated index parameter is the phoneme error rate between different phoneme sequences, determine the fifth length corresponding to the edit distance between the first phoneme sequence and the second phoneme sequence that participates in sequence conversion, and the sixth length corresponding to the longest phoneme sequence among the first phoneme sequence and the second phoneme sequence. Substituting the fifth length and the sixth length into the preset third formula, the third index parameter corresponding to the phoneme error rate is obtained, wherein the preset third formula is: phoneme error rate = fourth length / fifth length.

5. The scoring method of the recognition engine according to claim 1, characterized in that, Calculate multiple index parameters between the first phoneme sequence corresponding to the first recognized text and the second phoneme sequence corresponding to the second recognized text, including: When the current calculated index parameter is the phoneme length matching degree between different phoneme sequences, the first length ratio is obtained by dividing the seventh length corresponding to the second phoneme sequence by the fourth length corresponding to the first phoneme sequence. Substituting the first length proportion into the preset fourth formula, the fourth index parameter corresponding to the phoneme length matching degree is obtained, wherein the preset fourth formula is: phoneme length matching degree = 1.0 - min(abs(1.0 - first length proportion), 1.0).

6. The scoring method of the recognition engine according to claim 1, characterized in that, The multiple indicator parameters are aggregated and processed according to preset parameter weights to obtain the target score for the target audio recognized by the third-party text recognition engine, including: Obtain multiple target weight values ​​corresponding to the multiple indicator parameters, wherein the multiple target weight values ​​are dynamic values ​​determined based on the historical audio recognition accuracy of the third-party text recognition engine; The target score is obtained by weighting and summing the multiple target weight values ​​and multiple target indicator parameters.

7. The scoring method of the recognition engine according to claim 1, characterized in that, After summarizing and processing the multiple indicator parameters according to preset parameter weights to obtain the target score for the target audio recognized by the third-party text recognition engine, the method further includes: If the target score is less than a preset score threshold, it is determined that the second recognized text output by the third-party text recognition engine has a recognition anomaly, and the configuration of the third-party text recognition engine needs to be adjusted. If the target score is greater than or equal to a preset score threshold, it is determined that the second recognized text output by the third-party text recognition engine does not have any recognition anomalies, and the recognized text output by the third-party text recognition engine is allowed to be used as the reference text for audio interaction.

8. A scoring device for a recognition engine, characterized in that, include: The recognition module is used to input the target audio into a pre-trained speech model to obtain the first recognized text, and to obtain the second recognized text output by a third-party text recognition engine after processing the target audio. The calculation module is used to calculate multiple indicator parameters between the first phoneme sequence corresponding to the first identified text and the second phoneme sequence corresponding to the second identified text, wherein the multiple indicator parameters include at least one of the following: similarity between different phoneme sequences, proportion of common phonemes between different phoneme sequences, phoneme error rate between different phoneme sequences, and phoneme length matching degree between different phoneme sequences. The processing module is used to summarize and process the multiple indicator parameters according to preset parameter weights to obtain the target score of the target audio recognized by the third-party text recognition engine, wherein the target score is used to indicate the quality of the recognized text output by the third-party text recognition engine.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images