Evaluation method of a voice restoration system, electronic device, medium, and program product

Abstract

The application provides an evaluation method of a sound restoration system, an electronic device, a medium and a program product, and belongs to the technical field of audio processing. The evaluation method is characterized in that text recognition is performed on audio samples before and after restoration and the restored audio samples to realize objective evaluation of the sound restoration system, subjective evaluation of the sound restoration system is performed by using a speech recognition model to simulate the recognition result of artificial blind listening test, a model evaluation result is obtained, comprehensive evaluation of the sound restoration system is realized, and the real performance of the sound restoration system is truly reflected.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and more particularly to an evaluation method, electronic device, medium, and program product for a sound restoration system. Background Technology

[0002] In everyday interactions such as telephone and face-to-face communication, hearing-impaired individuals often experience problems such as unclear pronunciation, inaccurate articulation, and abnormal speech rhythm due to physiological reasons, which seriously affects their smooth daily communication. To address this issue, voice restoration technology has emerged, which can restore or correct distorted parts of digital speech, thereby improving the intelligibility of the speech.

[0003] With the widespread adoption of sound restoration technology, its effectiveness directly impacts the user experience for the hearing-impaired and the evaluation of such products. However, the industry currently lacks evaluation methods for sound restoration systems, thus failing to accurately reflect their true performance. Summary of the Invention

[0004] This invention provides an evaluation method, electronic device, medium, and program product for a sound restoration system, which addresses the shortcomings of existing technologies that lack evaluation schemes for sound restoration systems and cannot accurately reflect the true performance of sound restoration systems, thereby achieving an accurate reflection of the true performance of sound restoration systems.

[0005] This invention provides an evaluation method for a sound restoration system, comprising the following steps.

[0006] A sound restoration system was used to restore the sound of each audio sample in the evaluation sample set, resulting in a restored sample set. Text recognition is performed on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set to obtain text recognition results, and text evaluation results are determined based on the text recognition results; Each sample in the first sample set is input into the speech recognition model to obtain the recognition result output by the speech recognition model, and the model evaluation result is determined based on the recognition result; the first sample set is a sample set obtained by mixing and shuffling the evaluation sample set and the repair sample set, and the speech recognition model is trained with audio training samples and the recognition result labels corresponding to the audio training samples; Based on the text evaluation results and the model evaluation results, the evaluation results of the sound restoration system are determined.

[0007] According to the evaluation method of a sound restoration system provided by the present invention, the text evaluation result includes an objective intelligibility improvement index, and the text recognition result includes a first text recognition result and a second text recognition result; The step of performing text recognition on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set to obtain a text recognition result, and determining a text evaluation result based on the text recognition result, includes: An automatic recognition model is used to identify each audio sample in the evaluation sample set to obtain a first recognition result; The automatic recognition model is used to identify each repaired audio sample in the repaired sample set to obtain a second recognition result; Based on the first real-labeled data, a first accuracy rate corresponding to the first identification result and a second accuracy rate corresponding to the second identification result are determined, wherein the first real-labeled data is the real-labeled data corresponding to the evaluation sample set; The objective understandability improvement index is determined based on the first accuracy rate and the second accuracy rate.

[0008] According to an evaluation method for a sound restoration system provided by the present invention, the step of determining the objective intelligibility improvement index based on the first accuracy rate and the second accuracy rate includes: The difference between the second accuracy rate and the first accuracy rate is used as the objective understandability improvement index.

[0009] According to the evaluation method of a sound restoration system provided by the present invention, the model evaluation result includes a subjective intelligibility improvement index, and each recognition result includes a text content result. The step of inputting each sample in the first sample set into the speech recognition model to obtain the recognition result output by the speech recognition model, and determining the model evaluation result based on the recognition result, includes: The speech recognition model outputs a first text content result when the first audio sample is used as input, and a second text content result when the first repaired audio sample is used as input. The first repaired audio sample is the audio sample after the first audio sample has been repaired by the sound repair system. Based on the first real-world labeled data, determine the third accuracy rate corresponding to the first text content result and the fourth accuracy rate corresponding to the second text content result; The subjective comprehensibility improvement index is determined based on the third accuracy rate and the fourth accuracy rate; The first audio sample is any one of the audio samples in the evaluation sample set.

[0010] According to an evaluation method for a sound restoration system provided by the present invention, the step of determining the subjective intelligibility improvement index based on the third accuracy rate and the fourth accuracy rate includes: The difference between the fourth accuracy rate and the third accuracy rate is used as the subjective comprehensibility improvement index.

[0011] According to the evaluation method of the sound restoration system provided by the present invention, each recognition result further includes an evaluation value for evaluating whether the meaning of the sentence is understood, and the model evaluation result further includes a meaning understanding improvement value; The step of inputting each sample in the first sample set into the speech recognition model to obtain the recognition result output by the speech recognition model, and determining the model evaluation result based on the recognition result, further includes: Determine the first evaluation value corresponding to the first audio sample and the second evaluation value corresponding to the first repaired audio sample; The difference between the second evaluation value and the first evaluation value is used as the semantic understanding enhancement value.

[0012] According to the evaluation method of the sound restoration system provided by the present invention, the model evaluation result further includes the semantic restoration rate; The step of inputting each sample in the first sample set into the speech recognition model to obtain the recognition result output by the speech recognition model, and determining the model evaluation result based on the recognition result, further includes: Determine the first number of characters, which is the number of identical characters between the real annotation data corresponding to the first repaired audio sample and the second text content result; Determine the second number of characters, which is the total number of characters in the real labeled data corresponding to the first repaired audio sample; The ratio of the first number of characters to the second number of characters is taken as the sentence meaning repair rate.

[0013] According to the evaluation method of a sound restoration system provided by the present invention, the evaluation method of the sound restoration system further includes: Based on the evaluation sample set and the repair sample set, multiple audio pairs are constructed, each audio pair including an audio sample and a repair audio sample corresponding to the audio sample; For each audio pair, using the audio sample as a timbre reference, the timbre of the repaired audio sample is scored for similarity to obtain a timbre similarity score. For each audio sample in the audio pair and the repaired audio sample, the pronunciation clarity and rhythmic naturalness are scored respectively to obtain the first pronunciation clarity score and the first rhythmic naturalness score corresponding to the audio sample, and the second pronunciation clarity score and the second rhythmic naturalness score corresponding to the repaired audio sample. The step of determining the evaluation result of the sound restoration system based on the text evaluation result and the model evaluation result further includes: Output the first and second score values; The first score is determined by the first pronunciation clarity score and the first rhythmic naturalness score, and the second score is determined by the timbre similarity score, the second pronunciation clarity score, and the second rhythmic naturalness score.

[0014] According to the evaluation method of the sound restoration system provided by the present invention, the evaluation sample set includes multiple audio samples, and the multiple audio samples are recorded by sample recorders of different hearing impairment levels, different genders, and different age groups under different environmental factors.

[0015] The present invention also provides an evaluation device for a sound restoration system, comprising the following modules: The repair module is used to repair the audio samples in the evaluation sample set using a sound repair system, thereby obtaining a repaired sample set. The first evaluation module is used to perform text recognition on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set, obtain text recognition results, and determine text evaluation results based on the text recognition results; The second evaluation module is used to input each sample in the first sample set into the speech recognition model, obtain the recognition result output by the speech recognition model, and determine the model evaluation result based on the recognition result; the first sample set is a sample set obtained by mixing and shuffling the evaluation sample set and the repair sample set, and the speech recognition model is trained with audio training samples and the recognition result labels corresponding to the audio training samples; The determination module is used to determine the evaluation result of the sound restoration system based on the text evaluation result and the model evaluation result.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an evaluation method for any of the sound restoration systems described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the evaluation method of any of the above-described sound restoration systems.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements an evaluation method for any of the sound restoration systems described above.

[0019] This invention provides an evaluation method, electronic device, medium, and program product for a sound restoration system. By performing text recognition on audio samples before and after restoration and the restored audio samples, an objective evaluation of the sound restoration system is achieved. By using a speech recognition model to simulate the recognition results of a blind listening test, a subjective evaluation of the sound restoration system is performed, resulting in a model evaluation result. This achieves a comprehensive evaluation of the sound restoration system and thus truly reflects its actual performance. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts illustrating the evaluation method for the sound restoration system provided by the present invention.

[0022] Figure 2 This invention provides a flowchart illustrating the process of performing text recognition on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set, obtaining text recognition results, and determining text evaluation results based on the text recognition results.

[0023] Figure 3 This invention provides a flowchart illustrating the process of inputting each sample in a first sample set into a speech recognition model to obtain the recognition result output by the speech recognition model, and determining the model evaluation result based on the recognition result.

[0024] Figure 4 This is the second flowchart of the evaluation method for the sound restoration system provided by the present invention.

[0025] Figure 5 This is the third flowchart of the evaluation method for the sound restoration system provided by the present invention.

[0026] Figure 6 This is the fourth flowchart of the evaluation method for the sound restoration system provided by the present invention.

[0027] Figure 7 This is a schematic diagram of the overall principle of the evaluation method for the sound restoration system provided by the present invention.

[0028] Figure 8 This is a schematic diagram of the structure of the evaluation device for the sound restoration system provided by the present invention.

[0029] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0030] Figure label: 801: Repair module; 802: First evaluation module; 803: Second evaluation module; 804: Determination module; 910: Processor; 920: Communication interface; 930: Memory; 940: Communication bus. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] The evaluation method for a sound restoration system provided in this embodiment of the invention can be executed by any electronic device with data processing capabilities, such as a personal computer, server, cloud computing node, or specially configured evaluation terminal.

[0033] The following is combined with Figures 1 to 9 The present invention describes the evaluation method, electronic device, medium, and program product of the sound restoration system.

[0034] Figure 1 This is one of the flowcharts illustrating the evaluation method for the sound restoration system provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 101: Use a sound restoration system to restore the sound of each audio sample in the evaluation sample set to obtain a restored sample set.

[0035] In some embodiments, audio samples may be pre-collected voice datasets read by an electronic device from local storage, or they may be downloaded in real time from a remote database or a third-party voice acquisition platform via a network interface.

[0036] Specifically, after obtaining the set of evaluation samples, they are input into the audio restoration system to be evaluated. This audio restoration system is typically based on an acoustic model or speech processing engine trained using artificial intelligence algorithms. The audio restoration system performs acoustic feature extraction, noise reduction, distortion correction, or prosody reconstruction on each input audio sample, thereby outputting the corresponding restored audio data, thus obtaining the restored audio sample.

[0037] Step 102: Perform text recognition on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set to obtain the text recognition result, and determine the text evaluation result based on the text recognition result.

[0038] In this step, text recognition technology is used to process the audio samples before and after restoration. Specifically, the text in the audio samples and the restored audio samples is recognized to obtain the text recognition results. Text recognition can be achieved by calling a speech-to-text recognition interface deployed locally or in the cloud.

[0039] After obtaining the text recognition results for the audio samples and the audio samples to be restored, the restoration performance of the sound restoration system is evaluated using the text recognition results, and the text evaluation results are obtained.

[0040] The text evaluation results here refer to quantitative indicators derived purely from machine-recognized text data, without relying on human subjective auditory perception. By comparing and calculating the text recognition results before and after restoration, the performance of the sound restoration system can be objectively measured.

[0041] Step 103: Input each sample in the first sample set into the speech recognition model to obtain the recognition result output by the speech recognition model, and determine the model evaluation result based on the recognition result; the first sample set is the sample set obtained by mixing and shuffling the evaluation sample set and the repair sample set, and the speech recognition model is trained with the audio training samples and the recognition result labels corresponding to the audio training samples.

[0042] In this step, the order of the evaluation sample set and the repair sample set is shuffled using a random algorithm to generate the first sample set. This shuffling operation eliminates the control bias that occurs when the speech recognition model is used as a test subject for listening tests.

[0043] The speech recognition model simulates a blind listening test, collecting test results for each sample in the first sample set to obtain the recognition result. This blind listening test refers to an assessment conducted by the tester without knowing whether the currently played audio is the original audio or the restored audio. These test results are received and analyzed, extracting features reflecting the degree of human subjective auditory understanding, and then generating the model evaluation result through statistical data analysis. The model evaluation result represents the actual perceived quality of the speech output by the sound restoration system in real-world interactive scenarios.

[0044] In some embodiments, the speech recognition model is trained by taking audio training samples as input and using the recognition result labels corresponding to the audio training samples as the training target.

[0045] The labels for the recognition results corresponding to the audio training samples are obtained through self-revision after semantic understanding of the audio training samples.

[0046] Step 104: Based on the text evaluation results and model evaluation results, determine the evaluation results of the sound restoration system.

[0047] In this step, the text evaluation results and the model evaluation results serve as two performance metrics for evaluating the sound restoration system.

[0048] In this embodiment, text recognition is performed on the audio samples before and after restoration and the restored audio samples to achieve an objective evaluation of the sound restoration system. The recognition results of the speech recognition model are used to simulate the recognition results of the blind listening test to perform a subjective evaluation of the sound restoration system, and the model evaluation results are obtained. This achieves a comprehensive evaluation of the sound restoration system and thus truly reflects the real performance of the sound restoration system.

[0049] Because the evaluation results of a sound restoration system can accurately and comprehensively reflect its actual performance, they can be used to optimize the system and further improve its sound restoration effects. At the same time, this avoids relying solely on a single indicator or imprecise testing, which not only fails to guide effective system improvement but may also result in poor-quality audio output that could negatively impact the hearing-impaired community, exacerbating communication barriers.

[0050] In some embodiments, the text evaluation results include an objective intelligibility improvement index, and the text recognition results include a first text recognition result and a second text recognition result.

[0051] Among them, the objective comprehensibility improvement index refers to a quantitative index derived purely from machine-recognized text data without relying on human subjective hearing. It is used to describe whether the repaired audio sample can be understood and is used to evaluate the model's comprehension ability, whether the response content can understand the user's intention and follow the user's instructions, and whether it can correctly answer the user's questions and whether the answer is comprehensive.

[0052] like Figure 2 As shown, text recognition is performed on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set to obtain text recognition results. Based on the text recognition results, the text evaluation results are determined, including: Step 201: Use an automatic recognition model to identify each audio sample in the evaluation sample set to obtain the first recognition result.

[0053] In some embodiments, the automatic recognition model refers to an automatic speech recognition (ASR) model. Specifically, audio samples from the evaluation sample set are used as input data and fed into the automatic recognition model. The model performs mapping and decoding of acoustic features to text features, and outputs the corresponding text sequence as the first recognition result.

[0054] Step 202: Use an automatic recognition model to identify each audio sample in the repair sample set to obtain a second recognition result.

[0055] Similarly, the same automatic recognition model is used to identify each repaired audio sample in the repaired sample set to obtain a second recognition result.

[0056] Step 203: Based on the first real labeled data, determine the first accuracy rate corresponding to the first recognition result and the second accuracy rate corresponding to the second recognition result. The first real labeled data is the real labeled data corresponding to the evaluation sample set.

[0057] Understandably, the first set of real-text annotations is the actual reference text data corresponding to the audio sample being recorded, that is, the standard text actually read aloud by the recorder.

[0058] Specifically, the extracted first recognition result is compared and matched with the first real labeled data to calculate the first accuracy rate of recognition before character restoration; similarly, the second recognition result is compared and matched with the first real labeled data to calculate the second accuracy rate of recognition after character restoration.

[0059] Step 204: Determine the objective comprehensibility improvement index based on the first accuracy rate and the second accuracy rate.

[0060] In this embodiment, a data indicator that reflects the improvement in the model's understanding ability is quantified by comparing a first accuracy rate representing the accuracy of objective data before repair with a second accuracy rate representing the accuracy of objective data after repair. This is the objective comprehensibility improvement indicator.

[0061] By introducing an automatic recognition model to independently and consistently decode the audio samples before and after restoration, and the restored audio samples, and by using real labeled data as the benchmark anchor for data comparison, the first accuracy rate and the second accuracy rate are obtained, ultimately yielding an objective intelligibility improvement index. This process not only eliminates subjective errors that might be introduced by human intervention, making the recognition process fully automated, but also utilizes standardized data alignment methods to ensure the high objectivity of the output objective intelligibility improvement index.

[0062] In some embodiments, an objective comprehensibility improvement metric is determined based on a first accuracy rate and a second accuracy rate, including: The difference between the second and first accuracy rates is used as an indicator of improved objective understandability.

[0063] In this embodiment, the previously calculated values ​​of the first accuracy rate and the second accuracy rate are retrieved, and a subtraction operation is performed. Specifically, the first accuracy rate, representing the character accuracy result before repair, is subtracted from the second accuracy rate, which represents the character accuracy result after repair. The difference obtained through the subtraction operation is then set as the objective intelligibility improvement index.

[0064] In this embodiment, the computational complexity is reduced. There is no need to construct complex evaluation functions. The objective text intelligibility gain effect brought about by the sound restoration system after acoustic feature reconstruction can be intuitively and accurately quantified through basic difference extraction operations.

[0065] In some embodiments, the model evaluation results include a subjective comprehensibility improvement index, and each recognition result includes a text content result.

[0066] Among them, the subjective comprehensibility improvement index represents a quantitative indicator that describes the comprehensibility of the repaired audio samples output by the sound restoration system in real interactive scenarios.

[0067] The text content result is the text that is dictated or entered by the speech recognition model based on the subjective recognition and understanding of the content when the audio sample or the repaired audio sample is simulated by a blind listening test.

[0068] like Figure 3 As shown, each sample in the first sample set is input into the speech recognition model to obtain the recognition result output by the speech recognition model, and the model evaluation result is determined based on the recognition result, including: Step 301: Determine the first text content result output by the speech recognition model when the first audio sample is used as input, and the second text content result output by the speech recognition model when the first repaired audio sample is used as input. The first repaired audio sample is the audio sample after the first audio sample has been repaired by the sound repair system.

[0069] Specifically, after inputting the first audio sample, the speech recognition model records the text data based on its subjective recognition and understanding, identifying it as the first text content result. Similarly, the text data recognized and understood by the speech recognition model after inputting the first repaired audio sample is extracted and identified as the second text content result.

[0070] Step 302: Based on the first real labeled data, determine the third accuracy rate corresponding to the first text content result and the fourth accuracy rate corresponding to the second text content result.

[0071] Specifically, the first set of real-world labeled data is retrieved, and a string similarity or word error rate calculation algorithm is executed based on this data. Specifically, the first text content result is compared character-by-character with the first set of real-world labeled data to calculate a feature value reflecting the accuracy of human listening to the audio sample, i.e., the third accuracy rate. Similarly, using the same data comparison algorithm, the second text content result is matched with the same set of real-world labeled data to calculate a feature value reflecting the accuracy of human listening to the repaired audio sample, i.e., the fourth accuracy rate.

[0072] Step 303: Determine the subjective comprehensibility improvement index based on the third and fourth accuracy rates; where the first audio sample is any audio sample in the evaluation sample set.

[0073] After obtaining the third and fourth accuracy rates, the data features that characterize the accuracy of human dictation are compared or correlated to quantify and derive a comprehensive improvement data reflecting the actual hearing experience of the sound restoration system, namely the subjective intelligibility improvement index.

[0074] In this embodiment, by introducing first real-world labeled data as a benchmark, the first and second text content results are transformed into objective and quantifiable third and fourth accuracy rates. The final calculated subjective intelligibility improvement index, obtained during this process, maximally replicates the perceptual data attributes of the real human auditory system, thereby enabling subjective evaluation of the sound restoration system.

[0075] In some embodiments, the speech recognition model simulating a blind listening test should adhere to the following guidelines: For words that are pronounced but cannot be recognized and for which there is no corresponding Chinese character or pinyin, pre-selected symbols are used to mark them. Each audio does not need to be recognized repeatedly; one or two recognitions are sufficient. After the recognition results are collected, semantic calibration is required based on the labeled text and the recognition results to prevent inconsistencies between the recognized and understood content and the actual spoken content.

[0076] The training requirements for the speech recognition model are as follows: The number of evaluators providing audio training samples must be at least 2, and experienced audio testing users are required, who must be trained in advance.

[0077] For training scenarios using interactive corpora, where only one speaker's text is provided, and the content and meaning of another speaker are required, simple association based on the preceding context can be used for judgment. After the recognition results are collected, semantic calibration needs to be performed based on the labeled text and the recognition results to prevent discrepancies between the recognized and understood content and the actual spoken content.

[0078] Examples of interactive scene corpora are as follows: Speaker [A]: Where did you go these past two days?

[0079] Speaker [B]: I didn't go anywhere, I'm just lying at home.

[0080] Speaker [A]: ... Speaker [B]: ... In some embodiments, the subjective comprehensibility improvement metric is determined based on a third accuracy rate and a fourth accuracy rate, including: The difference between the fourth and third accuracy rates was used as an indicator of subjective intelligibility improvement.

[0081] In this embodiment, after calculating the third and fourth accuracy rates, a basic subtraction operation is performed. The third accuracy rate, representing the original defective speech accuracy before manual audiometry correction, is subtracted from the fourth accuracy rate, which represents the speech accuracy result after manual audiometry correction. After this subtraction operation, the corresponding difference is extracted as a subjective intelligibility improvement index.

[0082] In some embodiments, each identification result also includes an evaluation value for assessing whether the meaning of the sentence is understood, and the model evaluation result also includes a meaning understanding improvement value.

[0083] Among them, the semantic comprehension improvement value is used to describe the improvement in the degree of semantic comprehension before and after the repair.

[0084] like Figure 4 As shown, each sample in the first sample set is input into the speech recognition model to obtain the recognition result output by the speech recognition model, and the model evaluation result is determined based on the recognition result. The process also includes: Step 401: Determine the first evaluation value corresponding to the first audio sample and the second evaluation value corresponding to the first repaired audio sample.

[0085] In some embodiments, the evaluation value used to assess whether the meaning of the sentence is understood can be 0 or 1, where 1 indicates that the meaning of the sentence is understandable, 0 indicates a binary label indicating that the meaning of the sentence is incomprehensible, or a more granular quantitative score of the degree of understanding.

[0086] Specifically, these test data are analyzed, and the semantic evaluation data corresponding to the unrepaired first audio sample is identified and extracted as the first evaluation value; at the same time, the semantic evaluation data corresponding to the first restored audio sample after processing by the sound restoration system is identified and extracted as the second evaluation value.

[0087] Step 402: Use the difference between the second evaluation value and the first evaluation value as the sentence meaning comprehension improvement value.

[0088] In this embodiment, by calculating the sentence meaning comprehension improvement value, false repairs that are clear in the pronunciation of individual words but have broken overall sentence logic in actual applications are effectively identified and filtered out, making the model evaluation results more consistent with real cognitive logic and improving the comprehensiveness and scientific nature of the system evaluation dimensions.

[0089] In some embodiments, the model evaluation results also include the semantic repair rate.

[0090] Among them, the sentence meaning repair rate can be understood as the completeness rate of sentence repair.

[0091] like Figure 5 As shown, each sample in the first sample set is input into the speech recognition model to obtain the recognition result output by the speech recognition model, and the model evaluation result is determined based on the recognition result. The process also includes: Step 501: Determine the first number of characters. The first number of characters is the number of characters that are the same between the real annotation data corresponding to the first repaired audio sample and the second text content result.

[0092] Specifically, the real labeled data corresponding to the first restored audio sample is retrieved, and the second text content result obtained in the aforementioned steps is extracted. Subsequently, the two sets of text sequence data are compared character by character or word by word using a string matching algorithm or text alignment algorithm, and the number of characters or words that completely match in the same position or semantic correspondence is extracted. This statistically obtained matching value is determined as the first number of characters.

[0093] For example, assume that the real annotation data corresponding to the first repaired audio sample is "The weather is very good today" with a total of 6 characters, and the second text content result entered by the tester after blind listening is "The weather is really good today". After comparison by the algorithm, it is concluded that the 5 characters "今, 天, 天, 气, 好" are completely identical, so the value of the first text count is determined to be 5.

[0094] Step 502: Determine a second text count, where the second text count is the total number of characters in the real annotation data corresponding to the first repaired audio sample.

[0095] In this step, a string length calculation or character traversal statistics operation is performed on the real annotation data corresponding to the first repaired audio sample. Through this statistical operation, the absolute sum of all characters or words contained in the real annotation data is directly obtained, and this value is extracted and determined as the second text count. Following the aforementioned example, the total number of characters in the real annotation data "The weather is very good today" is 6, so the value of the second text count is determined to be 6.

[0096] Step 503: Take the ratio of the first text count to the second text count as the sentence meaning repair rate.

[0097] In this step, the division operation logic is invoked, the first text count representing the feature of the number of correctly transcribed characters in subjective dictation is taken as the numerator, and the second text count representing the feature of the reference total character count is taken as the denominator to perform the division operation. The proportional value obtained after the operation is directly assigned and output as the sentence meaning repair rate.

[0098] For example, the sentence meaning repair rate is 5 / 6 as in the aforementioned example, which is approximately 83.3%.

[0099] In this embodiment, the first text count of characters correctly transcribed in subjective blind listening and the objectively existing second text count are extracted, and the data ratio between the two is calculated using division logic. The success of sentence meaning repair, which was originally difficult to define, is represented by a ratio, so that the model evaluation result is more consistent with the real cognitive logic, and the comprehensiveness and scientific nature of the system evaluation dimension are improved.

[0100] In some embodiments, as Figure 6 shown, the evaluation method for the sound repair system further comprises: Step 601: Construct a plurality of audio pairs based on the evaluation sample set and the repaired sample set, where each audio pair includes one audio sample and a repaired audio sample corresponding to the audio sample.

[0101] Specifically, unrepaired audio samples are extracted from the evaluation sample set, and the corresponding processed repaired audio samples are retrieved from the repaired sample set using unique identifiers. These two related audio data segments from the same sound source are bound together in a storage structure to construct a data structure containing parallel comparison relationships, i.e., audio pairs. Multiple audio pair datasets are constructed by iteratively executing this data binding operation.

[0102] Step 602: For each audio pair, using the audio sample as a timbre reference, score the timbre similarity of the repaired audio sample to obtain a timbre similarity score.

[0103] Specifically, a pre-built timbre similarity evaluation model is used to score the similarity of timbre. Specifically, the unrestored audio sample is set as the baseline timbre reference, and the restoration audio is evaluated to determine whether it retains the original speaker's timbre characteristics. Based on this, a quantitative value is output and recorded as a timbre similarity score.

[0104] Step 603: For each audio pair, score the pronunciation clarity and rhythmic naturalness of the audio sample and the repaired audio sample respectively, and obtain the first pronunciation clarity score and the first rhythmic naturalness score corresponding to the audio sample, as well as the second pronunciation clarity score and the second rhythmic naturalness score corresponding to the repaired audio sample.

[0105] Specifically, scoring data for audio samples in two subdivisions—pronunciation clarity and naturalness of speech rhythm—are collected and used as the first pronunciation clarity score and the first naturalness of speech rhythm score, respectively. Similarly, scoring data for repaired audio samples in the same two dimensions are received and used as the second pronunciation clarity score and the second naturalness of speech rhythm score, respectively.

[0106] In some embodiments, a pre-built assessment model for pronunciation clarity and prosodic naturalness is used to score pronunciation clarity and prosodic naturalness, resulting in a first pronunciation clarity score and a first prosodic naturalness score, as well as a second pronunciation clarity score and a second prosodic naturalness score.

[0107] For example, each audio track is scored based on the evaluation form in Table 1: Table 1

[0108] In this embodiment, based on the above evaluation dimensions, evaluation schemes can be customized for different application scenarios, such as dialogue scenarios and face-to-face translation. The evaluation dimensions are comprehensive, covering technical performance and user experience. At the same time, the combination of objective indicators and subjective evaluation makes the results more reliable and provides a clear direction for model and experience optimization.

[0109] The evaluation results of the sound restoration system, based on the text evaluation results and the model evaluation results, also include: Output the first and second score values; The first score is determined by the first pronunciation clarity score and the first rhythmic naturalness score, while the second score is determined by the timbre similarity score, the second pronunciation clarity score, and the second rhythmic naturalness score.

[0110] In some embodiments, the first score is obtained by averaging the first pronunciation clarity score and the first rhythmic naturalness score, and the second score is obtained by averaging the timbre similarity score, the second pronunciation clarity score, and the second rhythmic naturalness score.

[0111] In some embodiments, the first score is obtained by weighted summation of the first pronunciation clarity score and the first rhythmic naturalness score, and the second score is obtained by weighted summation of the timbre similarity score, the second pronunciation clarity score, and the second rhythmic naturalness score.

[0112] In this embodiment, in order to further verify whether the model possesses behavioral characteristics such as tone of voice and interaction habits of a real person, the present invention introduces a multi-dimensional scoring mechanism including pronunciation, rhythm, and timbre, and reduces the dimensionality of these scattered feature values ​​through data fusion and integrates them into a single output first score value and second score value.

[0113] In this process, unstructured human subjective auditory perception of speech physical acoustic characteristics, such as vocal habits and tone, is transformed and aggregated into scores that can be directly compared by computers. This allows the final evaluation results to not only include quantitative tests of intelligibility but also to present the performance of the voice restoration system in maintaining biological characteristics and improving the naturalness of interaction through objective data indicators, thus completing the full-scenario evaluation system.

[0114] In some embodiments, the assessment sample set includes multiple audio samples, which are recorded by sample recorders of different hearing impairment levels, genders, and ages under different environmental factors.

[0115] Specifically, different levels of hearing impairment can be classified according to their causes into sensorineural hearing loss, such as damage to the auditory nerve, and conductive hearing loss, such as damage to the sound conduction pathway. The following classifications are made for different levels.

[0116] Level 1 hearing disability: Extremely severe damage to the structure and function of the auditory system, with an average hearing loss of ≥91 dBHL in the better ear. Without the help of hearing aids, the person cannot rely on hearing to communicate verbally, is extremely limited in activities such as comprehension and communication, and has extremely severe obstacles in participating in social life.

[0117] Level 2 hearing disability: Severe damage to the structure and function of the auditory system, with an average hearing loss of 81-90 dBHL in the better ear. Without the help of hearing aids, there is severe limitation in activities such as understanding and communication, and serious obstacles in participating in social life.

[0118] Level 3 hearing disability: Moderate to severe damage to the structure and function of the auditory system, with an average hearing loss of 61-80 dBHL in the better ear. Without the help of hearing aids, there is moderate limitation in activities such as comprehension and communication, and moderate impairment in participating in social life.

[0119] Level 4 hearing disability: Moderate damage to the structure and function of the auditory system, with an average hearing loss of 41-60 dBHL in the better ear. Without the help of hearing aids, there is mild limitation in activities such as comprehension and communication, and mild impairment in participating in social life.

[0120] Among them, different genders and different age groups can cover both male and female populations, with the male and female ratios each accounting for half; the survey age in the report mainly focuses on adults aged 40 and below, with other age groups accounting for 13% of those aged 41-50 and 6% of those aged 50 and above. The age group for evaluation here only requires adults.

[0121] Different environmental factors can be understood as focusing Chinese accents on Mandarin and not covering people with different accents.

[0122] In some embodiments, the assessment sample set uses recordings from disabled speakers.

[0123] Figure 7 This is a schematic diagram of the overall principle of the evaluation method for the sound restoration system provided by the present invention, as shown below. Figure 7 As shown, an evaluation sample set is constructed. During the evaluation process, ASR is used to perform text recognition on the audio before and after restoration, and then manual scoring is performed to form the evaluation results of the evaluation dimensions.

[0124] The evaluation apparatus for the sound restoration system provided by the present invention is described below. The evaluation apparatus for the sound restoration system described below can be referred to in correspondence with the evaluation method for the sound restoration system described above.

[0125] In some embodiments, such as Figure 8 As shown, an evaluation device for a sound restoration system is proposed, comprising the following modules: The repair module 801 is used to perform sound repair on each audio sample in the evaluation sample set using a sound repair system to obtain a repaired sample set. The first evaluation module 802 is used to perform text recognition on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set, obtain text recognition results, and determine text evaluation results based on the text recognition results. The second evaluation module 803 is used to input each sample in the first sample set into the speech recognition model, obtain the recognition result output by the speech recognition model, and determine the model evaluation result based on the recognition result; the first sample set is a sample set obtained by mixing and shuffling the evaluation sample set and the repair sample set, and the speech recognition model is trained with audio training samples and the recognition result labels corresponding to the audio training samples; The determination module 804 is used to determine the evaluation results of the sound restoration system based on the text evaluation results and the model evaluation results.

[0126] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute an evaluation method for the sound restoration system. The method includes: using the sound restoration system to restore the sound of each audio sample in the evaluation sample set, obtaining a restored sample set; performing text recognition on each audio sample in the evaluation sample set and each restored audio sample in the restored sample set, obtaining a text recognition result, and determining a text evaluation result based on the text recognition result; inputting each sample in the first sample set into a speech recognition model, obtaining a recognition result output by the speech recognition model, and determining a model evaluation result based on the recognition result; the first sample set is a sample set obtained by mixing and shuffling the evaluation sample set and the restored sample set, and the speech recognition model is trained using audio training samples and the recognition result labels corresponding to the audio training samples; and determining the evaluation result of the sound restoration system based on the text evaluation result and the model evaluation result.

[0127] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the evaluation method of the sound restoration system provided by the above methods. The method includes: using the sound restoration system to perform sound restoration on each audio sample in the evaluation sample set to obtain a restored sample set; performing text recognition on each audio sample in the evaluation sample set and each restored audio sample in the restored sample set to obtain a text recognition result, and determining a text evaluation result based on the text recognition result; inputting each sample in the first sample set into a speech recognition model to obtain a recognition result output by the speech recognition model, and determining a model evaluation result based on the recognition result; the first sample set is a sample set obtained by mixing and shuffling the evaluation sample set and the restored sample set, and the speech recognition model is trained using audio training samples and the recognition result labels corresponding to the audio training samples; and determining the evaluation result of the sound restoration system based on the text evaluation result and the model evaluation result.

[0129] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an evaluation method for the sound restoration system provided by the above methods. The method includes: performing sound restoration on each audio sample in a test sample set using the sound restoration system to obtain a restored sample set; performing text recognition on each audio sample in the test sample set and each restored audio sample in the restored sample set to obtain a text recognition result, and determining a text evaluation result based on the text recognition result; inputting each sample in a first sample set into a speech recognition model to obtain a recognition result output by the speech recognition model, and determining a model evaluation result based on the recognition result; the first sample set is a sample set obtained by mixing and shuffling the test sample set and the restored sample set, and the speech recognition model is trained using audio training samples and the recognition result labels corresponding to the audio training samples; and determining the evaluation result of the sound restoration system based on the text evaluation result and the model evaluation result.

[0130] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An evaluation method for a sound restoration system, characterized in that, include: A sound restoration system was used to restore the sound of each audio sample in the evaluation sample set, resulting in a restored sample set. Text recognition is performed on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set to obtain text recognition results, and text evaluation results are determined based on the text recognition results; Each sample in the first sample set is input into the speech recognition model to obtain the recognition result output by the speech recognition model, and the model evaluation result is determined based on the recognition result; the first sample set is a sample set obtained by mixing and shuffling the evaluation sample set and the repair sample set, and the speech recognition model is trained with the audio training samples and the recognition result labels of the simulated blind listening test corresponding to the audio training samples; Based on the text evaluation results and the model evaluation results, the evaluation results of the sound restoration system are determined; Based on the evaluation sample set and the repair sample set, multiple audio pairs are constructed, each audio pair including an audio sample and a repair audio sample corresponding to the audio sample; For each audio pair, using the audio sample as a timbre reference, the timbre of the repaired audio sample is scored for similarity to obtain a timbre similarity score. For each audio sample in the audio pair and the repaired audio sample, the pronunciation clarity and rhythmic naturalness are scored respectively to obtain the first pronunciation clarity score and the first rhythmic naturalness score corresponding to the audio sample, and the second pronunciation clarity score and the second rhythmic naturalness score corresponding to the repaired audio sample. The step of determining the evaluation result of the sound restoration system based on the text evaluation result and the model evaluation result further includes: Output the first and second score values; The first score is determined by the first pronunciation clarity score and the first rhythmic naturalness score, and the second score is determined by the timbre similarity score, the second pronunciation clarity score, and the second rhythmic naturalness score.

2. The evaluation method for the sound restoration system according to claim 1, characterized in that, The text evaluation results include an objective intelligibility improvement index, and the text recognition results include a first text recognition result and a second text recognition result. The step of performing text recognition on each audio sample in the evaluation sample set and each repaired audio sample in the repair sample set to obtain a text recognition result, and determining a text evaluation result based on the text recognition result, includes: An automatic recognition model is used to identify each audio sample in the evaluation sample set to obtain a first recognition result; The automatic recognition model is used to identify each repaired audio sample in the repaired sample set to obtain a second recognition result; Based on the first real-labeled data, a first accuracy rate corresponding to the first identification result and a second accuracy rate corresponding to the second identification result are determined, wherein the first real-labeled data is the real-labeled data corresponding to the evaluation sample set; The objective understandability improvement index is determined based on the first accuracy rate and the second accuracy rate.

3. The evaluation method for the sound restoration system according to claim 2, characterized in that, The determination of the objective comprehensibility improvement index based on the first accuracy rate and the second accuracy rate includes: The difference between the second accuracy rate and the first accuracy rate is used as the objective understandability improvement index.

4. The evaluation method for the sound restoration system according to claim 2, characterized in that, The model evaluation results include a subjective comprehensibility improvement index, and each recognition result includes a text content result. The step of inputting each sample in the first sample set into the speech recognition model to obtain the recognition result output by the speech recognition model, and determining the model evaluation result based on the recognition result, includes: The speech recognition model outputs a first text content result when the first audio sample is used as input, and a second text content result when the first repaired audio sample is used as input. The first repaired audio sample is the audio sample after the first audio sample has been repaired by the sound repair system. Based on the first real-world labeled data, determine the third accuracy rate corresponding to the first text content result and the fourth accuracy rate corresponding to the second text content result; The subjective comprehensibility improvement index is determined based on the third accuracy rate and the fourth accuracy rate; The first audio sample is any one of the audio samples in the evaluation sample set.

5. The evaluation method for the sound restoration system according to claim 4, characterized in that, The determination of the subjective comprehensibility improvement index based on the third accuracy rate and the fourth accuracy rate includes: The difference between the fourth accuracy rate and the third accuracy rate is used as the subjective comprehensibility improvement index.

6. The evaluation method for the sound restoration system according to claim 4, characterized in that, Each of the recognition results also includes an evaluation value for assessing whether the meaning of the sentence is understood, and the model evaluation result also includes a meaning understanding improvement value; The step of inputting each sample in the first sample set into the speech recognition model to obtain the recognition result output by the speech recognition model, and determining the model evaluation result based on the recognition result, further includes: Determine the first evaluation value corresponding to the first audio sample and the second evaluation value corresponding to the first repaired audio sample; The difference between the second evaluation value and the first evaluation value is used as the semantic understanding enhancement value.

7. The evaluation method for the sound restoration system according to claim 4, characterized in that, The model evaluation results also include the semantic repair rate; The step of inputting each sample in the first sample set into the speech recognition model to obtain the recognition result output by the speech recognition model, and determining the model evaluation result based on the recognition result, further includes: Determine the first number of characters, which is the number of identical characters between the real annotation data corresponding to the first repaired audio sample and the second text content result; Determine the second number of characters, which is the total number of characters in the real labeled data corresponding to the first repaired audio sample; The ratio of the first number of characters to the second number of characters is taken as the sentence meaning repair rate.

8. The evaluation method for the sound restoration system according to any one of claims 1 to 7, characterized in that, The assessment sample set includes multiple audio samples, which are recorded by sample recorders of different hearing impairment levels, genders, and ages under different environmental factors.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the evaluation method of the sound restoration system as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the evaluation method of the sound restoration system as described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the evaluation method of the sound restoration system as described in any one of claims 1 to 8.

Images