A method, device, equipment and storage medium for fine-tuning a speech recognition model

Abstract

The present invention discloses a speech recognition model fine-tuning processing method, device, equipment and storage medium, and relates to the field of rail transit and artificial intelligence technology. The present invention generates synthetic corpus for pre-recognition text, collects artificial corpus, and composes a mixed corpus with artificial corpus and synthetic corpus, and classifies and reads corpus from the synthetic corpus and artificial corpus in the mixed corpus, and sets parameters on the basis of a universal base model to perform model fine-tuning, and judges the stability and degree of optimization of the model by evaluating the convergence of parameters, and calculates wer and ser to quantitatively measure the accuracy and reliability of the model in the speech recognition task, so as to adjust the model fine-tuning strategy according to the evaluation results, and further optimize the model performance. The present invention successfully solves the problem of poor speech recognition effect in the field of rail transit, and also overcomes the problem of limited corpus quantity, and effectively saves the heavy workload brought by corpus collection and annotation.

Description

Technical Field

[0001] The present invention relates to the field of rail transportation and artificial intelligence technology, and more specifically to a speech recognition model fine-tuning processing method, device, equipment and storage medium. Background Art

[0002] In today's era, with the rapid advancement of digitalization and intelligence, voice recognition technology is like a bright pearl, shining brightly in many fields, playing an indispensable and vital role. From smart voice assistants on smartphones, to voice control functions in smart home systems, to the wide application in the field of smart customer service, voice recognition technology has been deeply integrated into people's daily lives and various production activities, bringing people unprecedented convenient experience and greatly improving production efficiency and quality of life.

[0003] However, when we focus on rail transit, a highly professional and specific vertical field, we find that the existing general speech recognition models often show many unsatisfactory performances in actual applications and it is difficult to achieve ideal application effects.

[0004] The construction of general speech recognition models is usually based on large-scale general data sets for training. These general data sets cover a wealth of speech samples from different fields and scenarios. The purpose is to enable the model to widely learn the characteristic patterns of various common speech contents, so as to have the ability to more accurately recognize speech in many aspects such as daily life and common work scenarios. For example, in daily social conversations, general office scene voice commands, etc., the general speech recognition model can indeed show a relatively good recognition accuracy, which provides people with a certain degree of convenience.

[0005] However, when we try to apply these speech recognition models trained on general data sets to the field of rail transit, the situation changes significantly. The recognition accuracy rate drops significantly, which seriously affects the effective application of this technology in this field.

[0006] In the field of rail transit, its language environment has distinct characteristics. First of all, there are a large number of professional terms, which constitute an important cornerstone of communication in the field of rail transit. For example, the line name is the basic identifier of the rail transit network. Each line has its own specific name, such as "Metro Line 1" and "Light Rail Line 2". These names frequently appear in the process of voice command transmission and information exchange; the same is true for station names. Each station has a unique title, and it needs to be accurately identified in the dispatching command, passenger service and other links, such as "People's Square Station" and "Beijing West Station". In addition, there are signal instructions, which are key elements to ensure the safety and order of rail transit operation. Professional signal instruction languages ​​such as "green light pass signal" and "red light stop signal" play a decisive role in the process of driving dispatch.

[0007] Moreover, the language expression in the field of rail transit is not random, but has specific patterns and norms. For example, in the dispatching and command link, the instructions issued by the dispatcher often follow a strict format and sequence, first clarifying the route, then pointing out the station, and finally giving specific operation requirements, such as "Metro Line 2, People's Square Station, prepare to depart"; in terms of equipment maintenance, maintenance personnel will also use specific professional terms and standardized expressions to describe equipment failures and maintenance measures when communicating with each other.

[0008] However, since the general speech recognition model is trained based on a general data set, it does not specifically learn and optimize for these unique language features in the rail transit field. Therefore, when faced with language content with specific patterns and norms in the rail transit field, the general speech recognition model is difficult to accurately capture its inherent language features, resulting in frequent recognition errors.

[0009] This phenomenon of recognition error may have serious impacts on many important aspects of rail transit. In terms of dispatching and command, if there is an error in voice recognition, it may lead to the miscommunication of dispatching instructions, which in turn affects the normal operation of the train and may even cause safety accidents; in the field of equipment maintenance, incorrect voice recognition may cause maintenance personnel to misunderstand the equipment failure and take incorrect maintenance measures, which will not only fail to effectively solve the equipment failure, but may also further damage the equipment; in terms of passenger service, if voice recognition is inaccurate, passengers may not be able to get accurate information feedback, affecting the passenger's travel experience, and may even cause passengers to miss the train.

[0010] In order to meet the urgent need for high-precision speech recognition in the rail transit field, an intuitive idea is to directly train the model in this field. However, this approach will face many difficult problems and require a huge cost.

[0011] The corpus data in the field of rail transit is highly professional and scarce. It is not easy to collect a large amount of high-quality data in a specific field, which requires a lot of time and resources. On the one hand, it is extremely difficult to obtain a sufficient number of voice samples covering all aspects and scenarios of rail transit, because the operating environment of rail transit is relatively closed, and the personnel involved are mainly professional staff and passengers, and the collection of voice samples needs to follow strict privacy policies and operating specifications; on the other hand, it also requires a lot of manpower and time costs to label and organize these collected voice samples to make them high-quality corpus data that can be used for model training.

[0012] Moreover, retraining a specialized speech recognition model requires strong computing power as support. Although the data scale in the rail transit field may be relatively small compared to general fields, due to its high professionalism, the accuracy requirements for model training are extremely high, so a large number of complex calculations are required during the training process, which requires high-performance computer equipment and powerful computing resources to match it.

[0013] At the same time, this process also requires the full participation of a professional technical team. The professional technical team must not only be familiar with the principles and processes of speech recognition technology, but also have a deep understanding of the professional knowledge in the field of rail transit, so that they can accurately analyze, process and train models for the collected corpus data. However, it is not easy to form such a team that has both professional technology and knowledge in the field of rail transit, and its labor cost is also quite high.

[0014] Even if huge costs are invested in training, since the rail transit field itself is constantly changing, new lines may be opened, stations may be added or renamed, and signal instructions may be adjusted. At the same time, data limitations still exist. For example, it may still be difficult to obtain sufficient voice samples in certain special scenarios. Therefore, even if a model is trained with a lot of investment, its stability and adaptability are difficult to guarantee in the long term, and inaccurate recognition may still occur when facing new changes in the rail transit field.

[0015] In summary, how to make full use of the existing advantages of the general speech recognition model and use effective fine-tuning methods to enable it to achieve efficient and accurate speech recognition in the field of rail transit while avoiding the high cost of training in the field of rail transit has become a key issue that needs to be urgently solved in the current speech recognition technology in the field of rail transit. The solution to this problem is not only of great significance for improving the operational efficiency, ensuring operational safety, and improving the quality of passenger service in the field of rail transit, but also has an important demonstration role in promoting the precise application of speech recognition technology in more vertical fields. Summary of the invention

[0016] In order to effectively overcome the various defects and shortcomings of the above-mentioned prior art, the present invention proposes a method for fine-tuning a speech recognition model, a device, related equipment and a storage medium thereof. The core purpose of the present invention is to explore how to fine-tune the general speech recognition model in an effective way, so that it can achieve efficient and accurate speech recognition effects in the specific field of rail transit, and simultaneously avoid the high cost of retraining in the field of rail transit and other related issues.

[0017] The present invention generates synthetic corpus for pre-recognized text using speech synthesis tools, collects on-site audio to obtain artificial corpus, forms a mixed corpus with artificial corpus and reads corpus from the synthetic corpus and artificial corpus in the mixed corpus by classification, provides rich data support for the speech recognition model, sets parameters on the basis of the general base model to fine-tune the model, judges the stability and optimization degree of the model by evaluating the convergence of parameters, calculates wer (word error rate) and ser (sentence error rate) to quantitatively measure the accuracy and reliability of the model in the speech recognition task, and adjusts the model fine-tuning strategy according to the evaluation results to further optimize the model performance. The present invention successfully solves the problem of poor speech recognition effect in the field of rail transit by fine-tuning the general model with mixed corpus, and also overcomes the problem of limited corpus quantity, effectively saving the heavy workload brought by corpus collection and annotation.

[0018] In order to solve the above problems existing in the prior art, the present invention is implemented through the following technical solutions.

[0019] A first aspect of the present invention provides a method for fine-tuning a speech recognition model, the method comprising the following steps: S1, processing the pre-recognized text, outputting the annotated text and the pre-synthesized speech text; S2, synthesizing the pre-synthesized speech text into an audio file; S3, classifying the annotated text and the synthesized audio according to the corresponding relationship and saving them into the synthesized corpus; S4. Acquire sound data in real scenarios, annotate them to clarify the semantics of the audio, and obtain manually annotated corpus; S5, classify and save the manually annotated corpus into the artificial corpus; S6. According to the speech recognition scenario, artificial corpus and synthetic corpus are retrieved from the synthetic corpus bureau and the artificial corpus according to business allocation, the corpus items are randomly shuffled to obtain a mixed corpus set, and the mixed corpus set is divided into a training set, a validation set and a test set; S7, training the speech recognition model using the training set, verifying the trained speech recognition model using the verification set, and fine-tuning the model parameters of the speech recognition model; S8. Use the fine-tuned speech recognition model to recognize the test set speech one by one, compare the recognized text results with the annotated text of the test set, and calculate the word error rate and sentence error rate of the adjusted speech recognition model; adjust the model parameter adjustment strategy according to the word error rate and sentence error rate to further optimize the speech recognition model performance.

[0020] Further preferably, in step S1, the pre-recognized text is professional terms and / or standardized expressions extracted from relevant standard documents, operating manuals and / or technical specifications of rail transit.

[0021] Further preferably, in step S1, the pre-recognized text is processed, including determining the text length corresponding to the synthesized speech according to the set synthesized speech duration, and cutting the recognized text into multiple segments according to the text length.

[0022] More preferably, when processing the pre-recognized text, when encountering segmentation punctuation in the text, the text is divided into multiple sentences.

[0023] Further preferably, in step S1, the pre-recognized text is processed, which also includes deleting all characters in the text except words, numbers and letters; then replacing the special pronunciation characters, kilometer sign value readings, speed limit value readings, digital readings and train number letter readings in sequence, retaining the text content before and after the replacement, using the text before the replacement as the annotated text, and the text after the replacement as the pre-synthesized speech text.

[0024] More preferably, in step S1, data cleaning is performed on the output annotated text and the pre-synthesized speech text, specifically, texts that do not contain Chinese characters and / or repeated texts are deleted.

[0025] Further preferably, in step S2, the pre-synthesized speech text is synthesized into an audio file, specifically, the synthesis speech rate is set according to the text length of the pre-synthesized speech text, the volume and pitch are randomly set, and the audio file is generated using a speech synthesis tool.

[0026] More preferably, the standard for setting the synthesis speed according to the text length of the pre-synthesized speech text is as follows: ; Where L represents the character length and S represents the synthetic speech speed.

[0027] More preferably, step S2 also includes a step of checking the synthesized audio file, specifically, checking the quality of the synthesized audio and re-synthesizing the audio with low quality.

[0028] More preferably, speech recognition is performed on the audio files synthesized in step S2 in sequence. If the recognition result is empty, it means that the quality of the audio file is low. Alternatively, the synthesis effect of the audio is checked by manual sampling, and speech synthesis is re-performed for audio files with low audio quality.

[0029] Further preferably, in step S4, obtaining sound data in a real scene includes collecting on-site audio and / or using an open source audio dataset.

[0030] Further preferably, in step S6, the time length ratio of the artificial corpus to the synthetic corpus is not less than 1:9.

[0031] Further preferably, in step S7, the Paraformer speech recognition model is selected as the universal base speech recognition model, and the fine-tuning strategy is determined according to the characteristics and requirements of the fine-tuning task. During the fine-tuning process, the loss and acc parameter changes of the speech recognition model are closely monitored, and the fine-tuning strategy is adjusted in time by analyzing the changes in the loss and acc parameters.

[0032] More preferably, in step S7, when fine-tuning the model parameters of the speech recognition model, the fine-tunable model parameters include any one or more combinations of batch_size, epoch and lr parameters.

[0033] The second aspect of the present invention provides a speech recognition model fine-tuning processing device based on the speech recognition model fine-tuning processing method described in the first aspect, the fine-tuning processing device includes a text processing module, a speech synthesis module, a synthetic corpus, an artificial corpus, an artificial corpus collection module, a model training module and a model verification analysis module; specifically, The text processing module is used to process the pre-recognized text and output the marked text and the pre-synthesized speech text; The speech synthesis module is used to synthesize the pre-synthesized speech text into an audio file; The synthetic corpus is used to classify and store the annotated text and the synthetic audio according to the corresponding relationship; The artificial corpus collection module is used to obtain sound data in real scenes, and the annotation can clarify the semantics of the audio to obtain artificially annotated corpus; The artificial corpus is used to classify and store artificial corpora; The model training module is used to retrieve artificial corpora and synthetic corpora from the synthetic corpus bureau and the artificial corpus according to business allocation according to the speech recognition scenario, randomly shuffle the corpus items to obtain a mixed corpus set, and divide the mixed corpus set into a training set, a verification set and a test set; use the training set to train the speech recognition model, use the verification set to verify the trained speech recognition model, and fine-tune the model parameters of the speech recognition model; The model verification and analysis module is used to use the speech recognition model fine-tuned by the model training module to recognize the test set speech one by one, compare the recognized text results with the annotated text of the test set, and calculate the word error rate and sentence error rate of the adjusted speech recognition model; adjust the model parameter adjustment strategy according to the word error rate and sentence error rate to further optimize the speech recognition model performance.

[0034] Further preferably, the pre-recognized text processed in the text processing module is professional terms and / or standardized expressions extracted from relevant standard documents, operating manuals and / or technical specifications of rail transit.

[0035] Further preferably, the text processing module determines the text length corresponding to the synthesized speech according to the set synthesized speech duration, cuts the recognized text into multiple segments according to the text length; when processing the pre-recognized text, encounters segmentation punctuation in the text and divides the text into multiple sentences.

[0036] Further preferably, the text processing module deletes all characters in the text except words, numbers and letters; then performs special pronunciation character replacement, kilometer sign value reading replacement, speed limit value reading replacement, number reading replacement and train number letter reading replacement in sequence, retaining the text content before and after the replacement, using the text before replacement as the annotated text, and the text after replacement as the pre-synthesized speech text.

[0037] More preferably, the text processing module performs data cleaning on the annotated text and the pre-synthesized speech text. Specifically, the text not containing Chinese characters and / or repeated text is deleted, and the annotated text and the pre-synthesized speech text after data cleaning are output.

[0038] Further preferably, in the speech synthesis module, the synthesis speech rate is set according to the text length of the pre-synthesized speech text, and the volume and pitch are randomly set to generate an audio file.

[0039] More preferably, in the speech synthesis module, the standard for setting the synthesis speed according to the text length of the pre-synthesized speech text is as follows: ; Where L represents the character length and S represents the synthetic speech speed.

[0040] More preferably, after the speech synthesis module synthesizes the audio file, the quality of the synthesized audio is checked, and speech synthesis is performed again for audio files with low audio quality; specifically, speech recognition is performed in sequence on the audio files output by the speech synthesis module using a speech recognition tool, and if the recognition result is empty, it indicates that the audio file quality is low; or the synthesis effect of the audio is checked by manual sampling, and speech synthesis is performed again for audio files with low audio quality.

[0041] Further preferably, the artificial corpus collection module collects live audio and / or uses an open source audio dataset.

[0042] Further preferably, the model training module retrieves artificial corpus and synthetic corpus from the synthetic corpus bureau and the artificial corpus according to business allocation based on the speech recognition scenario, and the time ratio of the retrieved artificial corpus to the synthetic corpus is not less than 1:9.

[0043] Further preferably, the model training module selects the Paraformer speech recognition model as the general base speech recognition model, determines the fine-tuning strategy according to the characteristics and requirements of the fine-tuning task, and closely monitors the loss and acc parameter changes of the speech recognition model during the fine-tuning process. By analyzing the changes in the loss and acc parameters, the fine-tuning strategy is adjusted in time.

[0044] More preferably, when the model training module fine-tunes the model parameters of the speech recognition model, the fine-tunable model parameters include any one or more combinations of batch_size, epoch and lr parameters.

[0045] The third aspect of the present invention provides a computer device, comprising a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute some or all of the steps described in the first aspect of the present invention.

[0046] The fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute some or all of the steps described in the first aspect of the present invention.

[0047] Compared with the prior art, the beneficial technical effects presented by the present invention are significantly reflected in many aspects: 1. The present invention ingeniously constructs a mixed corpus specifically used in the field of rail transit, and innovatively introduces both artificial and synthetic corpora. In actual speech recognition application scenarios, if you rely solely on synthetic corpora, there is often a thorny problem, that is, the fine-tuned speech recognition model cannot accurately recognize artificial speech. The introduction of artificial corpora in the present invention cleverly resolves this dilemma, allowing the speech recognition model to accurately and effectively recognize artificial speech, greatly improving the accuracy of recognition.

[0048] In addition, the present invention further provides a more accurate and highly valuable reference basis for the speech recognition model by scientifically and rationally controlling the duration ratio of artificial corpus and synthetic corpus. This precise ratio control enables the speech recognition model to fully absorb the advantages of the two corpora during learning and training, thereby better adapting to the complex and diverse speech environment in the field of rail transit and further improving the overall performance of the model.

[0049] 2. Another outstanding highlight of the present invention is that it uses relevant standard documents, operation manuals and technical specifications in the field of rail transit as a blueprint, carefully extracts professional terms and standardized expressions from them, and uses them as pre-recognition texts. Subsequently, these pre-recognition texts are subjected to speech synthesis operations with the help of existing speech synthesis technology.

[0050] In this way, rich and unique corpus resources are continuously injected into the field of rail transit. From the perspective of actual application scenarios in different regions and diversified business, this method greatly broadens the diversity of speech types. In addition, in order to facilitate the rapid and accurate retrieval of appropriate corpus according to specific needs in practical applications, the present invention also performs detailed classification and storage of synthesized speech. This measure not only improves the efficiency of corpus use, but also further optimizes the entire speech processing process, so that it can better serve various speech-related businesses in the field of rail transit.

[0051] 3. In the process of speech synthesis, the present invention always takes the pre-recognized text as the core orientation, and implements multi-step refined processing of the pre-recognized text, covering key links such as length control, preprocessing and data cleaning.

[0052] Through length control, it can ensure that the synthesized speech meets specific requirements in terms of duration, avoiding situations where it is too long or too short and affects the user experience; the preprocessing stage can perform preliminary optimization and adjustment on the pre-recognized text to make it more suitable for subsequent synthesis operations; data cleaning removes impurities, erroneous information, etc. in the text one by one, ensuring that the text data input into the speech synthesis system is pure and accurate.

[0053] After this series of rigorous processing steps, high-quality synthetic corpora are finally generated. These high-quality synthetic corpora lay a very solid foundation for the subsequent fine-tuning of the speech recognition model, so that the fine-tuned model can show more outstanding speech recognition performance with the support of high-quality corpora.

[0054] 4. The present invention adopts a method of fine-tuning the speech recognition model based on mixed corpus, which is systematic and scientific. Specifically, in the fine-tuning process, artificial and synthetic corpus are first retrieved from the carefully constructed corpus and randomly shuffled. This shuffling operation is intended to allow the model to be exposed to different types of corpus more comprehensively and evenly during the learning process, avoiding learning deviations caused by corpus order problems.

[0055] Then, select the appropriate universal model of the base and determine the fine-tuning strategy according to the specific situation. During the fine-tuning process, closely monitor the changes in parameters. Once abnormal changes in parameters or situations that are inconsistent with expectations are found, the fine-tuning strategy will be adjusted in a timely manner.

[0056] After the model is fine-tuned, a test experiment will be carefully organized to evaluate the fine-tuning effect based on a series of scientific and reasonable indicators. If the fine-tuning effect is not good during the evaluation process, the present invention will conduct an in-depth analysis of the problems in the corpus and parameters, and continuously optimize the relevant links to ensure that the overall effect of the model can be continuously improved so that it can better meet the speech recognition needs in the rail transit field.

[0057] 5. The present invention has successfully overcome the long-standing problem of poor speech recognition in the field of rail transit by ingeniously using mixed corpus to fine-tune the general model. In the past, due to the combined influence of multiple factors such as complex speech environment and limited corpus, the speech recognition effect was often unsatisfactory. The emergence of the present invention has effectively improved this situation, allowing speech recognition to reach a higher level in the field of rail transit.

[0058] At the same time, the present invention also overcomes the bottleneck problem of limited corpus quantity, and is no longer limited by the heavy workload brought by corpus collection and annotation in the traditional way. By making rational use of mixed corpus, not only a lot of manpower, material resources and time costs are saved, but also efficient training and optimization of speech recognition models can be achieved under limited resource conditions, further promoting the development of speech recognition technology in the field of rail transit. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of the speech recognition model fine-tuning processing method of the present invention; Figure 2 It is a structural schematic diagram of the speech recognition model fine-tuning processing device of the present invention; Figure 3 A flowchart of constructing a synthetic corpus in the speech recognition model fine-tuning processing method of the present invention; Figure 4 A flowchart of constructing an artificial corpus in the speech recognition model fine-tuning processing method of the present invention; Figure 5 The present invention is a flowchart of model fine-tuning in the speech recognition model fine-tuning processing method. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described in detail below in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example 1 As a preferred embodiment of the present invention, refer to the attached specification Figure 1 As shown, this embodiment discloses a method for fine-tuning a speech recognition model, and the method for fine-tuning includes the following steps: S1, processing the pre-recognized text, outputting the annotated text and the pre-synthesized speech text; In this initial step, the first task is to carry out comprehensive and detailed processing of the pre-recognition text. As the basic input information of the entire speech recognition process, the quality and standardization of the pre-recognition text plays a vital role in the smooth progress of subsequent steps and the performance of the final speech recognition model.

[0062] In this process, a series of advanced text processing technologies and algorithms can be used to deal with various problems that may exist in the pre-recognized text. For example, typos and grammatical errors in the text can be corrected, and some unclear or ambiguous expressions can be clarified and standardized. At the same time, part-of-speech tagging and syntactic analysis can be performed on the text according to specific language rules and speech recognition requirements in order to have a deeper understanding of the semantic structure of the text.

[0063] After this series of complex and sophisticated processing, the final output is the annotated text and pre-synthesized speech text. The annotated text is the product of deep processing and annotation based on the original pre-recognition text. It clearly marks the important information such as the part of speech and syntactic role of each word in the text, providing a more accurate semantic reference for subsequent model training and recognition. The pre-synthesized speech text is specially prepared for the subsequent speech synthesis step. On the basis of retaining the core semantics of the original text, it is formatted and optimized according to the specific requirements of speech synthesis so that it can be smoothly synthesized into an audio file.

[0064] S2, synthesizing the pre-synthesized speech text into an audio file; This step takes the pre-synthesized speech text output from the previous step and synthesizes it into an audio file. In this process, advanced speech synthesis technologies and tools are used. These technologies and tools have a high degree of accuracy and naturalness, and can accurately convert the pre-synthesized speech text into an audible audio form.

[0065] Specifically, the speech synthesis system will simulate the principles of human voice production based on the vocabulary, intonation, pauses and other information in the pre-synthesized speech text, combined with the pre-trained acoustic model and rhythm model, to generate natural and smooth audio signals. During the synthesis process, the timbre of the speech will be fully considered, and the appropriate timbre, such as male voice, female voice, child's voice, etc., will be selected according to the specific application scenario and user needs to ensure that the synthesized audio file is more in line with actual needs in terms of auditory perception.

[0066] S3, classifying the annotated text and the synthesized audio according to the corresponding relationship and saving them into the synthesized corpus; After completing the speech synthesis step, the next step is to classify the annotated text and synthesized audio according to a strict correspondence and save them into the synthetic corpus. As one of the important resource libraries for subsequent speech recognition model training, the scientificity and standardization of the synthetic corpus have a direct impact on the model training effect.

[0067] In this process, the annotated text and synthesized audio will be classified according to different classification standards. For example, you can classify according to the subject content of the text, and group the annotated text and synthesized audio related to the same subject into one category; you can also classify according to the timbre of the voice, and put the audio synthesized with the same timbre and its corresponding annotated text together; you can also classify according to the application scenario, and group the annotated text and synthesized audio of the same application scenario into one category. This classification and saving method can facilitate the subsequent training of the speech recognition model, and quickly and accurately retrieve the required synthetic corpus according to different training needs.

[0068] S4. Acquire sound data in real scenarios, annotate them to clarify the semantics of the audio, and obtain manually annotated corpus; At the same time, it is also necessary to obtain sound data in real scenarios and manually annotate them to obtain manually annotated corpus. Sound data in real scenarios comes from a wide range of sources, such as voice call records in the actual rail transit field, on-site meeting recordings, and voice interactions in daily work. These sound data are extremely authentic and diverse, and can fully reflect various situations in actual voice application scenarios.

[0069] After acquiring the sound data, professional annotators will annotate the semantics of the audio in detail according to clear annotation rules and standards. The annotation content includes but is not limited to the specific content expressed in the audio, the speaker's emotional state, the tone of voice, pauses and other information. Through this manual annotation method, the original simple sound data can be converted into manually annotated corpus with clear semantic information, providing rich materials for the subsequent construction of the artificial corpus and the training of the speech recognition model.

[0070] S5, classify and save the manually annotated corpus into the artificial corpus; After the acquisition of manually annotated corpus is completed, it is classified and saved in the artificial corpus. Similar to the synthetic corpus, the construction of the artificial corpus also needs to follow certain classification standards and specifications. For example, it can be classified according to the source scene of the sound data, and the manually annotated corpus from the same type of scene can be classified into one category; it can also be classified according to the professional field of the annotator, and the manually annotated corpus annotated by the annotators in the same professional field can be put together. This classification and saving method can facilitate the subsequent call of the artificial corpus according to different application scenarios and training needs. It is also conducive to the effective management and maintenance of the artificial corpus.

[0071] S6. According to the speech recognition scenario, artificial corpus and synthetic corpus are retrieved from the synthetic corpus bureau and the artificial corpus according to business allocation, the corpus items are randomly shuffled to obtain a mixed corpus set, and the mixed corpus set is divided into a training set, a validation set and a test set; According to different speech recognition scenarios, artificial and synthetic corpora are retrieved from synthetic corpora and artificial corpora respectively according to specific business allocations. This step needs to fully consider the specific needs of different speech recognition scenarios. For example, in some scenarios with high requirements for speech naturalness, more artificial corpora may need to be retrieved; and in scenarios with high requirements for speech synthesis speed, more synthetic corpora may need to be retrieved; the time ratio of artificial corpora to synthetic corpora should not be less than 1:9.

[0072] After the retrieval is completed, the corpus items are randomly shuffled. The purpose of this operation is to break the original order of the corpus, making the mixed corpus set more random and diverse, and avoiding model learning deviations caused by corpus order problems in the subsequent speech recognition model training process.

[0073] After the above operations, a mixed corpus is obtained. Subsequently, the mixed corpus is divided into a training set, a validation set, and a test set according to a certain ratio. The training set is mainly used for preliminary training of the speech recognition model, so that the model can learn the speech features and semantic information in the corpus; the validation set is used to verify the trained speech recognition model during the training process to check whether the model has problems such as overfitting; the test set is used to test the final performance of the model after the model training is completed to evaluate whether the model has met the expected performance requirements.

[0074] S7, training the speech recognition model using the training set, verifying the trained speech recognition model using the verification set, and fine-tuning the model parameters of the speech recognition model; The speech recognition model is trained using the training set. During the training process, the speech recognition model will continuously adjust its model parameters based on the corpus information in the training set to learn the characteristics and semantic information of the speech. This process takes a long time and a lot of computing resources, because the model needs to improve its performance gradually through continuous iterative learning. After completing the initial training using the training set, the trained speech recognition model is then verified using the validation set. The purpose of the verification is to check whether the model is overfitting, that is, whether the model is too dependent on the specific corpus in the training set and performs poorly when facing new corpus. If the model is found to be overfitting, the model parameters need to be adjusted, such as by adding regularization terms, adjusting the learning rate, etc., to improve the performance of the model. By fine-tuning the model parameters, the speech recognition model can better adapt to different speech recognition scenarios and improve the versatility and performance of the model.

[0075] S8. Use the fine-tuned speech recognition model to recognize the test set speech one by one, compare the recognized text results with the annotated text of the test set, and calculate the word error rate and sentence error rate of the adjusted speech recognition model; adjust the model parameter adjustment strategy according to the word error rate and sentence error rate to further optimize the performance of the speech recognition model; The fine-tuned speech recognition model is used to recognize the test set speech one by one. During the recognition process, the recognized text results are compared with the annotated text of the test set. Through this comparison, the word error rate and sentence error rate of the adjusted speech recognition model can be calculated. The word error rate and sentence error rate are important indicators for measuring the performance of the speech recognition model, and they directly reflect the accuracy of the model in recognizing speech.

[0076] Adjust the model parameter adjustment strategy based on the calculated spelling and sentence error rates. If the spelling and sentence error rates are high, it means that the performance of the model needs to be improved, and further in-depth analysis is needed to determine the cause of the problem, such as insufficient corpus, unreasonable model structure, or other reasons. Then, based on the analysis results, further optimize the performance of the speech recognition model, such as by increasing corpus, adjusting model structure, improving training methods, etc., so that the speech recognition model can meet higher performance requirements.

[0077] Through the above complete steps, the speech recognition model fine-tuning processing method disclosed in this embodiment can effectively train, verify, fine-tune, evaluate and optimize the performance of the speech recognition model, thereby improving the performance of the speech recognition model in rail transit scenarios.

[0078] Example 2 As another preferred embodiment of the present invention, refer to the attached specification Figure 3 As shown, this embodiment is a further detailed description and supplement to the technical solution of the present invention based on the above-mentioned embodiment 1. In this embodiment, the construction of the synthetic corpus in the speech recognition model fine-tuning processing method of the present invention is further described. The specific contents are as follows: Overview of the synthetic corpus construction process: When starting to build a synthetic corpus, a series of related operations need to be carried out in sequence. First, the pre-recognition text must be collected, and then various data preprocessing tasks are carried out for the pre-recognition text, including pre-recognition text data cleaning, replacement of special text readings, etc. Next, an audio file is synthesized based on the replaced text, and a speech recognition operation is performed on the synthesized audio file to determine whether the audio file meets the requirements, that is, to determine whether it is valid. If the requirements are not met, audio synthesis must be performed again; if the requirements are met, manual sampling inspection can be further carried out. Finally, the audio files that meet the requirements are associated with the corresponding annotated texts and stored in categories to form a synthetic corpus.

[0079] When collecting pre-recognition texts, the selected pre-recognition texts are all professional terms and standardized expressions carefully extracted from relevant standard documents, operation manuals, technical specifications and other materials in the field of rail transit. These text sources are highly professional and targeted, and can provide basic materials that fit actual application scenarios for subsequent speech synthesis and corpus construction.

[0080] When preprocessing the pre-recognized text, it is necessary to determine the corresponding text length based on the set synthesized speech duration. According to the determined text length, the pre-recognized text is cut and divided into multiple segments. Specifically, if there are segmentation punctuation marks in the text, the text is further divided into multiple sentences according to the punctuation marks. For example, to ensure that the synthesized audio duration does not exceed 18 seconds, after measurement and practical experience summary, the length of the pre-recognized text should be controlled within 80 words. In this process, once the text length exceeds 80 words, the text cutting operation is performed according to the above rules; when segmentation punctuation marks are encountered, the text is divided into multiple sentences so that subsequent speech synthesis and related processing can be performed more smoothly.

[0081] The purpose of the pre-identification text data cleaning phase is to purify the text and delete all characters except words, numbers and letters in the text. This operation can effectively remove possible interference information in the text, making the text content purer and more standardized, laying a good foundation for subsequent processing.

[0082] Replacement of special pronunciations of text: This step requires the completion of character replacement operations for various special pronunciations in sequence, including replacement of special pronunciation characters, replacement of kilometer value readings, replacement of speed limit readings, replacement of digital readings, and replacement of train number letter readings, etc. When performing these replacement operations, the text content before and after the replacement needs to be retained, wherein the text before the replacement is used as the annotated text for subsequent corresponding association with the synthesized audio file; and the text after the replacement is used as the pre-synthesized speech text to provide accurate input content for subsequent audio synthesis.

[0083] In addition, data cleaning can be performed again on the output annotated text and pre-synthesized speech text. The specific operation method is to delete those texts that do not contain Chinese characters and repeated texts, further improve the quality and standardization of the text, and ensure the accuracy and efficiency of subsequent processing.

[0084] When synthesizing an audio file, the synthesized speech rate should be set appropriately according to the length of the pre-synthesized speech text. At the same time, the volume and pitch should be set randomly, and then a professional speech synthesis tool should be used to generate an audio file.

[0085] For example, the synthesis speech rate can be set according to the text length of the pre-synthesized speech text according to the following standards: In the formula, L represents the character length, and S represents the synthesis speed. By accurately setting the synthesis speed according to the text length, the synthesized audio file can be more in line with actual needs in terms of speech speed, improving the naturalness and audibility of the audio.

[0086] Perform speech recognition on the synthesized audio file to determine whether the audio file meets the requirements. If the recognition result is empty, it means that the audio file is of low quality; or the audio synthesis effect can be checked by manual sampling. For those audio files that are judged to have low audio quality, the speech synthesis operation needs to be re-performed until an audio file that meets the requirements is generated.

[0087] Through the above series of rigorous and meticulous operational processes, the synthetic corpus can be constructed in a more scientific and standardized manner, providing high-quality and rich corpus resources for the subsequent fine-tuning of the speech recognition model, thereby effectively improving the application effect of the speech recognition model in the field of rail transit.

[0088] Example 3 As another preferred embodiment of the present invention, refer to the attached specification Figure 4 As shown, this embodiment is a further detailed description and supplement to the technical solution of the present invention based on the above-mentioned embodiment 1 or embodiment 2. In this embodiment, the construction of the artificial corpus in the speech recognition model fine-tuning processing method of the present invention is further described. The specific contents are as follows: Refer to the instruction manual Figure 4 As shown in the figure, the construction of the artificial corpus mainly covers the following key steps: First, the artificial corpus is collected. The collected artificial corpus is mainly based on the audio data in the real environment. If it is not possible to collect the on-site audio in actual operation, an open source audio data set can be used as an alternative. After the collection work is completed, the collected voice data needs to be cleaned to screen out high-quality corpus and eliminate those low-quality, unclear or erroneous parts to ensure that the corpus entering the subsequent processing stage has high reliability. Next, according to the specific needs of the rail transit field, the screened artificial corpus should be annotated and classified, and divided according to different application scenarios, such as dispatching command, station duty, etc. Finally, the classified corpus is stored in the artificial corpus.

[0089] When collecting artificial corpus, the source is mainly focused on audio data in real-life on-site environments. The scope of voice collection involved here is quite broad, and is not limited to the professional terms, instructions, and daily communication content used by train drivers, dispatchers, station attendants, and other personnel in the course of their work. During the collection process, it is important to ensure that the collected voice is clear and discernible, without unfavorable factors such as noise interference, so as to provide high-quality basic materials for the subsequent construction of the artificial corpus.

[0090] For example, when collecting voice data from train drivers, we need to include not only their professional instructions, such as train start, stop, accelerate, decelerate, and other related command terms, but also some of their routine communication in daily work, such as communicating with dispatchers about driving conditions, handing over work with station attendants, etc. In addition, no matter what type of voice, its sound quality must be clear and not affected by external noise, so as to accurately reflect the voice situation in the actual work scene, and provide reliable data support for the subsequent construction of artificial corpus and fine-tuning of speech recognition model.

[0091] Through the rigorous and standardized steps above, an artificial corpus can be constructed more scientifically and efficiently, thereby providing richer, more accurate and reliable corpus resources for the application of speech recognition models in the field of rail transit, and further improving the performance and application effect of speech recognition models.

[0092] Example 4 As another preferred embodiment of the present invention, refer to the attached specification Figure 5 As shown, this embodiment is a further detailed description and supplement to the technical solution of the present invention based on the above-mentioned embodiment 1, embodiment 2 or embodiment 3. In this embodiment, the specific process of model fine-tuning in the speech recognition model fine-tuning processing method of the present invention is further described. The specific contents are as follows: Refer to the instruction manual Figure 5 As shown in Figure 1, the specific process of model fine-tuning includes the following steps: Mixed corpus preparation and division: Before fine-tuning the model, we must first retrieve artificial and synthetic corpora from the built corpus according to business classification based on the specific speech recognition scenario. Then, in order to make the corpus more random and avoid deviations in the model during learning, the retrieved corpus items need to be randomly shuffled.

[0093] After completing the above operations, the preprocessed mixed corpus needs to be divided into training set, validation set and test set. When dividing, a reasonable ratio of the three should be set, which is set to 8:1:1 this time. In addition, special attention should be paid to ensure that the length of the training set is not less than 10 hours, so that the model can be provided with sufficient learning materials and better grasp the speech features and semantic information during the training process.

[0094] Select a universal base model and determine the fine-tuning strategy: Select the Paraformer speech recognition model as the universal base model used for this fine-tuning. After deciding to use this model, it is necessary to carefully determine the appropriate fine-tuning strategy based on the characteristics of the fine-tuning task itself and specific needs. For example, make reasonable adjustments to key parameters such as batch_size (batch size), epoch (number of training rounds), and lr (learning rate) of the model.

[0095] During the fine-tuning process, we should pay close attention to the changes in the key indicator parameters of the model, such as loss (loss value), acc (accuracy), etc. Through in-depth and detailed analysis of these indicators, we can promptly discover possible problems in the model training process, and then adjust the fine-tuning strategy in time to ensure that the model always develops in the optimal direction to achieve better fine-tuning results.

[0096] Test experiments and effect evaluation after model fine-tuning: After the model fine-tuning is completed, corresponding test experiments must be carried out to evaluate the effect of model fine-tuning. The specific operations are as follows: First, the fine-tuned model is used to recognize the test set speech one by one. Then, the recognized text results are carefully compared with the originally annotated text of the test set. Through this comparison, the wer (word error rate, i.e., the wrong character rate) and ser (sentence error rate, i.e., the wrong sentence rate) of the fine-tuned model can be accurately calculated. These indicators are of great significance for us to fully understand the effect of model fine-tuning, and can provide a strong basis for further improvement and optimization of the model in the future.

[0097] Analyze and optimize based on the fine-tuning effect: If the calculated WER of the fine-tuning model is less than 5%, it means that the fine-tuning work has achieved relatively good results, which means that the model can convert speech into text relatively accurately and can basically meet the needs of actual on-site applications.

[0098] However, if wer is greater than or equal to 5%, it means that the fine-tuning effect is not good. In this case, it is necessary to conduct an in-depth and detailed analysis of the corpus and the fine-tuning parameters to find the root cause of the problem.

[0099] From the perspective of corpus, there may be many reasons for poor fine-tuning results. For example, the corpus quality may not be high, and there may be noise interference, which affects the model's accurate recognition of speech; or there may be incorrect annotations, causing the model to receive wrong information during the learning process; or the corpus may lack language expressions in specific fields, resulting in the model being unable to fully grasp the speech characteristics of the relevant fields.

[0100] From the parameter perspective, there are also many factors that may affect the fine-tuning effect. For example, if the learning rate is set too high or too low, the model may not be able to learn and converge effectively; if the batch size is not appropriate, it will also affect the training efficiency and effect of the model; furthermore, if the number of training rounds is insufficient, the model may not be able to fully learn the speech features and semantic information in the corpus. Different parameter settings will have completely different effects on the fine-tuning and performance of the model, so these parameters need to be carefully adjusted and optimized.

[0101] By continuously trying and optimizing the above-mentioned links, the model's spelling error rate can be gradually reduced until a satisfactory fine-tuning effect is achieved, so that the speech recognition model can better adapt to the needs of actual application scenarios.

[0102] Through the above detailed explanation and optimization measures of the specific process of model fine-tuning, the speech recognition model can be fine-tuned more scientifically and effectively, improving its performance in speech recognition tasks, and thus better serving the application needs in related fields.

[0103] Example 5 As another preferred embodiment of the present invention, refer to the attached specification Figure 1 , Attachment Figure 3 , Attachment Figure 4 and attached Figure 5 As shown, this embodiment provides a method for fine-tuning a speech recognition model. First, a synthetic corpus and an artificial corpus need to be constructed. The construction order of the synthetic corpus and the artificial corpus is not limited. The synthetic corpus and the artificial corpus form a mixed corpus, and the mixed corpus is used as a data set for training a speech recognition model. Artificial corpus and synthetic corpus are retrieved from the mixed corpus according to speech recognition scenarios and business allocations, and the corpus items are randomly shuffled to obtain a mixed corpus set, and the mixed corpus set is divided into a training set, a verification set, and a test set; the speech recognition model is trained using the training set, and the speech recognition model obtained by the training is verified using the verification set, and the model parameters of the speech recognition model are fine-tuned; the fine-tuned speech recognition model is used to recognize the test set speech one by one, and the recognized text results are compared with the annotated text of the test set, and the word error rate and sentence error rate of the adjusted speech recognition model are calculated; the model parameter adjustment strategy is adjusted according to the word error rate and sentence error rate to further optimize the performance of the speech recognition model.

[0104] In this embodiment, when constructing a synthetic corpus, the construction of the synthetic corpus is guided by pre-recognition text. Specifically, according to the speech recognition scenario, relevant pre-recognition text data (such as train operation, scheduling and control commands, etc.) is collected, and the text content is read through technologies such as OCR. Building a comprehensive and diverse synthetic corpus is crucial to improving the accuracy and reliability of the speech recognition model.

[0105] Collecting pre-recognition text is an important part of this process. Professional terms and standardized expressions can be extracted from relevant standard documents, operating manuals, technical specifications and other documents or other data of rail transit to provide accurate industry-specific voice materials for the synthetic corpus. The collected pre-recognition text is then optimized and classified, including but not limited to data cleaning, special reading replacement and other methods. Finally, the corpus text is reviewed and checked to further improve its quality.

[0106] As an example of this embodiment, the construction of the synthetic corpus includes the following steps: S11, read the text classification and content, ensure that the length of the synthesized speech does not exceed 18 seconds, so the length of the synthesized text should be within 80 words, and it will be divided into multiple sentences when there are segmentation punctuation marks, and the text will be segmented if it exceeds 80 words; Step S12, preprocessing the text in step S11. Specifically, all characters except Chinese characters, numbers and letters in the text are deleted. Then, special reading characters, kilometer reading, speed limit reading, number reading and train number letter reading are replaced in sequence. The text content before and after the replacement is retained, and the text before the replacement is used as the annotation text, while the text after the replacement is used for speech synthesis; Step S13: In order to avoid unknown effects caused by invalid characters, the text in step S12 needs to be cleaned. On the one hand, texts that do not contain Chinese characters at all should be deleted, because such texts may have no practical significance for the overall corpus construction and may even interfere with subsequent processing. On the other hand, duplicate texts should also be removed. The duplicate texts mentioned here cover the duplicate content in the corpus. A large amount of duplicate content will not only take up unnecessary storage space, but also affect the effect of later model fine-tuning. Step S14, reading the text in step S13 in sequence, setting the synthesis speed according to the length of the text, randomly setting the volume and pitch, and calling the speech synthesis interface to generate an audio file; The formula for setting the synthesis rate based on the text length is as follows, where L represents the character length and S represents the synthesis rate: ; Step S15: Perform speech recognition on the audio files generated in step S14 in sequence. If the recognition result is empty, it means that there is a problem with the audio file. In addition, the synthesis effect of the audio file can be checked by manual sampling, and speech synthesis can be re-performed for invalid audio. After the inspection is completed, the audio file and the annotated text are saved in different classification folders of the synthesized corpus according to the text classification.

[0107] In the above voice synthesis process, the use of voice cloning technology can inject richer and more unique corpus resources into the rail transit field. Starting from different regions and business directions, it can increase the diversity of voice types. Simulate the complex voice environment on-site, automatically adjust the corpus text in synthesis step S11 through a random algorithm, and conduct recognition checks and manual sampling on the synthesized voice, including but not limited to voice clarity and naturalness, content integrity, and accuracy of polyphonic characters. After checking and confirming that there are no errors, save the synthesized voice into the synthesis corpus according to the corresponding classification, so as to enrich the resources of the synthesis corpus. In actual application scenarios, appropriate corpus can be retrieved from the corpus according to the classification based on different needs.

[0108] As another example of this embodiment, the construction of the synthesis corpus includes the following steps: S11. According to the voice recognition scenario, collect relevant pre-recognition text data (such as train operation, dispatching, and control commands, etc.), and read the text content through technologies such as ocr to construct a comprehensive and diverse synthesis corpus, which is crucial for improving the accuracy and reliability of the voice recognition model; S12. After extracting the preliminary text, it is necessary to check the expression accuracy of the text content to ensure that there are no problems in aspects such as typos, grammar errors, garbled display, or special symbols, and review and correct professional terms, format specifications, and semantic clarity; S13. The text data cleaning aims to perform a series of processing on the text data in step S12 to improve its quality and usability. The specific operations include paragraph clause separation, removal of empty text, removal of duplicate content, removal of punctuation marks, and standardization processing, etc., to ensure that the number of words in each text does not exceed the limit; S14. Since some numbers and letters in the rail transit field have special pronunciations, replace the pre-synthesized text in step S33 according to the actual pronunciation and configuration requirements. For example, replace "G1809" with "high one eight zero nine", and at the same time retain the text before and after replacement, which are used as the annotated text and the pre-synthesized text respectively; S15. Use a personalized voice synthesis tool to clone a variety of voices from different regions and characters, and sequentially perform voice synthesis on the pre-synthesized text in step S14 through a random algorithm; S16. Use the voice recognition model to test the synthesized audio in step S15. If there is no recognized text output, the task audio is invalid, and synthesize this audio again; S17. In order to ensure the quality of the synthesized audio, adopt the method of manual random sampling for inspection, focusing on the clarity and fluency of the audio to ensure that each voice is natural and coherent. In addition, it is also necessary to evaluate the authenticity of the voice to ensure that the synthesis effect can meet the expected standards; S18. The corpus that meets the spot check conditions is classified according to different scenarios and stored in the synthetic corpus for later use in fine-tuning of the speech recognition model.

[0109] As an example of this embodiment, the construction of the artificial corpus includes the following steps: S21. The corpus collection phase mainly collects voice data for actual application scenarios, which can better meet specific language needs and provide more targeted and practical resources for speech recognition model learning and application; S22. If the conditions for voice collection are not available on site, and it is impossible to collect voice in the actual scene, open source voice datasets can be used as an alternative, such as AISHELL. S23, using the VAD (Voice Activity Detection) model to remove the silent and non-voice parts of the corpus collected in step S21, extracting effective vocal segments, and saving them in segments as audio files, ensuring that the file length does not exceed 18 seconds, performing preliminary speech recognition on the audio files, checking their recognition effect and audio quality, and deleting blank audio files; S24, performing manual annotation based on the speech recognition text in step S23 to provide more accurate language information and annotation data for the speech recognition model, and deleting low-quality audio files during the annotation process to ensure the quality and reliability of the annotation data; S25. Classify and save the annotated corpus to facilitate subsequent management and use. Classification based on different application scenarios, language types and other factors can improve the retrieval efficiency and usability of the corpus. For open source data sets, there is no need for annotation and classification processing, and they can be directly saved to the artificial corpus for use as supplementary resources.

[0110] The above-mentioned synthetic corpus and artificial corpus are combined to form a mixed corpus. The purpose of adding artificial corpus is to improve the effect of artificial speech recognition and avoid the model being unable to recognize artificial speech normally due to synthetic corpus. Among them, the time ratio of artificial corpus to synthetic corpus is not less than 1:9, so as to provide a more accurate reference standard for the model and ensure that the model can accurately recognize artificial speech.

[0111] As an example of this embodiment, a method for fine-tuning a speech recognition model based on the mixed corpus includes the following steps: S31. According to the actual application scenarios and target tasks of speech recognition, comprehensively evaluate the performance, accuracy, stability and other factors of the universal base model, and select the universal base model that best suits the current application; S32. Considering the characteristics and needs of actual application scenarios, mixed corpora are obtained from artificial corpora and synthetic corpora according to corpus classification. If artificial corpora for relevant scenarios do not exist, open source corpus datasets are provided, and the time-to-length ratio is guaranteed to be no less than 1:9. On the basis of ensuring a certain amount of high-quality artificial corpora, a large amount of synthetic corpora are fully utilized to enrich the data and improve the generalization ability of the model. S33. Before fine-tuning begins, fine-tuning parameters need to be set according to the characteristics of the general base model and the size of the corpus. These parameters may include learning rate, batch size, number of training rounds, etc. Different model and corpus combinations may require different parameter settings to achieve the best fine-tuning effect; S34. During the fine-tuning process, closely observe the fitting degree of parameters such as loss. Loss reflects the gap between the model's predicted value and the true value. As the training continues, the loss should gradually decrease and stabilize. By observing the changes in loss, you can determine whether the model training is proceeding normally, and whether you need to adjust parameters or stop training early; S35. After fine-tuning, use the new model to perform speech recognition on the test set and calculate the wer (word error rate) and ser (sentence error rate) of the test set. wer (word error rate) measures the error level of the model at the word level, and ser (sentence error rate) measures the error level of the model at the sentence level. These indicators can objectively reflect the optimization effect of the model; S36. Evaluate the recognition effect of the fine-tuned model based on wer (word error rate). The lower the wer (word error rate), the higher the recognition accuracy of the model. You can compare wer (word error rate) with other models or benchmarks to determine whether the fine-tuned model meets the needs of actual applications; S37. Export the fine-tuned and optimized model and perform quantization. Quantization can reduce the size of the model and improve the running speed and deployment efficiency of the model.

[0112] As another example of this embodiment, a method for fine-tuning a speech recognition model based on the mixed corpus includes the following steps: S31. According to the speech recognition scenario, artificial and synthetic corpora are retrieved from the corpus according to business classification, and the corpus items are randomly shuffled. The pre-processed mixed corpus is divided into a training set, a validation set, and a test set. The ratio of the three is set to 8:1:1, and the training set duration is guaranteed to be no less than 10 hours; S32. Select the Paraformer speech recognition model as the general base model for fine-tuning, and determine the appropriate fine-tuning strategy according to the characteristics and requirements of the fine-tuning task, such as adjusting the model's batch_size, epoch, lr and other parameters. During the fine-tuning process, closely monitor the changes in the model's loss, acc and other parameters, and adjust the fine-tuning strategy in a timely manner through the analysis of these indicators to ensure that the model develops in the optimal direction; S33. After the model is fine-tuned, a test experiment must be conducted. First, the fine-tuned model is used to recognize the test set speech one by one. Then, the recognized text results are compared with the annotated text of the test set, and the wer (word error rate) and ser (sentence error rate) of the fine-tuned model are calculated. These indicators can help us understand the effect of model fine-tuning so as to further improve and optimize it; S34. If the wer of the fine-tuned model is less than 5%, it means that the fine-tuning has achieved good results and can convert speech into text more accurately to meet the actual on-site application needs. Otherwise, it means that the fine-tuning effect is not good, and it is necessary to conduct in-depth analysis based on the corpus and fine-tuning parameters to find the root cause of the problem. From the perspective of the corpus, it may be that the corpus is of low quality, insufficient quantity or insufficient representativeness. For example, there may be noise, incorrect annotations or lack of language expression in a specific field in the corpus; from the perspective of parameters, it may be that the learning rate is too high or too low, the batch size is not appropriate, the number of training rounds is insufficient, etc. Different parameter settings will have different effects on the fine-tuning and performance of the model, and need to be carefully adjusted and optimized. Through continuous trial and optimization, the model's word error rate can be gradually reduced until a good fine-tuning effect is achieved.

[0113] Example 6 As another preferred embodiment of the present invention, this embodiment is a speech recognition model fine-tuning processing device based on the speech recognition model fine-tuning processing method described in the above-mentioned embodiment 1, embodiment 2, embodiment 3, embodiment 4 or embodiment 5, and the speech recognition model fine-tuning processing device is described in the embodiment 1, embodiment 2, embodiment 3, embodiment 4 or embodiment 5. Figure 2 As shown, the fine-tuning processing device includes a text processing module, a speech synthesis module, a synthetic corpus, an artificial corpus, an artificial corpus collection module, a model training module and a model verification and analysis module; specifically, The text processing module is used to process the pre-recognized text and output the marked text and the pre-synthesized speech text; The speech synthesis module is used to synthesize the pre-synthesized speech text into an audio file; The synthetic corpus is used to classify and store the annotated text and the synthetic audio according to the corresponding relationship; The artificial corpus collection module is used to obtain sound data in real scenes, and the annotation can clarify the semantics of the audio to obtain artificially annotated corpus; The artificial corpus is used to classify and store artificial corpora; The model training module is used to retrieve artificial corpora and synthetic corpora from the synthetic corpus bureau and the artificial corpus according to business allocation according to the speech recognition scenario, randomly shuffle the corpus items to obtain a mixed corpus set, and divide the mixed corpus set into a training set, a verification set and a test set; use the training set to train the speech recognition model, use the verification set to verify the trained speech recognition model, and fine-tune the model parameters of the speech recognition model; The model verification and analysis module is used to use the speech recognition model fine-tuned by the model training module to recognize the test set speech one by one, compare the recognized text results with the annotated text of the test set, and calculate the word error rate and sentence error rate of the adjusted speech recognition model; adjust the model parameter adjustment strategy according to the word error rate and sentence error rate to further optimize the speech recognition model performance.

[0114] As an example of this embodiment, the pre-recognized text processed in the text processing module is professional terms and / or standardized expressions extracted from relevant standard documents, operation manuals and / or technical specifications of rail transit.

[0115] Furthermore, the text processing module determines the text length corresponding to the synthesized speech according to the set duration of the synthesized speech, and cuts the recognized text according to the length of the text and divides it into multiple segments; when the pre-recognized text is processed, the text is divided into multiple sentences when segmentation punctuation is encountered in the text. The text processing module deletes all characters except words, numbers and letters in the text; then, it replaces the special reading characters, the kilometer value reading, the speed limit value reading, the number reading and the train number letter reading in turn, retains the text content before and after the replacement, and uses the text before the replacement as the annotated text and the text after the replacement as the pre-synthesized speech text. The text processing module cleans the annotated text and the pre-synthesized speech text. Specifically, the text that does not contain Chinese characters and / or repeated text is deleted, and the cleaned annotated text and pre-synthesized speech text are output.

[0116] As another example of this embodiment, in the speech synthesis module, the synthesis rate is set according to the text length of the pre-synthesized speech text, the volume and pitch are randomly set, and an audio file is generated. In the speech synthesis module, the standard for setting the synthesis rate according to the text length of the pre-synthesized speech text is as follows: ; Where L represents the character length and S represents the synthetic speech speed.

[0117] After the speech synthesis module synthesizes the audio file, the quality of the synthesized audio is checked, and speech synthesis is performed again for audio files with low audio quality; specifically, speech recognition is performed on the audio files output by the speech synthesis module in sequence using a speech recognition tool. If the recognition result is empty, it indicates that the audio file quality is low, or the synthesis effect of the audio is checked by manual sampling, and speech synthesis is performed again for audio files with low audio quality.

[0118] As another example of this embodiment, the artificial corpus collection module collects live audio and / or uses an open source audio data set.

[0119] As another example of this embodiment, the model training module retrieves artificial and synthetic corpora from the synthetic corpus bureau and the artificial corpus according to the business allocation based on the speech recognition scenario, and the time ratio of the retrieved artificial and synthetic corpora is not less than 1:9. The model training module selects the Paraformer speech recognition model as the general base speech recognition model, determines the fine-tuning strategy according to the characteristics and requirements of the fine-tuning task, and closely monitors the loss and acc parameter changes of the speech recognition model during the fine-tuning process. By analyzing the changes in the loss and acc parameters, the fine-tuning strategy is adjusted in time. When the model training module fine-tunes the model parameters of the speech recognition model, the fine-tunable model parameters include any one or more combinations of batch_size, epoch and lr parameters.

[0120] Example 7 In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned speech recognition model fine-tuning processing method when executing the computer program.

[0121] In this embodiment, the processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0122] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the methods in the above method embodiments 1-5.

[0123] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The one or more units are stored in the memory, and when executed by the processor, perform the method in the above-mentioned embodiment 1, embodiment 2, embodiment 3, embodiment 4 or embodiment 5.

[0125] Example 8 As another preferred embodiment of the present invention, this embodiment discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the above-mentioned embodiment 1, embodiment 2, embodiment 3, embodiment 4 or embodiment 5 are implemented.

Claims

1. A speech recognition model fine-tuning method, characterized in that: The fine-tuning processing method includes the following steps: S1, processing the pre-recognized text, outputting the annotated text and the pre-synthesized speech text; S2, synthesizing the pre-synthesized speech text into an audio file; S3, classifying the annotated text and the synthesized audio according to the corresponding relationship and saving them into the synthesized corpus; S4. Acquire sound data in real scenarios, annotate them to clarify the semantics of the audio, and obtain manually annotated corpus; S5, classify and save the manually annotated corpus into the artificial corpus; S6. According to the speech recognition scenario, artificial corpus and synthetic corpus are retrieved from the synthetic corpus bureau and the artificial corpus according to business allocation, the corpus items are randomly shuffled to obtain a mixed corpus set, and the mixed corpus set is divided into a training set, a validation set and a test set; S7, training the speech recognition model using the training set, verifying the trained speech recognition model using the verification set, and fine-tuning the model parameters of the speech recognition model; S8. Use the fine-tuned speech recognition model to recognize the test set speech one by one, compare the recognized text results with the annotated text of the test set, and calculate the word error rate and sentence error rate of the adjusted speech recognition model; adjust the model parameter adjustment strategy according to the word error rate and sentence error rate to further optimize the speech recognition model performance.

2. A method for fine-tuning a speech recognition model as claimed in claim 1, characterized in that: In step S1, the pre-recognized text is professional terms and / or standardized expressions extracted from relevant standard documents, operation manuals and / or technical specifications of rail transit.

3. A method for fine-tuning a speech recognition model as claimed in claim 1, characterized in that: In step S1, the pre-recognized text is processed, including determining the text length corresponding to the synthesized speech according to the set synthesized speech duration, and cutting the recognized text into multiple segments according to the text length.

4. A method for fine-tuning a speech recognition model as claimed in claim 3, characterized in that: When processing the pre-recognized text, when encountering segmentation punctuation in the text, the text is divided into multiple sentences.

5. A method for fine-tuning a speech recognition model as claimed in claim 3 or 4, characterized in that: In step S1, the pre-recognized text is processed, which also includes deleting all characters except words, numbers and letters in the text; then replacing the special pronunciation characters, kilometer value reading, speed limit value reading, number reading and train number letter reading in turn, retaining the text content before and after the replacement, using the text before replacement as the annotated text, and the text after replacement as the pre-synthesized speech text.

6. A method for fine-tuning a speech recognition model according to any one of claims 1 to 4, characterized in that: In step S1, data cleaning is performed on the output annotated text and the pre-synthesized speech text. Specifically, texts that do not contain Chinese characters and / or repeated texts are deleted.

7. A method for fine-tuning a speech recognition model according to any one of claims 1 to 4, characterized in that: In step S2, the pre-synthesized speech text is synthesized into an audio file, specifically, the synthesis speed is set according to the text length of the pre-synthesized speech text, the volume and pitch are randomly set, and the audio file is generated using a speech synthesis tool.

8. A method for fine-tuning a speech recognition model as claimed in claim 7, characterized in that: The standard for setting the synthesis rate according to the text length of the pre-synthesized speech text is as follows: ; Where L represents the character length and S represents the synthetic speech speed.

9. A method for fine-tuning a speech recognition model according to any one of claims 1 to 4, characterized in that: Step S2 also includes a step of checking the synthesized audio file, specifically, checking the quality of the synthesized audio, and re-synthesizing the audio with low quality.

10. A method for fine-tuning a speech recognition model as claimed in claim 9, characterized in that: The audio files synthesized in step S2 are subjected to speech recognition in sequence. If the recognition result is empty, it indicates that the audio file has low quality. Alternatively, the synthesis effect of the audio is checked by manual sampling. For audio files with low audio quality, speech synthesis is performed again.

11. A method for fine-tuning a speech recognition model according to any one of claims 1 to 4, characterized in that: In step S4, obtaining sound data in a real scene includes collecting on-site audio and / or using an open source audio dataset.

12. A method for fine-tuning a speech recognition model according to any one of claims 1 to 4, characterized in that: In step S6, the time ratio of artificial corpus to synthetic corpus is not less than 1:

9.

13. A method for fine-tuning a speech recognition model according to any one of claims 1 to 4, characterized in that: In step S7, the Paraformer speech recognition model is selected as the general base speech recognition model, and the fine-tuning strategy is determined according to the characteristics and requirements of the fine-tuning task. During the fine-tuning process, the loss and acc parameter changes of the speech recognition model are closely monitored, and the fine-tuning strategy is adjusted in time by analyzing the changes in loss and acc parameters.

14. A method for fine-tuning a speech recognition model as claimed in claim 13, characterized in that: In step S7, when fine-tuning the model parameters of the speech recognition model, the fine-tunable model parameters include any one or more combinations of batch_size, epoch and lr parameters.

15. A speech recognition model fine-tuning processing device based on a speech recognition model fine-tuning processing method according to any one of claims 1 to 14, the fine-tuning processing device comprising a text processing module, a speech synthesis module, a synthetic corpus, an artificial corpus, an artificial corpus collection module, a model training module and a model verification and analysis module; specifically, The text processing module is used to process the pre-recognized text and output the marked text and the pre-synthesized speech text; The speech synthesis module is used to synthesize the pre-synthesized speech text into an audio file; The synthetic corpus is used to classify and store the annotated text and the synthetic audio according to the corresponding relationship; The artificial corpus collection module is used to obtain sound data in real scenes, and the annotation can clarify the semantics of the audio to obtain artificially annotated corpus; The artificial corpus is used to classify and store artificial corpora; The model training module is used to retrieve artificial corpora and synthetic corpora from the synthetic corpus bureau and the artificial corpus according to business allocation according to the speech recognition scenario, randomly shuffle the corpus items to obtain a mixed corpus set, and divide the mixed corpus set into a training set, a verification set and a test set; use the training set to train the speech recognition model, use the verification set to verify the trained speech recognition model, and fine-tune the model parameters of the speech recognition model; The model verification and analysis module is used to use the speech recognition model fine-tuned by the model training module to recognize the test set speech one by one, compare the recognized text results with the annotated text of the test set, and calculate the word error rate and sentence error rate of the adjusted speech recognition model; adjust the model parameter adjustment strategy according to the word error rate and sentence error rate to further optimize the speech recognition model performance.

16. A computer device, characterized in that: It includes a processor, an input device, an output device and a memory, which are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the speech recognition model fine-tuning processing method as described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the speech recognition model fine-tuning processing method as described in any one of claims 1 to 14.

Images