Speech Recognition Training Method, Device, Equipment and Medium Based on Industrial and Mining Fields

Abstract

An embodiment of the present application provides a voice recognition training method, device, equipment and medium based on the industrial and mining field, which relates to the technical field of voice recognition. The method includes: creating an industrial and mining voice library; collecting voice data and saving it to a voice database; pre-training an audio recognition model using the voice data in the voice database; training the audio recognition model according to the industrial and mining voice library by means of transfer learning; evaluating the prediction performance of the audio recognition model according to set indexes; being able to adapt to different dialects and professional terms, and can be flexibly applied in various industrial and mining environments, thereby improving the accuracy rate and enhancing the prediction result.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology. Specifically, it relates to a speech recognition training method, device, equipment, and medium based on the industrial and mining field. Background Art

[0002] With the continuous expansion of the informatization and intelligent systems in coal preparation plants, the complexity of human-computer interaction has been increasing continuously, which has correspondingly increased the learning and usage costs. In this situation, the application of the speech interaction system is of crucial significance for reducing the system usage threshold. With its natural and convenient interface, speech recognition technology has already become a significant advantage in human-computer interaction. Although the current progress of artificial intelligence technology has promoted the wide popularization of speech recognition in many industries, in specific fields such as coal preparation, due to the uniqueness of professional terms, general speech recognition models often fail to achieve a high accuracy rate. In addition, the problem of the speaker's dialect accent will also cause the recognition accuracy rate to decrease. These factors have all limited the effective application of speech recognition technology in the coal preparation field to a certain extent. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a speech recognition training method, device, equipment, and medium based on the industrial and mining field, so as to solve the problem that the speech recognition model in the current coal preparation field often fails to achieve a high accuracy rate.

[0004] In the first aspect, the embodiments of this application provide a speech recognition training method based on the industrial and mining field, including:

[0005] Create an industrial and mining speech library; where the industrial and mining speech library includes professional terms and dialect accents in a set field;

[0006] Collect speech data and save it to the speech database;

[0007] Use the speech data in the speech database to pre-train the audio recognition model;

[0008] Through the way of transfer learning, train the audio recognition model according to the working condition speech library;

[0009] Evaluate the prediction performance of the audio recognition model according to the set indicators.

[0010] In the above implementation process, create an industrial and mining speech library; collect speech data and save it to the speech database; use the speech data in the speech database to pre-train the audio recognition model; through the way of transfer learning, train the audio recognition model according to the working condition speech library; evaluate the prediction performance of the audio recognition model according to the set indicators; can adapt to different dialects and professional terms, and can be flexibly applied in a variety of industrial and mining environments, thereby improving the accuracy rate and enhancing the prediction results.

[0011] Further, the creation of the industrial and mining voice library includes:

[0012] Construct an industrial and mining voice library, where the industrial and mining voice library includes an index query text library, a system operation text library, a knowledge Q&A text library, a fault diagnosis text library, a safety regulation text library, an equipment maintenance text library, an environmental detection text library, and an emergency plan text library.

[0013] In the above implementation process, the model obtained by training with a working condition voice library containing different dialects and professional terms can be flexibly applied in various industrial and mining environments.

[0014] Further, the collection of voice data and saving it to the voice database includes:

[0015] Record the voice data of the user;

[0016] Automatically recognize the voice data to obtain a text transcription result;

[0017] Receive the user's feedback evaluation and dialect category of the transcription result, complete the collection process of the voice data, and save it to the voice database.

[0018] In the above implementation process, collecting and organizing the audio data for training the speech recognition model and its corresponding text information ensures the diversity and quality of the voice, providing a basis for subsequent voice training.

[0019] Further, the training of the audio recognition model according to the working condition voice library by means of transfer learning includes:

[0020] Fine-tune the pre-trained audio recognition model using the industrial and mining voice library;

[0021] Based on the voice data, obtain a preliminary prediction result without backpropagation. Sample and generate a new embedding vector based on the difference between the preliminary prediction result and the true label, and calculate the loss in different ways for model training optimization;

[0022] Transfer the knowledge of the large pre-trained model to the small model through knowledge distillation;

[0023] Perform streaming processing on the speech recognition model based on the Transformer self-attention mechanism to process voice data in real time;

[0024] Among them, during the training process, a variety of optimization techniques are used to improve the performance of the model. Among them, the optimization techniques include the Adam optimizer and learning rate decay.

[0025] In the above implementation process, the model is pre-trained using general speech data, and then, through transfer learning, the model is applied to a specific domain and further trained to adapt to the professional terms and dialect accents of that domain.

[0026] Furthermore, obtaining the preliminary prediction result based on the speech data without backpropagation includes:

[0027] Input the acoustic features of a set number of frames. The encoder uses the self-attention module to extract the key information in the acoustic features and generate corresponding hidden vectors;

[0028] The predictor predicts the number of words based on the CIF continuous integration and emission mechanism to obtain the predicted number of words;

[0029] Calculate the mean absolute error loss between the predicted number of words and the true number of words. The formula for the mean absolute error loss is:

[0030] ;

[0031] where N represents the true number of words, n represents the total number of samples, represents the loss function of the true number of words, represents the loss function of the predicted number of words;

[0032] Measure the error situation of word number prediction by calculating the average of the absolute values of the differences between the two, and obtain the preliminary prediction result;

[0033] Calculate the Hamming distance between the preliminary prediction result and the true result of the first path to determine the sampling quantity.

[0034] In the above implementation process, by obtaining the preliminary prediction result, the model training steps can be further carried out according to the difference between the preliminary prediction result and the true label.

[0035] Furthermore, sampling based on the difference between the preliminary prediction result and the true label to generate new embedding vectors, and calculating losses in different ways for model training optimization includes:

[0036] Based on the determined sampling quantity, perform sampling operations using the acoustic embedding vectors and the character embedding vectors of the labels to generate sampled embedding vectors;

[0037] Use the generated sampled embedding vectors to generate prediction results, and calculate the cross-entropy loss between the predicted probability distribution corresponding to the prediction results and the true labels, which represents the cumulative difference between the predicted probability and the true label probability in logarithmic form;

[0038] Construct positive and negative sample pairs by comparing the prediction results, and calculate the minimum word error rate loss based on the positive and negative samples.

[0039] In the above implementation process, predictions are made through different mechanisms and various loss functions are calculated, aiming to comprehensively utilize different strategies to optimize the model output and make it closer to the correct label.

[0040] Furthermore, based on the speech data, a preliminary prediction result is obtained without backpropagation. New embedding vectors are sampled and generated based on the difference between the preliminary prediction result and the true label, and losses are calculated in different ways for model training optimization, including:

[0041] The energy of the high-frequency part is enhanced through pre-emphasis, and a first-order digital filter is used:

[0042] y(n) = x(n) - ax(n - 1);

[0043] where x(n) is the original speech signal, a represents the filter coefficient, x(n - 1) represents the historical data of the input speech signal, y(n) is the pre-emphasized signal, and is a coefficient close to 1;

[0044] The speech signal is segmented into short frames;

[0045] A Hamming window function is used to add a window function to each frame;

[0046] The Fourier transform is performed on each windowed frame to convert the time-domain signal into a frequency-domain signal, obtaining the spectrum of each frame;

[0047] The obtained linear frequencies are converted to Mel frequencies, a Mel filter bank is constructed, the energy of each filter is calculated, and the FBanks Mel filter bank coefficient features of the speech signal are obtained;

[0048] A self-attention module is used to capture the association between each unit and other units in the sequence;

[0049] The FBanks features are a sequence represented by vectors. The degree of association between each vector and all other vectors is calculated through the self-attention module to generate a new feature representation;

[0050] The predictor predicts the number of target characters and extracts the acoustic feature vectors corresponding to the target characters. The output is a floating-point number between 0 and 1, and the output values are accumulated to predict the number of target characters;

[0051] The sampler receives the acoustic feature vectors output from the predictor and the vector representation corresponding to the target characters;

[0052] According to the semantic information and position information in the target character vector, relevant parts are selected from the acoustic feature vectors to integrate the acoustic information and semantic information. After the sampling operation of the generalized linear model GLM, the sampler transforms the acoustic feature vectors and the target character vectors into feature vectors containing rich semantic information;

[0053] For offline and streaming scenarios, the decoder is divided into two structures: an offline recognition decoder and a streaming recognition decoder;

[0054] The offline recognition decoder adopts a bidirectional SAN-M structure, with a local modeling module added on the basis of the standard Self-attention layer;

[0055] The streaming recognition decoder structure adopts a unidirectional SAN-M structure. It performs chunking operations on the speech features, and after sending them into the Decoder for modeling, it enters the predictor to predict the number of output tokens for each chunk respectively;

[0056] When the Decoder receives the number of tokens and the hidden layer representation, it predicts the output of each chunk based on the SCAMA streaming attention mechanism;

[0057] The model is optimized through a loss function. The loss function consists of three parts: cross-entropy, minimum word error rate loss MWER, and mean absolute error MAE loss function.

[0058] In the above implementation process, the backpropagation algorithm is used to update the parameters of the encoder, predictor, and each module involved in sampling and generating the final prediction result in the model, gradually improving the prediction ability of the model and making the final output prediction result closer to the true label.

[0059] Furthermore, evaluating the prediction performance of the model according to the set metrics includes:

[0060] For the recognition result of each test speech sample and the corresponding correct text, the edit distance is calculated using the dynamic programming algorithm to calculate the word error rate; among them, the edit distance includes the number of substitution, insertion, and deletion operations, which is the minimum number of steps required to convert the recognition result into the correct text through the edit distance;

[0061] A term dictionary is constructed, the number of sample term entries is calculated, and the number of detected terms is calculated. The ratio of the number of detected terms to the total number is calculated to obtain the term recall rate;

[0062] The average inference speed of the sample is calculated;

[0063] For the dialects in the set region, the word error rate of each dialect speech recognition system is calculated to obtain the dialect recognition accuracy;

[0064] The prediction performance of the model is evaluated according to the word error rate, term recall rate, inference speed, and dialect recognition accuracy.

[0065] In the above implementation process, during the entire model training process, considering the MAE loss, cross-entropy loss, and MWER loss comprehensively, the prediction ability of the model is improved, making the finally output prediction result closer to the true label.

[0066] In a second aspect, an embodiment of the present application provides a voice recognition training device based on the industrial and mining field, including:

[0067] A voice library creation module for creating an industrial and mining voice library; wherein, the industrial and mining voice library includes professional terms and dialect accents in a set field;

[0068] A voice acquisition module for acquiring voice data and saving it to a voice database;

[0069] A pre-training module for pre-training an audio recognition model using the voice data in the voice database;

[0070] A model training module for training the audio recognition model according to the industrial and mining voice library by means of transfer learning;

[0071] A performance prediction module for evaluating the prediction performance of the audio recognition model according to set metrics.

[0072] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0073] A processor, a memory, and a bus, the processor is connected to the memory through the bus, and the memory stores computer-readable instructions, which, when executed by the processor, are used to implement the voice recognition training method based on the industrial and mining field as described above.

[0074] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a server, it implements the voice recognition training method based on the industrial and mining field as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0076] Figure 1 It is a schematic flowchart of a voice recognition training method based on the industrial and mining field provided by an embodiment of the present application;

[0077] Figure 2Schematic diagram of the training process of a voice recognition training method based on the industrial and mining field provided by an embodiment of the present application;

[0078] Figure 3 Schematic diagram of the structure of a voice recognition training device based on the industrial and mining field provided by an embodiment of the present application;

[0079] Figure 4 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0080] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0081] It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first", "second", etc. are only used for differential description and cannot be understood as indicating or implying relative importance.

[0082] Please refer to Figure 1 and Figure 2 , Figure 1 Schematic diagram of the process of a voice recognition training method based on the industrial and mining field provided by an embodiment of the present application. The voice recognition training method based on the industrial and mining field includes:

[0083] 100. Create an industrial and mining voice library; wherein, the industrial and mining voice library includes professional terms and dialect accents in a set field.

[0084] It can be understood that the set field can be set according to specific requirements. In the embodiment of the present application, the industrial and mining field is taken as an example.

[0085] Specifically, construct an industrial and mining voice library, wherein the industrial and mining voice library includes an index query text library, a system operation text library, a knowledge Q&A text library, a fault diagnosis text library, a safety regulation text library, an equipment maintenance text library, an environmental detection text library, and an emergency plan text library; thereby, training the model through the working condition voice library containing different dialects and professional terms, so that the obtained model can be flexibly applied in various industrial and mining environments.

[0086] It can be understood that in the industrial and mining environment, professional terms often appear in conversations, such as "gangue", "slime", "flotation", "heavy medium washing", and "ash content", etc.

[0087] Thus, the industrial and mining voice library in the embodiment of the present application is divided into eight categories, and each category covers 100 carefully selected text entries to ensure comprehensive coverage of the specific needs of the industrial and mining industry.

[0088] Among them, index query: includes query statements related to key performance indicators in industrial and mining production processes, such as "What is the coal output today?" or "The current gas concentration standard". Among them, system operation: involves instructions for operating industrial and mining equipment or software systems, such as "Start the conveyor belt" or "Adjust the speed of the crusher". Among them, knowledge Q&A: provides Q&A pairs about industrial and mining industry knowledge, such as "What is the principle of heavy medium washing?" or "What are the main components of gangue?". Among them, fault diagnosis: includes the diagnosis process when a fault occurs, such as "What could be the reasons for the conveyor belt to stop running?" or "How to handle abnormal noise of the crusher?". Among them, safety regulations: involve safety operation regulations and guidelines for industrial and mining, such as "Safety precautions for using electrical equipment in the mine" or "Action steps during emergency evacuation". Among them, equipment maintenance: includes guidance on daily maintenance and upkeep of equipment, such as "Regularly check the fastening bolts of the mine car track" or "Clean and lubricate the pump station". Among them, environmental monitoring: includes statements related to monitoring the industrial and mining environment, such as "Check whether the dust concentration exceeds the standard" or "Measure the temperature and humidity in the mine". Among them, emergency plan: involves emergency response measures in case of emergencies, such as "Emergency procedures in case of gas leakage" or "Emergency evacuation route in case of mine water inflow".

[0089] Above, through such classification and quantity setting, the industrial and mining voice library can provide rich and scenario-based voices for the speech recognition system, thereby improving the recognition accuracy and practicality of the system in specific fields.

[0090] 200. Collect voice data and save it to the voice database.

[0091] 210. Record the user's voice data.

[0092] 220. Automatically recognize the voice data to obtain the text transcription result.

[0093] 230. Receive the user's feedback evaluation and dialect category of the transcription result, complete the collection process of the voice data, and save it to the voice database.

[0094] Exemplarily, the user's voice data can be collected through the mobile phone voice collection system. The user can easily record audio through the mobile phone microphone; the user can dynamically manage the voice library according to the preset text and record their own voice samples.

[0095] After the user completes each voice recording, the system will quickly provide the transcription result; the user can then evaluate the recognition effect as "good", "medium", or "bad" according to the recognition accuracy; after the evaluation, the user only needs to select the corresponding dialect category to complete the collection process of the voice.

[0096] As described above, high-quality speech label data is cleaned according to the evaluation results, and the audio data and its corresponding text information for training the speech recognition model are collected and sorted, ensuring the diversity and quality of the speech and providing a basis for subsequent speech training.

[0097] 300. Use the speech data in the speech database to pre-train the audio recognition model.

[0098] Optionally, use a large amount of diverse speech data to pre-train the Transformer-based audio recognition model. This data includes multiple languages, dialects, and different users to ensure that the model can learn general speech features.

[0099] 400. Train the audio recognition model according to the working condition speech library through transfer learning.

[0100] 410. Fine-tune the pre-trained audio recognition model using the industrial and mining speech library.

[0101] Optionally, apply the pre-trained audio recognition model to data in a specific field. At this stage, use the speech library of professional terms and relevant dialects in the industrial and mining industry to fine-tune the audio recognition model so that it can better recognize the speech features in these specific fields.

[0102] 420. Based on the speech data, obtain the preliminary prediction result without backpropagation. Sample and generate new embedding vectors based on the difference between the preliminary prediction result and the true label, and calculate the loss in different ways for model training optimization.

[0103] 430. Transfer the knowledge of the large pre-trained model to the small model through knowledge distillation.

[0104] Exemplarily, use a large amount of training data and computing resources to train a large and high-performance teacher model. The teacher model has a complex structure and a large number of parameters, and can capture the subtle features and patterns in the data. Extracting useful knowledge from the teacher model can be achieved in various ways, such as extracting the output (soft labels), feature representations of the intermediate layer, or attention weights of the teacher model. In addition, use this knowledge to train a smaller student model. During the training process, the student model not only has to learn the direct mapping from the input data to the output, but also has to imitate the output or feature representation of the teacher model. This is usually achieved by minimizing the difference between the student model and the teacher model (such as cross-entropy loss, KL divergence, etc.).

[0105] To transfer the knowledge of the teacher model to the student model, a distillation loss function is defined. This function measures the difference between the student model and the teacher model and encourages the student model to imitate the output or behavior of the teacher model. Thus, model compression and acceleration are achieved by transferring the knowledge of the teacher model to the student model in an easily learnable form.

[0106] 440. Perform streaming processing on the speech recognition model based on the Transformer self-attention mechanism to process speech data in real time.

[0107] Exemplarily, the original Transformer self-attention mechanism needs to calculate the attention scores between all elements in the entire input sequence, which is not feasible in streaming processing. Therefore, a streaming self-attention mechanism is designed to enable it to calculate the attention scores step by step, relying only on the processed input data.

[0108] In streaming processing, the model needs to gradually receive and process new input data. Through an incremental encoding method, the model can update the current encoding state based on the previous encoding results and the new input data.

[0109] To balance real-time performance and accuracy, the input data is divided into multiple blocks (or called windows) and these blocks are processed one by one; after each block is processed, the window is moved and the next block is continued to be processed. Thus, the model is allowed to capture long-range dependencies across blocks while maintaining real-time performance.

[0110] Among them, during the training process, multiple optimization techniques are adopted to improve the performance of the model. Among them, the optimization techniques include the Adam optimizer and learning rate decay.

[0111] Based on the above embodiments, the embodiments of the present application can be further specified as: The obtaining of the preliminary prediction result according to the speech data without performing backpropagation includes:

[0112] Input the acoustic features of the set number of frames, extract the key information in the acoustic features through the encoder using the self-attention module, and generate the corresponding hidden vector; use the predictor to predict the number of words based on the CIF continuous integration and emission mechanism to obtain the predicted number of words; calculate the mean absolute error loss between the predicted number of words and the true number of words. The formula for the mean absolute error loss is:

[0113] ;

[0114] where N represents the true number of words, n represents the total number of samples, represents the loss function of the true number of words, represents the loss function of the predicted number of words.

[0115] The error situation of word count prediction is measured by calculating the average value of the absolute value of the difference between the two, and a preliminary prediction result is obtained; the Hamming distance between the preliminary prediction result of the first path and the true result is calculated to determine the sampling quantity. It can be understood that the Hamming distance measures the number of different characters at the corresponding positions of two equal-length strings. The sampling quantity is determined through this distance. For example, if the Hamming distance is large, it may mean that the prediction result is quite different from the true result, and accordingly, a relatively large sampling quantity can be set according to certain rules, and vice versa for a relatively small sampling quantity.

[0116] Based on the above embodiments, the embodiments of the present application can be further specified as: sampling to generate new embedding vectors based on the difference between the preliminary prediction result and the true label, and calculating losses in different ways for model training optimization, including:

[0117] Based on the determined sampling quantity, sampling operations are performed using the acoustic embedding vectors and the character embedding vectors of the labels to generate sampled embedding vectors; the generated sampled embedding vectors are used to generate prediction results, and the cross-entropy loss between the predicted probability distribution corresponding to the prediction result and the true label is calculated to represent the cumulative difference between the predicted probability and the true label probability in logarithmic form; positive and negative sample pairs are constructed by comparing the prediction results, and the minimum word error rate loss MWER is calculated based on the positive and negative samples.

[0118] It can be understood that the cross-entropy loss is often used to measure the difference between the predicted probability distribution and the true probability distribution (which can be regarded as the true label represented by one-hot encoding) in a classification task, prompting the model to output a probability distribution that is more in line with the real situation.

[0119] It can be understood that the MWER loss function mainly focuses on the word error rate between the predicted sequence and the true sequence, measures the difference between the two by considering editing operations such as substitution, insertion, and deletion, and guides the model to reduce the word error rate in sequence tasks such as text generation, and optimize the output result to make it closer to the true label sequence.

[0120] As described above, during the model training process, the MAE loss, CE loss, and MWER loss are comprehensively considered, and the backpropagation algorithm is used to update the parameters of the encoder, predictor, and each module involved in sampling and generating the final prediction result in the model, gradually improving the prediction ability of the model and making the finally output prediction result closer to the true label.

[0121] Based on the above embodiments, the embodiments of the present application can be further specified as: obtaining a preliminary prediction result according to the speech data without performing backpropagation, sampling to generate new embedding vectors based on the difference between the preliminary prediction result and the true label, and calculating losses in different ways for model training optimization, including:

[0122] Feature Extractor: The energy of the high-frequency part is enhanced through pre-emphasis. It should be noted that the energy of the high-frequency part of the speech signal is usually low, and the energy of the high-frequency part can be enhanced through pre-emphasis. Specifically, a first-order digital filter is used:

[0123] y(n) = x(n) - ax(n - 1);

[0124] where x(n) is the original speech signal, a represents the filter coefficient, x(n - 1) represents the historical data of the input speech signal, and y(n) is the pre-emphasized signal, which is a coefficient close to 1.

[0125] Segment the speech signal into short frames: The speech signal is segmented into short frames because the speech signal is a non-stationary signal, but can be approximately regarded as a stationary signal in a short period. Exemplarily, the frame length is 20 milliseconds, and the frame shift (the interval between two adjacent frames) is 10 milliseconds.

[0126] Apply a Hamming window function to each frame: A window function is applied to each frame to reduce spectral leakage and make the spectrum of each frame smoother.

[0127] Perform Fourier transform on each windowed frame to convert the time-domain signal into a frequency-domain signal, obtaining the spectrum of each frame; convert the obtained linear frequency to Mel frequency, construct a Mel filter bank, calculate the energy of each filter, and obtain the FBanks Mel filter bank coefficient features of the speech signal; use a self-attention module to capture the association between each unit and other units in the sequence, that is, the association with other units except this unit; Optionally, 20 Mel filters.

[0128] Encoder: Composed of a self-attention module, the self-attention module highlights the most important semantic parts in the sequence by capturing the association between each unit and other units. Among them, the FBanks feature is a sequence X = [x1, x2,..., x_n] represented by vectors. The association degree between each vector and all other vectors (that is, other vectors except this vector) is calculated through the self-attention module to generate a new feature representation Z = [z1, z2,..., z_n].

[0129] Predictor: It consists of a two-layer deep neural network and is used to predict the number of target words and extract the acoustic vectors corresponding to the target words. By predicting the number of target words and extracting the acoustic feature vectors corresponding to the target words, the output is a floating-point number between 0 and 1, and the output values are accumulated to predict the number of target words; the acoustic vectors are obtained through the CIF (Continuous integrate and fire, a mechanism for sequence conversion tasks such as speech recognition) module. CIF integrates the sequentially arriving acoustic information one by one. When the integrated information reaches the recognition threshold, the integrated information is emitted for subsequent recognition.

[0130] Sampler: The sampler receives the acoustic feature vectors output from the predictor and the vector representations corresponding to the target words, and uses the Glancing Language Model (GLM) for sampling operations. According to the semantic information and position information in the target word vectors, relevant parts are selected from the acoustic feature vectors to integrate the acoustic information and semantic information. After the sampling operation of the generalized linear model GLM, the sampler transforms the acoustic feature vectors and the target word vectors into feature vectors containing rich semantic information. Among them, this newly generated feature vector not only contains the acoustic features of the speech but also incorporates the semantic information of the target words, thus being able to better express the correspondence between speech and text, providing more valuable input for the subsequent decoder module, enhancing the model's ability to model the context, and helping to improve the accuracy and performance of speech recognition.

[0131] Decoder: For offline and streaming scenarios, the decoder is divided into two structures: an offline recognition decoder and a streaming recognition decoder; the offline recognition decoder adopts a bidirectional SAN-M structure. The bidirectional structure can better capture the context information in the text, thus generating text that conforms to language logic more accurately. The SAN-M structure adds a local modeling module Memory Block on the basis of the standard Self-attention layer, enhancing the local modeling ability of Self-attention, enabling it to better handle the local features and long-term dependencies in the speech signal, and thus improving the accuracy of speech recognition.

[0132] Streaming recognition decoder structure: It is implemented by adopting a unidirectional SAN-M and combining a SCAMA-based streaming attention mechanism. Specifically, first, the speech features are chunked and fed into the decoder for modeling, and then enter the predictor to predict the number of output tokens for each chunk respectively; after receiving the number of tokens and the hidden layer representation, the decoder predicts the output of each chunk based on the SCAMA (Streaming Chunk-Aware Multihead Attention) streaming attention mechanism; the combination of the unidirectional SAN-M structure and the SCAMA streaming attention mechanism can not only meet the real-time requirements of streaming recognition but also effectively utilize the information in the speech signal to improve the recognition accuracy.

[0133] It can be understood that for bidirectional information processing: In the SAN-M network structure, if a bidirectional attention mechanism or a bidirectional recurrent neural network (such as bidirectional LSTM) is adopted, then the network can consider both the forward and backward information of the input sequence, thereby enhancing the model's ability to capture context information. This bidirectional information processing mechanism usually achieves better performance in sequence processing tasks such as speech recognition. For unidirectional information processing: On the contrary, if the SAN-M network structure only adopts a unidirectional attention mechanism or a unidirectional recurrent neural network, then the network will only be able to consider the forward information of the input sequence. Although this unidirectional information processing mechanism is relatively simple, it can still achieve good performance in some cases, especially when the context information of the input sequence is not particularly important.

[0134] The model is optimized through a loss function. Specifically, the loss function consists of three parts: cross-entropy, minimum word error rate loss MWER, and mean absolute error MAE loss function. The MAE loss is used to train the accuracy of the predictor.

[0135] 500. Evaluate the prediction performance of the audio recognition model according to the set metrics.

[0136] 510. For the recognition result of each test speech sample and the corresponding correct text, use the dynamic programming algorithm to calculate the edit distance to calculate the word error rate; among them, the edit distance includes the number of substitution, insertion, and deletion operations, which is the minimum number of steps required to convert the recognition result into the correct text through the edit distance.

[0137] 520. Construct a term dictionary, calculate the number of sample term entries, and check out the number. Calculate the ratio of the checked-out number to the total number to obtain the term recall rate.

[0138] Exemplarily, collect relevant terms from professional literature, databases, domain experts, or other reliable sources, organize the collected terms to ensure that each term is unique and accurate, and store the organized terms in an easily accessible data structure such as a list, dictionary, or database; preprocess the sample text, such as word segmentation and stop word removal, extract all the lemmas from the preprocessed sample, and count the number of all lemmas in the sample; match the preprocessed sample lemmas with the term dictionary to find the term lemmas that appear in the sample, and count the number of the matched term lemmas; calculate the recall rate using the ratio of the number of detected terms to the total number of terms in the term dictionary.

[0139] 530. Calculate the average inference speed of the sample.

[0140] Exemplarily, sum up the inference times of all input samples; divide the sum of the inference times by the number of samples to obtain the average inference time; use the above formula, divide the number of samples for inference by the average inference time to obtain the average inference speed.

[0141] 540. For the dialects in the set region, calculate the word error rate of each dialect speech recognition system to obtain the dialect recognition accuracy.

[0142] Exemplarily, train a dialect speech recognition model, recognize the collected dialect speech data, count the number of replaced, deleted, and inserted characters in the recognition results, divide the sum of the characters of the three (replacement, deletion, and insertion) by the total number of characters to obtain the word error rate. The word error rate is an important indicator for evaluating the performance of the dialect speech recognition system. Optionally, the recognition accuracy can be calculated by the following formula: Recognition accuracy = 1 - word error rate.

[0143] 550. Evaluate the prediction performance of the model based on the word error rate, term recall rate, inference speed, and dialect recognition accuracy.

[0144] Exemplarily, the evaluation weights of the word error rate, term recall rate, inference speed, and dialect recognition accuracy can be set according to requirements, and the performance of the model can be obtained based on the evaluation weights.

[0145] As described above, the embodiments of the present application create an industrial and mining voice library; collect voice data and save it to the voice database; pre-train the audio recognition model using the voice data in the voice database; train the audio recognition model according to the industrial and mining voice library by means of transfer learning; evaluate the prediction performance of the audio recognition model according to the set indicators; can adapt to different dialects and professional terms, and can be flexibly applied in various industrial and mining environments, thereby improving the accuracy and enhancing the prediction results.

[0146] As described above, the embodiments of the present application focus on creating a voice database specifically for the industrial and mining industries, including industry-specific terms and expressions, to improve the accuracy and applicability of the voice recognition system in this field, including the construction strategy of the voice database, term collection, voice screening and classification, and voice annotation process; ensuring the quality and coverage of the voice database, and how to dynamically update the voice database to adapt to industry changes.

[0147] The embodiments of the present application can support the collection, annotation, evaluation, and feedback of dialects to improve the dialect recognition ability of the voice recognition system. The user interface design, dialect annotation process, evaluation mechanism, and feedback loop of the system; using user feedback to optimize the voice recognition model to ensure the quality and usability of the collected data.

[0148] The embodiments of the present application use transfer learning technology to train the voice recognition model to adapt to voice data in specific fields, especially in the industrial and mining fields, including the selection of pre-trained models, domain adaptation strategies, model fine-tuning techniques, etc.; reducing the distribution difference between the source domain and the target domain to improve the generalization ability of the model in the target domain.

[0149] In addition, the embodiments of the present application have the following advantages: it can adapt to different dialects and professional terms, which means it can be flexibly applied in various industrial and mining environments, not only limited to the coal preparation industry, but also extended to other fields that require the recognition of specific dialects or professional terms. By simplifying the interaction process, the system reduces the learning curve of users, enabling users without a technical background to easily get started, improving the popularity and user satisfaction of the system. The architecture based on the Transformer model has good scalability and can continuously optimize and expand the functions of the system with the progress of technology and the accumulation of data. It has the ability of self-learning and self-optimization and can continuously improve according to user feedback and newly collected data to adapt to changing language usage habits and technological developments.

[0150] The above steps are not strictly executed in the order described by the numbers and should be understood as an overall solution.

[0151] In a second aspect, based on the above embodiments, Figure 3 FIG. is a schematic structural diagram of a voice recognition training device based on the industrial and mining fields provided by the embodiments of the present application. Referring to Figure 3 , the voice recognition training device based on the industrial and mining fields provided in this embodiment specifically includes: a voice database creation module 301, a voice collection module 302, a pre-training module 303, a model training module 304, and a performance prediction module 305.

[0152] Among them, the voice library creation module 301 is used to create an industrial and mining voice library; among them, the industrial and mining voice library includes professional terms and dialect accents in the set field; the voice acquisition module 302 acquires voice data and saves it to the voice database; the pre-training module 303 is used to pre-train the audio recognition model using the voice data in the voice database; the model training module 304 is used to train the audio recognition model according to the industrial and mining voice library by means of transfer learning; the performance prediction module 305 is used to evaluate the prediction performance of the audio recognition model according to the set metrics.

[0153] As described above, the embodiments of the present application create an industrial and mining voice library; acquire voice data and save it to the voice database; pre-train the audio recognition model using the voice data in the voice database; train the audio recognition model according to the industrial and mining voice library by means of transfer learning; evaluate the prediction performance of the audio recognition model according to the set metrics; can adapt to different dialects and professional terms, and can be flexibly applied in various industrial and mining environments, thereby improving the accuracy and enhancing the prediction results.

[0154] The voice recognition training device based on the industrial and mining field provided by the embodiments of the present application can be used to execute the voice recognition training method based on the industrial and mining field provided by the above embodiments, and has the corresponding functions and beneficial effects.

[0155] In a third aspect, the embodiments of the present application further provide an electronic device, and the electronic device can integrate the voice recognition training device based on the industrial and mining field provided by the embodiments of the present application. Figure 4 It is a schematic structural diagram of an electronic device provided by the embodiments of the present application. Refer to Figure 4 , the electronic device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the voice recognition training method based on the industrial and mining field provided by the above embodiments. Among them, the input device 43, the output device 44, the memory 42, and the processor 41 can be connected through a bus or other means, Figure 4 Taking connection through a bus as an example.

[0156] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions, and modules stored in the memory 42, that is, implements the above-mentioned voice recognition training method based on the industrial and mining field.

[0157] The above-provided electronic device can be used to execute the voice recognition training method based on the industrial and mining field provided by the above embodiments, and has the corresponding functions and beneficial effects.

[0158] Fourthly, an embodiment of the present application further provides a computer-readable storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned voice recognition training method based on the industrial and mining field, and can achieve the same beneficial effects.

[0159] Certainly, the computer-executable instructions included in the storage medium provided by the embodiment of the present application are not limited to the above-mentioned voice recognition training method based on the industrial and mining field, and can also execute related operations in the voice recognition training method based on the industrial and mining field provided by any embodiment of the present application.

[0160] Fifthly, an embodiment of the present application further provides a computer program product. The methods described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM (Open Application Model), or other programmable devices.

[0161] The computer program or instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc; or it can be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0162] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0163] In addition, in each embodiment of the present application, the various functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0164] If the described function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0165] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application. It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0166] As described above, the above are only specific implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and all should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0167] It should be noted that in this text, relative terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

Claims

1. A voice recognition training method based on the industrial and mining field, characterized in that, The method includes: Creating an industrial and mining voice library; wherein, the industrial and mining voice library includes professional terms and dialect accents in a set field; Collecting voice data and saving it to a voice database; Pre-training an audio recognition model using the voice data in the voice database; Training the audio recognition model according to the working condition voice library by means of transfer learning; Evaluating the prediction performance of the audio recognition model according to set metrics; The training of the audio recognition model according to the working condition voice library by means of transfer learning includes: Fine-tuning the pre-trained audio recognition model using the industrial and mining voice library; Based on the voice data, obtaining a preliminary prediction result without backpropagation, sampling and generating a new embedding vector based on the difference between the preliminary prediction result and the true label, and calculating losses in different ways for model training optimization; Transferring the knowledge of a large pre-trained model to a small model through knowledge distillation; Performing streaming processing on the speech recognition model based on the Transformer self-attention mechanism to process voice data in real time; Wherein, during the training process, a variety of optimization techniques are adopted to improve the performance of the model, and the optimization techniques include the Adam optimizer and learning rate decay.

2. The voice recognition training method based on the industrial and mining field according to claim 1, wherein, The creating of the industrial and mining voice library includes: Constructing an industrial and mining voice library, wherein the industrial and mining voice library includes an index query text library, a system operation text library, a knowledge Q&A text library, a fault diagnosis text library, a safety regulation text library, an equipment maintenance text library, an environment detection text library, and an emergency plan text library.

3. The voice recognition training method based on the industrial and mining field according to claim 1, wherein The collecting of voice data and saving it to a voice database includes: Recording the voice data of the user; Automatically recognizing the voice data to obtain a text transcription result; Receiving the feedback evaluation and dialect category of the user on the transcription result, completing the collection process of the voice data and saving it to the voice database.

4. The voice recognition training method based on the industrial and mining field according to claim 1, wherein The obtaining of a preliminary prediction result based on the voice data without backpropagation includes: Inputting set-frame acoustic features, using a self-attention module in the encoder to extract key information in the acoustic features, and generating corresponding hidden vectors; Using a predictor to predict the number of words based on the CIF continuous integration and emission mechanism to obtain the predicted number of words; Calculating the mean absolute error loss between the predicted number of words and the true number of words, and the formula for the mean absolute error loss is: ； Where N represents the true number of words and n represents the total number of samples, is the loss function representing the true number of words, is the loss function representing the predicted number of words; Measuring the error situation of word number prediction by calculating the average of the absolute values of the differences between the two to obtain a preliminary prediction result; Calculating the Hamming distance between the preliminary prediction result of the first path and the true result to determine the sampling quantity.

5. The voice recognition training method based on the industrial and mining field according to claim 1, wherein The sampling and generating of a new embedding vector based on the difference between the preliminary prediction result and the true label, and calculating losses in different ways for model training optimization includes: Based on the determined sampling quantity, performing a sampling operation using the acoustic embedding vector and the character embedding vector of the label to generate a sampled embedding vector; Using the generated sampled embedding vector to generate a prediction result, calculating the cross-entropy loss between the predicted probability distribution corresponding to the prediction result and the true label, and representing the difference accumulation between the predicted probability and the true label probability in logarithmic form; Constructing positive and negative sample pairs by comparing the prediction results, and calculating the minimum word error rate loss based on the positive and negative samples.

6. The voice recognition training method based on the industrial and mining field according to claim 1, wherein Based on the speech data, obtain a preliminary prediction result without performing backpropagation. Sample and generate new embedding vectors based on the difference between the preliminary prediction result and the true label, and calculate the loss in different ways for model training optimization, including: Boost the energy of the high-frequency part through pre-emphasis, using a first-order digital filter: y(n) = x(n) - ax(n-1); where x(n) is the original speech signal, a represents the filter coefficient, x(n-1) represents the historical data of the input speech signal, and y(n) is the pre-emphasized signal, which is a coefficient close to 1; Segment the speech signal into short frames; Add a window function to each frame using a Hamming window function; Perform a Fourier transform on each windowed frame to convert the time-domain signal into a frequency-domain signal and obtain the spectrum of each frame; Convert the obtained linear frequency to Mel frequency, construct a Mel filter bank, calculate the energy of each filter, and obtain the FBanks Mel filter bank coefficient features of the speech signal; Adopt a self-attention module to capture the association between each unit and other units in the sequence; The FBanks features are a sequence represented by vectors. Calculate the degree of association between each vector and all other vectors through the self-attention module to generate a new feature representation; Predict the number of target characters and extract the acoustic feature vectors corresponding to the target characters through a predictor. The output is a floating-point number between 0 and 1, and the output values are accumulated to predict the number of target characters; Receive the acoustic feature vectors output from the predictor and the vector representation corresponding to the target characters through a sampler; According to the semantic information and position information in the target character vector, select the relevant part from the acoustic feature vectors, integrate the acoustic information and semantic information, and after the sampling operation of the generalized linear model GLM, the sampler transforms the acoustic feature vectors and the target character vectors into feature vectors containing rich semantic information; For offline and streaming, the decoder is divided into two structures: an offline recognition decoder and a streaming recognition decoder; The offline recognition decoder adopts a bidirectional SAN-M structure, adding a local modeling module on the basis of the standard Self-attention layer; The streaming recognition decoder structure adopts a unidirectional SAN-M structure. Perform a chunk operation on the speech features, send them into the Decoderr for modeling, and then enter the predictor to predict the output token number of each chunk respectively; When the Decoder receives the token number and the hidden layer representation, predict the output of each chunk based on the SCAMA streaming attention mechanism; Optimize the model through a loss function, where the loss function consists of three parts: cross-entropy, minimum word error rate loss MWER, and mean absolute error MAE loss function.

7. The voice recognition training method based on the industrial and mining field according to claim 1, characterized in that, Evaluate the prediction performance of the model according to the set metrics, including: For the recognition result of each test speech sample and the corresponding correct text, use the dynamic programming algorithm to calculate the edit distance to calculate the word error rate; among them, the edit distance includes the number of substitution, insertion, and deletion operations, which is the minimum number of steps required to convert the recognition result into the correct text through the edit distance; Construct a term dictionary, calculate the number of sample term entries and the number of detected terms, calculate the ratio of the number of detected terms to the total number, and obtain the term recall rate; Calculate the average inference speed of the sample; For the dialects in the set area, calculate the character error rate of each dialect speech recognition system to obtain the dialect recognition accuracy; Evaluate the prediction performance of the model according to the character error rate, term recall rate, inference speed, and dialect recognition accuracy.

8. A voice recognition training device based on the industrial and mining field, characterized in that, Including: A voice database creation module for creating an industrial and mining voice database; wherein, the industrial and mining voice database includes professional terms and dialect accents in the set field; A voice acquisition module that acquires voice data and saves it to the voice database; A pre-training module for pre-training the audio recognition model using the voice data in the voice database; A model training module for training the audio recognition model according to the industrial and mining voice database by means of transfer learning; A performance prediction module for evaluating the prediction performance of the audio recognition model according to the set metrics; The training of the audio recognition model according to the industrial and mining voice database by means of transfer learning includes: Fine-tuning the pre-trained audio recognition model using the industrial and mining voice database; Based on the voice data, obtain a preliminary prediction result without backpropagation, sample and generate a new embedding vector based on the difference between the preliminary prediction result and the true label, and calculate the loss in different ways for model training optimization; Transfer the knowledge of the large pre-trained model to the small model through knowledge distillation; Perform streaming processing on the speech recognition model based on the Transformer self-attention mechanism to process speech data in real time; Among them, during the training process, multiple optimization techniques are adopted to improve the performance of the model. Among them, the optimization techniques include the Adam optimizer and learning rate decay.

9. An electronic device, characterized in that, Including: A processor, a memory, and a bus. The processor is connected to the memory through the bus. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the speech recognition training method based on the industrial and mining field according to any one of claims 1-7.

Images