Speech recognition training method, device and equipment based on industrial and mining field and medium

Abstract

The embodiment of the invention provides a voice recognition training method, device and equipment based on the industrial and mining field and a medium, and relates to the technical field of voice recognition. Acquiring voice data and storing the voice data in a voice database; pre-training the audio recognition model by using the voice data of the voice database; training the audio recognition model according to the working condition voice library in a transfer learning mode; evaluating the prediction performance of the audio recognition model according to a set index; the method can adapt to different dialects and terminologies, and can be flexibly applied to various industrial and mining environments, so that the accuracy is improved, and the prediction result is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and more specifically, to a speech recognition training method, device, equipment and medium based on the industrial and mining fields. Background Art

[0002] With the continuous expansion of information and intelligent systems in coal preparation plants, the complexity of human-computer interaction continues to rise, which increases the cost of learning and use accordingly. In this case, the application of voice interaction systems is extremely critical to lowering the threshold for using the system. Speech recognition technology has become a significant advantage in human-computer interaction with its natural and convenient interface. Although the current progress in artificial intelligence technology has promoted the widespread popularization of speech recognition in many industries, in specific fields such as coal preparation, due to the uniqueness of professional terms, general speech recognition models often find it difficult to achieve a high accuracy rate. In addition, the speaker's dialect accent will also reduce the recognition accuracy. These factors have, to a certain extent, limited the effective application of speech recognition technology in the field of coal preparation. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a speech recognition training method, device, equipment and medium based on the industrial and mining fields, so as to solve the problem that the current speech recognition models in the coal preparation field often find it difficult to achieve a high accuracy rate.

[0004] In a first aspect, the present application embodiment provides a speech recognition training method based on the industrial and mining fields, including: Creating an industrial and mining speech database; wherein the industrial and mining speech database includes professional terms and dialect accents in a set field; Collect voice data and save it to the voice database; Pre-train the audio recognition model using speech data from a speech database; Through transfer learning, the audio recognition model is trained based on the working condition voice library; Evaluate the prediction performance of the audio recognition model based on the set metrics.

[0005] In the above implementation process, an industrial and mining voice database is created; voice data is collected and saved in the voice database; the audio recognition model is pre-trained using the voice data in the voice database; the audio recognition model is trained based on the working voice database through transfer learning; the prediction performance of the audio recognition model is evaluated based on set indicators; it can adapt to different dialects and professional terms and can be flexibly applied in a variety of industrial and mining environments, thereby improving accuracy and improving prediction results.

[0006] Furthermore, the creation of the industrial and mining speech library includes: Construct an industrial and mining voice database, which includes an indicator query text library, a system operation text library, a knowledge question and answer text library, a fault diagnosis text library, a safety regulations text library, an equipment maintenance text library, an environmental detection text library and an emergency plan text library.

[0007] In the above implementation process, the model is trained by using a working speech library containing different dialects and professional terms, so that the obtained model can be flexibly applied in a variety of industrial and mining environments.

[0008] Furthermore, the collecting of voice data and saving it to a voice database includes: Recording user’s voice data; Automatically identify voice data and obtain text transcription results; Receive user feedback on the transcription results and dialect category, complete the voice data collection process and save it to the voice database.

[0009] In the above implementation process, the audio data and its corresponding text information used to train the speech recognition model are collected and organized, which ensures the diversity and quality of the speech and provides a basis for subsequent speech training.

[0010] Furthermore, the audio recognition model is trained according to the working condition voice library by means of transfer learning, including: Fine-tune the pre-trained audio recognition model using the industrial and mining speech library; Based on the speech data, a preliminary prediction result is obtained without back propagation. A new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization. Transfer knowledge from large pre-trained models to small models through knowledge distillation; Streaming the speech recognition model based on the Transformer self-attention mechanism to process speech data in real time; During the training process, a variety of optimization techniques are used to improve the performance of the model, including Adam optimizer and learning rate decay.

[0011] In the above implementation process, the model is pre-trained using general speech data, and then the model is applied to a specific field through transfer learning, and further trained to adapt to the professional terminology and dialect accent of the field.

[0012] Further, obtaining a preliminary prediction result based on the speech data without performing back propagation includes: Input the acoustic features of a set number of frames, use the self-attention module through the encoder to extract the key information in the acoustic features, and generate the corresponding latent vector; The predictor is used to predict the word count based on the CIF continuous integration and emission mechanism to obtain the predicted word count; Calculate the mean absolute error loss between the predicted word count and the actual word count. The mean absolute error loss calculation formula is: ; Among them, N represents the real number of words, n represents the total number of samples, represents the loss function of the true word count, Represents the loss function for predicting the number of words; The error of word count prediction is measured by calculating the average of the absolute values ​​of the differences between the two values, and a preliminary prediction result is obtained; Calculate the Hamming distance between the preliminary prediction and the true result of the first path to determine the number of samples.

[0013] In the above implementation process, a preliminary prediction result is obtained so that the model training step can be further performed according to the difference between the preliminary prediction result and the true label.

[0014] Furthermore, the new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization, including: Based on the determined number of samples, the acoustic embedding vector and the character embedding vector of the label are used to perform sampling operations to generate a sampling embedding vector; Generate predictions using the generated sample embedding vectors, calculate the cross entropy loss between the predicted probability distribution and the true label corresponding to the prediction, and express the cumulative difference between the predicted probability and the true label probability in logarithmic form; By comparing the prediction results, positive and negative sample pairs are constructed, and the minimum word error rate loss is calculated based on the positive and negative samples.

[0015] In the above implementation process, predictions are made through different mechanisms and multiple loss functions are calculated, aiming to comprehensively utilize different strategies to optimize the model output and make it closer to the correct label.

[0016] Furthermore, the method obtains a preliminary prediction result based on the speech data without back propagation, generates a new embedding vector based on the difference sampling between the preliminary prediction result and the true label, and calculates the loss for model training optimization in different ways, including: By pre-emphasis to improve the energy of the high frequency part, a first-order digital filter is used: y(n) = x(n) - ax(n-1); Among them, x(n) is the original speech signal, a represents the filter coefficient, x(n-1) represents the historical data of the input speech signal, and y(n) is the pre-emphasized signal, which is a coefficient close to 1; Split the speech signal into short frames; Use the Hamming window function to add a window function to each frame; Perform Fourier transform on each windowed frame to convert the time domain signal into a frequency domain signal to obtain the spectrum of each frame; Convert the obtained linear frequency into Mel frequency, construct a Mel filter bank, calculate the energy of each filter, and obtain the FBanks Mel filter bank coefficient characteristics of the speech signal; Use a self-attention module to capture the association between each unit and other units in the sequence; The FBanks feature is a sequence represented by a vector. The self-attention module calculates the degree of association between each vector and all other vectors to generate a new feature representation. The predictor predicts the number of target characters and extracts the acoustic feature vectors corresponding to the target characters. The output is a floating point number between 0 and 1. The output values ​​are accumulated to predict the number of target characters. receiving, through a sampler, an acoustic feature vector outputted from the predictor and a vector representation corresponding to the target text; According to the semantic information and position information in the target text vector, the relevant part is selected from the acoustic feature vector, and the acoustic information and semantic information are integrated. After the sampling operation of the generalized linear model GLM, the sampler transforms the acoustic feature vector and the target text vector into a feature vector containing rich semantic information; For offline and streaming, the decoder is divided into two structures: offline recognition decoder and streaming recognition decoder; The offline recognition decoder adopts a bidirectional SAN-M structure, which adds a local modeling module based on the standard Self-attention layer; The streaming recognition decoder structure adopts a unidirectional SAN-M structure, which divides the speech features into chunks, sends them to Decoderr for modeling, and then enters the predictor to predict the number of output tokens for each chunk. After receiving the token number and hidden layer representation, the Decoder predicts the output of each chunk based on the SCAMA streaming attention mechanism; The model is optimized through the loss function, where the loss function consists of three parts: cross entropy, minimum word error rate loss MWER and mean absolute error MAE loss function.

[0017] In the above implementation process, the back-propagation algorithm is used to update the encoder, predictor and parameters of each module involved in sampling and generating the final prediction results in the model, so as to gradually improve the prediction ability of the model and make the final output prediction results closer to the true label.

[0018] Furthermore, the predictive performance of the evaluation model according to the set index includes: For each test speech sample recognition result and the corresponding correct text, the edit distance is calculated using a dynamic programming algorithm to calculate the word error rate; the edit distance includes the number of substitution, insertion and deletion operations, and the minimum number of steps required to convert the recognition result into the correct text through the edit distance; Construct a term dictionary, calculate the number of sample term entries, and the number of detected terms, calculate the ratio of the detected number to the total number, and obtain the term recall rate; Calculate the average inference speed of samples; For the dialects of the set region, the word error rate of the speech recognition system of each dialect is calculated to obtain the dialect recognition accuracy; The prediction performance of the model was evaluated based on word error rate, term recall, inference speed, and dialect recognition accuracy.

[0019] In the above implementation process, during the entire model training process, MAE loss, cross entropy loss and MWER loss are comprehensively considered to improve the prediction ability of the model and make the final output prediction result closer to the true label.

[0020] In a second aspect, the embodiment of the present application provides a speech recognition training device based on the industrial and mining fields, including: A voice library creation module is used to create an industrial and mining voice library; wherein the industrial and mining voice library includes professional terms and dialect accents in a set field; Voice collection module, collects voice data and saves it to the voice database; A pre-training module, used to pre-train the audio recognition model using speech data from a speech database; Model training module, used to train the audio recognition model based on the working condition voice library by means of transfer learning; The performance prediction module is used to evaluate the prediction performance of the audio recognition model according to the set indicators.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, including: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the speech recognition training method based on the industrial and mining fields as described above.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a server, the speech recognition training method based on the industrial and mining fields as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of a speech recognition training method based on the industrial and mining fields provided in an embodiment of the present application; Figure 2 A schematic diagram of a training process of a speech recognition training method based on the industrial and mining fields provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a speech recognition training device based on the industrial and mining fields provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0027] Please see Figure 1 and Figure 2 , Figure 1 A flow chart of a speech recognition training method based on the industrial and mining fields provided in an embodiment of the present application. The speech recognition training method based on the industrial and mining fields comprises: 100. Create an industrial and mining voice database; wherein the industrial and mining voice database includes professional terms and dialect accents in a set field.

[0028] It is understandable that the setting field can be set according to specific needs. The embodiment of the present application takes the industrial and mining field as an example.

[0029] Specifically, an industrial and mining voice database is constructed, wherein the industrial and mining voice database includes an indicator query text library, a system operation text library, a knowledge question and answer text library, a fault diagnosis text library, a safety regulation text library, an equipment maintenance text library, an environmental detection text library, and an emergency plan text library; thus, the model is trained by using an operating voice library containing different dialects and professional terms, and the obtained model can be flexibly applied in a variety of industrial and mining environments.

[0030] Understandably, in an industrial and mining environment, technical terms such as “gangue”, “slime”, “flotation”, “heavy media washing” and “ash” often appear in conversation.

[0031] Therefore, the industrial and mining speech database of the embodiment of the present application is divided into eight categories, each of which covers 100 carefully selected text entries to ensure comprehensive coverage of the specific needs of the industrial and mining industry.

[0032] Among them, indicator query: contains query statements related to key performance indicators in the industrial and mining production process, such as "What is today's coal production?" or "Current gas concentration standard". Among them, system operation: involves instructions for operating industrial and mining equipment or software systems, such as "Start the conveyor belt" or "Adjust the crusher speed". Among them, knowledge questions and answers: provide question and answer pairs about industrial and mining industry knowledge, such as "What is the principle of heavy medium washing?" or "What are the main components of gangue?" Among them, fault diagnosis: contains the diagnosis process when a fault occurs, such as "What may be the reason for the conveyor belt to stop running?" or "How to deal with abnormal crusher noise?" Among them, safety regulations: involve regulations and guidelines for safe operation of industrial and mining, such as "Safety precautions for using electrical equipment in mines" or "Action steps for emergency evacuation". Among them, equipment maintenance: includes guidance on daily maintenance and maintenance of equipment, such as "Regularly check the fastening bolts of the mine car track" or "Clean and lubricate the pump station". Among them, environmental monitoring: statements related to monitoring the industrial and mining environment, such as "Check whether the dust concentration exceeds the standard" or "Measure the temperature and humidity in the mine." Among them, emergency plans: involve emergency response measures in emergency situations, such as "emergency procedures in the event of a gas leak" or "emergency evacuation routes in the event of mine flooding."

[0033] As described above, through such classification and quantity setting, the industrial and mining voice database can provide rich, scenario-based voices for the voice recognition system, thereby improving the recognition accuracy and practicality of the system in specific fields.

[0034] 200. Collect voice data and save it to a voice database.

[0035] 210. Record the user's voice data.

[0036] 220. Automatically identify voice data and obtain text transcription results.

[0037] 230. Receive user feedback and evaluation of the transcription result and the dialect category, complete the collection process of the voice data and save it to the voice database.

[0038] For example, the user's voice data can be collected through the mobile phone voice collection system, and the user can easily record audio through the mobile phone microphone; the user can dynamically manage the voice library based on preset text and record his own voice samples.

[0039] After the user completes each voice recording, the system will quickly provide the transcription results; the user can then rate the recognition effect as "good", "medium" or "poor" based on the accuracy of the recognition; after the evaluation, the user only needs to select the corresponding dialect category to complete the collection process of the voice.

[0040] In the above, high-quality speech label data is cleaned out according to the evaluation results, and the audio data and its corresponding text information used to train the speech recognition model are collected and organized, ensuring the diversity and quality of the speech and providing a basis for subsequent speech training.

[0041] 300. Use speech data from a speech database to pre-train the audio recognition model.

[0042] Optionally, pre-train the Transformer-based audio recognition model using large and diverse speech data from multiple languages, dialects, and different users to ensure that the model learns common speech features.

[0043] 400. Through transfer learning, the audio recognition model is trained according to the working condition voice library.

[0044] 410. Use the industrial and mining speech library to fine-tune the pre-trained audio recognition model.

[0045] Optionally, the pre-trained audio recognition model is applied to data in a specific field. At this stage, the audio recognition model is fine-tuned using the specialized terminology of the mining industry and the speech library of the relevant dialect so that it can better recognize the speech features of these specific fields.

[0046] 420. Based on the speech data, a preliminary prediction result is obtained without back propagation. A new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization.

[0047] 430. Transfer knowledge from large pre-trained models to small models through knowledge distillation.

[0048] For example, a large amount of training data and computing resources are used to train a large and high-performance teacher model. The teacher model has a complex structure and a large number of parameters, which can capture the subtle features and patterns in the data; useful knowledge is extracted from the teacher model, which can be achieved in many ways, such as extracting the output of the teacher model (soft label), the feature representation of the intermediate layer, or the attention weight. In addition, this knowledge is used to train a smaller student model. During the training process, the student model not only learns the direct mapping from input data to output, but also imitates the output or feature representation of the teacher model. This is usually achieved by minimizing the difference between the student model and the teacher model (such as cross entropy loss, KL divergence, etc.).

[0049] In order to transfer the knowledge of the teacher model to the student model, a distillation loss function is defined. This function measures the difference between the student model and the teacher model and encourages the student model to imitate the output or behavior of the teacher model. Thus, by transferring the knowledge of the teacher model to the student model in an easy-to-learn form, the model is compressed and accelerated.

[0050] 440. Perform streaming processing on the speech recognition model based on the Transformer self-attention mechanism to process speech data in real time.

[0051] For example, the original Transformer self-attention mechanism needs to calculate the attention scores between all elements in the entire input sequence, which is not feasible in streaming processing. Therefore, a streaming self-attention mechanism is designed that can gradually calculate the attention scores, relying only on the input data that has been processed.

[0052] In streaming processing, the model needs to gradually receive and process new input data. Through the incremental encoding method, the model can update the current encoding state based on the previous encoding results and the new input data.

[0053] In order to balance real-time performance and accuracy, the input data is divided into multiple blocks (or windows) and processed one by one. After each block is processed, the window is moved and the next block is processed. This allows the model to capture long-distance dependencies across blocks while maintaining real-time performance.

[0054] During the training process, a variety of optimization techniques are used to improve the performance of the model, including Adam optimizer and learning rate decay.

[0055] On the basis of the above embodiment, the embodiment of the present application can also be embodied as follows: the step of obtaining a preliminary prediction result according to the speech data without performing back propagation includes: Input the acoustic features of the set number of frames, use the self-attention module through the encoder to extract the key information in the acoustic features, and generate the corresponding latent vector; use the predictor to predict the number of words based on the CIF continuous integration and emission mechanism to obtain the predicted number of words; calculate the mean absolute error loss between the predicted number of words and the actual number of words. The mean absolute error loss calculation formula is: ; Among them, N represents the real number of words, n represents the total number of samples, represents the loss function of the true word count, Represents the loss function for predicting word count.

[0056] The error of word count prediction is measured by calculating the average of the absolute values ​​of the difference between the two to obtain a preliminary prediction result; the Hamming distance between the preliminary prediction result of the first path and the actual result is calculated to determine the number of samples. It can be understood that the Hamming distance measures the number of different characters in the corresponding positions of two equal-length strings, and the number of samples is determined by this distance. For example, if the Hamming distance is large, it may mean that the prediction result is very different from the actual result. Accordingly, a larger number of samples can be set according to certain rules, and vice versa.

[0057] Based on the above embodiment, the embodiment of the present application can also be embodied as follows: the new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization, including: Based on the determined number of samples, the acoustic embedding vector and the character embedding vector of the label are used for sampling operations to generate a sampling embedding vector; the generated sampling embedding vector is used to generate a prediction result, and the cross entropy loss between the predicted probability distribution and the true label corresponding to the prediction result is calculated, and the difference between the predicted probability and the true label probability is expressed in logarithmic form; positive and negative sample pairs are constructed by comparing the prediction results, and the minimum word error rate loss MWER is calculated based on the positive and negative samples.

[0058] Understandably, cross entropy loss is often used to measure the difference between the predicted probability distribution and the true probability distribution (here it can be regarded as the true label represented by one-hot encoding) in classification tasks, so as to encourage the model to output a probability distribution that is more in line with the actual situation.

[0059] Understandably, the MWER loss function focuses on the word error rate between the predicted sequence and the true sequence, measures the difference between the two by considering editing operations such as replacement, insertion, and deletion, guides the model to reduce the word error rate in sequence tasks such as generating text, and optimizes the output results to make them closer to the true label sequence.

[0060] As mentioned above, during the model training process, MAE loss, CE loss, and MWER loss are comprehensively considered, and the back-propagation algorithm is used to update the encoder, predictor, and parameters of each module involved in sampling and generating the final prediction results in the model, so as to gradually improve the prediction ability of the model and make the final output prediction result closer to the true label.

[0061] On the basis of the above embodiment, the embodiment of the present application can also be concretized as follows: the preliminary prediction result is obtained according to the speech data without back propagation, a new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization, including: Feature extractor: Pre-emphasis is used to increase the energy of the high-frequency part. It should be noted that the high-frequency part of the speech signal usually has low energy. Pre-emphasis can increase the energy of the high-frequency part. Specifically, a first-order digital filter is used: y(n) = x(n) - ax(n-1); Among them, x(n) is the original speech signal, a represents the filter coefficient, x(n-1) represents the historical data of the input speech signal, and y(n) is the pre-emphasized signal, which is a coefficient close to 1.

[0062] Divide the speech signal into short frames: Divide the speech signal into short frames because the speech signal is a non-stationary signal, but it can be approximately regarded as a stationary signal in a short time. For example, the frame length is 20 milliseconds and the frame shift (the interval between two adjacent frames) is 10 milliseconds.

[0063] Use the Hamming window function to add a window function to each frame: Add a window function to each frame to reduce spectrum leakage and make the spectrum of each frame smoother.

[0064] Perform Fourier transform on each windowed frame to convert the time domain signal into frequency domain signal and obtain the spectrum of each frame; convert the obtained linear frequency into Mel frequency, construct Mel filter bank, calculate the energy of each filter, and obtain the FBanks Mel filter bank coefficient characteristics of the speech signal; use the self-attention module to capture the association between each unit in the sequence and other units, that is, the association with other units except this unit; optionally, 20 Mel filters.

[0065] Encoder: It is composed of a self-attention module, which highlights the most important semantic part of the sequence by capturing the association between each unit in the sequence and other units. Among them, the FBanks feature is a sequence X=[x1, x2, ..., x_n] represented by a vector. The self-attention module calculates the degree of association between each vector and all other vectors (that is, other vectors except this vector) to generate a new feature representation Z=[z1, z2, ..., z_n].

[0066] Predictor: It consists of two layers of deep neural network, which is used to predict the number of target characters and extract the acoustic vector corresponding to the target characters. The predictor predicts the number of target characters and extracts the acoustic feature vector corresponding to the target characters. The output is a floating point number between 0 and 1. The output value is accumulated to predict the number of target characters. The acoustic vector is obtained through the CIF (Continuous integrate and fire, a mechanism used for sequence conversion tasks such as speech recognition) module. CIF integrates the acoustic information that arrives successively. When the amount of integrated information reaches the recognition threshold, the integrated information is released for subsequent recognition.

[0067] Sampler: The sampler receives the acoustic feature vector output from the predictor and the vector representation corresponding to the target text, and uses the Glancing Language Model (GLM) to perform sampling operations. According to the semantic information and position information in the target text vector, the relevant parts are selected from the acoustic feature vector, and the acoustic information and semantic information are integrated. After the sampling operation of the generalized linear model GLM, the sampler transforms the acoustic feature vector and the target text vector into a feature vector containing rich semantic information. Among them, this newly generated feature vector not only contains the acoustic features of the speech, but also incorporates the semantic information of the target text, so that it can better express the correspondence between speech and text, provide more valuable input for the subsequent decoder module, enhance the model's ability to model context, and help improve the accuracy and performance of speech recognition.

[0068] Decoder: For offline and streaming, the decoder is divided into two structures: offline recognition decoder and streaming recognition decoder; the offline recognition decoder adopts a bidirectional SAN-M structure, which can better capture the contextual information in the text, thereby more accurately generating text that conforms to the language logic. The SAN-M structure adds a local modeling module Memory Block on the basis of the standard Self-attention layer, enhancing the local modeling capability of Self-attention, enabling it to better process local features and long-term dependencies in speech signals, thereby improving the accuracy of speech recognition.

[0069] Streaming recognition decoder structure: It adopts unidirectional SAN-M and combines it with the streaming attention mechanism based on SCAMA. Specifically, the speech features are first divided into chunks, sent to the decoder for modeling, and then enter the predictor to predict the number of output tokens of each chunk; after receiving the number of tokens and the hidden layer representation, the decoder predicts the output of each chunk based on the SCAMA (Streaming Chunk-Aware Multihead Attention) streaming attention mechanism; the unidirectional SAN-M structure combined with the SCAMA streaming attention mechanism can not only meet the real-time requirements of streaming recognition, but also effectively utilize the information in the speech signal to improve the recognition accuracy.

[0070] Understandably, bidirectional information processing: In the SAN-M network structure, if a bidirectional attention mechanism or a bidirectional recurrent neural network (such as a bidirectional LSTM) is used, the network can simultaneously consider the forward and backward information of the input sequence, thereby improving the model's ability to capture contextual information. This bidirectional information processing mechanism can usually achieve better performance in sequence processing tasks such as speech recognition. Unidirectional information processing: On the contrary, if the SAN-M network structure only uses a unidirectional attention mechanism or a unidirectional recurrent neural network, the network will only be able to consider the forward information of the input sequence. Although this unidirectional information processing mechanism is relatively simple, it can still achieve good performance in some cases, especially when the contextual information of the input sequence is not particularly important.

[0071] The model is optimized through the loss function, where the loss function consists of three parts: cross entropy, minimum word error rate loss MWER and mean absolute error MAE loss function. MAE loss is used to train the predictor accuracy.

[0072] 500. Evaluate the prediction performance of the audio recognition model according to the set indicators.

[0073] 510. For each recognition result of the test speech sample and the corresponding correct text, a dynamic programming algorithm is used to calculate the edit distance to calculate the word error rate; wherein the edit distance includes the number of replacement, insertion and deletion operations, and the minimum number of steps required to convert the recognition result into the correct text through the edit distance.

[0074] 520. Build a term dictionary, calculate the number of sample term entries, and the number of detected terms, calculate the ratio of the detected number to the total number, and obtain the term recall rate.

[0075] Exemplarily, relevant terms are collected from professional literature, databases, domain experts or other reliable sources, the collected terms are sorted to ensure that each term is unique and accurate, and the sorted terms are stored in an easily accessible data structure, such as a list, dictionary or database; the sample text is preprocessed, such as word segmentation, stop word removal, etc., all entries are extracted from the preprocessed sample, and the number of all entries in the sample is counted; the preprocessed sample entries are matched with the term dictionary to find the term entries that appear in the sample, and the number of matched term entries is counted; the recall rate is calculated using the ratio of the number of detected terms to the total number of terms in the term dictionary.

[0076] 530. Calculate the average inference speed of samples.

[0077] Exemplarily, the inference time of all input samples is summarized; the summarized inference time is divided by the number of samples to obtain the average inference time; using the above formula, the number of inference samples is divided by the average inference time to obtain the average inference speed.

[0078] 540. For the dialects of the set region, the word error rate of the speech recognition system of each dialect is calculated to obtain the dialect recognition accuracy.

[0079] Exemplarily, a dialect speech recognition model is trained, the collected dialect speech data is recognized, the number of replaced, deleted and inserted characters in the recognition results is counted, the sum of the number of replaced, deleted and inserted characters is divided by the total number of characters to obtain the word error rate. The word error rate is an important indicator for evaluating the performance of the dialect speech recognition system. Optionally, the recognition accuracy can be calculated by the following formula: Recognition accuracy = 1-word error rate.

[0080] 550. Evaluate the prediction performance of the model based on word error rate, term recall rate, inference speed and dialect recognition accuracy.

[0081] For example, the evaluation weights of word error rate, term recall rate, inference speed and dialect recognition accuracy can be set according to requirements, and the performance of the model can be obtained based on the evaluation weights.

[0082] As described above, the embodiment of the present application creates an industrial and mining voice database; collects voice data and saves it to the voice database; uses the voice data of the voice database to pre-train the audio recognition model; trains the audio recognition model according to the working voice database through transfer learning; evaluates the prediction performance of the audio recognition model according to set indicators; can adapt to different dialects and professional terms, and can be flexibly applied in a variety of industrial and mining environments, thereby improving the accuracy and improving the prediction results.

[0083] As mentioned above, the embodiments of the present application focus on creating a speech library specifically for the industrial and mining industry, which contains industry-specific terms and expressions to improve the accuracy and applicability of the speech recognition system in this field, including speech library construction strategies, terminology collection, speech screening and classification, and speech annotation processes; ensuring the quality and coverage of the speech library, and how to dynamically update the speech library to adapt to industry changes.

[0084] The embodiments of this application can support the collection, annotation, evaluation and feedback of dialects to improve the recognition ability of the speech recognition system for dialects. The system's user interface design, dialect annotation process, evaluation mechanism and feedback loop; use user feedback to optimize the speech recognition model to ensure the quality and availability of the collected data.

[0085] The embodiment of the present application uses transfer learning technology to train the speech recognition model so that it can adapt to the speech data of specific fields, especially in the industrial and mining fields, including the selection of pre-trained models, domain adaptation strategies, model fine-tuning techniques, etc.; reducing the distribution differences between the source domain and the target domain to improve the generalization ability of the model in the target domain.

[0086] In addition, the embodiments of the present application have the following advantages: It can adapt to different dialects and professional terms, which means that it can be flexibly applied in a variety of industrial and mining environments, not only limited to the coal preparation industry, but also extended to other fields that require specific dialects or professional terminology recognition. By simplifying the interaction process, the system reduces the user's learning curve, making it easy for users with non-technical backgrounds to get started, and improving the popularity and user satisfaction of the system. The architecture based on the Transformer model has good scalability, and can continuously optimize and expand the functions of the system with the advancement of technology and the accumulation of data. It has the ability of self-learning and self-optimization, and can continuously improve based on user feedback and newly collected data to adapt to the ever-changing language usage habits and technological development.

[0087] The above steps are not to be performed in a strict order as described in the numbers, but should be understood as an overall solution.

[0088] In the second aspect, based on the above embodiments, Figure 3 A schematic diagram of the structure of a speech recognition training device based on the industrial and mining fields provided in an embodiment of the present application. Figure 3 The speech recognition training device based on the industrial and mining fields provided in this embodiment specifically includes: a speech library creation module 301, a speech collection module 302, a pre-training module 303, a model training module 304 and a performance prediction module 305.

[0089] Among them, the voice library creation module 301 is used to create an industrial and mining voice library; wherein, the industrial and mining voice library includes professional terms and dialect accents in a set field; the voice acquisition module 302 collects voice data and saves it to a voice database; the pre-training module 303 is used to pre-train the audio recognition model using the voice data in the voice database; the model training module 304 is used to train the audio recognition model according to the working condition voice library through transfer learning; the performance prediction module 305 is used to evaluate the prediction performance of the audio recognition model according to set indicators.

[0090] As described above, the embodiment of the present application creates an industrial and mining voice database; collects voice data and saves it to the voice database; uses the voice data of the voice database to pre-train the audio recognition model; trains the audio recognition model according to the working voice database through transfer learning; evaluates the prediction performance of the audio recognition model according to set indicators; can adapt to different dialects and professional terms, and can be flexibly applied in a variety of industrial and mining environments, thereby improving the accuracy and improving the prediction results.

[0091] The speech recognition training device based on the industrial and mining fields provided in the embodiment of the present application can be used to execute the speech recognition training method based on the industrial and mining fields provided in the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0092] In a third aspect, an embodiment of the present application further provides an electronic device, which can integrate the speech recognition training device based on the industrial and mining fields provided in an embodiment of the present application. Figure 4 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 4 The electronic device includes: an input device 43, an output device 44, a memory 42 and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the speech recognition training method based on the industrial and mining fields as provided in the above embodiment. The input device 43, the output device 44, the memory 42 and the processor 41 can be connected by a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0093] The processor 41 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 42, that is, realizes the above-mentioned speech recognition training method based on the industrial and mining fields.

[0094] The electronic device provided above can be used to execute the speech recognition training method based on the industrial and mining fields provided in the above embodiment, and has corresponding functions and beneficial effects.

[0095] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the speech recognition training method based on the industrial and mining fields as described above, and can achieve the same beneficial effects.

[0096] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the speech recognition training method based on the industrial and mining fields as described above, and can also execute related operations in the speech recognition training method based on the industrial and mining fields provided in any embodiment of the present application.

[0097] In a fifth aspect, the embodiments of the present application also provide a computer program product, and the methods described in the various embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the processes or functions described in the various embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM (Open Application Model) or other programmable device.

[0098] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0099] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0101] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0102] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0104] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A speech recognition training method based on the industrial and mining fields, characterized in that: The method comprises: Creating an industrial and mining speech database; wherein the industrial and mining speech database includes professional terms and dialect accents in a set field; Collect voice data and save it to the voice database; Pre-train the audio recognition model using speech data from a speech database; Through transfer learning, the audio recognition model is trained based on the working condition voice library; Evaluate the prediction performance of the audio recognition model based on the set metrics; The audio recognition model is trained according to the working condition voice library by means of transfer learning, including: Fine-tune the pre-trained audio recognition model using the industrial and mining speech library; Based on the speech data, a preliminary prediction result is obtained without back propagation. A new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization. Transfer knowledge from large pre-trained models to small models through knowledge distillation; Streaming the speech recognition model based on the Transformer self-attention mechanism to process speech data in real time; During the training process, a variety of optimization techniques are used to improve the performance of the model, including Adam optimizer and learning rate decay.

2. The speech recognition training method based on the industrial and mining fields according to claim 1 is characterized in that: The step of creating a mining speech library comprises: Construct an industrial and mining voice database, which includes an indicator query text library, a system operation text library, a knowledge question and answer text library, a fault diagnosis text library, a safety regulations text library, an equipment maintenance text library, an environmental detection text library and an emergency plan text library.

3. The speech recognition training method based on the industrial and mining fields according to claim 1 is characterized in that: The collecting of voice data and saving it to a voice database comprises: Recording user’s voice data; Automatically identify voice data and obtain text transcription results; Receive user feedback on the transcription results and dialect category, complete the voice data collection process and save it to the voice database.

4. The speech recognition training method based on the industrial and mining fields according to claim 1 is characterized in that: The method of obtaining a preliminary prediction result based on the speech data without performing back propagation includes: Input the acoustic features of a set number of frames, use the self-attention module through the encoder to extract the key information in the acoustic features, and generate the corresponding latent vector; The predictor is used to predict the word count based on the CIF continuous integration and emission mechanism to obtain the predicted word count; Calculate the mean absolute error loss between the predicted word count and the actual word count. The mean absolute error loss calculation formula is: ； Among them, N represents the real number of words, n represents the total number of samples, represents the loss function of the true word count, Represents the loss function for predicting the number of words; The error of word count prediction is measured by calculating the average of the absolute values ​​of the differences between the two values, and a preliminary prediction result is obtained; Calculate the Hamming distance between the preliminary prediction and the true result of the first path to determine the number of samples.

5. The speech recognition training method based on the industrial and mining fields according to claim 1 is characterized in that: The method generates a new embedding vector based on the difference sampling between the preliminary prediction result and the true label, and calculates the loss for model training optimization in different ways, including: Based on the determined number of samples, the acoustic embedding vector and the character embedding vector of the label are used to perform sampling operations to generate a sampling embedding vector; Generate predictions using the generated sample embedding vectors, calculate the cross entropy loss between the predicted probability distribution and the true label corresponding to the prediction, and express the cumulative difference between the predicted probability and the true label probability in logarithmic form; By comparing the prediction results, positive and negative sample pairs are constructed, and the minimum word error rate loss is calculated based on the positive and negative samples.

6. The speech recognition training method based on the industrial and mining fields according to claim 1 is characterized in that: According to the speech data, a preliminary prediction result is obtained without back propagation, a new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization, including: By pre-emphasis to improve the energy of the high frequency part, a first-order digital filter is used: y(n) = x(n) - ax(n-1); Among them, x(n) is the original speech signal, a represents the filter coefficient, x(n-1) represents the historical data of the input speech signal, and y(n) is the pre-emphasized signal, which is a coefficient close to 1; Split the speech signal into short frames; Use the Hamming window function to add a window function to each frame; Perform Fourier transform on each windowed frame to convert the time domain signal into a frequency domain signal to obtain the spectrum of each frame; Convert the obtained linear frequency into Mel frequency, construct a Mel filter bank, calculate the energy of each filter, and obtain the FBanks Mel filter bank coefficient characteristics of the speech signal; Use a self-attention module to capture the association between each unit and other units in the sequence; The FBanks feature is a sequence represented by a vector. The self-attention module calculates the degree of association between each vector and all other vectors to generate a new feature representation. The predictor predicts the number of target characters and extracts the acoustic feature vectors corresponding to the target characters. The output is a floating point number between 0 and 1. The output values ​​are accumulated to predict the number of target characters. receiving, through a sampler, an acoustic feature vector outputted from the predictor and a vector representation corresponding to the target text; According to the semantic information and position information in the target text vector, the relevant part is selected from the acoustic feature vector, and the acoustic information and semantic information are integrated. After the sampling operation of the generalized linear model GLM, the sampler transforms the acoustic feature vector and the target text vector into a feature vector containing rich semantic information; For offline and streaming, the decoder is divided into two structures: offline recognition decoder and streaming recognition decoder; The offline recognition decoder adopts a bidirectional SAN-M structure, which adds a local modeling module based on the standard Self-attention layer; The streaming recognition decoder structure adopts a unidirectional SAN-M structure, which divides the speech features into chunks, sends them to Decoderr for modeling, and then enters the predictor to predict the number of output tokens for each chunk. After receiving the token number and hidden layer representation, the Decoder predicts the output of each chunk based on the SCAMA streaming attention mechanism; The model is optimized through the loss function, where the loss function consists of three parts: cross entropy, minimum word error rate loss MWER and mean absolute error MAE loss function.

7. The speech recognition training method based on the industrial and mining fields according to claim 1 is characterized in that: The predictive performance of the model evaluated according to the set indicators includes: For each test speech sample recognition result and the corresponding correct text, the edit distance is calculated using a dynamic programming algorithm to calculate the word error rate; the edit distance includes the number of substitution, insertion and deletion operations, and the minimum number of steps required to convert the recognition result into the correct text through the edit distance; Construct a term dictionary, calculate the number of sample term entries, and the number of detected terms, calculate the ratio of the detected number to the total number, and obtain the term recall rate; Calculate the average inference speed of samples; For the dialects of the set region, the word error rate of the speech recognition system of each dialect is calculated to obtain the dialect recognition accuracy; The prediction performance of the model was evaluated based on word error rate, term recall, inference speed, and dialect recognition accuracy.

8. A speech recognition training device based on the industrial and mining fields, characterized in that: include: A voice library creation module is used to create an industrial and mining voice library; wherein the industrial and mining voice library includes professional terms and dialect accents in a set field; Voice collection module, collects voice data and saves it to the voice database; A pre-training module, used to pre-train the audio recognition model using speech data from a speech database; Model training module, used to train the audio recognition model based on the working condition voice library by means of transfer learning; A performance prediction module, used to evaluate the prediction performance of the audio recognition model according to set indicators; The audio recognition model is trained according to the working condition voice library by means of transfer learning, including: Fine-tune the pre-trained audio recognition model using the industrial and mining speech library; Based on the speech data, a preliminary prediction result is obtained without back propagation. A new embedding vector is generated based on the difference sampling between the preliminary prediction result and the true label, and the loss is calculated in different ways for model training optimization. Transfer knowledge from large pre-trained models to small models through knowledge distillation; Streaming the speech recognition model based on the Transformer self-attention mechanism to process speech data in real time; During the training process, a variety of optimization techniques are used to improve the performance of the model, including Adam optimizer and learning rate decay.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the speech recognition training method based on the industrial and mining fields as described in any one of claims 1 to 7.

Images