Train driver voice recognition method, equipment, medium and product

Abstract

The invention discloses a train driver voice recognition method and device, a medium and a product, and relates to the technical field of deep learning and voice recognition, and the method comprises the steps: obtaining the voice information of a train driver; performing denoising processing, preprocessing and fast Fourier transform on the voice information to obtain a first spectrum feature; inputting the first spectrum feature into a special vocabulary recognition model to obtain a special vocabulary recognition result; inputting the first spectrum feature into a general vocabulary recognition model to obtain a general vocabulary recognition result; and carrying out fusion processing on the special vocabulary recognition result and the general vocabulary recognition result to obtain a voice recognition result of the train driver. The voice recognition result of the train driver is obtained by fusing the recognition results of the special vocabulary recognition model and the general vocabulary recognition model, and the fusion processing effectively makes up for the limitation of the general model on the recognition in the railway field, thereby improving the accuracy of the voice recognition of the train driver.

Description

Technical Field

[0001] The present application relates to the field of deep learning and speech recognition technology, and in particular to a train driver speech recognition method, device, medium and product. Background Art

[0002] Train drivers are an important part of the rail transit system, and the completion status of train drivers' driving tasks is closely related to driving safety. Surveys and statistics on train driving safety accidents show that most accidents are related to human factors. At present, the monitoring and management tasks of train drivers mainly revolve around drivers' fatigue status and dangerous behaviors, and lack monitoring of skill-based errors in non-fatigue states. In order to avoid skill-based errors by train drivers, my country's train drivers need to perform pointing calls and call responses in addition to driving tasks during driving to improve the train drivers' concentration on driving tasks. By using speech recognition technology to identify the voice information of train drivers during driving, the monitoring of skill-based errors of train drivers in non-fatigue states can be achieved, thereby improving the level of driving safety.

[0003] Speech recognition technology is responsible for converting speech content into corresponding text information. With the development of machine learning and deep learning technologies, the accuracy and speed of speech recognition technology have been greatly improved. At present, a large number of end-to-end speech recognition models have appeared in the domestic and foreign markets. However, these models are mainly applicable to general fields, but have limitations in practical applications in the railway field. Summary of the invention

[0004] The purpose of this application is to provide a train driver speech recognition method, device, medium and product, which can effectively make up for the limitations of general models in railway recognition.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a train driver speech recognition method, comprising:

[0007] Get the train driver's voice information;

[0008] Performing denoising and preprocessing on the voice information to obtain first voice information;

[0009] Performing a fast Fourier transform on the first speech information to obtain a first frequency spectrum feature;

[0010] Inputting the first spectrum feature into a proprietary vocabulary recognition model to obtain a proprietary vocabulary recognition result; the proprietary vocabulary recognition model includes a proprietary syllable recognition model and a syllable conversion model; the proprietary syllable recognition model is used to perform syllable recognition processing on the first spectrum feature to obtain a proprietary syllable recognition result; the syllable conversion model is used to perform syllable conversion processing on the proprietary syllable recognition result to obtain the proprietary vocabulary recognition result;

[0011] The first spectrum feature is input into a general vocabulary recognition model to obtain a general vocabulary recognition result; the specific vocabulary recognition result and the general vocabulary recognition result are fused to obtain a speech recognition result of the train driver.

[0012] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned train driver speech recognition method.

[0013] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned train driver speech recognition method.

[0014] In a fourth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned train driver speech recognition method when executed by a processor.

[0015] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0016] The present application provides a train driver speech recognition method, device, medium and product, the train driver speech recognition method comprising: acquiring the train driver's speech information; performing denoising and preprocessing on the speech information to obtain first speech information; performing fast Fourier transform on the first speech information to obtain first spectrum features; inputting the first spectrum features into a proprietary vocabulary recognition model to obtain a proprietary vocabulary recognition result; the proprietary vocabulary recognition model comprises a proprietary syllable recognition model and a syllable conversion model; the proprietary syllable recognition model is used to perform syllable recognition processing on the first spectrum features to obtain a proprietary syllable recognition result; the syllable conversion model is used to perform syllable conversion processing on the proprietary syllable recognition result to obtain a proprietary vocabulary recognition result; the first spectrum features are input into a general vocabulary recognition model to obtain a general vocabulary recognition result; the proprietary vocabulary recognition result and the general vocabulary recognition result are fused to obtain the train driver's speech recognition result.

[0017] The present application obtains the train driver's speech recognition result by integrating the recognition results of a proprietary vocabulary recognition model with a general vocabulary recognition model. This fusion process effectively makes up for the limitations of the general model in the railway field, thereby improving the accuracy of train driver speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is an application environment diagram of a train driver speech recognition method in one embodiment of the present application;

[0020] Figure 2 A flow chart of a train driver speech recognition method provided in one embodiment of the present application;

[0021] Figure 3 This is a detailed flowchart of the preprocessing and fast Fourier transform steps in one embodiment of the present application;

[0022] Figure 4 Schematic diagram of the structure of a proprietary vocabulary recognition model in an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of the basic principle of the soft voting method in one embodiment of the present application;

[0024] Figure 6 A module diagram of a train driver speech recognition system provided by an embodiment of the present application;

[0025] Figure 7 This is a schematic diagram of the interface of the user interaction layer in one embodiment of the present application; Figure 7 (a) is the initialization interface; Figure 7 (b) Speech recognition task interface;

[0026] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] Since the train driver's voice information contains not only common words but also a large number of railway-specific words, when faced with railway-specific words, the existing speech recognition model has a low recognition accuracy and is usually trained based on English data, and has poor support for Chinese information. Based on this, the embodiment of the present application provides a train driver speech recognition method.

[0030] The train driver speech recognition method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the voice information of the train driver to the server 104. After the server 104 receives the voice information of the train driver, the server 104 performs denoising, preprocessing and fast Fourier transform on the voice information to obtain a first spectral feature; inputs the first spectral feature into a proprietary vocabulary recognition model to obtain a proprietary vocabulary recognition result; inputs the first spectral feature into a general vocabulary recognition model to obtain a general vocabulary recognition result; fuses the proprietary vocabulary recognition result and the general vocabulary recognition result to obtain the train driver's speech recognition result. The server 104 can feedback the obtained train driver's speech recognition result to the terminal 102. In addition, in some embodiments, the train driver speech recognition method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform speech recognition processing on the train driver's voice information, or the server 104 can obtain the train driver's voice information from the data storage system and perform speech recognition processing on the train driver's voice information.

[0031] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0032] In an exemplary embodiment, Figure 2 As shown, a train driver speech recognition method is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 206. Among them:

[0033] Step 201, obtaining the voice information of the train driver.

[0034] Step 202: perform denoising and preprocessing on the voice information to obtain first voice information.

[0035] The denoising process is to set the denoising parameter value range in the Wiener filter according to the acquired running state of the current train, and input the voice information into the Wiener filter after the corresponding setting, so as to obtain the denoised voice information; the running state includes four states: default, uniform speed, deceleration or parking; the pre-processing is to perform pre-emphasis, framing and windowing operations on the denoised voice information in sequence to obtain the first voice information, such as Figure 3 shown.

[0036] Step 203, fast Fourier transform is performed on the first speech information, so that the first speech information is converted from the time domain to the frequency domain, and a first spectrum feature including a spectrogram and a log-mel spectrogram (the result of taking the logarithm of the Mel spectrogram) is obtained, such as Figure 3 shown.

[0037] Step 204: input the first frequency spectrum feature into a proprietary vocabulary recognition model to obtain a proprietary vocabulary recognition result; the proprietary vocabulary recognition model includes a proprietary syllable recognition model and a syllable conversion model.

[0038] Furthermore, if Figure 4 As shown, the first spectrum feature is input into the proprietary vocabulary recognition model to obtain the proprietary vocabulary recognition result, which specifically includes:

[0039] Step 2041: Use a proprietary syllable recognition model to perform syllable recognition processing on the first frequency spectrum feature to obtain a proprietary syllable recognition result; wherein the determination process of the proprietary syllable recognition model specifically includes:

[0040] a. construct a proprietary syllable training data set; each sample of the proprietary syllable training data set includes a second spectral feature and a corresponding syllable label; the second spectral feature is obtained by preprocessing and fast Fourier transforming the standard call phrase of the train driver; the syllable label is obtained by labeling the second spectral feature.

[0041] b. Construct a proprietary syllable recognition network; the proprietary syllable recognition network is a network structure based on the CNN-GRU-CTC framework, which includes several layers of convolutional layers, gated recurrent units (GRU), fully connected layers and connected temporal classifiers (CTC) connected in sequence; first, use several layers of one-dimensional convolutional layers to extract the time domain features and time correlation in the second spectral features, for example, set 12 one-dimensional convolutional layers, each convolution kernel size is 3, the activation function is ReLU, and at the same time, in view of the characteristics of the train driver's voice information with a fast speaking speed and a short voice length, the present application uses a dilated convolution in the last two convolutional layers to expand the network receptive field; then, it can be A gated recurrent unit (GRU) with a layer of 128 hidden units is used as a timing module to further extract the deep timing features of the speech information; finally, in the speech recognition task of the train driver, due to the inconsistency between the second spectral features and the corresponding syllable labels, the deep timing features output by the gated recurrent unit cannot be aligned with the manually annotated syllable labels. Therefore, this application adopts a connection-based temporal classifier (CTC) as a forced alignment module. When the input sequence and output sequence of the network cannot form a simple one-to-one alignment relationship, the deep timing features of the train driver's speech information and the syllable labels are forced to be aligned by extending the label length and traversing all possible labels.

[0042] c. Input each sample of the proprietary syllable training data set into the proprietary syllable recognition network, update the parameters of the proprietary syllable recognition network, and obtain the proprietary syllable recognition model. Moreover, in the application stage of the proprietary syllable recognition model, for a given first spectrum feature, after feature extraction of CNN and GRU, CTC decoding, softmax mapping and decoding search, the first spectrum feature is mapped to the output probability of the proprietary syllable recognition result.

[0043] Step 2042: Use a syllable conversion model to perform syllable conversion processing on the proprietary syllable recognition result to obtain the proprietary vocabulary recognition result.

[0044] Furthermore, the syllable conversion model includes a pronunciation dictionary and an n-gram language model; performing syllable conversion processing on the proprietary syllable recognition result to obtain the proprietary vocabulary recognition result specifically includes: using the pronunciation dictionary to convert the proprietary syllable recognition result into all corresponding text sequences; using the n-gram language model to calculate the probability values ​​of all text sequences to obtain the proprietary vocabulary recognition result; assuming that the text sequence S = {w1 ,w 2 ,...,w i}, according to the conditional probability, the occurrence probability of the text sequence S can be expressed as:

[0045] P(S)=P(w 1 ,w 2 ,...,w n )=P(w 1 )*P(w 2 |w 1 )*...*P(w i |w 1 ,w 2 ,...,w i-1 );

[0046] Among them, w i is the i-th word in the text sequence, the conditional probability P(w i |w i-n+1 ,...,w i-1 ) is usually estimated by counting the frequency of a word in a large corpus, specifically:

[0047]

[0048] Among them, count represents the number of times a word appears in the train driving corpus.

[0049] For example, suppose the result of the proprietary syllable recognition is "zhuyi xian su liu shi gong li". By consulting the pronunciation dictionary, a corresponding possible text sequence "Note that the speed limit is 60 kilometers" can be found from the corpus. The following is the process of calculating the output probability of this text sequence using the 3-gram model:

[0050] P(Note that the speed limit is 60 kilometers) = P(Note)*P(Speed ​​limit|Note)*P(60|Note the speed limit)*P(Kilometers|Note the speed limit is 60);

[0051] Among them, P(Attention) is the conditional probability of the word "Attention", P(Speed ​​Limit|Attention) is the conditional probability of the word "Speed ​​Limit", P(60|Attention Speed ​​Limit) is the conditional probability of the word "60" appearing, and P(Kilometer|Attention Speed ​​Limit 60) is the conditional probability of the word "Kilometer". The specific calculation method of each conditional probability is as follows:

[0052]

[0053] Step 205: input the first spectrum feature into a general vocabulary recognition model to obtain a general vocabulary recognition result.

[0054] The process of determining the general vocabulary recognition model specifically includes:

[0055] a. Construct a universal vocabulary recognition network; the universal vocabulary recognition network is built based on the Whisper framework. Specifically, the Whisper network is a speech recognition network open sourced by OpenAI in 2022, with Transformer as the infrastructure; the network takes a log-Mel spectrogram with a time step of 3000 and a channel number of 80 as input, and the main structure of the network consists of an input layer consisting of two layers of one-dimensional convolutional networks and an encoder-decoder layer based on Transformer; the two convolutional layers of the input layer are similar to a pyramid structure, and their main function is to reduce the dimension of the data, reduce the temporal resolution, and extract local information for subsequent feature extraction; the multi-layer stacked multi-head self-attention mechanism and multi-layer perceptron in the encoder are used to capture the correlation between each position in the input sequence without being restricted by the length of the sequence, and the deep features are spliced ​​in the last encoding layer as the query and value input of the decoder's multi-head self-attention mechanism; in addition, the decoder also receives information from the encoder to convert and generate a target sequence, and finally outputs a coherent text sequence.

[0056] b. Construct a general vocabulary training data set; each sample of the general vocabulary training data set includes a third spectral feature and a corresponding text label; the third spectral feature is obtained after preprocessing and fast Fourier transforming the Chinese speech data; the text label is obtained after annotating the third spectral feature.

[0057] c. Based on the transfer learning technology, the samples in the general vocabulary training data set are input into the general vocabulary recognition network, and the parameters of the general vocabulary recognition network are updated so that the network parameters are shifted toward the Chinese speech recognition task to obtain a general vocabulary recognition model.

[0058] Furthermore, the universal vocabulary recognition network includes a preset layer (e.g., a fully connected layer), and uses the LoRa algorithm (Low-Rank Adaptation) to fine-tune the network parameters of the Whisper model in a low-resource environment to obtain a universal vocabulary recognition model. Specifically:

[0059] Initialize the preset layer; the initialization is used to convert the original pre-trained weight matrix W of the preset layer 0 It is represented as the product of two trainable low-rank matrices A and B, where the dimension of A is d×r and the dimension of B is r×d (r<<d, d is the dimension of the original pre-trained weight matrix);

[0060] Perform the following iterations on the samples in the general vocabulary training dataset:

[0061] The current sample of the general vocabulary training data set is input into the general vocabulary recognition network corresponding to the current iteration number to obtain the initial general vocabulary recognition result corresponding to the current sample; the current sample of the general vocabulary training data set is a sample that has not been input into the general vocabulary recognition network corresponding to the current iteration number; the general vocabulary recognition network corresponding to the current iteration number is the general vocabulary recognition network initialized by the preset layer or the general vocabulary recognition network updated by the previous iteration number.

[0062] Calculate the loss value between the initial common vocabulary recognition result corresponding to the current sample and the corresponding text label, update the parameters of the trainable low-rank matrices A and B through back propagation, obtain the updated trainable low-rank matrices A and B, and combine the updated trainable low-rank matrices A and B with the original pre-trained weight matrix W 0 , get the updated weight matrix W after the current number of iterations, expressed as:

[0063] W=W 0 +BA.

[0064] Determine whether the loss value meets the stopping condition. If so, determine the universal vocabulary recognition model according to the weight matrix W updated after the current iteration number; if not, update the universal vocabulary recognition network corresponding to the current iteration number according to the weight matrix W updated after the current iteration number, and return "input the current sample of the universal vocabulary training data set into the universal vocabulary recognition network corresponding to the current iteration number, and obtain the initial universal vocabulary recognition result corresponding to the current sample".

[0065] Since Whisper is a network trained with multilingual data, and the common vocabulary recognition task in the train driver voice information recognition scenario is mainly based on Chinese vocabulary, the existing Whisper network needs to be fine-tuned. The most basic fine-tuning method is fine-tuning, which directly uses the target data set to train the network without changing the network structure. During the fine-tuning process, it is necessary to freeze the parameter values ​​of the original model, and add new layers to the original model to learn new features. The parameters of the unfrozen network layers and the added new network layers are trained to achieve the purpose of parameter migration, but there is a serious waste of computing resources in the traditional fine-tuning method. Therefore, this application uses the LoRa algorithm to fine-tune the model, and only trains the introduced low-rank matrices A and B without updating the original pre-trained weight matrix W 0 , since the number of parameters of the low-rank matrices A and B is much smaller than the original pre-trained weight matrix W 0 ,Therefore, the computational cost and storage requirements of fine-tuning are greatly reduced. The fine-tuning effect is shown in Table 1 below:

[0066] Table 1 Fine-tuning effect table

[0067] Fine-tuning the algorithm Word Error Rate (WER) Training parameter ratio Training time Not fine-tuned 40.9％ - - Fine-tuning 7.51％ 100％ 26 hours LoRa 10.8％ 2.89％ 18 hours

[0068] Step 206, fusing the special vocabulary recognition result and the general vocabulary recognition result to obtain the train driver's speech recognition result, specifically including:

[0069] Step 2061, performing hot word weighting processing on the proprietary vocabulary recognition result to obtain the weighted proprietary vocabulary recognition result.

[0070] Step 2062, performing hot word weighting processing on the common vocabulary recognition result to obtain the weighted common vocabulary recognition result.

[0071] Step 2063, using a soft voting method, a fusion calculation is performed on the weighted proprietary vocabulary recognition result and the weighted general vocabulary recognition result to obtain a predicted speech recognition result.

[0072] Step 2064, based on the predicted speech recognition result, obtain the speech recognition result of the train driver.

[0073] It should be noted that if Figure 5 As shown in the figure, the specific vocabulary recognition model and the general vocabulary recognition model are regarded as two trained classifiers h 1 and h 2 , each classifier makes a judgment on the classification task Q respectively, and makes a decision by combining the judgment results of each classifier and outputs the final judgment result. Assume that the class label of the classification task Q is {q 1 ,q 2 ,...,q j}, the input sample is x, then the output results of the above two classifiers can be expressed as:

[0074]

[0075] in, It means that for the input sample x, the classifier h i Get the class label q j The predicted output probability.

[0076] In the process of soft voting, the weighted average of the output probabilities of the two classifiers is used as the final output probability value of each class label, and finally the class label with the highest probability is selected as the final output result, as shown below:

[0077]

[0078] Among them, h j (x) represents the class label q jThe weighted average probability of h(x) represents the set of final output probability values ​​of all class labels, and the class label with the largest probability value is selected as the final output result y.

[0079] Furthermore, considering that the train driver's calling content is still mainly proprietary words, supplemented by common words, this means that most of the time when performing voting decisions, the proprietary word recognition model should be biased. Therefore, after the probability calculation and before the soft voting decision, a hot word weighting module is added, and the hot word weight value is used as the metric for voting decisions. Its core function is to give higher parameter weights to specific words in specific scenarios.

[0080] This application assigns weights to hot words based on the frequency of occurrence of each proprietary word in the train driver's call text library, and divides the words in the text library into four categories according to word frequency: high-frequency words, medium-high-frequency words, medium-frequency words, and low-frequency words. The division method and weights are shown in Table 2 below:

[0081] Table 2 Division method and weight table

[0082] Special vocabulary Word frequency Weight High frequency words >70 4 Medium and high frequency words 40～70 3 Medium frequency words 10～40 2 Low frequency words <10 1

[0083] After obtaining the hot word weights λ∈{1,2,3,4} of the proprietary vocabulary, the soft voting method can be adjusted to amplify the output probability of specific hot words in the proprietary vocabulary recognition model, that is:

[0084]

[0085] in, It represents the proprietary vocabulary recognition model h after hot word empowerment 1 For the class label q j The output probability, λ j Indicates that it belongs to the class label q j The hot word weight. j′ (x) represents the class label q j The weighted average probability after hot word weighting, h′(x) represents the set of final output probability values ​​of all class labels after hot word weighting, and the class label with the largest probability value is selected as the final output result y.

[0086] For example, suppose that for a certain train driver speech recognition task Q, the class labels include {“main segment”, “segment by segment”}. 1 The recognition results and the general vocabulary recognition model h 2 The recognition results are as follows:

[0087]

[0088] According to the traditional soft voting method, the prediction result after model fusion is:

[0089]

[0090] Obviously, “segment by segment” with a higher final output probability is selected as the final prediction result.

[0091] However, considering that train drivers use more proprietary vocabulary in driving scenarios, in order to shift the model integration towards the proprietary vocabulary recognition model and reduce the interference of general vocabulary on proprietary vocabulary, this application uses hot word weighting to give specific vocabulary higher parameter weights in specific scenarios. In the above recognition task, "main segment" is a medium-high frequency proprietary vocabulary that appears in the corpus, and its weight is 3; while "paragraph by paragraph" belongs to the general vocabulary, and its weight is 1. After hot word weighting, the proprietary vocabulary recognition model h 1 The output is:

[0092]

[0093] Therefore, the output probability of the model after hot word weighting is:

[0094]

[0095] Obviously, after hot word empowerment, the recognition effect of proprietary vocabulary is enhanced, and the final output result of the fusion model is "main diagnosis".

[0096] The present application also provides an application scenario, which applies the above-mentioned train driver voice recognition method. Specifically: the train driver voice recognition method provided in this embodiment can be applied in the train driver voice recognition scenario. The train driver voice recognition scenario includes a train driver voice recognition link and a voice recognition result display link; the train driver voice recognition method provided in this embodiment belongs to the train driver voice recognition link.

[0097] Based on the same inventive concept, the embodiment of the present application also provides a train driver voice recognition system for implementing the train driver voice recognition method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more train driver voice recognition system embodiments provided below can refer to the limitations of the train driver voice recognition method above, and will not be repeated here.

[0098] In an exemplary embodiment, Figure 6 As shown, a train driver speech recognition system is provided, comprising:

[0099] User interaction layer: The train driver inputs the original voice information into the system through the user interaction layer, which supports two voice information input modes: microphone and local wav file, to ensure that the train can take into account both the driver's on-the-way voice recognition and offline voice recognition. The user interaction layer interface in this application is as follows Figure 7 As shown, it includes an initialization interface and a speech recognition task interface.

[0100] Business logic layer: The voice information is passed to the voice denoising module through the business logic layer.

[0101] Basic service layer; the speech denoising module denoises the speech signal; the speech recognition module receives the output from the speech denoising module and recognizes the denoised speech information. After preprocessing and feature extraction, the denoised speech information is input into the proprietary vocabulary recognition model and the general vocabulary recognition model of the dual-model integration to perform the proprietary vocabulary recognition task and the general vocabulary recognition task respectively. After recognition and voting, the speech recognition result of the train driver is output, and the speech recognition result is displayed on the user interaction layer interface. The output result after speech recognition is recorded and saved as a recognition log.

[0102] This application uses Python language to implement the overall design and development of the system on the Pycharm platform. It is mainly divided into graphical interface design and business module design. The graphical interface design of the speech recognition system is mainly based on the PyQt framework. The business module can be divided into three modules: speech denoising, proprietary vocabulary recognition and general vocabulary recognition. The training and development of the proprietary syllable recognition network based on CNN-GRU-CTC is mainly implemented through the Keras deep learning framework, the training and development of the general vocabulary recognition network based on transfer learning is mainly implemented through PyTorch and Transformer machine learning libraries, and speech signal preprocessing and denoising are mainly implemented through the librosa library. The deep learning training system is Ubuntu 22.04.2LTS, and the processor is Xeon(R)w5-3423 CPU@2.10GHz×12, the graphics card used for training is NVIDIARTX6000AdaGeneration with 48GB video memory. The graphical user interface development system is Windows10, and the processor is Core(TM)i7-6700K CPU@4GHz×4.

[0103] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for train driver voice processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a train driver voice recognition method is implemented.

[0104] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0106] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0107] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0108] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0109] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0110] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A train driver speech recognition method, characterized in that: The train driver speech recognition method comprises: Get the train driver's voice information; Performing denoising and preprocessing on the voice information to obtain first voice information; Performing a fast Fourier transform on the first speech information to obtain a first frequency spectrum feature; Inputting the first spectrum feature into a proprietary vocabulary recognition model to obtain a proprietary vocabulary recognition result; the proprietary vocabulary recognition model includes a proprietary syllable recognition model and a syllable conversion model; the proprietary syllable recognition model is used to perform syllable recognition processing on the first spectrum feature to obtain a proprietary syllable recognition result; the syllable conversion model is used to perform syllable conversion processing on the proprietary syllable recognition result to obtain the proprietary vocabulary recognition result; Inputting the first spectrum feature into a general vocabulary recognition model to obtain a general vocabulary recognition result; The proprietary vocabulary recognition result and the general vocabulary recognition result are fused to obtain the train driver's speech recognition result.

2. The train driver speech recognition method according to claim 1, characterized in that: The determination process of the proprietary syllable recognition model specifically includes: Constructing a proprietary syllable training data set; each sample of the proprietary syllable training data set includes a second spectrum feature and a corresponding syllable label; the second spectrum feature is obtained by preprocessing and fast Fourier transforming the standard calling phrase of the train driver; the syllable label is obtained by labeling the second spectrum feature; Constructing a proprietary syllable recognition network; the proprietary syllable recognition network includes a plurality of convolutional layers, a gated recurrent unit, a fully connected layer, and a connection time series classifier connected in sequence; wherein the last two convolutional layers in the plurality of convolutional layers are atrous convolutional layers; Each sample of the proprietary syllable training data set is input into a proprietary syllable recognition network, and the parameters of the proprietary syllable recognition network are updated to obtain a proprietary syllable recognition model.

3. The train driver speech recognition method according to claim 1, characterized in that: The determination process of the general vocabulary recognition model specifically includes: Constructing a universal vocabulary recognition network; the universal vocabulary recognition network is constructed based on the Whisper framework; Constructing a general vocabulary training data set; each sample of the general vocabulary training data set includes a third spectral feature and a corresponding text label; the third spectral feature is obtained by preprocessing and fast Fourier transforming the Chinese speech data; the text label is obtained by annotating the third spectral feature; The samples in the general vocabulary training data set are input into the general vocabulary recognition network, the parameters of the general vocabulary recognition network are updated, and the general vocabulary recognition model is obtained.

4. The train driver speech recognition method according to claim 3, characterized in that: The universal vocabulary recognition network includes a preset layer; the samples in the universal vocabulary training data set are input into the universal vocabulary recognition network, the parameters of the universal vocabulary recognition network are updated, and the universal vocabulary recognition model is obtained, which specifically includes: Initializing the preset layer; the initialization is used to express the original pre-trained weight matrix W0 of the preset layer as the product of two trainable low-rank matrices A and B; Perform the following iterations on the samples in the general vocabulary training dataset: Inputting a current sample of the general vocabulary training data set into the general vocabulary recognition network corresponding to the current iteration number to obtain an initial general vocabulary recognition result corresponding to the current sample; the current sample of the general vocabulary training data set is a sample that has not been input into the general vocabulary recognition network corresponding to the current iteration number; the general vocabulary recognition network corresponding to the current iteration number is a general vocabulary recognition network initialized by a preset layer or a general vocabulary recognition network updated by the previous iteration number; Calculate the loss value between the initial common vocabulary recognition result corresponding to the current sample and the corresponding text label, update the parameters of the trainable low-rank matrices A and B through back-propagation, obtain the updated trainable low-rank matrices A and B, and combine the updated trainable low-rank matrices A and B with the original pre-trained weight matrix W0 to obtain the updated weight matrix W for the current number of iterations; Determine whether the loss value meets the stopping condition. If so, determine the universal vocabulary recognition model according to the weight matrix W updated after the current iteration number; if not, update the universal vocabulary recognition network corresponding to the current iteration number according to the weight matrix W updated after the current iteration number, and add 1 to the current iteration number, and return to "input the current sample of the universal vocabulary training data set into the universal vocabulary recognition network corresponding to the current iteration number, and obtain the initial universal vocabulary recognition result corresponding to the current sample".

5. The train driver speech recognition method according to claim 1, characterized in that: The syllable conversion model includes a pronunciation dictionary and an n-gram language model. The syllable conversion model performs syllable conversion processing on the proprietary syllable recognition result to obtain the proprietary vocabulary recognition result, which specifically includes: Using the pronunciation dictionary, converting the proprietary syllable recognition results into all corresponding text sequences; The probability values ​​of all text sequences are calculated using the n-gram language model to obtain the proprietary vocabulary recognition result.

6. The train driver speech recognition method according to claim 1, characterized in that: The special vocabulary recognition result and the general vocabulary recognition result are fused to obtain the train driver's speech recognition result, which specifically includes: Perform hot word weighting processing on the proprietary vocabulary recognition results to obtain the weighted proprietary vocabulary recognition results; Perform hot word weighting processing on the common vocabulary recognition results to obtain the weighted common vocabulary recognition results; Using a soft voting method, the weighted proprietary vocabulary recognition results and the weighted general vocabulary recognition results are fused and calculated to obtain a predicted speech recognition result; Based on the predicted speech recognition result, the speech recognition result of the train driver is obtained.

7. The train driver speech recognition method according to claim 1, characterized in that: The performing denoising and preprocessing on the voice information to obtain the first voice information specifically includes: According to the acquired running state of the current train, the denoising parameter value range in the Wiener filter is set accordingly, and the voice information is input into the Wiener filter after the corresponding setting to obtain the denoised voice information; the running state includes four states: default, uniform speed, deceleration or parking; The denoised voice information is preprocessed to obtain first voice information; the preprocessing includes pre-emphasis, framing and windowing.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the train driver speech recognition method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the train driver speech recognition method described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the train driver speech recognition method described in any one of claims 1 to 7 is implemented.

Images