A method, medium and device for cabin voice recognition based on small samples

Abstract

The present invention provides a cabin voice recognition method, medium and device based on small samples, which relate to the field of speech recognition technology. The method trains an initial speech recognition model based on a composite loss function to obtain a target speech recognition model, can optimize the model from multiple dimensions, and only needs to use a small amount of sample data to train the model. The composite loss function combines multiple loss functions, so that the model can be adjusted from multiple directions at each update, thereby accelerating the convergence process. In addition, the reinforcement learning loss function in the composite loss function is a loss function determined based on the model prediction distribution and character error rate, which can better handle phenomena such as swallowing or ligatures, so that the model generates a sequence that is closer to the actual output; the target cabin voice is input into the target speech recognition model to obtain the predicted text corresponding to the target cabin voice, which improves the speed of cabin voice recognition and ensures that the predicted text can accurately reflect the actual content of the cabin voice.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a method, medium and device for cabin voice recognition based on a small sample. Background Art

[0002] Speech recognition technology is crucial in pilot assessment. By using speech recognition technology to identify human voices in the cabin during flight, it is not only possible to determine whether the pilot has complied with operating procedures and standard call-out processes, but also to evaluate the pilot's speech clarity, providing an objective and systematic method to evaluate pilot performance and help improve flight safety and service quality. In existing technologies, using speech recognition technology to identify human voices in the cabin during flight mainly relies on a trained speech recognition model. The speech recognition model is usually trained using a single cross-entropy loss function. The audio containing human voices collected in the cabin is input into the trained speech recognition model to obtain the predicted text output by the speech recognition model. The predicted text is then compared with the standard text, and the pilot's performance is evaluated based on the comparison results.

[0003] However, the above method also has the following technical problems:

[0004] The cross-entropy loss function focuses primarily on classification accuracy at each time step, which can be understood as focusing only on matching local probability distributions. Therefore, a speech recognition model trained using a single cross-entropy loss function may overfit noise or detailed features in the training data with small sample sizes, resulting in insufficient generalization ability. Training the speech recognition model requires a large amount of sample data, and the gradient signal provided by the cross-entropy loss function comes only from the classification error at each time step, making it difficult for the model to converge quickly during training. Furthermore, the cross-entropy loss function aims to minimize the difference between the predicted distribution and the true distribution. Therefore, a speech recognition model trained using a single cross-entropy loss function tends to select candidate words with the highest probability in the context. This can be understood as inferring missing or ambiguous content based on contextual information. For example, when swallowed or ligatured words appear in audio, the speech recognition model may infer the most likely character or word based on the surrounding context. This may cause swallowed or ligatured words to be mistakenly recognized as other more common words, resulting in a discrepancy between the text output by the speech recognition model and the actual content, failing to accurately reflect the actual content of the audio. Summary of the Invention

[0005] In view of the above technical problems, the technical solution adopted by the present invention is:

[0006] According to a first aspect of the present invention, a method for cabin voice recognition based on a small sample is provided, the method comprising the following steps:

[0007] S1. Train the initial speech recognition model using a target sample dataset D to obtain a target speech recognition model, where D includes a plurality of target sample audios and the actual text corresponding to each target sample audio. The composite loss function L used to train the initial speech recognition model meets the following conditions:

[0008] L = α × L num +β×L cer +γ×L cross-entroy +μ×L ctc , L num is the length loss function, L cer is the reinforcement learning loss function, L cross-entroy is the cross entropy loss function, L ctc is the CTC loss function, α is L num The corresponding weight parameter, β is L cer The corresponding weight parameter, γ is L cross-entroy The corresponding weight parameter, μ is L ctc The corresponding weight parameter; L cer Meet the following conditions:

[0009] L cer =E x-D [E p(y1|x) [cer(y1, y)]], x is the target sample audio in D, y is the actual text corresponding to x, y1 is the text predicted by the target speech recognition model, E x-D [ ] indicates the expected value calculation for all x in D; E p(y1|x) [cer(y1, y)] means that under the condition of given input x, based on the probability distribution of the target speech recognition model output, the expected value of the character error rate between y1 and y is calculated; p(y1|x) is the probability that the target speech recognition model outputs y1 under the condition of given input x; cer(y1, y) is the character error rate between y1 and y.

[0010] S2. Input the target cabin sound into the target speech recognition model to obtain predicted text corresponding to the target cabin sound, where the predicted text is the text output by the target speech recognition model, and the target cabin sound is an audio clip containing human voice in the initial cabin sound, where the initial cabin sound is audio collected from the cabin of the target aircraft.

[0011] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the aforementioned method.

[0012] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the computer program.

[0013] The present invention has at least the following beneficial effects:

[0014] The present invention provides a cabin voice recognition method, medium and device based on small samples. The method trains an initial speech recognition model based on a composite loss function to obtain a target speech recognition model, and inputs the target cabin sound into the target speech recognition model to obtain a predicted text corresponding to the target cabin sound. The composite loss function is composed of four loss functions, namely, a length loss function, a reinforcement learning loss function, a cross entropy loss function and a CTC loss function, and weight parameters corresponding to the four loss functions. It can be seen that the present invention can optimize the model from multiple dimensions. This multi-objective joint optimization method can achieve higher data utilization in the case of small samples. It only needs to use a small amount of sample data to train the model, and does not need to use a large amount of sample data to train the model, thereby reducing dependence on large amounts of data, thereby improving the generalization ability of the model, and the composite loss function is combined Combining multiple loss functions provides richer and more diverse gradient signals, allowing the model to adjust in multiple directions at each update, thereby accelerating the convergence process. In addition, compared with using a single cross-entropy loss function, the composite loss function introduces other types of losses, which can provide additional constraints for the model and prevent the model from relying entirely on context for prediction. Moreover, the reinforcement learning loss function in the composite loss function is a loss function determined based on the model's predicted distribution and character error rate. By minimizing the character error rate, it can better handle phenomena such as swallowing or hyphenating words, allowing the model to generate sequences that are closer to the actual output, reducing errors caused by context speculation, and making the text output by the model closer to the actual content in the audio. This not only significantly improves the speed of cabin voice recognition, but also ensures that the predicted text can accurately reflect the actual content of the cabin sound. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a method for cabin voice recognition based on a small sample size is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar tasks and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] An embodiment of the present invention provides a method for cabin voice recognition based on a small sample, the method comprising the following steps: Figure 1 As shown:

[0020] S1. Train the initial speech recognition model using a target sample dataset D to obtain a target speech recognition model, where D includes a plurality of target sample audios and the actual text corresponding to each target sample audio. The composite loss function L used to train the initial speech recognition model meets the following conditions:

[0021] L = α × L num +β×L cer +γ×L cross-entroy +μ×L ctc , where L num is the length loss function, L cer is the reinforcement learning loss function, L cross-entroy is the cross entropy loss function, L ctc is the CTC loss function, α is L num The corresponding weight parameter, β is L cer The corresponding weight parameter, γ is L cross-entroy The corresponding weight parameter, μ is L ctc The corresponding weight parameters are parameters used to adjust the degree of influence of the corresponding loss components on the total loss. Those skilled in the art know that the specific values ​​of α, β, γ and μ are set by those skilled in the art according to actual needs and will not be repeated here.

[0022] Specifically, the length loss function is a loss function that can measure the difference between the sequence length predicted by the model and the actual sequence length, such as: mean square error, mean absolute error; in an embodiment of the present invention, the length loss function can be understood as a loss function that can measure the difference between the text length predicted by the initial speech recognition model and the actual text length.

[0023] Specifically, L cer Meet the following conditions:

[0024] L cer =E x-D [E p(y1|x) [cer(y1, y)]], x is the target sample audio in D, y is the actual text corresponding to x, and y1 is the text predicted by the target speech recognition model.

[0025] Concrete, E x-D [ ] indicates the expected value calculation for all x in D; E p(y1|x) [cer(y1, y)] means that under the condition of given input x, according to the probability distribution of the target speech recognition model output, the expected value of the character error rate between y1 and y is calculated; p(y1|x) is the probability that the target speech recognition model outputs y1 under the condition of given input x; cer(y1, y) is the character error rate between y1 and y; E x-D [E p(y1|x) [cer(y1, y)]] can be understood as indicating that, given D, the expected value of the character error rate is calculated through the prediction distribution of the target speech recognition model; that is, E x-D [E p(y1|x) [cer(y1,y)]] meets the following conditions:

[0026] E x-D [E p(y1|x) [cer(y1,y)]]=1 / n×∑ n j=1 E p(y1|xj) [cer(y1, yj)], xj is the jth target sample audio in D, j ranges from 1 to n, n is the number of target sample audios in D, and yj is the actual text corresponding to xj;

[0027] E p(y1|xj) [cer(y1,yj)]=∑ yj (p(y1|xj)×cer(y1,yj)), cer(y1,yj) is the character error rate between y1 and yj.

[0028] Specifically, in D, the target sample audio is the input feature, and the actual text corresponding to the target sample audio is the label corresponding to the input feature.

[0029] Specifically, the actual text corresponding to the target sample audio can be understood as a written representation of the content in the target sample audio.

[0030] S2. Input the target cabin sound into the target speech recognition model to obtain the predicted text corresponding to the target cabin sound. The predicted text is the text output by the target speech recognition model, which can be understood as the character sequence output by the target speech recognition model. The target cabin sound is an audio segment containing human voice in the initial cabin sound. The initial cabin sound is audio collected from the cabin of the target aircraft, and the audio contains human voice and background noise. Among them, those skilled in the art know that any method of obtaining an audio segment containing human voice in audio in the prior art falls within the scope of protection of the present invention and will not be repeated here.

[0031] Through the above steps, the initial speech recognition model is trained based on the composite loss function. The composite loss function is composed of four loss functions, namely, length loss function, reinforcement learning loss function, cross entropy loss function and CTC loss function, and the weight parameters corresponding to these four loss functions. The model can be optimized from multiple dimensions. This multi-objective joint optimization method can achieve higher data utilization in the case of small samples. Only a small amount of sample data is needed to train the model, and there is no need to use a large amount of sample data to train the model, reducing dependence on large amounts of data, thereby improving the generalization ability of the model. In addition, the composite loss function provides richer and more diverse gradient signals by combining multiple loss functions, so that the model can be adjusted from multiple directions each time it is updated, thereby accelerating Convergence process. In addition, compared with the use of a single cross-entropy loss function, the composite loss function introduces other types of losses, which can provide additional constraints for the model and avoid the model from relying entirely on the context for prediction. Moreover, the reinforcement learning loss function in the composite loss function is a loss function determined based on the model prediction distribution and the character error rate. By minimizing the character error rate, it can better handle phenomena such as swallowing or ligatures, so that the model generates sequences that are closer to the actual output, reducing the error caused by context speculation, so that the text output by the model is closer to the actual content in the audio, which not only significantly improves the speed of cabin voice recognition, but also ensures that the predicted text can accurately reflect the actual content in the cabin voice, avoiding non-standard pronunciations in the cabin voice (such as swallowing or ligatures) being mistakenly recognized as correct text.

[0032] Specifically, the background noise includes engine noise, wind noise, rain noise, thunder noise, etc., wherein the target aircraft is an aircraft predetermined by those skilled in the art and will not be described in detail here.

[0033] In a specific embodiment, the initial cabin sound is the clearest channel of audio selected from the multi-channel audio collected in the cabin of the target aircraft, and the audio includes human voice and background noise. Among them, those skilled in the art know that any method of selecting the clearest channel of audio from multi-channel audio in the prior art falls within the scope of protection of the present invention, for example: by evaluating the signal-to-noise ratio and speech clarity of each channel, the clearest channel of audio is selected, which will not be repeated here.

[0034] Specifically, the target cabin sound also includes background noise. It can be understood that the target cabin sound is an audio segment containing human voice intercepted from the initial cabin sound.

[0035] In a specific embodiment, the target speech recognition model includes: an audio feature fusion module, an encoding module, a word count recognition module and a decoding module, wherein the encoding module includes several CNN layers and several encoders, the word count recognition module includes a convolutional layer and a fully connected neural network, and the decoding module includes several non-autoregressive decoders.

[0036] Specifically, the encoder in the encoding module is the encoder in the Transformer architecture.

[0037] Specifically, the word count recognition module can be understood as a neural network model specially trained for the word count recognition task.

[0038] Specifically, in the process of the target speech recognition model generating the predicted text corresponding to the target cabin sound, it is necessary to perform feature extraction on the target cabin sound to obtain the Fbank feature of the target cabin sound, and input the Fbank feature of the target cabin sound into the audio feature fusion module to obtain the intermediate audio feature corresponding to the target cabin sound. The audio feature fusion module is used to splice the continuous Fbank feature vectors of each a frames in the Fbank feature into a feature vector, and the splicing is non-overlapping (that is, a frames are moved after each splicing), where a is the preset number of splicing frames; for example: if the Fbank feature includes 100 frames of Fbank feature vectors, the feature dimension of each frame of Fbank feature vector is 80 dimensions, and a is 5, then every 5 frames of continuous Fbank feature vectors are spliced ​​into a 400-dimensional feature vector in a non-overlapping manner, and a total of 20 400-dimensional feature vectors are obtained, that is, the intermediate audio feature is composed of these 20 400-dimensional feature vectors.

[0039] In a specific embodiment, after obtaining the Fbank features of the target cabin sound, before inputting the Fbank features of the target cabin sound into the audio feature fusion module to obtain the intermediate audio features corresponding to the target cabin sound, it also includes: normalizing the Fbank features based on the cepstral coefficients so that the Fbank features of the target cabin sound are updated.

[0040] Through the above steps, each Fbank feature vector usually represents the spectral features within a short time window (such as 10 milliseconds). By splicing continuous Fbank feature vectors together, dependencies on longer time scales can be captured without losing key information. The spliced ​​feature vector not only contains the information of a single time point, but also integrates the contextual information of adjacent time points. Since adjacent Fbank feature vectors usually have high similarity, the splicing operation can effectively reduce these redundant information without significantly affecting the understanding of the content in the audio. Therefore, the audio feature fusion module splices the continuous feature vectors of each a frame in the Fbank feature into a feature vector to obtain intermediate audio features, reducing the amount of data for subsequent processing and reducing the computational complexity, thereby speeding up the processing speed without significantly affecting the understanding of the content in the audio.

[0041] Optional, a=5.

[0042] Specifically, the intermediate audio features corresponding to the target cabin sound are input into the encoding module, first processed by several CNN layers and then by several encoders to obtain the feature vector matrix corresponding to the target cabin sound; it can be understood as: first extracting local features through several CNN layers, then performing global feature modeling through several encoders, and finally obtaining the feature vector matrix.

[0043] Through the above steps, the CNN layer can efficiently extract local features from the input data, such as short-term features such as phonemes and tones. At the same time, the computational complexity is greatly reduced through parameter sharing and parallel computing. Compared with using a coding module that only includes several encoders to obtain a feature vector matrix, using a coding module that includes several CNN layers and several encoders, first extracting local features through several CNN layers, and then performing global feature modeling through several encoders, and finally obtaining a feature vector matrix, is conducive to improving the efficiency of obtaining the feature vector matrix.

[0044] Specifically, the feature vector matrix corresponding to the target cabin sound is input into the word count recognition module to obtain the number of characters corresponding to the target cabin sound. The word count recognition module predicts the length of the corresponding text, that is, the number of characters, by analyzing the information in the feature vector matrix.

[0045] Specifically, the eigenvector matrix corresponding to the target cabin sound and the number of characters corresponding to the target cabin sound are input into the decoding module to obtain the predicted text corresponding to the target cabin sound. Obtaining the predicted text based on the eigenvector matrix and the number of characters can better control the length of the output text and avoid generating text that is too long or too short, which is conducive to improving the accuracy of the predicted text. It can also avoid repeated adjustments or erroneous generation caused by uncertain text length during the decoding process, further improving the decoding efficiency.

[0046] Specifically, most existing speech recognition models are based on the Transformer architecture, which includes an encoding component and a decoding component. The encoding component includes several encoders for converting the input speech signal into feature representations, and the decoding component includes several autoregressive decoders for converting the feature representations into text outputs. Transformer architecture models can effectively capture contextual information. In particular, when processing audio containing non-standard pronunciations (such as swallowed words or ligatures), they may tend to make inferences based on the context, which may cause the output text to differ from the actual content, which may affect the accuracy of the evaluation. In addition, the autoregressive decoder works by generating output text word by word, that is, predicting a word or character at each step and using it as input for the next prediction. This process requires waiting for the previous word to be generated before starting the prediction of the next word, and cannot be processed in parallel, resulting in a slower overall decoding speed and, in turn, a slower speech recognition speed.

[0047] Through the above steps, the target cabin sound is input into the target speech recognition model to obtain the predicted text corresponding to the target cabin sound, wherein the target speech recognition model includes: an audio feature fusion module, an encoding module, a word count recognition module and a decoding module, the encoding module includes several CNN layers and several encoders, the word count recognition module includes a convolutional layer and a fully connected neural network, and the decoding module includes several non-autoregressive decoders; in the process of the target speech recognition model generating the predicted text corresponding to the target cabin sound, the Fbank feature of the target cabin sound is input into the audio feature fusion module to obtain the intermediate audio feature corresponding to the target cabin sound, and the audio feature fusion module is used to combine the Fbank The continuous Fbank feature vectors of each a frames in the feature are spliced ​​into a feature vector, and there is no overlap during splicing. a is the preset number of splicing frames, which reduces the amount of data for subsequent processing, reduces the computational complexity, and speeds up the processing speed. The non-autoregressive decoder allows parallel generation of output text, generating the entire output sequence at one time rather than generating it word by word, reducing context-based speculation, ensuring that the output text is closer to the actual content, and greatly improving the decoding speed. It not only significantly improves the speed of cabin sound recognition, but also ensures that the predicted text can accurately reflect the actual content in the cabin sound, avoiding the misidentification of non-standard pronunciations in the cabin sound (such as swallowed words or ligatures) as correct text.

[0048] In a specific embodiment, the following steps are also included to obtain D:

[0049] S01, based on the initial sample data set Y, obtain the initial noise audio data set B = {B1, B2, ..., B e ,……,B f}, B e H e The corresponding initial noise audio data list, He is the e-th preset aircraft type in the preset aircraft type list H, H={H1, H2, ..., H e ,……,H f}, the value of e ranges from 1 to f, where f is the number of preset aircraft types, wherein Y includes several initial sample audios corresponding to each preset aircraft type and the actual text corresponding to each initial sample audio, and the initial noise audio data list includes several initial noise audios and the external environment type corresponding to each initial noise audio.

[0050] Specifically, the initial sample audio is audio collected from the cabin of an aircraft of its corresponding preset aircraft type, and the audio includes human voice and background noise. The preset aircraft type is an aircraft type predetermined by those skilled in the art, such as Boeing 737 and Airbus A320, and will not be further described here.

[0051] Specifically, the actual text corresponding to the initial sample audio can be understood as a written representation of the content in the initial sample audio.

[0052] Specifically, the initial noise audio is audio separated from the initial sample audio and containing only background noise.

[0053] Specifically, the external environment type of the initial noise audio is the external environment type of the aircraft when the corresponding initial sample audio is collected, such as strong wind, heavy rain, thunder and lightning, hail, etc.

[0054] Specifically, in Y, the initial sample audio is the input feature, and the actual text corresponding to the initial sample audio is the label corresponding to the input feature.

[0055] S02, for B1, B2, ..., B e ,……,B f The external environment types corresponding to all initial noise audios in the _ are deduplicated to obtain a key environment type list C = {C1, C2, ..., C r ,……,C s}, C r is the rth key environment type, r ranges from 1 to s, and s is the number of key environment types.

[0056] S03, if B e The external environment type corresponding to all initial noise audio in C r If they are different, then C r As H e The corresponding target environment type.

[0057] S04, H e Corresponding aircraft feature list G 0e 、H e The corresponding key noise audio G 1 e and H e The corresponding target environment type is input into the intermediate noise audio generation model to obtain H e Corresponding intermediate noise audio, and constructing an intermediate noise audio list, the intermediate noise audio list includes several intermediate noise audio corresponding to several preset aircraft types. Those skilled in the art know that the number of intermediate noise audio in the intermediate noise audio list can be set by those skilled in the art according to actual needs, and will not be repeated here.

[0058] Specifically, G 0 e ={G 0 e1 , G 0 e2 ,……,G 0 er ,……,G 0 es}, G 0 er H e Corresponding aircraft K r The corresponding eigenvalue, K r is the rth noise-related feature in the noise-related feature list K, K={K1, K2, ..., K r ,……,K s}, r ranges from 1 to s, where s is the number of noise-related features. Noise-related features are noise-related features of the aircraft, such as the shape of the aircraft cabin, the size of the aircraft, the type of the aircraft engine, the operating characteristics of the aircraft engine, and the sound insulation materials used in the aircraft.

[0059] Specifically, G 1 e B e Any initial noise audio in .

[0060] Specifically, based on H e Corresponding aircraft feature list G 0 e 、H e The corresponding key noise audio G 1 e and H e H obtained by the corresponding target environment type e The external environment type of the corresponding intermediate noise audio is the same as the target environment type.

[0061] S05. Fuse any one of the intermediate human voice audios in the intermediate human voice audio list with any one of the intermediate noise audios in the intermediate noise audio list to obtain an incremental sample audio, and use the actual text corresponding to the intermediate human voice audio as the actual text corresponding to the incremental sample audio to construct an incremental sample data set Z, wherein Z includes several incremental sample audios and the actual text corresponding to each incremental sample audio, and the intermediate human voice audio list includes several intermediate human voice audios; it can be understood as: repeating step S05, gradually generating multiple incremental sample audios and their corresponding actual texts, and finally forming Z.

[0062] Specifically, the intermediate human voice audio is audio that is separated from the initial sample audio and contains only human voice.

[0063] In a specific embodiment, the intermediate human voice audio is an audio that only contains human voice and is preset by those skilled in the art.

[0064] S06. Take the union of Z and Y as D.

[0065] Specifically, the intermediate noise audio generation model is a model obtained by training a neural network model using an intermediate sample data set.

[0066] Specifically, the intermediate sample data set includes several intermediate sample data, including G 0 e , G 1 e , G 1 e Corresponding intermediate environment type G 2 e and G 2 e The corresponding specified noise audio, where G 2 e For B e Among all the external environment types in G 1 e The corresponding external environment types are different; G 2 e The corresponding specified noise audio is B e Among all the initial noise audios in , any one of the corresponding external environment types is the same as G 2 e Same initial noise audio.

[0067] Specifically, in the intermediate sample data, G 0 e , G 1 e and G 2 e is the input feature, G2 e The corresponding specified noise audio is the label of the input feature.

[0068] Specifically, data augmentation involves performing a series of specific transformations and expansions on the original data to generate new samples, thereby increasing the diversity and quantity of training data. The goal is to improve the model's generalization capabilities and enable the model to better adapt to unknown input data. In the field of speech recognition technology, commonly used data augmentation methods include adding noise to the original speech signal and changing the playback speed to generate more diverse training samples. However, when these methods are applied to cabin vocal data augmentation, simple operations such as adding noise and changing the playback speed cannot accurately simulate the complex noise characteristics of the cabin of different aircraft types and the impact of the external environment on internal sounds. Therefore, the training samples generated by these data augmentation methods may not be accurate enough to truly reflect the actual situation. Using these data augmentation methods to train speech recognition models will affect the robustness and performance of the speech recognition models.

[0069] Through the above steps, an initial noise audio dataset is obtained based on the initial sample dataset, a target environment type is obtained based on the initial noise audio dataset and the key environment type, and an aircraft feature list corresponding to the preset aircraft type, a key noise audio corresponding to the preset aircraft type, and a target environment type corresponding to the preset aircraft type are input into an intermediate noise audio generation model to obtain intermediate noise audio corresponding to the preset aircraft type and construct an intermediate noise audio list, wherein the intermediate noise audio list includes several intermediate noise audios corresponding to several preset aircraft types, and any intermediate human voice audio in the intermediate human voice audio list and any intermediate noise audio in the intermediate noise audio list are fused to obtain an incremental sample audio, and the actual text corresponding to the intermediate human voice audio is used as the incremental sample audio pair. The actual text corresponding to the incremental sample dataset is used to construct the incremental sample dataset, and the union of the incremental sample dataset and the initial sample dataset is used as the target sample dataset to achieve cabin voice data enhancement. The aircraft characteristics corresponding to different aircraft types and the impact of the external environment on the internal sound are taken into account, so that the generated incremental sample audio is more accurate and can truly reflect the actual situation. Therefore, the target sample dataset is used to train the initial speech recognition model to obtain the target speech recognition model, which is conducive to improving the robustness of the target speech recognition model, and the elements in the intermediate human voice audio list and the intermediate noise audio list can be flexibly adjusted according to different needs to generate incremental sample audio that meets specific requirements. It also improves the diversity of the data, helps to reduce the risk of overfitting of the target speech recognition model, and improves the generalization ability of the target speech recognition model.

[0070] In a specific embodiment, the following steps are further included after step S05:

[0071] S001. Input the aircraft feature list P corresponding to the target aircraft type, the associated aircraft feature list Q corresponding to P, and the associated noise audio corresponding to Q into a designated noise audio generation model to obtain the designated noise audio corresponding to the target aircraft type, and construct a designated noise audio list. The designated noise audio list includes several designated noise audios corresponding to several target aircraft types. Those skilled in the art know that the number of designated noise audios in the designated noise audio list can be set by those skilled in the art according to actual needs, and will not be repeated here.

[0072] Specifically, the target aircraft type is an aircraft type that is different from all preset aircraft types.

[0073] Specifically, P={P1, P2, ..., P r ,……,P s}, P r K is the aircraft type corresponding to the target aircraft r The corresponding eigenvalues.

[0074] Specifically, Q is G 0 1, G 0 2, ..., G 0 e ,……,G 0 f The aircraft feature list with the greatest feature similarity between and P, wherein those skilled in the art know that any method of obtaining the feature similarity between two feature lists in the prior art falls within the protection scope of the present invention, for example: respectively performing vectorization processing on the two feature lists to obtain two feature vectors, and taking the vector similarity between the two feature vectors as the feature similarity between the two feature lists, which will not be repeated here.

[0075] Specifically, the greater the feature similarity, the more similar the corresponding two aircraft feature lists are.

[0076] Specifically, the associated noise audio corresponding to Q is any initial noise audio in the initial noise audio data list corresponding to Q.

[0077] S002. Fuse any one of the intermediate human voice audios in the intermediate human voice audio list with any one of the specified noise audios in the specified noise audio list to obtain an expanded sample audio, and use the actual text corresponding to the intermediate human voice audio as the actual text corresponding to the expanded sample audio to construct an expanded sample data set T, where T includes several expanded sample audios and the actual text corresponding to each expanded sample audio; it can be understood as: repeating step S002 to gradually generate multiple expanded sample audios and their corresponding actual texts, and finally forming T.

[0078] S003. Take the union of T and Y as D.

[0079] In a specific embodiment, the following steps are included after step S002:

[0080] S0021. Let D be the union of Z, T, and Y.

[0081] Specifically, the specified noise audio generation model is a model obtained by training a neural network model using a specified sample data set.

[0082] Specifically, the specified sample data set includes several specified sample data, and the specified sample data includes G 0 e , G 0 e Corresponding associated aircraft feature list V 0 e 、V 0 e The corresponding first correlated noise audio V 1 e and V 1 e The corresponding second correlated noise audio, where V 0 e G 0 1, G 0 2, ..., G 0 e-1 , G 0 e+1 , G 0 e+2 ,……,G 0 f Zhong and G 0 e The list of aircraft features with the greatest feature similarity between them, G 0 e-1 H e-1 Corresponding aircraft feature list, H e-1 The e-1th preset aircraft type in H, G 0 e+1 H e+1Corresponding aircraft feature list, H e+1 is the e+1th preset aircraft type in H, G 0 e+2 H e+2 Corresponding aircraft feature list, H e+2 is the e+2th preset aircraft type in H, V 1 e V 0 e Any initial noise audio in the corresponding initial noise audio list, V 1 e The corresponding second correlated noise audio is B e All initial noise in the audio frequency is related to V 1 e The initial noise audio with the greatest audio feature similarity between the two audios, wherein those skilled in the art know that any method of obtaining the audio feature similarity between two audios in the prior art falls within the protection scope of the present invention, for example: respectively obtaining the feature vectors of the two audios, and taking the similarity between the two feature vectors as the audio feature similarity between the two audios, which will not be repeated here.

[0083] Through the above steps, based on the feature similarity between the aircraft feature lists, the associated aircraft feature list corresponding to the preset aircraft type is determined, the audio feature similarity between the initial noise audio corresponding to the preset aircraft type and the first associated noise audio corresponding to the associated aircraft feature list is determined, the second associated noise audio corresponding to the first associated noise audio is determined, and based on the preset aircraft feature list, the associated aircraft feature list corresponding to the preset aircraft feature list, the first associated noise audio corresponding to the associated aircraft feature list, and the second associated noise audio corresponding to the first associated noise audio, designated sample data is constructed to obtain a designated sample data set, and the designated sample data set is used to train a neural network model to obtain a designated noise audio generation model, so that the designated noise audio generated by the designated noise audio generation model can be more accurate and can truly reflect the actual situation.

[0084] Specifically, the greater the audio feature similarity, the more similar the corresponding initial noise audio and the first associated noise audio are.

[0085] Specifically, in the specified sample data, G 0 e 、V 0 e and V 1 e is the input feature, V 1 e The corresponding second associated noise audio is the label of the input feature.

[0086] Through the above steps, based on the feature similarity between the aircraft feature list corresponding to the target aircraft type and the aircraft feature list corresponding to the preset aircraft type, the associated aircraft feature list corresponding to the aircraft feature list corresponding to the target aircraft type is determined, the aircraft feature list corresponding to the target aircraft type, the associated aircraft feature list corresponding to the aircraft feature list and the associated noise audio corresponding to the associated aircraft feature list are input into the specified noise audio generation model to obtain the specified noise audio corresponding to the target aircraft type, and a specified noise audio list is constructed, the specified noise audio list includes several specified noise audios corresponding to several target aircraft types, any one of the intermediate human voice audios in the intermediate human voice audio list and any one of the specified noise audios in the specified noise audio list are fused to obtain an expanded sample audio, and the actual text corresponding to the intermediate human voice audio is used as the expanded sample audio. The actual text corresponding to the sample audio is used to construct an expanded sample dataset, and the union of the expanded sample dataset and the initial sample dataset is used as the target sample dataset to achieve cabin voice data enhancement. The impact of aircraft characteristics corresponding to different aircraft types on internal sounds is taken into account, so that the generated expanded sample audio is more accurate and can truly reflect the actual situation. Moreover, for target aircraft types that are different from any preset aircraft types, effective noise simulation can be performed based on existing knowledge to generate their corresponding specified noise audio. Furthermore, the corresponding expanded sample audio is obtained, which improves the diversity of the data. Therefore, using the target sample dataset to train the initial speech recognition model to obtain the target speech recognition model is conducive to improving the robustness of the target speech recognition model, and also helps to reduce the risk of overfitting of the target speech recognition model, and can improve the generalization ability of the target speech recognition model.

[0087] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store a computer program related to a method in the method embodiment. The computer program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0088] An embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the above embodiment when executing the computer program.

[0089] An embodiment of the present invention further provides a computer program product comprising program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0090] The present invention provides a cabin voice recognition method, medium and device based on small samples. The method trains an initial speech recognition model based on a composite loss function to obtain a target speech recognition model, and inputs the target cabin sound into the target speech recognition model to obtain a predicted text corresponding to the target cabin sound. The composite loss function is composed of four loss functions, namely, a length loss function, a reinforcement learning loss function, a cross entropy loss function and a CTC loss function, and weight parameters corresponding to the four loss functions. It can be seen that the present invention can optimize the model from multiple dimensions. This multi-objective joint optimization method can achieve higher data utilization in the case of small samples. It only needs to use a small amount of sample data to train the model, and there is no need to use a large amount of sample data to train the model, thereby reducing dependence on large amounts of data, thereby improving the generalization ability of the model, and the composite loss function provides a richer and diversified gradient signals, so that the model can adjust from multiple directions at each update, thereby accelerating the convergence process. In addition, compared with the use of a single cross-entropy loss function, the composite loss function introduces other types of losses, which can provide additional constraints for the model and prevent the model from relying entirely on the context for prediction. Moreover, the reinforcement learning loss function in the composite loss function is a loss function determined based on the model's predicted distribution and the character error rate. By minimizing the character error rate, it can better handle phenomena such as swallowing or ligatures, allowing the model to generate sequences closer to the actual output, reducing the error caused by context speculation, and making the text output by the model closer to the actual content in the audio. This not only significantly improves the speed of cabin voice recognition, but also ensures that the predicted text can accurately reflect the actual content in the cabin sound, avoiding misidentification of non-standard pronunciations in the cabin sound (such as swallowing or ligatures) as correct text.

[0091] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A cabin voice recognition method based on small samples, characterized in that: The method comprises the following steps: S1. Train the initial speech recognition model using a target sample dataset D to obtain a target speech recognition model, where D includes a plurality of target sample audios and the actual text corresponding to each target sample audio. The composite loss function L used to train the initial speech recognition model meets the following conditions: L = α × L num +β×L cer +γ×L cross-entroy +μ×L ctc , L num is the length loss function, L cer is the reinforcement learning loss function, L cross-entroy is the cross entropy loss function, L ctc is the CTC loss function, α is L num The corresponding weight parameter, β is L cer The corresponding weight parameter, γ is L cross-entroy The corresponding weight parameter, μ is L ctc The corresponding weight parameter; L cer Meet the following conditions: L cer =E x-D [E p(y1|x) [cer(y1, y)]], x is the target sample audio in D, y is the actual text corresponding to x, y1 is the text predicted by the target speech recognition model, E x-D [ ] indicates the expected value calculation for all x in D; E p(y1|x) [cer(y1, y)] means that under the condition of given input x, according to the probability distribution of the target speech recognition model output, the expected value of the character error rate between y1 and y is calculated; p(y1|x) is the probability that the target speech recognition model outputs y1 under the condition of given input x; cer(y1, y) is the character error rate between y1 and y; S2. Input the target cabin sound into the target speech recognition model to obtain the predicted text corresponding to the target cabin sound, where the predicted text is the text output by the target speech recognition model, and the target cabin sound is an audio clip containing human voice in the initial cabin sound, and the initial cabin sound is audio collected from the cabin of the target aircraft; wherein the target speech recognition model includes: an audio feature fusion module, an encoding module, a word count recognition module and a decoding module, wherein the encoding module includes several CNN layers and several encoders, the word count recognition module includes a convolutional layer and a fully connected neural network, and the decoding module includes several non-autoregressive decoders.

2. The cabin voice recognition method based on small samples according to claim 1, characterized in that: The length loss function is a loss function that measures the difference between the sequence length predicted by the model and the actual sequence length.

3. The cabin voice recognition method based on small samples according to claim 1, characterized in that: In D, the target sample audio is the input feature, and the actual text corresponding to the target sample audio is the label corresponding to the input feature.

4. The cabin voice recognition method based on small samples according to claim 1, characterized in that: The encoder in the encoding module is the encoder in the Transformer architecture.

5. The cabin voice recognition method based on small samples according to claim 1, characterized in that: The intermediate audio features corresponding to the target cabin sound are input into the encoding module, first processed by several CNN layers and then by several encoders to obtain the feature vector matrix corresponding to the target cabin sound.

6. The cabin voice recognition method based on small samples according to claim 5, characterized in that: The feature vector matrix corresponding to the target cabin sound is input into the word count recognition module to obtain the number of characters corresponding to the target cabin sound; the feature vector matrix corresponding to the target cabin sound and the number of characters corresponding to the target cabin sound are input into the decoding module to obtain the predicted text corresponding to the target cabin sound.

7. A non-transitory computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the cabin voice recognition method based on a small sample as described in any one of claims 1 to 6.

8. An electronic device comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for cabin voice recognition based on a small sample size as described in any one of claims 1 to 6 is implemented.

Images