Regional spoken language intelligent translation method based on deep learning

Abstract

The invention relates to the technical field of language processing, in particular to a regional spoken language intelligent translation method based on deep learning, which comprises the following steps: selecting a target region, collecting spoken language audio in the target region, cleaning the collected spoken language audio, labeling translations on the cleaned spoken language audio, and creating a spoken language translation library, placing the cleaned spoken language audio marked with the translation in a spoken language translation library for storage; traversing the spoken language translation library, and performing noise reduction processing on each spoken language audio in the spoken language translation library; a Transform architecture model is selected, and spoken language audios stored in the spoken language translation library subjected to cleaning, labeling and noise reduction are imported into the model to be trained; judging the availability of the model after the model training is finished; according to the method, the spoken language audio is subjected to cleaning, noise reduction and labeling, and the spoken language audio is imported into the Transform architecture model to be trained, so that a continuously optimized regional spoken language translation method is realized, and an auxiliary effect is provided for regional spoken language translation work.

Description

Technical Field

[0001] The present invention relates to the field of language processing technology, and in particular to a regional spoken language intelligent translation method based on deep learning. Background Art

[0002] There are certain regional differences in spoken English between China's current standardized spoken English and interactions with native English speakers, making it difficult for users on both sides to understand each other in conversation scenarios.

[0003] The invention patent with application number 202211439914.9 discloses an English translation system based on machine learning, including: a database for storing a translation set, the translation set includes a mapping set of English-standard language-common language, a speech receiving module for receiving the speech of the user and performing noise reduction processing on the speech; a translation module for translating between English and the standard language; a selection module for selecting the common language corresponding to the standard Chinese language in the database; an output module for outputting the common language; a learning module for receiving the user's language, learning the user's common language habits, establishing a corresponding relationship with the standard language, and storing it in the database.

[0004] The application aims to solve the problem that "current machine translation systems are diverse, including many English-Chinese systems, but the translated language is relatively stiff and cannot be translated according to the user's common voice habits and speech, and cannot meet current practical translation requirements."

[0005] However, current regional spoken language translation technologies are mostly based on fixed databases, and their translation capabilities are capped. As regional spoken languages ​​are likely to evolve over time, existing translation technologies are gradually becoming ineffective.

[0006] To this end, a regional spoken language intelligent translation method based on deep learning was proposed. Summary of the Invention

[0007] In response to the above-mentioned shortcomings of the existing technology, the present invention provides a regional spoken language intelligent translation method based on deep learning, which solves the technical problems raised in the above-mentioned background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] The regional spoken language intelligent translation method based on deep learning includes:

[0010] Select a target area, collect spoken audio in the target area, clean the collected spoken audio, annotate the cleaned spoken audio with translation, create a spoken translation library, and store the cleaned and annotated spoken audio in the spoken translation library; traverse the spoken translation library and perform noise reduction on each spoken audio in the spoken translation library;

[0011] The spoken audio noise reduction processing logic is expressed as:

[0012]

[0013] Where: is the spoken audio after noise reduction; U is the total number of spoken audio frames; IFFT[·] is the inverse fast Fourier transform operation; FFT[·] is the fast Fourier transform operation; ξ v (k) is the prior signal-to-noise ratio; is the estimated value of noise power spectral density; is the power spectral density; y v (n) is the noisy frequency signal of the vth frame; δ(·) is the discrete impulse function used to splice the frames according to the frame shift rule;

[0014] In the discrete impulse function δ(·), n represents the discrete time point index, M represents the frame shift size during framing, and the unit is the number of discrete time points;

[0015] The prior signal-to-noise ratio ξ v (k) is calculated by the following formula:

[0016]

[0017] Where: α is the smoothing factor; The frequency domain representation of the clean speech signal estimated for the previous frame; Perform fast Fourier transform on the previous audio frame to obtain the corresponding power spectrum density in the frequency domain;

[0018] Where, the smoothing factor is 0≤α≤1, and max(·) means taking the maximum value in the brackets;

[0019] Spoken audio noise reduction effectively improves the clarity of spoken audio, providing more accurate data for training Transformer architecture models.

[0020] Select the Transformer architecture model, and import the spoken audio stored in the oral translation library that has completed cleaning, annotation, and noise reduction into the model for training; after the model training is completed, determine the availability of the model; if the model availability determination result is no, jump to the model training stage; if the model availability determination result is yes, receive the spoken audio, import the spoken audio into the model for processing, and output the translation result of the spoken audio through the model; set the Transformer architecture model update cycle, and update the Transformer architecture model in real time based on the update cycle.

[0021] Furthermore, the target area is customized by the user end. When collecting spoken audio in the target area, any number of users are selected in the target area, and the daily conversation audios of the selected users are collected and recorded as spoken audio.

[0022] When performing the storage operation on the spoken audio marked with the translation, the spoken audio is segmented based on the corresponding translation of the spoken audio, so that the spoken audio is segmented into multiple audio segments, each audio segment corresponding to a single word or phrase in the translation;

[0023] When performing the cleaning operation on the spoken audio, the cleaning logic is applied to continuously clean the spoken audio so that each spoken audio on which the translation is annotated is different.

[0024] Furthermore, the cleaning logic is expressed as:

[0025]

[0026] Where: S1 is the similarity of the time domain features of the two spoken audio segments; is the average amplitude of spoken audio A and B; N is the number of equally spaced sampling points of spoken audio; a[n] is the time domain sampling sequence of spoken audio A; b[n] is the time domain sampling sequence of spoken audio B;

[0027] Represents the standard deviation of the amplitude of spoken audio A and B, denoted as σ A , σ B ;

[0028] The logic for calculating the similarity of the frequency domain features of two spoken audio segments is as follows:

[0029]

[0030] Where: S2 is the similarity of the frequency domain features of the two spoken audio segments; is the frequency domain feature vector of spoken audio A, the frequency domain feature vector of spoken audio B;

[0031] Combine the sum and average of S1 and S2 as the similarity between the two spoken audio segments. Determine the two spoken audio segments with a similarity ≥ 99% as cleaning targets, and perform the cleaning operation on any of the spoken audio segments in the cleaning target.

[0032] The frequency domain feature vector of the spoken audio A is calculated in the same manner as the frequency domain feature vector of the spoken audio B;

[0033] When obtaining the frequency domain eigenvector of spoken audio A, the spoken audio is subjected to fast Fourier transform to obtain the frequency domain amplitude spectrum, the number of frequency points is recorded as M, the frequency domain amplitude spectrum of the spoken audio is recorded as A[m] (m=1, 2, ..., M), and the amplitude proportion of frequency point m is recorded as

[0034] but

[0035] Furthermore, when the spoken audio stored in the spoken translation library is trained in the model, the stored spoken audio is evenly divided into three groups, which are recorded as a priori group and a test group;

[0036] The model training process is:

[0037] Phase 1: Separate the spoken audio and its annotated translations from the prior group to obtain two sets, denoted as the training set and the validation set. The model is trained based on the training set and the validation set.

[0038] Phase 2: After the prior group completes training, the same separation operation is performed on the test group, resulting in two sets, designated as the test set and the validation set. The test set is based on the model's output translation, which is then compared with the validation set to determine if the model is viable.

[0039]

[0040] Where: ε is the judgment value; G(U OUT ∩U CFI ) is the total number of characters or words in the intersection of the model output translation set and the validation set in the second stage; G(U CFI ) is the total number of characters or words in the verification set in the second stage;

[0041] Among them, the smaller the judgment value ε, the higher the availability of the model. When the judgment value ε≤0.01, the model is judged to be available.

[0042] When the Transformer architecture model determines whether it is available or not, the spoken audio stored in the spoken translation library is synchronously updated and imported into the model again for training, and this process continues until the determination result is yes;

[0043] During the model re-run phase, the user side independently coordinates the number of layers, the number of hidden layer neurons, the size of the convolution kernel, the step size, and the learning rate.

[0044] Furthermore, the Transformer architecture model update cycle is customized by the user end. During the Transformer architecture model update cycle, the translation results output by the model and the corresponding spoken audio are subjected to noise reduction processing simultaneously. After the noise reduction processing, the spoken audio corresponding to the translation result is compared with each spoken audio in the spoken translation library for similarity. If the comparison result shows that there is no similar item, the corresponding spoken audio is marked with the translation result and stored. When the next Transformer architecture model update cycle arrives, it is imported into the spoken translation library. If the comparison result shows that there is a similar item, the comparison ends.

[0045] Among them, the similarity comparison logic between the spoken audio corresponding to the translation result and each spoken audio in the spoken translation library is the same as the cleaning logic of the spoken audio.

[0046] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects:

[0047] The present invention provides a regional spoken language intelligent translation method based on deep learning. During the execution of this method, the spoken language audio is cleaned, denoised, and annotated, and then the spoken language audio is imported into a Transformer architecture model for training, thereby realizing a method for regional spoken language translation that can be continuously optimized, providing auxiliary effects for regional spoken language translation work.

[0048] During the application phase of the Transformer architecture model, available spoken audio is collected within a specified update cycle to expand the spoken translation library when the next update cycle arrives, providing training support for the Transformer architecture model. This ensures that the Transformer architecture model is continuously improved and optimized, and can successfully complete regional spoken translation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0050] Figure 1 Flowchart of the regional spoken language intelligent translation method based on deep learning. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] The present invention will be further described below with reference to the embodiments.

[0053] Example:

[0054] The regional spoken language intelligent translation method based on deep learning in this embodiment is as follows: Figure 1 Shown, including:

[0055] Selecting a target area, collecting spoken audio in the target area, cleaning the collected spoken audio, annotating the cleaned spoken audio with translations, creating a spoken translation library, and storing the cleaned and annotated spoken audio in the spoken translation library;

[0056] The target area is customized by the user. When collecting spoken audio in the target area, any number of users are selected in the target area, and the daily conversation audio of the selected users is collected and recorded as spoken audio.

[0057] When performing a storage operation on the spoken audio marked with the translation, the spoken audio is segmented based on the corresponding translation, so that the spoken audio is segmented into multiple audio segments, each audio segment corresponding to a single word or phrase in the translation;

[0058] When performing the cleaning operation on the spoken audio, the cleaning logic is applied to continuously clean the spoken audio so that each spoken audio for translation annotation is different.

[0059] The cleaning logic is expressed as:

[0060]

[0061] Where: S1 is the similarity of the time domain features of the two spoken audio segments; is the average amplitude of spoken audio A and B; N is the number of equally spaced sampling points of spoken audio; a[n] is the time domain sampling sequence of spoken audio A; b[n] is the time domain sampling sequence of spoken audio B;

[0062] Represents the standard deviation of the amplitude of spoken audio A and B, denoted as σ A , σ B ;

[0063] The logic for calculating the similarity of the frequency domain features of two spoken audio segments is as follows:

[0064]

[0065] Where: S2 is the similarity of the frequency domain features of the two spoken audio segments; is the frequency domain feature vector of spoken audio A, the frequency domain feature vector of spoken audio B;

[0066] Combine the sum and average of S1 and S2 as the similarity between the two spoken audio segments. Determine the two spoken audio segments with a similarity ≥ 99% as cleaning targets, and perform the cleaning operation on any of the spoken audio segments in the cleaning target.

[0067] The frequency domain feature vector of spoken audio A is calculated in the same way as the frequency domain feature vector of spoken audio B;

[0068] When obtaining the frequency domain eigenvector of spoken audio A, the spoken audio is subjected to fast Fourier transform to obtain the frequency domain amplitude spectrum, the number of frequency points is recorded as M, the frequency domain amplitude spectrum of the spoken audio is recorded as A[m] (m=1, 2, ..., M), and the amplitude proportion of frequency point m is recorded as

[0069] but

[0070] The above logical formula provides specified cleaning logic support for spoken audio, ensuring that the spoken audio can be imported into the Transformer architecture model with higher quality based on the noise reduction processing described below. This provides a better training environment for the Transformer architecture model and ensures its basic performance.

[0071] Traverse the spoken translation library and perform noise reduction on each spoken audio in the spoken translation library;

[0072] The logic of spoken audio noise reduction processing is expressed as:

[0073]

[0074] Where: is the spoken audio after noise reduction; U is the total number of spoken audio frames; IFFT[·] is the inverse fast Fourier transform operation; FFT[·] is the fast Fourier transform operation; ξ v (k) is the prior signal-to-noise ratio; is the estimated value of noise power spectral density; is the power spectral density; y v (n) is the noisy frequency signal of the vth frame; δ(·) is the discrete impulse function used to splice the frames according to the frame shift rule;

[0075] In the discrete impulse function δ(·), n represents the discrete time point index, M represents the frame shift size during framing, and the unit is the number of discrete time points;

[0076] Prior signal-to-noise ratio ξ v (k) is calculated by the following formula:

[0077]

[0078] Where: α is the smoothing factor; The frequency domain representation of the clean speech signal estimated for the previous frame; Perform fast Fourier transform on the previous audio frame to obtain the corresponding power spectrum density in the frequency domain;

[0079] Where, the smoothing factor is 0≤α≤1, and max(·) means taking the maximum value in the brackets;

[0080] Through the above logical formula, the noise reduction processing logic of spoken audio is limited, thereby improving the quality of spoken audio.

[0081] Select the Transformer architecture model and import the spoken audio stored in the spoken translation library that has been cleaned, annotated, and denoised into the model for training;

[0082] When the spoken audio stored in the spoken translation library is trained in the model, the stored spoken audio is evenly divided into three groups, which are recorded as the prior group and the test group;

[0083] The model training process is:

[0084] Phase 1: Separate the spoken audio and its annotated translations from the prior group to obtain two sets, denoted as the training set and the validation set. The model is trained based on the training set and the validation set.

[0085] Phase 2: After the prior group completes training, the same separation operation is performed on the test group, resulting in two sets, designated as the test set and the validation set. The test set is based on the model's output translation, which is then compared with the validation set to determine if the model is viable.

[0086]

[0087] Where: ε is the judgment value; G(U OUT ∩U CFI ) is the total number of characters or words in the intersection of the model output translation set and the validation set in the second stage; G(U CFI ) is the total number of characters or words in the verification set in the second stage;

[0088] Among them, the smaller the judgment value ε, the higher the availability of the model. When the judgment value ε≤0.01, the model is judged to be available.

[0089] Through the above logical formula, the judgment logic of whether the Transformer architecture model is available is given, thereby providing support for the further application of the Transformer architecture model.

[0090] After the model training is completed, the availability of the model is determined;

[0091] When the Transformer architecture model determines whether it is available or not, the spoken audio stored in the spoken translation library is updated synchronously and imported into the model again for training. This process continues until the result is yes.

[0092] During the model re-run phase, the user side independently coordinates the number of layers, the number of hidden layer neurons, the size of the convolution kernel, the step size, and the learning rate.

[0093] If the model availability is judged as negative, the process will jump to the model training phase.

[0094] If the model availability is determined to be yes, the spoken audio is received, the spoken audio is fed into the model for processing, and the model outputs a translation result of the spoken audio;

[0095] Set the Transformer architecture model update cycle and update the Transformer architecture model in real time based on the update cycle;

[0096] The Transformer architecture model update cycle is customized by the user. During the Transformer architecture model update cycle, the translation results and the corresponding spoken audio output by the model are subjected to noise reduction processing simultaneously. After noise reduction processing, the translation results corresponding to the spoken audio are compared with each spoken audio in the spoken translation library for similarity. If the comparison result shows that there is no similar item, the corresponding spoken audio is annotated with the translation result and stored. When the next Transformer architecture model update cycle arrives, it is imported into the spoken translation library. If the comparison result shows that there is a similar item, the comparison ends.

[0097] Among them, the similarity comparison logic between the spoken audio corresponding to the translation result and each spoken audio in the spoken translation library is the same as the cleaning logic of the spoken audio.

[0098] In this embodiment, when the method in the above embodiment is applied to actual scenarios, it helps the translator to have an in-depth understanding of the language habits, unique expressions and cultural background of the region, so that the translation is more in line with the local spoken style and avoids stiff literal translation. Secondly, it allows the translator to sort out the logic step by step, accurately grasp the intention of the discourse, and convey the original meaning more fluently and accurately during interpretation. Furthermore, it helps the translator accumulate regional characteristic corpus and improve the ability to cope with various oral scenarios. In addition, based on this method, it can enhance the smoothness of communication between different regions, reduce misunderstandings caused by differences in spoken language, and effectively promote communication and interaction between regions in culture, trade and other aspects.

[0099] In summary, during the execution of the method in the above embodiment, the spoken audio is cleaned, denoised, and labeled, and the spoken audio is imported into the Transformer architecture model for training, thereby realizing a method for continuously optimizing regional spoken translation and providing auxiliary effects for regional spoken translation work; in the application stage of the Transformer architecture model, the available spoken audio is collected within the specified update cycle to expand the spoken translation library when the next update cycle arrives, providing training support for the Transformer architecture model, thereby ensuring that the Transformer architecture model is continuously improved and optimized, and well completing the regional spoken translation task.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A regional spoken language intelligent translation method based on deep learning, characterized by: include: Selecting a target area, collecting spoken audio in the target area, cleaning the collected spoken audio, annotating the cleaned spoken audio with translations, creating a spoken translation library, and storing the cleaned and annotated spoken audio in the spoken translation library; Traverse the spoken translation library and perform noise reduction on each spoken audio in the spoken translation library; Select the Transformer architecture model and import the spoken audio stored in the spoken translation library that has been cleaned, annotated, and denoised into the model for training; After the model training is completed, the availability of the model is determined; If the model availability is judged as negative, the process will jump to the model training phase. If the model availability is determined to be yes, the spoken audio is received, the spoken audio is fed into the model for processing, and the model outputs a translation result of the spoken audio; Set the Transformer architecture model update cycle and update the Transformer architecture model in real time based on the update cycle.

2. The regional spoken language intelligent translation method based on deep learning according to claim 1 is characterized in that The target area is customized by the user terminal. When collecting spoken audio in the target area, any number of users are selected in the target area, and the daily conversation audio of the selected users is collected and recorded as spoken audio; When performing the storage operation on the spoken audio marked with the translation, the spoken audio is segmented based on the corresponding translation of the spoken audio, so that the spoken audio is segmented into multiple audio segments, each audio segment corresponding to a single word or phrase in the translation; When performing the cleaning operation on the spoken audio, the cleaning logic is applied to continuously clean the spoken audio so that each spoken audio on which the translation is annotated is different.

3. The regional spoken language intelligent translation method based on deep learning according to claim 2 is characterized in that The cleaning logic is expressed as: Where: S1 is the similarity of the time domain features of the two spoken audio segments; is the average amplitude of spoken audio A and B; N is the number of equally spaced sampling points of spoken audio; a[n] is the time domain sampling sequence of spoken audio A; b[n] is the time domain sampling sequence of spoken audio B; Represents the standard deviation of the amplitude of spoken audio A and B, denoted as σ A , σ B ; The logic for calculating the similarity of the frequency domain features of two spoken audio segments is as follows: Where: S2 is the similarity of the frequency domain features of the two spoken audio segments; is the frequency domain feature vector of spoken audio A, the frequency domain feature vector of spoken audio B; The sum and average of S1 and S2 are combined as the similarity of the two spoken audio segments. The two spoken audio segments with a similarity ≥ 99% are determined as cleaning targets, and the cleaning operation is performed on any spoken audio segment in the cleaning target.

4. The regional spoken language intelligent translation method based on deep learning according to claim 3 is characterized in that The frequency domain feature vector of the spoken audio A is calculated in the same manner as the frequency domain feature vector of the spoken audio B; When obtaining the frequency domain eigenvector of spoken audio A, the spoken audio is subjected to fast Fourier transform to obtain the frequency domain amplitude spectrum, the number of frequency points is recorded as M, the frequency domain amplitude spectrum of the spoken audio is recorded as A[m] (m=1, 2, ..., M), and the amplitude proportion of frequency point m is recorded as but 5. The regional spoken language intelligent translation method based on deep learning according to claim 1 is characterized in that The spoken audio noise reduction processing logic is expressed as: Where: is the spoken audio after noise reduction; U is the total number of spoken audio frames; IFFT[·] is the inverse fast Fourier transform operation; FFT[·] is the fast Fourier transform operation; ξ v (k) is the prior signal-to-noise ratio; is the estimated value of noise power spectral density; is the power spectral density; y v (n) is the noisy frequency signal of the vth frame; δ(·) is the discrete impulse function used to splice the frames according to the frame shift rule; In the discrete impulse function δ(·), n represents the discrete time point index, M represents the frame shift size during frame division, and the unit is the number of discrete time points.

6. The regional spoken language intelligent translation method based on deep learning according to claim 5 is characterized in that: The prior signal-to-noise ratio ξ v (k) is calculated by the following formula: Where: α is the smoothing factor; The frequency domain representation of the clean speech signal estimated for the previous frame; Perform fast Fourier transform on the previous audio frame to obtain the corresponding power spectrum density in the frequency domain; Wherein, the smoothing factor is 0≤α≤1, and max(·) means taking the maximum value in the brackets.

7. The regional spoken language intelligent translation method based on deep learning according to claim 1 is characterized in that: When the spoken audio stored in the spoken translation library is trained in the model, the stored spoken audio is evenly divided into three groups, which are recorded as a priori group and a test group; The model training process is: Phase 1: Separate the spoken audio and its annotated translations from the prior group to obtain two sets, denoted as the training set and the validation set. The model is trained based on the training set and the validation set. Phase 2: After the prior group completes training, the same separation operation is performed on the test group, resulting in two sets, designated as the test set and the validation set. The test set is based on the model's output translation, which is then compared with the validation set to determine if the model is viable. Where: ε is the judgment value; G ( U OUT ∩U CFI ) is the total number of characters or words in the intersection of the model output translation set and the validation set in the second stage; G(U CFI ) is the total number of characters or words in the verification set in the second stage; Among them, the smaller the judgment value ε is, the higher the availability of the model is. When the judgment value ε≤0.01, the model is judged to be available.

8. The regional spoken language intelligent translation method based on deep learning according to claim 7 is characterized in that: When the Transformer architecture model determines whether it is available or not, the spoken audio stored in the spoken translation library is synchronously updated and imported into the model again for training, and this process continues until the determination result is yes; During the model re-run phase, the user side independently coordinates the number of layers, the number of hidden layer neurons, the size of the convolution kernel, the step size, and the learning rate.

9. The regional spoken language intelligent translation method based on deep learning according to claim 1 is characterized in that: The Transformer architecture model update cycle is customized by the user end. During the Transformer architecture model update cycle, the translation results output by the model and the corresponding spoken audio are subjected to noise reduction processing simultaneously. After the noise reduction processing, the translation result corresponding to the spoken audio is compared with each spoken audio in the spoken translation library for similarity. If the comparison result shows that there is no similar item, the corresponding spoken audio is marked with the translation result and stored. When the next Transformer architecture model update cycle arrives, it is imported into the spoken translation library. If the comparison result shows that there is a similar item, the comparison ends. Among them, the similarity comparison logic between the spoken audio corresponding to the translation result and each spoken audio in the spoken translation library is the same as the cleaning logic of the spoken audio.

Images