Artificial intelligence-based voice translation method and device, computer device and medium

Abstract

The application is suitable for the field of digital medical technology, and particularly relates to a voice translation method and device based on artificial intelligence, computer equipment and medium. The application uses an acoustic encoder to perform feature coding on target voice to obtain an acoustic feature sequence, uses a boundary predictor to perform boundary prediction on the acoustic feature sequence to obtain a probability value of each feature value being predicted as a boundary and use the probability value as a weight of the corresponding feature value, performs weighted summation on all feature values to obtain acoustic contraction features, uses a semantic encoder to extract semantic features in the acoustic contraction features, uses a decoder to decode the semantic features to obtain target translation text in a preset target language, contracts the acoustic features to eliminate the length gap problem between voice features and text features, effectively inherits the knowledge of a pre-trained model when performing voice translation, improves the accuracy of the target translation text, and improves the work efficiency and work quality of doctors in the field of digital medical technology.

Description

TECHNICAL FIELD

[0001] The application is suitable for the field of digital medical technology, and particularly relates to a voice translation method and device based on artificial intelligence, a computer device and a medium. BACKGROUND

[0002] With the development of artificial intelligence technology, voice translation technology can directly translate voice into text in another language, and has a wide range of applications in the fields of finance, medicine, education and the like. For example, in the field of digital medical technology, with the development of digital medical technology, in order to facilitate the management of patient information and improve the work efficiency of doctors, hospitals with more patients generally develop internal medical systems for doctors to fill in patient medical records, check patient examination results, handle medical orders, discharge and the like. When the language of the doctor and the language required by the medical record text are inconsistent, it will be difficult for the doctor to efficiently and accurately complete the medical record filling, and based on the voice translation technology, the voice of the medical record diagnosed by the doctor can be directly translated into medical record text in another language, which can significantly improve the work efficiency and work quality of the doctor, and has important significance for improving the efficiency of digital services and intelligent interactions in the medical process.

[0003] The existing voice translation model generally completes the voice-to-text translation task based on a pre-trained automatic speech recognition model and a pre-trained machine translation model. However, since the voice features are usually much longer than the corresponding text features, there is a length gap between the voice and the text that cannot be ignored, which makes it difficult for the voice translation model to effectively inherit the pre-trained knowledge of the automatic speech recognition model and the machine translation model, resulting in low accuracy of the voice translation result, which cannot meet the accuracy requirement of the medical record text translation.

[0004] Therefore, in the medical record filling scene in the field of digital medical technology, how to improve the accuracy of the voice translation result and thus improve the accuracy of the medical record text has become a problem to be solved. SUMMARY

[0005] Therefore, the embodiments of the present application provide a voice translation method and device based on artificial intelligence, a computer device and a medium to solve the problem of low accuracy of the existing voice translation result, which cannot meet the accuracy requirement of the medical record text translation.

[0006] In a first aspect, the embodiments of the present application provide a voice translation method based on artificial intelligence, which comprises:

[0007] obtaining a target voice to be translated, using a trained acoustic encoder to encode the target voice, and obtaining an acoustic feature sequence composed of feature values arranged in sequence;

[0008] perform boundary prediction on the acoustic feature sequence using the trained boundary predictor to obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary;

[0009] perform weighted summation on all the feature values according to the weight corresponding to each feature value, to obtain acoustic contracted features.

[0010] perform feature encoding on the acoustic contracted features using the trained semantic encoder to obtain semantic features, and perform decoding on the semantic features using the trained decoder to obtain a target translation text corresponding to the target speech in a preset target language.

[0011] In a second aspect, an embodiment of the present application provides a speech translation device based on artificial intelligence, which comprises:

[0012] a feature encoding module, configured to obtain target speech to be translated, and perform feature encoding on the target speech using a trained acoustic encoder to obtain an acoustic feature sequence composed of feature values arranged in sequence;

[0013] a boundary prediction module, configured to perform boundary prediction on the acoustic feature sequence using the trained boundary predictor to obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary;

[0014] a feature contraction module, configured to take the probability value corresponding to each feature value as the weight of the corresponding feature value, and perform weighted summation on all the feature values according to the weight corresponding to each feature value, to obtain acoustic contracted features.

[0015] a speech translation module, configured to perform feature encoding on the acoustic contracted features using the trained semantic encoder to obtain semantic features, and perform decoding on the semantic features using the trained decoder to obtain a target translation text corresponding to the target speech in a preset target language.

[0016] In a third aspect, an embodiment of the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the speech translation method according to the first aspect when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the speech translation method according to the first aspect.

[0018] Compared with the prior art, the embodiment of the present application has the beneficial effects that: by acquiring the target speech to be translated, using the trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence composed of feature values arranged in sequence, using the trained boundary predictor to perform boundary prediction on the acoustic feature sequence to obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary, taking the probability value corresponding to each feature value as the weight of the corresponding feature value, performing weighted summation on all feature values according to the weight corresponding to each feature value to obtain acoustic contraction features, using the trained semantic encoder to perform feature encoding on the acoustic contraction features to obtain semantic features, and using the trained decoder to decode the semantic features to obtain a target translation text corresponding to the target speech in a preset target language, by contracting the acoustic features, the length difference problem between the speech features and the text features is eliminated, so that the pre-trained model knowledge can be effectively inherited when performing speech translation on the target speech, the accuracy of the target translation text is improved, and the work efficiency and work quality of doctors are improved in the field of digital medical technology. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor.

[0020] Figure 1 is an application environment schematic diagram of a speech translation method based on artificial intelligence provided by the first embodiment of the present application;

[0021] Figure 2 is a flowchart of a speech translation method based on artificial intelligence provided by the first embodiment of the present application;

[0022] Figure 3 is a structural schematic diagram of a speech translation device based on artificial intelligence provided by the second embodiment of the present application;

[0023] Figure 4 is a structural schematic diagram of a computer device provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0025] It will be understood that the terms "comprises" and / or "comprising," when used in this specification, include the presence of one or more features, integers, steps, operations, elements, and / or components described in the specification, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0026] It will be understood that the term "and / or," when used in the specification and in the following claims, is intended to mean one or more of the associated listed items can be present, and includes the possibilities of one or more of the associated listed items being present, and all possible combinations of the associated listed items.

[0027] As used in the description of the application and the following claims, the term "if" can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0028] In addition, the description in the specification of the application and the appended claims, the terms "first," "second," "third," etc. are used merely as labels, and are not intended to signify relative importance or significance.

[0029] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in other embodiments" or "in still other embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise specified. Furthermore, the terms "comprise," "comprises," "comprising," and other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0030] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results.

[0031] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0032] It should be understood that the size of the serial number of each step in the following embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0033] In order to illustrate the technical solutions of the present application, the following will be explained by specific embodiments.

[0034] The embodiment of the present application provides a voice translation method based on artificial intelligence, which can be applied in the application environment such as Figure 1 , wherein the client and the server communicate. The client includes but is not limited to palmtop computer, desktop computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cloud computer device, personal digital assistant (PDA) and other computer devices. The server can be realized by an independent server or a server cluster composed of multiple servers.

[0035] Referring to Figure 2 , it is a flowchart of a voice translation method based on artificial intelligence provided by the embodiment of the present application, and the voice translation method can be applied to the client in Figure 1 , and the voice translation method can include the following steps:

[0036] Step S201, acquiring the target voice to be translated, using a trained acoustic encoder to encode the target voice, and obtaining an acoustic feature sequence composed of feature values.

[0037] The voice translation technology can directly translate voice into text in another language, and has wide application in the fields of finance, medical treatment, education and the like. For example, in the field of digital medical technology, if the language of the doctor is inconsistent with the language required by the medical record text when the doctor fills in the medical record, it will cause the doctor to be difficult to efficiently and accurately complete the medical record filling. Therefore, the embodiment is applied to the medical record filling scene in the field of digital medical technology, and the voice translation technology is used to directly translate the target voice diagnosed by the doctor into target translation text in another language, so as to fill in the medical record according to the target translation text in a unified language, thereby improving the work efficiency and work quality of the doctor.

[0038] Specifically, the target voice to be translated can be medical record data recorded by the doctor through language expression of the diagnosis process and diagnosis result of the patient, the trained acoustic encoder can perform feature encoding on the input target voice, and extract an acoustic feature sequence of the target voice, wherein the acoustic feature sequence is composed of a plurality of feature values arranged to represent the acoustic features of the target voice as a basis for translating the target voice.

[0039] The above steps of obtaining the target voice to be translated, using the trained acoustic encoder to perform feature encoding on the target voice, and obtaining the acoustic feature sequence composed of the feature values arranged to extract the acoustic features of the target voice provide a feature basis for translating the target voice.

[0040] In step S202, the trained boundary predictor is used to perform boundary prediction on the acoustic feature sequence to obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary.

[0041] In the embodiment, the voice translation target is to translate the target voice into target translation text in another language based on the pre-trained model. However, since the voice features are usually much longer than the corresponding text features, there is a length difference between the voice and the text that cannot be ignored, which makes it difficult for the embodiment to effectively use the knowledge of the pre-trained model, resulting in low accuracy of the voice translation result.

[0042] Therefore, the embodiment reduces the length of the acoustic feature sequence to contract the acoustic feature sequence of the target voice, thereby reducing the length difference between the voice features and the text features corresponding to the voice translation result, so that the knowledge of the pre-trained model can be effectively used to improve the accuracy of the voice translation result.

[0043] Specifically, the trained boundary predictor is used to perform boundary prediction on the acoustic feature sequence to obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary, and the probability value is used as a basis for length contraction of the acoustic feature sequence.

[0044] The step of predicting the boundary of the acoustic feature sequence by using the trained boundary predictor to obtain the probability value of each feature value in the acoustic feature sequence being predicted as a boundary, takes the probability value of the feature value being predicted as a boundary as the basis for length contraction of the acoustic feature sequence, so as to reduce the length difference between the speech feature and the text feature corresponding to the speech translation result, thereby effectively using the knowledge of the pre-trained model to improve the accuracy of the speech translation result.

[0045] In step S203, the probability value corresponding to each feature value is taken as the weight of the corresponding feature value, and all feature values are weighted and summed according to the weight corresponding to each feature value to obtain acoustic contraction features.

[0046] In the embodiment, the greater the probability value of the feature value being predicted as a boundary, the higher the importance of the feature value in the length contraction process of the acoustic feature sequence. Therefore, the probability value corresponding to each feature value is taken as the weight of the corresponding feature value, and all feature values are weighted and summed according to the weight corresponding to each feature value to obtain a weighted sum result, and the weighted sum result is determined as the acoustic contraction feature. The length of the acoustic contraction feature is less than the length of the corresponding acoustic feature sequence, thereby reducing the length difference between the speech feature and the text feature corresponding to the speech translation result, improving the ability to use the knowledge of the pre-trained model, and further improving the accuracy of the subsequent speech translation result.

[0047] The step of taking the probability value corresponding to each feature value as the weight of the corresponding feature value and weighting and summing all feature values according to the weight corresponding to each feature value to obtain acoustic contraction features takes the probability value of the feature value being predicted as a boundary as the weight of the corresponding feature value to weight and sum all feature values, and obtains length-reduced acoustic contraction features as the basis for subsequent speech translation, thereby improving the ability to use the knowledge of the pre-trained model and further improving the accuracy of the speech translation result.

[0048] In step S204, the trained semantic encoder is used to encode the acoustic contraction features to obtain semantic features, and the trained decoder is used to decode the semantic features to obtain the target translation text corresponding to the target speech in the preset target language.

[0049] The trained acoustic encoder extracts the acoustic features of the target speech. Since the speech translation target is to translate the target speech into a target translation text in another language, it is necessary to further extract the semantic feature information in the target speech to combine the acoustic feature information and the semantic feature information to recognize and translate the target speech. The target speech is translated into the corresponding target translation text in the preset target language. In the medical record filling scene in the digital medical technology field, the target translation text is the corresponding medical record text in the target language.

[0050] The preset target language can be set according to actual conditions. For example, the target speech expressed by the doctor can be English. If the language of the medical record required to be filled in is Chinese, the preset target language is Chinese. Correspondingly, the speech translation target is to translate the target speech in English into the target translation text in Chinese; if the language of the medical record required to be filled in is Japanese, the preset target language is Japanese. Correspondingly, the speech translation target is to translate the target speech in English into the target translation text in Japanese.

[0051] Specifically, this embodiment uses a semantic encoder to encode the acoustic contraction features to obtain semantic features. The semantic features are obtained by semantic extraction based on the acoustic contraction features, and include both acoustic feature information and semantic feature information in the target speech. The trained decoder is then used to decode the semantic features to obtain a target translation text corresponding to the target speech in a preset target language.

[0052] Optionally, the training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder, and the trained decoder includes:

[0053] Obtain a pre-trained acoustic encoder, a pre-trained semantic encoder, a pre-trained decoder, a boundary predictor, several speech samples, and the actual translation text corresponding to each speech sample;

[0054] For any speech sample, use the pre-trained acoustic encoder to perform feature encoding on the speech sample to obtain an acoustic feature sequence sample consisting of feature value samples;

[0055] Use the boundary predictor to predict the boundary of the acoustic feature sequence sample, and obtain the probability value of each feature value sample in the acoustic feature sequence sample being predicted as a boundary;

[0056] The probability value corresponding to each eigenvalue sample is used as the weight of the corresponding eigenvalue sample, and all eigenvalue samples are weighted and summed according to the weight corresponding to each eigenvalue sample to obtain the acoustic contraction feature sample;

[0057] Use a pre-trained semantic encoder to encode the acoustic contraction feature sample to obtain a semantic feature sample, and use a pre-trained decoder to decode the semantic feature sample to obtain a target translation text sample corresponding to the speech sample in a preset target language;

[0058] According to the model loss calculated according to the actual translation text and the corresponding target translation text sample, the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor are corrected until the model loss converges, and the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder are obtained.

[0059] In the embodiment, the voice translation target is to translate the target voice into target translation text in another language based on the pre-trained model, the trained boundary predictor is a key model for length contraction of the acoustic feature sequence, in order to improve the accuracy and training efficiency of the boundary predictor by training, the pre-trained acoustic encoder, the pre-trained semantic encoder and the pre-trained decoder are obtained first, the boundary predictor is trained based on the pre-training knowledge of the pre-trained acoustic encoder, the pre-trained semantic encoder and the pre-trained decoder, and the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder and the pre-trained decoder are fine-tuned, so as to improve the accuracy of the voice translation result.

[0060] Specifically, the embodiment obtains a plurality of voice samples as training samples, obtains the actual translation text corresponding to each voice sample as a training label, obtains the corresponding target translation text sample by recognizing and translating the voice sample, and calculates the model loss according to the actual translation text and the corresponding target translation text sample to correct the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor until the model loss converges, thereby obtaining the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder as the model basis for recognizing and translating the target voice, and improving the accuracy of the voice translation result corresponding to the target voice.

[0061] Optionally, the training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder further comprises:

[0062] obtaining the boundary feature value sample in the acoustic feature sequence sample, and determining the real probability value of each feature value sample in the acoustic feature sequence sample predicted as a boundary according to the boundary feature value sample;

[0063] calculating the boundary prediction loss according to the probability value and the real probability value of each feature value sample predicted as a boundary;

[0064] Correspondingly, parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor are corrected according to the model loss until the model loss converges, to obtain a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder and a trained decoder, including:

[0065] A sum of the boundary prediction loss and the model loss is calculated to obtain a model total loss;

[0066] Parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor are corrected according to the model total loss until the model total loss converges, to obtain a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder and a trained decoder.

[0067] In order to improve the accuracy of the trained boundary predictor in boundary prediction of the acoustic feature sequence, the embodiment obtains boundary feature value samples in the acoustic feature sequence sample, and determines a real probability value of each feature value sample in the acoustic feature sequence sample being predicted as a boundary according to the boundary feature value samples. The closer the probability value of the feature value being predicted as a boundary is to the real probability value, the higher the accuracy of the boundary predictor is. Therefore, the boundary prediction loss is calculated according to the probability value of all feature value samples being predicted as a boundary and the real probability value, and the sum of the boundary prediction loss and the model loss is taken as the model total loss to correct the model parameters, to obtain the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder, so as to improve the accuracy of the model training.

[0068] Optionally, obtaining the boundary feature value samples in the acoustic feature sequence sample includes:

[0069] The trained connection time sequence classifier is obtained, and the trained connection time sequence classifier is used to perform boundary prediction on the acoustic feature sequence sample to obtain a prediction probability value of each feature value sample in the acoustic feature sequence sample being predicted as a boundary.

[0070] For any feature value sample in the acoustic feature sample, the prediction probability value corresponding to the feature value sample is compared with a preset threshold to obtain a comparison result of the corresponding feature value sample.

[0071] If the comparison result is that the prediction probability value is greater than the preset threshold, the feature value sample is determined as a boundary feature value sample.

[0072] All feature value samples in the acoustic feature sequence sample are traversed to obtain all boundary feature value samples in the acoustic feature sequence sample.

[0073] The trained connection time sequence classifier can perform boundary prediction on the acoustic feature sequence sample, and obtain a prediction probability value of each feature value sample in the acoustic feature sequence sample being predicted as a boundary. The greater the prediction probability value is, the higher the possibility of the corresponding feature value being a boundary is. Then, the prediction probability value corresponding to the feature value sample is compared with a preset threshold value, and a comparison result of the corresponding feature value sample is obtained. Correspondingly, if the comparison result is that the prediction probability value is greater than the preset threshold value, it is determined that the feature value sample is a boundary feature value sample. By traversing all the feature value samples in the acoustic feature sequence sample, all the boundary feature value samples in the acoustic feature sequence sample can be obtained as the training label of the boundary predictor, so as to improve the prediction accuracy of the boundary predictor.

[0074] Optionally, the model loss is calculated according to the actual translation text and the corresponding target translation text sample, and includes:

[0075] The actual translation text is converted into a vector to obtain a first text vector corresponding to the actual translation text;

[0076] The target translation text sample is converted into a vector to obtain a second text vector corresponding to the target translation text sample;

[0077] A first similarity between the first text vector and the second text vector is calculated, a first difference between the first similarity and a first preset value is calculated, and the first difference is determined as the model loss.

[0078] The higher the similarity between the actual translation text and the corresponding target translation text sample is, the higher the accuracy of the model is. Therefore, the similarity between the actual translation text and the corresponding target translation text sample can be calculated as the basis for model training.

[0079] In order to facilitate the calculation of the similarity between the texts, the actual translation text and the target translation text sample are first converted into vectors to obtain a first text vector corresponding to the actual translation text and a second text vector corresponding to the target translation text sample. Then, a first similarity between the first text vector and the second text vector is calculated, a first difference between the first similarity and a first preset value is calculated, and the first difference is determined as the model loss.

[0080] The preset value can be set according to the actual situation. For example, if the value range of the first similarity is within the range of [0, 1], the preset value can be set to 1.

[0081] Optionally, the boundary prediction loss is calculated according to the probability value and the true probability value of all the feature value samples being predicted as a boundary, and includes:

[0082] For any feature value sample, calculate the second similarity between the probability value predicted by the feature value sample and the true probability value, calculate the second difference between the second similarity and the second preset value, and determine the second difference as the boundary prediction sub-loss of the corresponding feature value sample;

[0083] Iterate through all the feature value samples to obtain the boundary prediction sub-loss corresponding to each feature value sample;

[0084] Sum the boundary prediction sub-losses corresponding to all feature value samples to obtain a summation result, and determine the summation result as the boundary prediction loss.

[0085] The higher the similarity between the actual translation text and the corresponding target translation text sample, the higher the accuracy of the model, therefore, the embodiment calculates the second similarity between the probability value predicted by the feature value sample and the true probability value, calculates the second difference between the second similarity and the second preset value, determines the second difference as the boundary prediction sub-loss of the corresponding feature value sample, and sums the boundary prediction sub-losses corresponding to all feature value samples to obtain a summation result, and determines the summation result as the boundary prediction loss.

[0086] The above step of using the trained semantic encoder to perform feature encoding on the acoustic contraction feature to obtain semantic features, and using the trained decoder to decode the semantic features to obtain the target translation text corresponding to the target speech in the preset target language, extracts acoustic feature information and semantic feature information in the target speech, and decodes the semantic features which have undergone feature length contraction to obtain the target translation text corresponding to the target speech in the preset target language as the speech translation result of the target speech, thereby improving the accuracy of the speech translation result.

[0087] The embodiment of the application obtains a target speech to be translated, uses a trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence composed of feature values, uses a trained boundary predictor to perform boundary prediction on the acoustic feature sequence to obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary, takes the probability value corresponding to each feature value as the weight of the corresponding feature value, performs weighted summation on all feature values according to the weight corresponding to each feature value to obtain acoustic contraction features, uses a trained semantic encoder to perform feature encoding on the acoustic contraction features to obtain semantic features, and uses a trained decoder to decode the semantic features to obtain a target translation text corresponding to the target speech in a preset target language. By contracting the acoustic features, the length difference problem between the speech features and the text features is eliminated, the pre-trained model knowledge can be effectively inherited when performing speech translation on the target speech, the accuracy of the target translation text is improved, and the work efficiency and work quality of doctors in the field of digital medical technology are improved.

[0088] The speech translation method corresponding to the above embodiment, Figure 3 The structure block diagram of the speech translation device based on artificial intelligence provided by the second embodiment of the present application is given, only the parts related to the embodiments of the present application are shown for the convenience of description.

[0089] Referring to Figure 3 The speech translation device comprises:

[0090] The feature encoding module 31 is configured to obtain the target speech to be translated, encode the target speech using the trained acoustic encoder, and obtain an acoustic feature sequence composed of feature values arranged in sequence;

[0091] The boundary prediction module 32 is configured to predict the boundaries of the acoustic feature sequence using the trained boundary predictor, and obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary;

[0092] The feature contraction module 33 is configured to use the probability value corresponding to each feature value as the weight of the corresponding feature value, and perform weighted summation on all feature values according to the weight corresponding to each feature value to obtain acoustic contraction features;

[0093] The speech translation module 34 is configured to encode the acoustic contraction features using the trained semantic encoder to obtain semantic features, and decode the semantic features using the trained decoder to obtain the target translation text corresponding to the target speech in the preset target language.

[0094] Optionally, the speech translation device further comprises:

[0095] The sample acquisition module is configured to obtain the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, the boundary predictor, and a plurality of speech samples and actual translation texts corresponding to each speech sample;

[0096] The sample feature encoding module is configured to encode any speech sample using the pre-trained acoustic encoder to obtain an acoustic feature sequence sample composed of feature value samples;

[0097] The sample boundary prediction module is configured to predict the boundaries of the acoustic feature sequence sample using the boundary predictor to obtain a probability value of each feature value sample in the acoustic feature sequence sample being predicted as a boundary;

[0098] The sample feature contraction module is configured to use the probability value corresponding to each feature value sample as the weight of the corresponding feature value sample, and perform weighted summation on all feature value samples according to the weight corresponding to each feature value sample to obtain acoustic contraction feature samples;

[0099] The sample speech translation module is configured to encode the acoustic feature sample using a pre-trained semantic encoder to obtain a semantic feature sample, and decode the semantic feature sample using a pre-trained decoder to obtain a target translation text sample corresponding to the speech sample in a preset target language.

[0100] The parameter correction module is configured to calculate a model loss based on the actual translation text and the corresponding target translation text sample, correct parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, and the boundary predictor based on the model loss until the model loss converges, and obtain a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder, and a trained decoder.

[0101] Optionally, the parameter correction module includes:

[0102] The first vector conversion submodule is configured to convert the actual translation text into a first text vector corresponding to the actual translation text.

[0103] The second vector conversion submodule is configured to convert the target translation text sample into a second text vector corresponding to the target translation text sample.

[0104] The model loss calculation submodule is configured to calculate a first similarity between the first text vector and the second text vector, calculate a first difference between the first similarity and a first preset value, and determine the first difference as the model loss.

[0105] Optionally, the speech translation device further includes:

[0106] The real probability value determination module is configured to obtain a boundary feature value sample in the acoustic feature sequence sample, and determine a real probability value of each feature value sample in the acoustic feature sequence sample being predicted as a boundary based on the boundary feature value sample.

[0107] The boundary prediction loss calculation module is configured to calculate a boundary prediction loss based on the probability value and the real probability value of each feature value sample being predicted as a boundary.

[0108] Correspondingly, the parameter correction module includes:

[0109] The model total loss calculation submodule is configured to calculate a sum of the boundary prediction loss and the model loss to obtain a model total loss.

[0110] The parameter correction submodule is configured to correct parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, and the boundary predictor based on the model total loss until the model total loss converges, and obtain a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder, and a trained decoder.

[0111] Optionally, the true probability value determining module includes:

[0112] The prediction probability value determination submodule is used to obtain a trained connection time series classifier, use the trained connection time series classifier to perform boundary prediction on the acoustic feature sequence samples, and obtain the prediction probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary;

[0113] The probability value comparison submodule is used to compare the predicted probability value corresponding to any eigenvalue sample in the acoustic feature samples with a preset threshold value to obtain a comparison result of the corresponding eigenvalue sample;

[0114] A first boundary determination submodule, configured to determine that the eigenvalue sample is a boundary eigenvalue sample if the comparison result shows that the predicted probability value is greater than a preset threshold;

[0115] The second boundary determination submodule is used to traverse all eigenvalue samples in the acoustic feature sequence samples to obtain all boundary eigenvalue samples in the acoustic feature sequence samples.

[0116] Optionally, the boundary prediction loss calculation module includes:

[0117] A first sub-loss calculation submodule is configured to calculate, for any eigenvalue sample, a second similarity between a probability value of the eigenvalue sample predicted as a boundary and a true probability value, calculate a second difference between the second similarity and a second preset value, and determine the second difference as a boundary prediction sub-loss for the corresponding eigenvalue sample;

[0118] The second sub-loss calculation sub-module is used to traverse all eigenvalue samples and obtain the boundary prediction sub-loss corresponding to each eigenvalue sample;

[0119] The boundary prediction loss calculation submodule is used to sum the boundary prediction sub-losses corresponding to all eigenvalue samples, obtain the summation result, and determine the summation result as the boundary prediction loss.

[0120] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0121] Figure 4 This is a schematic diagram of the structure of a computer device provided in the third embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4The computer device can include, but is not limited to, a processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor implementing the steps in any of the above voice translation method embodiments when executing the computer program.

[0122] The computer device can include, but is not limited to, a processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor implementing the steps in any of the above voice translation method embodiments when executing the computer program. Figure 4 The computer device is merely an example and does not constitute a limitation on the computer device, which can include more or fewer components than shown, or combine some components, or include different components, such as a network interface, a display screen, and an input device, etc.

[0123] The processor can be a CPU, and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0124] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be a memory of the computer device, and the internal memory provides an environment for the operation of the operating system and the computer readable instructions in the readable storage medium. The readable storage medium can be a hard disk of the computer device, and in other embodiments, can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory can include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of a computer program, etc. The memory can also be used to temporarily store data that has been output or will be output.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above device can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the above-mentioned embodiment methods, which can be realized by a computer program to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable medium cannot be electric carrier signal and telecommunication signal.

[0126] The present application realizes all or part of the processes in the above-mentioned embodiment methods, which can also be completed by a computer program product. When the computer program product runs on the computer equipment, it makes the computer equipment execute the steps in the above-mentioned embodiment methods.

[0127] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0128] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0129] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other manners. For example, the described apparatus / computer device embodiments are merely schematic. For example, the division of the modules or units can be different, and each can include a plurality of sub-units. Some or all of the modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e. may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0131] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A speech translation method based on artificial intelligence, characterized in that: The speech translation method comprises: Obtaining a target speech to be translated, and performing feature encoding on the target speech using a trained acoustic encoder to obtain an acoustic feature sequence consisting of an arrangement of feature values; Using a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, and obtaining a probability value of each feature value in the acoustic feature sequence being predicted as a boundary; The probability value corresponding to each eigenvalue is used as the weight of the corresponding eigenvalue, and all eigenvalues ​​are weighted and summed according to the weight corresponding to each eigenvalue to obtain the acoustic contraction feature; Using a trained semantic encoder to perform feature encoding on the acoustic contraction feature to obtain a semantic feature, and using a trained decoder to decode the semantic feature to obtain a target translation text corresponding to the target speech in a preset target language; The training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder, and the trained decoder includes: Obtain a pre-trained acoustic encoder, a pre-trained semantic encoder, a pre-trained decoder, a boundary predictor, several speech samples, and the actual translation text corresponding to each speech sample; For any speech sample, use the pre-trained acoustic encoder to perform feature encoding on the speech sample to obtain an acoustic feature sequence sample consisting of feature value samples; Using the boundary predictor to perform boundary prediction on the acoustic feature sequence samples, and obtaining a probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary; The probability value corresponding to each eigenvalue sample is used as the weight of the corresponding eigenvalue sample, and all eigenvalue samples are weighted and summed according to the weight corresponding to each eigenvalue sample to obtain the acoustic contraction feature sample; Using a pre-trained semantic encoder to perform feature encoding on the acoustic contraction feature sample to obtain a semantic feature sample, and using a pre-trained decoder to decode the semantic feature sample to obtain a target translation text sample corresponding to the speech sample in a preset target language; A model loss is calculated based on the actual translation text and the corresponding target translation text sample, and parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, and the boundary predictor are corrected based on the model loss until the model loss converges, thereby obtaining a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder, and a trained decoder.

2. The speech translation method according to claim 1, wherein Calculating the model loss according to the actual translation text and the corresponding target translation text sample includes: Performing vector conversion on the actual translation text to obtain a first text vector corresponding to the actual translation text; Performing vector conversion on the target translation text sample to obtain a second text vector corresponding to the target translation text sample; A first similarity between the first text vector and the second text vector is calculated, a first difference between the first similarity and a first preset value is calculated, and the first difference is determined as the model loss.

3. The speech translation method according to claim 1, wherein: The training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder, and the trained decoder further includes: Obtaining boundary feature value samples in the acoustic feature sequence samples, and determining, based on the boundary feature value samples, a true probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary; Calculate the boundary prediction loss based on the probability value of all the feature value samples predicted to be the boundary and the true probability value; Correspondingly, modifying the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, and the boundary predictor according to the model loss until the model loss converges, to obtain the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder, and the trained decoder includes: Calculating the sum of the boundary prediction loss and the model loss to obtain the total model loss; The parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, and the boundary predictor are modified according to the total loss of the model until the total loss of the model converges, thereby obtaining a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder, and a trained decoder.

4. The speech translation method according to claim 3, wherein: The obtaining of boundary feature value samples in the acoustic feature sequence samples includes: Obtaining a trained connection time series classifier, and using the trained connection time series classifier to perform boundary prediction on the acoustic feature sequence samples to obtain a predicted probability value that each feature value sample in the acoustic feature sequence samples is predicted to be a boundary; For any of the eigenvalue samples in the acoustic feature samples, comparing the predicted probability value corresponding to the eigenvalue sample with a preset threshold to obtain a comparison result corresponding to the eigenvalue sample; If the comparison result is that the predicted probability value is greater than the preset threshold, determining that the eigenvalue sample is a boundary eigenvalue sample; All eigenvalue samples in the acoustic feature sequence samples are traversed to obtain all boundary eigenvalue samples in the acoustic feature sequence samples.

5. The speech translation method according to claim 3, wherein: Calculating the boundary prediction loss based on the probability values ​​and true probability values ​​of all the feature value samples predicted as boundaries includes: For any of the eigenvalue samples, calculating a second similarity between a probability value of the eigenvalue sample predicted to be a boundary and a true probability value, calculating a second difference between the second similarity and a second preset value, and determining the second difference as a boundary prediction sub-loss corresponding to the eigenvalue sample; Traverse all eigenvalue samples and obtain the boundary predictor loss corresponding to each eigenvalue sample; The boundary prediction sub-losses corresponding to all eigenvalue samples are summed to obtain a summation result, and the summation result is determined to be the boundary prediction loss.

6. A speech translation device based on artificial intelligence, characterized in that: The speech translation device comprises: A feature encoding module is used to obtain the target speech to be translated, and perform feature encoding on the target speech using a trained acoustic encoder to obtain an acoustic feature sequence consisting of an arrangement of feature values; A boundary prediction module, configured to use a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, and obtain a probability value that each feature value in the acoustic feature sequence is predicted to be a boundary; The feature contraction module is used to take the probability value corresponding to each eigenvalue as the weight of the corresponding eigenvalue, and perform weighted summation on all eigenvalues ​​according to the weight corresponding to each eigenvalue to obtain the acoustic contraction feature; A speech translation module is configured to perform feature encoding on the acoustic contraction features using a trained semantic encoder to obtain semantic features, and to decode the semantic features using a trained decoder to obtain a target translation text corresponding to the target speech in a preset target language; The sample acquisition module is used to obtain a pre-trained acoustic encoder, a pre-trained semantic encoder, a pre-trained decoder, a boundary predictor, and several speech samples and the actual translation text corresponding to each speech sample; A sample feature encoding module is used to perform feature encoding on any speech sample using the pre-trained acoustic encoder to obtain an acoustic feature sequence sample consisting of feature value samples; A sample boundary prediction module, configured to use the boundary predictor to perform boundary prediction on the acoustic feature sequence samples, and obtain a probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary; The sample feature shrinkage module is used to use the probability value corresponding to each eigenvalue sample as the weight of the corresponding eigenvalue sample, and perform weighted summation on all eigenvalue samples according to the weight corresponding to each eigenvalue sample to obtain an acoustic shrinkage feature sample; A sample speech translation module is configured to perform feature encoding on the acoustic contraction feature sample using a pre-trained semantic encoder to obtain a semantic feature sample, and to decode the semantic feature sample using a pre-trained decoder to obtain a target translation text sample corresponding to the speech sample in a preset target language; A parameter correction module is used to calculate a model loss based on the actual translation text and the corresponding target translation text sample, and correct the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, and the boundary predictor according to the model loss until the model loss converges, thereby obtaining a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder, and a trained decoder.

7. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the speech translation method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech translation method according to any one of claims 1 to 5 is implemented.

Images