Speech translation method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention is suitable for the technical field of digital medical treatment, and particularly relates to a speech translation method and device based on artificial intelligence, computer equipment and a medium. According to the method, an acoustic encoder is used for carrying out feature coding on target voice to obtain an acoustic feature sequence, a boundary predictor is used for carrying out boundary prediction on the acoustic feature sequence to obtain a probability value that each feature value is predicted as a boundary and serves as a weight of the corresponding feature value, and weighted summation is carried out on all the feature values to obtain an acoustic contraction feature. A semantic encoder is used for extracting semantic features in acoustic contraction features, a decoder is used for decoding the semantic features to obtain a target translation text in a preset target language, and the acoustic features are contracted, so that the problem of length difference between the voice features and the text features is eliminated; when speech translation is carried out, pre-trained model knowledge is effectively inherited, the accuracy of the target translation text is improved, and the working efficiency and the working quality of doctors are improved in the technical field of digital medical treatment.

Description

Technical Field

[0001] The present invention is applicable to the field of digital medical technology, and in particular relates to a speech translation method, device, computer equipment and medium based on artificial intelligence. Background Art

[0002] With the development of artificial intelligence technology, speech translation technology can directly translate speech into text in another language, and has wide applications in finance, medical care, education and other fields. For example, in the field of digital medical technology, with the development of digital medical technology, in order to facilitate the management of patient information and improve the work efficiency of doctors, hospitals with a large number of patients generally develop internal medical systems for doctors to fill in patient medical records, view patient examination results, handle medical orders, discharge and other operations. When the doctor's language is inconsistent with the language required by the medical record text, it will make it difficult for the doctor to complete the medical record efficiently and accurately. Based on speech translation technology, the medical record voice of the doctor's diagnosis can be directly translated into the medical record text in another language, which can significantly improve the doctor's work efficiency and work quality, and is of great significance to improving the efficiency of digital services, intelligent interaction and other businesses in the medical process.

[0003] Existing speech translation models usually complete speech-to-text translation tasks based on pre-trained automatic speech recognition models and pre-trained machine translation models. However, since speech features are usually much longer than the corresponding text features, there is a length gap between speech and text that is difficult to ignore, which makes it difficult for speech translation models to effectively inherit the pre-trained knowledge of automatic speech recognition models and machine translation models. As a result, the accuracy of speech translation results is low and cannot meet the translation accuracy requirements of medical record texts.

[0004] Therefore, in the medical record filling scenario in the field of digital medical technology, how to improve the accuracy of speech translation results and thus improve the accuracy of medical record text has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a speech translation method, apparatus, computer equipment and medium based on artificial intelligence to solve the problem that the accuracy of existing speech translation results is low and cannot meet the translation accuracy requirements of medical record texts.

[0006] In a first aspect, an embodiment of the present invention provides a speech translation method based on artificial intelligence, the speech translation method comprising: Obtaining a target speech to be translated, and using a trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence consisting of feature value arrangements; Using a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, and obtaining a probability value of each feature value in the acoustic feature sequence being predicted as a boundary; The probability value corresponding to each eigenvalue is used as the weight of the corresponding eigenvalue, and all eigenvalues ​​are weighted summed according to the weight corresponding to each eigenvalue to obtain the acoustic contraction feature; The acoustic contraction feature is feature encoded using a trained semantic encoder to obtain a semantic feature, and the semantic feature is decoded using a trained decoder to obtain a target translation text corresponding to the target speech in a preset target language.

[0007] In a second aspect, an embodiment of the present invention provides a speech translation device based on artificial intelligence, the speech translation device comprising: A feature encoding module is used to obtain a target speech to be translated, and use a trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence composed of feature value arrangements; A boundary prediction module, used to use a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, and obtain a probability value that each feature value in the acoustic feature sequence is predicted to be a boundary; A feature contraction module is used to use the probability value corresponding to each eigenvalue as the weight of the corresponding eigenvalue, and perform weighted summation on all eigenvalues ​​according to the weight corresponding to each eigenvalue to obtain an acoustic contraction feature; The speech translation module is used to use a trained semantic encoder to perform feature encoding on the acoustic contraction feature to obtain a semantic feature, and use a trained decoder to decode the semantic feature to obtain a target translation text corresponding to the target speech in a preset target language.

[0008] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech translation method as described in the first aspect when executing the computer program.

[0009] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech translation method as described in the first aspect is implemented.

[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by acquiring a target speech to be translated, using a trained acoustic encoder to perform feature encoding on the target speech, obtaining an acoustic feature sequence composed of feature value arrangements, using a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, obtaining a probability value that each feature value in the acoustic feature sequence is predicted to be a boundary, using the probability value corresponding to each feature value as the weight of the corresponding feature value, performing weighted summation on all feature values ​​according to the weight corresponding to each feature value, obtaining an acoustic contraction feature, using a trained semantic encoder to perform feature encoding on the acoustic contraction feature, obtaining a semantic feature, using a trained decoder to decode the semantic feature, obtaining a target translation text corresponding to the target speech in a preset target language, and by shrinking the acoustic features, eliminating the length difference problem between speech features and text features, so that the pre-trained model knowledge can be effectively inherited when performing speech translation on the target speech, thereby improving the accuracy of the target translation text and improving the work efficiency and work quality of doctors in the field of digital medical technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 This is a schematic diagram of an application environment of a speech translation method based on artificial intelligence provided in Embodiment 1 of the present invention; Figure 2 It is a flowchart of a speech translation method based on artificial intelligence provided in Embodiment 1 of the present invention; Figure 3 1 is a schematic diagram of the structure of a speech translation device based on artificial intelligence provided in Embodiment 2 of the present invention; Figure 4 It is a structural diagram of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0014] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0015] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0016] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0017] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0019] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0020] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0021] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0022] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0023] The first embodiment of the present invention provides a speech translation method based on artificial intelligence, which can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other computer devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0024] See also Figure 2 , is a flow chart of a speech translation method based on artificial intelligence provided by the first embodiment of the present invention, and the above-mentioned speech translation method can be applied to Figure 1 The client in the speech translation method may include the following steps: Step S201, obtaining a target speech to be translated, and using a trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence composed of feature value arrangements.

[0025] Among them, speech translation technology can directly translate speech into text in another language, and has wide applications in finance, medical care, education and other fields. For example, in the field of digital medical technology, when a doctor fills in a patient's medical record, if the doctor's language is inconsistent with the language required by the medical record text, it will make it difficult for the doctor to complete the medical record filling efficiently and accurately. Therefore, this embodiment is applied to the medical record filling scenario in the field of digital medical technology. The target speech of the doctor's diagnosis is directly translated into a target translation text in another language through speech translation technology, so that the medical record is filled in according to the target translation text in a unified language, thereby improving the doctor's work efficiency and work quality.

[0026] Specifically, the target speech to be translated can be the medical record data in which the doctor records the patient's diagnosis process and diagnosis results through language expression. The trained acoustic encoder can perform feature encoding on the input target speech and extract the acoustic feature sequence of the target speech, wherein the acoustic feature sequence is composed of a plurality of feature value arrangements, which are used to characterize the acoustic features of the target speech and serve as the basis for translating the target speech.

[0027] The above steps of obtaining the target speech to be translated, using the trained acoustic encoder to perform feature encoding on the target speech, obtaining an acoustic feature sequence composed of feature value arrangements, extracting the acoustic features of the target speech, and obtaining the acoustic feature sequence provide a feature basis for translating the target speech.

[0028] Step S202: Use the trained boundary predictor to perform boundary prediction on the acoustic feature sequence to obtain a probability value of each feature value in the acoustic feature sequence being predicted as a boundary.

[0029] Among them, the speech translation goal of this embodiment is to translate the target speech into a target translation text in another language based on a pre-trained model. However, since speech features are usually much longer than corresponding text features, there is a length gap between speech and text that is difficult to ignore, which makes it difficult for this embodiment to effectively draw on the pre-trained model knowledge, resulting in low accuracy of the speech translation results.

[0030] Therefore, this embodiment shrinks the acoustic feature sequence of the target speech by reducing the length of the acoustic feature sequence, thereby reducing the length difference between the speech features and the text features corresponding to the speech translation results, so that the pre-trained model knowledge can be effectively used to improve the accuracy of the speech translation results.

[0031] Specifically, the trained boundary predictor is used to predict the boundary of the acoustic feature sequence, and the probability value of each feature value in the acoustic feature sequence being predicted as a boundary is obtained, and the probability value is used as the basis for length contraction of the acoustic feature sequence.

[0032] The above-mentioned step of using the trained boundary predictor to predict the boundary of the acoustic feature sequence to obtain the probability value of each feature value in the acoustic feature sequence being predicted as a boundary, uses the probability value of the feature value predicted as a boundary as the basis for length contraction of the acoustic feature sequence to reduce the length difference between the speech feature and the text feature corresponding to the speech translation result, thereby effectively drawing on the pre-trained model knowledge to improve the accuracy of the speech translation result.

[0033] Step S203: taking the probability value corresponding to each eigenvalue as the weight of the corresponding eigenvalue, performing weighted summation on all eigenvalues ​​according to the weight corresponding to each eigenvalue, and obtaining the acoustic contraction feature.

[0034] Among them, in this embodiment, the larger the probability value of the eigenvalue predicted as the boundary, the higher the importance of the eigenvalue in the length contraction process of the acoustic feature sequence. Therefore, the probability value corresponding to each eigenvalue is used as the weight of the corresponding eigenvalue, and all the eigenvalues ​​are weightedly summed according to the weight corresponding to each eigenvalue to obtain the weighted summation result, and the weighted summation result is determined as the acoustic contraction feature. The length of the acoustic contraction feature is smaller than the length of the corresponding acoustic feature sequence, thereby reducing the length difference between the speech feature and the text feature corresponding to the speech translation result, improving the ability to draw on the pre-trained model knowledge, and thereby improving the accuracy of the subsequent speech translation results.

[0035] In the above step, the probability value corresponding to each eigenvalue is used as the weight of the corresponding eigenvalue, and all the eigenvalues ​​are weighted summed according to the weight corresponding to each eigenvalue to obtain the acoustic contraction feature. The probability value of the eigenvalue predicted as the boundary is used as the weight of the corresponding eigenvalue to perform weighted summation on all the eigenvalues ​​to obtain the acoustic contraction feature with reduced length as the basis for subsequent speech translation, which improves the ability to draw on pre-trained model knowledge and thereby improves the accuracy of the speech translation results.

[0036] Step S204, using the trained semantic encoder to perform feature encoding on the acoustic contraction feature to obtain semantic features, and using the trained decoder to decode the semantic features to obtain a target translation text corresponding to the target speech in a preset target language.

[0037] Among them, the trained acoustic encoder extracts the acoustic features of the target speech. Since the goal of speech translation is to translate the target speech into a target translation text in another language, it is necessary to further extract the semantic feature information in the target speech, so as to combine the acoustic feature information and the semantic feature information to recognize and translate the target speech, and translate the target speech into the corresponding target translation text in the preset target language. In the medical record filling scenario in the field of digital medical technology, the target translation text is the corresponding medical record text in the target language.

[0038] The preset target language can be set according to actual conditions. For example, the target voice expressed by the doctor can be English. If the language of the medical record required to be filled in is Chinese, the preset target language is Chinese. Correspondingly, the speech translation target is to translate the target voice in English into the target translation text in Chinese; if the language of the medical record required to be filled in is Japanese, the preset target language is Japanese. Correspondingly, the speech translation target is to translate the target voice in English into the target translation text in Japanese.

[0039] Specifically, this embodiment uses a semantic encoder to perform feature encoding on the acoustic contraction feature to obtain a semantic feature. The semantic feature is obtained by semantic extraction based on the acoustic contraction feature, and includes both acoustic feature information and semantic feature information in the target speech. The semantic feature is decoded using a trained decoder to obtain a target translation text corresponding to the target speech in a preset target language.

[0040] Optionally, the training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder, and the trained decoder includes: Obtain a pre-trained acoustic encoder, a pre-trained semantic encoder, a pre-trained decoder, a boundary predictor, and a number of speech samples and the actual translation text corresponding to each speech sample; For any speech sample, use the pre-trained acoustic encoder to perform feature encoding on the speech sample to obtain an acoustic feature sequence sample consisting of feature value samples; Using the boundary predictor to perform boundary prediction on the acoustic feature sequence samples, and obtaining the probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary; The probability value corresponding to each eigenvalue sample is used as the weight of the corresponding eigenvalue sample, and all eigenvalue samples are weighted summed according to the weight corresponding to each eigenvalue sample to obtain an acoustic contraction feature sample; Using a pre-trained semantic encoder to perform feature encoding on the acoustic contraction feature sample to obtain a semantic feature sample, and using a pre-trained decoder to decode the semantic feature sample to obtain a target translation text sample corresponding to the speech sample in a preset target language; The model loss is calculated according to the actual translation text and the corresponding target translation text samples, and the parameters of the pre-trained acoustic encoder, pre-trained semantic encoder, pre-trained decoder and boundary predictor are corrected according to the model loss until the model loss converges, thereby obtaining the trained acoustic encoder, trained boundary predictor, trained semantic encoder and trained decoder.

[0041] Among them, the speech translation target of this embodiment is to translate the target speech into a target translation text in another language based on a pre-trained model, wherein the trained boundary predictor is a key model for length contraction of acoustic feature sequences. In order to improve the accuracy and training efficiency of the boundary predictor through training, this embodiment first obtains a pre-trained acoustic encoder, a pre-trained semantic encoder and a pre-trained decoder, and trains the boundary predictor based on the pre-trained knowledge of the pre-trained acoustic encoder, the pre-trained semantic encoder and the pre-trained decoder, and fine-tunes the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder and the pre-trained decoder, thereby improving the accuracy of the speech translation result.

[0042] Specifically, this embodiment obtains several speech samples as training samples, obtains the actual translation text corresponding to each speech sample as a training label, recognizes and translates the speech samples to obtain corresponding target translation text samples, and calculates the model loss based on the actual translation text and the corresponding target translation text samples to correct the parameters of the pre-trained acoustic encoder, pre-trained semantic encoder, pre-trained decoder and boundary predictor until the model loss converges, and obtains the trained acoustic encoder, trained boundary predictor, trained semantic encoder and trained decoder as the model basis for recognizing and translating the target speech to be translated, thereby improving the accuracy of the speech translation result corresponding to the target speech.

[0043] Optionally, the training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder, and the trained decoder further includes: Obtain boundary feature value samples in the acoustic feature sequence samples, and determine the true probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary according to the boundary feature value samples; Calculate the boundary prediction loss based on the probability value of all feature value samples predicted as boundaries and the true probability value; Correspondingly, the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor are modified according to the model loss until the model loss converges, and the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder are obtained, including: Calculate the sum of boundary prediction loss and model loss to get the total model loss; According to the total loss of the model, the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor are modified until the total loss of the model converges, thereby obtaining the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder.

[0044] Among them, in order to improve the accuracy of boundary prediction of acoustic feature sequences by the trained boundary predictor, this embodiment obtains boundary feature value samples in the acoustic feature sequence samples, and determines the true probability value of each feature value sample in the acoustic feature sequence samples predicted as a boundary based on the boundary feature value samples. The closer the probability value of the feature value predicted as a boundary and the true probability value are, the higher the accuracy of the boundary predictor is characterized. Therefore, the boundary prediction loss is calculated based on the probability values ​​of all feature value samples predicted as boundaries and the true probability values, and the sum of the boundary prediction loss and the model loss is taken as the total model loss to correct the model parameters, so as to obtain a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder and a trained decoder, so as to improve the accuracy of model training.

[0045] Optionally, obtaining a boundary feature value sample in the acoustic feature sequence sample includes: Obtain a trained connection time series classifier, use the trained connection time series classifier to perform boundary prediction on the acoustic feature sequence samples, and obtain a prediction probability value that each feature value sample in the acoustic feature sequence samples is predicted to be a boundary; For any eigenvalue sample in the acoustic feature samples, the predicted probability value corresponding to the eigenvalue sample is compared with a preset threshold to obtain a comparison result of the corresponding eigenvalue sample; If the comparison result is that the predicted probability value is greater than the preset threshold, the eigenvalue sample is determined to be a boundary eigenvalue sample; All eigenvalue samples in the acoustic feature sequence samples are traversed to obtain all boundary eigenvalue samples in the acoustic feature sequence samples.

[0046] Among them, the trained connection time series classifier can perform boundary prediction on the acoustic feature sequence samples, and obtain the predicted probability value of each eigenvalue sample in the acoustic feature sequence samples being predicted as a boundary. The larger the predicted probability value, the higher the possibility that the corresponding eigenvalue is a boundary. The predicted probability value corresponding to the eigenvalue sample is compared with the preset threshold, and the comparison result of the corresponding eigenvalue sample can be obtained. Correspondingly, if the comparison result is that the predicted probability value is greater than the preset threshold, the eigenvalue sample is determined to be a boundary eigenvalue sample. By traversing all the eigenvalue samples in the acoustic feature sequence samples, all the boundary eigenvalue samples in the acoustic feature sequence samples can be obtained as training labels for the boundary predictor to improve the prediction accuracy of the boundary predictor.

[0047] Optionally, the model loss is calculated based on the actual translation text and the corresponding target translation text samples, including: Performing vector conversion on the actual translation text to obtain a first text vector corresponding to the actual translation text; Performing vector conversion on the target translation text sample to obtain a second text vector corresponding to the target translation text sample; A first similarity between the first text vector and the second text vector is calculated, a first difference between the first similarity and a first preset value is calculated, and the first difference is determined as a model loss.

[0048] The higher the similarity between the actual translated text and the corresponding target translated text sample, the higher the accuracy of the representation model. Therefore, the similarity between the actual translated text and the corresponding target translated text sample can be calculated as the basis for model training.

[0049] In order to facilitate the calculation of the similarity between texts, this embodiment first performs vector conversion on the actual translation text and the target translation text sample to obtain a first text vector corresponding to the actual translation text and a second text vector corresponding to the target translation text sample, then calculates the first similarity between the first text vector and the second text vector, and calculates the first difference between the first similarity and a first preset value, and determines the first difference as the model loss.

[0050] The preset value may be set according to actual conditions. For example, if the value range of the first similarity is in the range of [0, 1], the preset value may be set to 1.

[0051] Optionally, the boundary prediction loss is calculated based on the probability values ​​of all feature value samples predicted as boundaries and the true probability values, including: For any eigenvalue sample, calculate a second similarity between a probability value of the eigenvalue sample predicted as a boundary and a true probability value, calculate a second difference between the second similarity and a second preset value, and determine the second difference as a boundary prediction sub-loss of the corresponding eigenvalue sample; Traverse all eigenvalue samples and obtain the boundary predictor loss corresponding to each eigenvalue sample; The boundary prediction sub-losses corresponding to all eigenvalue samples are summed to obtain a sum result, and the sum result is determined to be the boundary prediction loss.

[0052] Among them, the higher the similarity between the actual translation text and the corresponding target translation text sample, the higher the accuracy of the characterization model. Therefore, this embodiment calculates the second similarity between the probability value of the feature value sample predicted as the boundary and the true probability value, calculates the second difference between the second similarity and the second preset value, determines the second difference as the boundary prediction sub-loss of the corresponding feature value sample, and sums the boundary prediction sub-losses corresponding to all feature value samples to obtain the summation result, and determines the summation result as the boundary prediction loss.

[0053] The above-mentioned steps of using a trained semantic encoder to encode the acoustic contraction feature to obtain a semantic feature, using a trained decoder to decode the semantic feature, and obtaining a target translation text corresponding to the target speech in a preset target language, extracting both the acoustic feature information and the semantic feature information in the target speech, and decoding the semantic feature with feature length contraction, obtaining the target translation text corresponding to the target speech in the preset target language as the speech translation result of the target speech, and improving the accuracy of the speech translation result.

[0054] The embodiment of the present invention obtains a target speech to be translated, uses a trained acoustic encoder to perform feature encoding on the target speech, obtains an acoustic feature sequence composed of feature value arrangements, uses a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, obtains a probability value that each feature value in the acoustic feature sequence is predicted to be a boundary, uses the probability value corresponding to each feature value as the weight of the corresponding feature value, performs weighted summation on all feature values ​​according to the weight corresponding to each feature value, obtains an acoustic contraction feature, uses a trained semantic encoder to perform feature encoding on the acoustic contraction feature, obtains a semantic feature, uses a trained decoder to decode the semantic feature, obtains a target translation text corresponding to the target speech in a preset target language, and by shrinking the acoustic features, eliminates the length difference problem between speech features and text features, so that the pre-trained model knowledge can be effectively inherited when performing speech translation on the target speech, thereby improving the accuracy of the target translation text and improving the work efficiency and work quality of doctors in the field of digital medical technology.

[0055] Corresponding to the speech translation method of the above embodiment, Figure 3 A structural block diagram of an artificial intelligence-based speech translation device provided in Embodiment 2 of the present invention is given. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0056] See also Figure 3 , the speech translation device comprises: The feature encoding module 31 is used to obtain the target speech to be translated, and use the trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence composed of feature value arrangements; A boundary prediction module 32 is used to use a trained boundary predictor to perform boundary prediction on the acoustic feature sequence to obtain a probability value that each feature value in the acoustic feature sequence is predicted to be a boundary; A feature contraction module 33 is used to use the probability value corresponding to each feature value as the weight of the corresponding feature value, and perform weighted summation on all feature values ​​according to the weight corresponding to each feature value to obtain an acoustic contraction feature; The speech translation module 34 is used to perform feature encoding on the acoustic contraction feature using a trained semantic encoder to obtain semantic features, and to decode the semantic features using a trained decoder to obtain a target translation text corresponding to the target speech in a preset target language.

[0057] Optionally, the speech translation device further includes: A sample acquisition module, used to acquire a pre-trained acoustic encoder, a pre-trained semantic encoder, a pre-trained decoder, a boundary predictor, and a number of speech samples and the actual translation text corresponding to each speech sample; A sample feature encoding module is used to perform feature encoding on any speech sample using a pre-trained acoustic encoder to obtain an acoustic feature sequence sample consisting of feature value samples; A sample boundary prediction module is used to use a boundary predictor to perform boundary prediction on acoustic feature sequence samples, and obtain a probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary; A sample feature shrinkage module is used to use the probability value corresponding to each eigenvalue sample as the weight of the corresponding eigenvalue sample, and perform weighted summation on all eigenvalue samples according to the weight corresponding to each eigenvalue sample to obtain an acoustic shrinkage feature sample; A sample speech translation module is used to perform feature encoding on the acoustic contraction feature sample using a pre-trained semantic encoder to obtain a semantic feature sample, and to decode the semantic feature sample using a pre-trained decoder to obtain a target translation text sample corresponding to the speech sample in a preset target language; The parameter correction module is used to calculate the model loss according to the actual translation text and the corresponding target translation text sample, and correct the parameters of the pre-trained acoustic encoder, pre-trained semantic encoder, pre-trained decoder and boundary predictor according to the model loss until the model loss converges, thereby obtaining the trained acoustic encoder, trained boundary predictor, trained semantic encoder and trained decoder.

[0058] Optionally, the parameter correction module includes: A first vector conversion submodule, used to perform vector conversion on the actual translation text to obtain a first text vector corresponding to the actual translation text; A second vector conversion submodule, used to perform vector conversion on the target translation text sample to obtain a second text vector corresponding to the target translation text sample; The model loss calculation submodule is used to calculate a first similarity between the first text vector and the second text vector, calculate a first difference between the first similarity and a first preset value, and determine the first difference as the model loss.

[0059] Optionally, the speech translation device further includes: A true probability value determination module is used to obtain boundary feature value samples in the acoustic feature sequence samples, and determine the true probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary according to the boundary feature value samples; A boundary prediction loss calculation module is used to calculate the boundary prediction loss based on the probability values ​​and true probability values ​​of all feature value samples predicted as boundaries; Correspondingly, the above parameter correction module includes: The model total loss calculation submodule is used to calculate the sum of the boundary prediction loss and the model loss to obtain the model total loss; The parameter correction submodule is used to correct the parameters of the pre-trained acoustic encoder, pre-trained semantic encoder, pre-trained decoder and boundary predictor according to the total loss of the model until the total loss of the model converges to obtain the trained acoustic encoder, trained boundary predictor, trained semantic encoder and trained decoder.

[0060] Optionally, the true probability value determination module includes: A prediction probability value determination submodule is used to obtain a trained connection time series classifier, use the trained connection time series classifier to perform boundary prediction on the acoustic feature sequence samples, and obtain a prediction probability value that each feature value sample in the acoustic feature sequence samples is predicted to be a boundary; The probability value comparison submodule is used to compare the predicted probability value corresponding to any eigenvalue sample in the acoustic feature samples with a preset threshold value to obtain a comparison result of the corresponding eigenvalue sample; A first boundary determination submodule, configured to determine that the feature value sample is a boundary feature value sample if the comparison result is that the predicted probability value is greater than a preset threshold; The second boundary determination submodule is used to traverse all feature value samples in the acoustic feature sequence samples to obtain all boundary feature value samples in the acoustic feature sequence samples.

[0061] Optionally, the above-mentioned boundary prediction loss calculation module includes: A first sub-loss calculation submodule is used to calculate, for any eigenvalue sample, a second similarity between a probability value of the eigenvalue sample predicted as a boundary and a true probability value, calculate a second difference between the second similarity and a second preset value, and determine the second difference as a boundary prediction sub-loss of the corresponding eigenvalue sample; The second sub-loss calculation sub-module is used to traverse all eigenvalue samples and obtain the boundary prediction sub-loss corresponding to each eigenvalue sample; The boundary prediction loss calculation submodule is used to sum the boundary prediction sub-losses corresponding to all eigenvalue samples to obtain a sum result, and determine the sum result as the boundary prediction loss.

[0062] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0063] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned speech translation method embodiments are implemented.

[0064] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0065] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0066] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0067] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0068] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0069] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0071] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A speech translation method based on artificial intelligence, characterized in that: The speech translation method comprises: Obtaining a target speech to be translated, and using a trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence consisting of feature value arrangements; Using a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, and obtaining a probability value of each feature value in the acoustic feature sequence being predicted as a boundary; The probability value corresponding to each eigenvalue is used as the weight of the corresponding eigenvalue, and all eigenvalues ​​are weighted summed according to the weight corresponding to each eigenvalue to obtain the acoustic contraction feature; The acoustic contraction feature is feature encoded using a trained semantic encoder to obtain a semantic feature, and the semantic feature is decoded using a trained decoder to obtain a target translation text corresponding to the target speech in a preset target language.

2. The speech translation method according to claim 1, characterized in that: The training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder includes: Obtain a pre-trained acoustic encoder, a pre-trained semantic encoder, a pre-trained decoder, a boundary predictor, and a number of speech samples and the actual translation text corresponding to each speech sample; For any speech sample, use the pre-trained acoustic encoder to perform feature encoding on the speech sample to obtain an acoustic feature sequence sample consisting of feature value samples; Using the boundary predictor to perform boundary prediction on the acoustic feature sequence samples, and obtaining a probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary; The probability value corresponding to each eigenvalue sample is used as the weight of the corresponding eigenvalue sample, and all eigenvalue samples are weighted summed according to the weight corresponding to each eigenvalue sample to obtain an acoustic contraction feature sample; Using a pre-trained semantic encoder to perform feature encoding on the acoustic contraction feature sample to obtain a semantic feature sample, and using a pre-trained decoder to decode the semantic feature sample to obtain a target translation text sample corresponding to the speech sample in a preset target language; The model loss is calculated according to the actual translation text and the corresponding target translation text sample, and the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor are corrected according to the model loss until the model loss converges, thereby obtaining a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder and a trained decoder.

3. The speech translation method according to claim 2, characterized in that: The calculating the model loss according to the actual translation text and the corresponding target translation text sample comprises: Performing vector conversion on the actual translation text to obtain a first text vector corresponding to the actual translation text; Performing vector conversion on the target translation text sample to obtain a second text vector corresponding to the target translation text sample; A first similarity between the first text vector and the second text vector is calculated, a first difference between the first similarity and a first preset value is calculated, and the first difference is determined as the model loss.

4. The speech translation method according to claim 2, characterized in that: The training process of the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder and the trained decoder further includes: Acquire boundary feature value samples in the acoustic feature sequence samples, and determine, according to the boundary feature value samples, a true probability value of each feature value sample in the acoustic feature sequence samples being predicted as a boundary; Calculate the boundary prediction loss based on the probability values ​​of all the feature value samples predicted as boundaries and the true probability values; Correspondingly, modifying the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder, and the boundary predictor according to the model loss until the model loss converges, and obtaining the trained acoustic encoder, the trained boundary predictor, the trained semantic encoder, and the trained decoder includes: Calculating the sum of the boundary prediction loss and the model loss to obtain the total model loss; The parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor are corrected according to the total loss of the model until the total loss of the model converges, thereby obtaining a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder and a trained decoder.

5. The speech translation method according to claim 4, characterized in that: The obtaining of the boundary feature value samples in the acoustic feature sequence samples comprises: Obtaining a trained connection time series classifier, and using the trained connection time series classifier to perform boundary prediction on the acoustic feature sequence samples to obtain a predicted probability value that each feature value sample in the acoustic feature sequence samples is predicted to be a boundary; For any of the eigenvalue samples in the acoustic feature samples, comparing the predicted probability value corresponding to the eigenvalue sample with a preset threshold value to obtain a comparison result corresponding to the eigenvalue sample; If the comparison result is that the predicted probability value is greater than the preset threshold, determining that the eigenvalue sample is a boundary eigenvalue sample; All feature value samples in the acoustic feature sequence samples are traversed to obtain all boundary feature value samples in the acoustic feature sequence samples.

6. The speech translation method according to claim 4, characterized in that: The calculation of the boundary prediction loss according to the probability values ​​and true probability values ​​of all the feature value samples predicted as boundaries includes: For any of the eigenvalue samples, calculating a second similarity between a probability value of the eigenvalue sample predicted as a boundary and a true probability value, calculating a second difference between the second similarity and a second preset value, and determining the second difference as a boundary prediction sub-loss corresponding to the eigenvalue sample; Traverse all eigenvalue samples and obtain the boundary predictor loss corresponding to each eigenvalue sample; The boundary prediction sub-losses corresponding to all eigenvalue samples are summed to obtain a sum result, and the sum result is determined to be the boundary prediction loss.

7. A speech translation device based on artificial intelligence, characterized in that: The speech translation device comprises: A feature encoding module is used to obtain a target speech to be translated, and use a trained acoustic encoder to perform feature encoding on the target speech to obtain an acoustic feature sequence composed of feature value arrangements; A boundary prediction module, used to use a trained boundary predictor to perform boundary prediction on the acoustic feature sequence, and obtain a probability value that each feature value in the acoustic feature sequence is predicted to be a boundary; A feature contraction module is used to take the probability value corresponding to each eigenvalue as the weight of the corresponding eigenvalue, and perform weighted summation on all eigenvalues ​​according to the weight corresponding to each eigenvalue to obtain an acoustic contraction feature; The speech translation module is used to use a trained semantic encoder to perform feature encoding on the acoustic contraction feature to obtain a semantic feature, and use a trained decoder to decode the semantic feature to obtain a target translation text corresponding to the target speech in a preset target language.

8. The speech translation device according to claim 7, characterized in that: The speech translation device also includes: A sample acquisition module, used to acquire a pre-trained acoustic encoder, a pre-trained semantic encoder, a pre-trained decoder, a boundary predictor, and a number of speech samples and the actual translation text corresponding to each speech sample; A sample feature encoding module, for performing feature encoding on any speech sample using the pre-trained acoustic encoder to obtain an acoustic feature sequence sample consisting of feature value samples; A sample boundary prediction module, used to use the boundary predictor to perform boundary prediction on the acoustic feature sequence samples, and obtain a probability value that each of the feature value samples in the acoustic feature sequence samples is predicted to be a boundary; A sample feature contraction module is used to use the probability value corresponding to each eigenvalue sample as the weight of the corresponding eigenvalue sample, and perform weighted summation on all eigenvalue samples according to the weight corresponding to each eigenvalue sample to obtain an acoustic contraction feature sample; A sample speech translation module is used to perform feature encoding on the acoustic contraction feature sample using a pre-trained semantic encoder to obtain a semantic feature sample, and to decode the semantic feature sample using a pre-trained decoder to obtain a target translation text sample corresponding to the speech sample in a preset target language; A parameter correction module is used to calculate the model loss according to the actual translation text and the corresponding target translation text sample, and correct the parameters of the pre-trained acoustic encoder, the pre-trained semantic encoder, the pre-trained decoder and the boundary predictor according to the model loss until the model loss converges, thereby obtaining a trained acoustic encoder, a trained boundary predictor, a trained semantic encoder and a trained decoder.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the speech translation method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech translation method according to any one of claims 1 to 6 is implemented.

Images