Speech translation method and device, storage medium and program product
By constructing a multimodal terminology database and leveraging the weighting advantage of speech modalities during the translation process, the problem of low translation quality of professional terms in existing technologies has been solved, achieving higher translation accuracy and terminology comprehension.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- IFLYTEK CO LTD
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-08
AI Technical Summary
Existing speech translation methods have low translation quality for specialized terminology.
A multimodal terminology database is constructed to store the multimodal information of each term. Target terms related to source language speech segments are retrieved from the multimodal terminology database, and translation is performed based on the modal information of each target term and the pre-configured weights of different modalities, especially with the weight of speech modalities being greater than that of non-speech modalities.
It improved the quality of translation of professional terms, enhanced the understanding of terms during the translation process, eliminated homonyms, and improved the accuracy of translation.
Smart Images
Figure CN121997950A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech translation method, device, storage medium, and program product. Background Technology
[0002] Speech translation refers to translating spoken language into text in a target language. Current speech translation methods generally have low translation quality for specialized terminology. Summary of the Invention
[0003] In view of the above problems, this application provides a speech translation method, device, storage medium, and program product to improve the translation quality of specialized terminology. The specific solution is as follows:
[0004] The first aspect of this application provides a speech translation method, the method comprising:
[0005] Obtain the source language audio segment to be translated;
[0006] Retrieve terms related to the source language speech segment from a multimodal terminology database; the multimodal information of each term in the multimodal terminology database includes: source language text, target language text, and source language speech;
[0007] If a target term related to the source language speech segment is retrieved, the source language speech segment is translated based on the modal information of the target term and the pre-configured weights of different modalities to obtain a target language text segment; wherein the weight of the speech modality is greater than the weight of the non-speech modality.
[0008] In one possible implementation, retrieving terms related to the source language speech segment from a multimodal terminology database includes:
[0009] At least based on the source language speech fragment and its transcribed text, retrieve terms related to the source language speech fragment from a multimodal terminology database.
[0010] In one possible implementation, retrieving terms related to the source language speech segment from a multimodal terminology database based on the source language speech segment and its transcribed text includes:
[0011] For each term in the multimodal terminology database, using the length of the source language speech of the term as the target length, a sub-speech segment of the target length is determined within the source language speech segment; for each sub-speech segment, whether the term is related to the source language speech segment is determined using any of the following methods:
[0012] Calculate the first similarity between the sub-speech segment and the source language speech of the term, and the second similarity between the transcribed text of the sub-speech segment and the source language text of the term. Weight the first similarity and the second similarity and fuse them to obtain the comprehensive similarity between the sub-speech segment and the term. The term with the largest comprehensive similarity to the sub-speech segment and greater than the threshold is identified as the target term related to the source language speech segment.
[0013] Alternatively, calculate the first similarity between the sub-speech segment and the source language speech of the term, and determine the term with the largest first similarity to the sub-speech segment that is greater than a first threshold as the target term related to the source language speech segment; otherwise, calculate the second similarity between the transcribed text of the sub-speech segment and the source language text of the term, and determine the term with the largest second similarity to the sub-speech segment that is greater than a second threshold as the target term related to the source language speech segment;
[0014] Alternatively, obtain a first multimodal feature determined at least based on the sub-speech segment and its transcribed text, and a second multimodal feature determined at least based on the source language speech and source language text of the term; calculate the similarity between the first multimodal feature and the second multimodal term feature; and determine the term with the highest similarity to the first multimodal feature and greater than a fifth threshold as the target term related to the source language speech segment.
[0015] In one possible implementation, the source language speech segment is translated based on the modal information of the target term and pre-configured weights of different modalities, including:
[0016] The source language speech segment is encoded to obtain an encoded feature sequence;
[0017] Encode each modal information of the target term to obtain the encoding features of each modality of the target term;
[0018] The encoded features of each modality of the target term are weighted and fused to obtain the multimodal features of the target term;
[0019] The encoded feature sequence is decoded based on the multimodal features of the target term to obtain the target language text fragment.
[0020] In one possible implementation, the multimodal information of each term also includes the related image of the term;
[0021] The text modality has a greater weight than the image modality.
[0022] In one possible implementation, the source language speech segment is translated based on the modal information of the target term and pre-configured weights of different modalities, including:
[0023] The sub-speech segments related to the target term in the source language speech segment are replaced with the source language speech of the target term to obtain the updated source language speech segment;
[0024] Based on the modal information of the target term and the pre-configured weights of different modalities, the updated source language speech segment is translated to obtain the target language text segment.
[0025] In one possible implementation, the process of retrieving terms related to the source language speech segment from a multimodal terminology database and translating the source language speech segment is implemented using a large model, which is trained in the following manner:
[0026] The large model is pre-trained based on multiple tasks to obtain a pre-trained large model; the multiple tasks include: text translation task, terminology speech recognition task, speech-image matching task, and text-image matching task.
[0027] The pre-trained large model is fine-tuned based on the speech translation task to obtain a well-trained large model.
[0028] In one possible implementation, fine-tuning the pre-trained large model based on a speech translation task includes:
[0029] The parameters of the large model are updated by taking the source language speech samples and the multimodal information of the related terms as input, and taking the target language text output by the large model as close as possible to the target language text label of the source language speech samples.
[0030] In the target language text labels of the source language speech samples, a preset label is added before the term, indicating that the term is followed by the preset label.
[0031] A second aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the speech translation method of the first aspect or any implementation thereof.
[0032] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0033] The memory is used to store computer programs;
[0034] The processor is used to execute the computer program so that the electronic device can implement the speech translation method of the first aspect or any implementation thereof.
[0035] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the speech translation method described in the first aspect or any implementation thereof.
[0036] By employing the above technical solutions, the speech translation method, device, storage medium, and program product provided in this application retrieve terms related to source language speech segments from a multimodal terminology database. If a target term related to the source language speech segment to be translated is found, the source language speech segment is translated based on the modal information of the target term and the pre-configured weights of different modalities to obtain a target language text segment; wherein the weight of the speech modality is greater than the weight of the non-speech modality. This application constructs a multimodal terminology database, which stores the multimodal information of each term. For a source language speech segment to be translated, before translation, target terms related to the source language speech segment to be translated are searched in the multimodal terminology database. After the target term is found, the source language speech segment is translated based on the modal information of the target term and the pre-configured weights of different modalities, thereby enhancing the understanding of terms during the translation process and improving the translation quality. Attached Figure Description
[0037] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0038] Figure 1 A flowchart illustrating an implementation of the speech translation method provided in this application;
[0039] Figure 2 A flowchart illustrating an implementation method for retrieving terms related to source language speech segments from a multimodal terminology database based on source language speech segments and their transcribed texts, as provided in this application.
[0040] Figure 3 Another implementation flowchart for retrieving terms related to source language speech segments from a multimodal terminology database based on source language speech segments and their transcribed texts, provided for this application;
[0041] Figure 4 A flowchart illustrating another implementation method for retrieving terms related to source language speech segments from a multimodal terminology database based on source language speech segments and their transcribed texts, as provided in this application.
[0042] Figure 5A flowchart illustrating an implementation of translating source language speech segments based on modal information of target terms and pre-configured weights of different modalities, as provided in this application.
[0043] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation
[0044] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.
[0045] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.
[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.
[0047] This application is made in order to improve the accuracy of the translation of technical terms (hereinafter referred to as "terms").
[0048] The speech translation function of this application can be used in simultaneous interpretation scenarios, as well as other translation scenarios, such as translating a user-specified speech segment.
[0049] like Figure 1 The diagram shown is a flowchart of one implementation of the speech translation method provided in this application, which may include:
[0050] Step S101: Obtain the source language speech segment to be translated.
[0051] The source language speech segment to be translated can be a user-specified speech segment, or it can be a segment of a user-specified speech segment. The speech translation method of this application is used to translate each segment of the speech segment.
[0052] In simultaneous interpretation scenarios, while the speaker is speaking in the source language, the audio acquisition device will collect the speech in real time. For each segment of source language speech of a preset duration that is collected, the speech translation method of this application will be called to translate the collected source language speech segment.
[0053] Step S102: Retrieve terms related to the source language speech segment from the multimodal terminology database. The multimodal information of each term in the multimodal terminology database includes: source language text, target language text, and source language speech.
[0054] You can use source language speech segments as search criteria to retrieve terms related to source language speech segments from a multimodal terminology database.
[0055] Alternatively, the transcribed text of the source language speech segment can be used as a search criterion to retrieve terms related to the source language speech segment from a multimodal terminology database.
[0056] Optionally, in the multimodal terminology database, the paralinguistic information of the source language speech for different terms may be the same or different. Paralinguistic information may include, but is not limited to, at least one of the following: speaker gender, accent, speech rate, etc.
[0057] The multimodal terminology database stores multimodal information on terms specific to a particular field. A specific field may include, but is not limited to, any of the following: medicine, science and technology, etc.
[0058] A multimodal terminology database can record the multimodal information of each term using structured information. For each term, a separate record is used to record the multimodal information. Each term record can include: term ID, source language text, target language text, and source language speech. As an example, the source language speech of a term can be stored as a file path or an embedding vector (such as audio coding features) to reduce the space occupied by the multimodal terminology database. The source language speech of a term can be found based on its file path. By storing the term's embedding vector, computational overhead during subsequent term retrieval is avoided.
[0059] Step S103: If a target term related to the source language speech segment is retrieved, the source language speech segment is translated based on the modal information of the target term and the pre-configured weights of different modalities to obtain a target language text segment; wherein the weight of the speech modality is greater than the weight of the non-speech modality.
[0060] There may be one or more target terms associated with a source language speech segment. In the case of multiple target terms, there is no overlap between the sub-speech segments of the source language speech segment corresponding to different target terms.
[0061] In the process of translating source language speech segments by referring to the modal information of the target term, the attention paid to different modal information of the target term varies according to the pre-configured weights of different modalities, with the highest attention paid to the information of the speech modality.
[0062] This application constructs a multimodal terminology database, which stores the multimodal information of each term. For a source language speech segment to be translated, before translation, the database is used to search for target terms related to the source language speech segment. After the target terms are found, the source language speech segment is translated based on the modal information of the target terms and the pre-configured weights of different modalities. Because the weight of the speech modality is relatively large, the translation process can better distinguish homophones, eliminate homophonic ambiguity, and enhance the understanding of terms in the translation process, thereby improving the accuracy of term translation, i.e., improving the quality of term translation.
[0063] If no target terms related to the source language speech segment are found, it means that the speech content of the source language speech segment does not contain terms. Therefore, the translation process does not need to refer to terminology information and can directly translate the source language speech segment.
[0064] Furthermore, in the multimodal terminology database, the multimodal information for each term can also include term-related images (denoted as term-related images). If the image contains text, that text is in the source language. For medical terms, related images may include, but are not limited to, anatomical diagrams, equipment illustrations, etc. For scientific and technical terms, related images may include, but are not limited to, schematic diagrams, interface diagrams, etc. Providing additional semantic constraints through visual information can enhance contextual understanding and further improve the accuracy of terminology translation.
[0065] Correspondingly, the text modality has a greater weight than the image modality.
[0066] In an optional embodiment, one implementation of retrieving terms related to source language speech segments from a multimodal terminology database can be:
[0067] At least based on the source language speech fragments and their transcribed texts, retrieve terms related to the source language speech fragments from a multimodal terminology database.
[0068] In other words, terms related to the source language speech segment are retrieved from a multimodal terminology database using information from at least two modalities of the source language speech segment as retrieval criteria.
[0069] Optionally, the flowchart of an implementation method for retrieving terms related to source language speech segments from a multimodal terminology database based on source language speech segments and their transcribed text provided in this application is as follows: Figure 2 As shown, it may include:
[0070] Step S201: For each term in the multimodal terminology database, with the length of the source language speech of the term as the target length, determine the sub-speech segment of the target length in the source language speech segment.
[0071] Optionally, assuming the target length is c, a sliding window with a preset step size (e.g., 1 speech frame) and a window size of c can be used to divide the source language speech segment into multiple sub-speech segments of length c.
[0072] Step S202: For each sub-speech segment, calculate the first similarity between the sub-speech segment and the source language speech of the term, and the second similarity between the transcribed text of the sub-speech segment and the source language text of the term.
[0073] The sub-speech segment can be encoded using the encoding module of the large model to obtain the first speech coding feature; the source language speech of the term can be encoded using the encoding module of the large model to obtain the second speech coding feature; the cosine similarity between the first speech coding feature and the second speech coding feature can be calculated, and the cosine similarity can be normalized to obtain the first similarity.
[0074] A speech segment can be transcribed using a speech recognition model to obtain the transcribed text. The edit distance between the transcribed text and the source language text of the term can be calculated and normalized to obtain a second similarity. Alternatively, the cosine similarity between the first text encoding feature of the transcribed text and the second text encoding feature of the source language text of the term can be calculated and normalized to obtain a second similarity. The transcribed text of the sub-speech segment can be encoded using the encoding module of a large model to obtain the first text encoding feature, and the source language text of the term can be encoded using the encoding module of the large model to obtain the second text encoding feature.
[0075] The second speech coding feature and the second text coding feature can be pre-stored in the multimodal terminology database, which can reduce the computational load of the term search process.
[0076] Step S203: Weighted fusion of the first similarity and the second similarity to obtain the comprehensive similarity between the sub-speech segment and the term.
[0077] The weight of the first similarity score is greater than the weight of the second similarity score. The first and second similarity scores can be weighted and summed to obtain the overall similarity score between the sub-speech segment and the term.
[0078] Step S204: The term with the highest comprehensive similarity to the sub-speech segment and greater than a preset threshold is identified as the target term related to the sub-speech segment, that is, the target term related to the source language speech segment.
[0079] Furthermore, if a video segment associated with the source language segment to be translated is also obtained simultaneously, keyframes can be extracted from the sub-video segment corresponding to the sub-speech segment. The third similarity between the keyframe and the related image of the term can be calculated. Based on this, step 203 can be replaced by: weighted fusion (e.g., weighted summation) of the first, second, and third similarities to obtain the comprehensive similarity between the sub-speech segment and the term. The weight of the second similarity is greater than the weight of the third similarity.
[0080] Optionally, another implementation flowchart of retrieving terms related to source language speech segments from a multimodal terminology database based on source language speech segments and their transcribed text provided in this application embodiment is shown below. Figure 3 As shown, it may include:
[0081] Step S301: For each term in the multimodal terminology database, with the length of the source language speech of the term as the target length, determine the sub-speech segment of the target length in the source language speech segment.
[0082] For the specific implementation process, please refer to step S201, which will not be repeated here.
[0083] Step S302: For each sub-speech segment, calculate the first similarity between the sub-speech segment and the source language speech of the term (denoted as the first similarity between the sub-speech segment and the term), and determine the term with the largest first similarity to the sub-speech segment and greater than the first threshold as the target term related to the source language speech segment.
[0084] The calculation process for the first similarity is described in the aforementioned embodiments and will not be repeated here.
[0085] Step S303: If there is no term with the highest first similarity to the sub-speech segment and greater than the first threshold, calculate the second similarity between the transcribed text of the sub-speech segment and the source language text of the term (denoted as the second similarity between the sub-speech segment and the term), and determine the term with the highest second similarity to the sub-speech segment and greater than the second threshold as the target term related to the source language speech segment.
[0086] The calculation process for the second similarity is described in the aforementioned embodiments and will not be repeated here.
[0087] Furthermore, if there is no term with the highest second similarity to the sub-speech segment and greater than the second threshold, it is determined that there is no target term related to the source language speech segment.
[0088] Alternatively, further, if a source language speech segment to be translated is obtained along with its associated video segment, and if no term with the highest second similarity score greater than the second threshold exists for that sub-speech segment, keyframes can be extracted from the corresponding sub-video segment. A third similarity score (denoted as the third similarity score between the sub-speech segment and the term) between the keyframe and the related image of the term can be calculated. The term with the highest third similarity score greater than the third threshold is identified as the target term associated with the source language speech segment. If no term with the highest third similarity score greater than the third threshold exists for that sub-speech segment, it is determined that no target term is associated with the source language speech segment.
[0089] Optionally, another implementation flowchart of retrieving terms related to source language speech segments from a multimodal terminology database based on source language speech segments and their transcribed texts provided in this application is shown below. Figure 4 As shown, it may include:
[0090] Step S401: For each term in the multimodal terminology database, with the length of the source language speech of the term as the target length, determine the sub-speech segment of the target length in the source language speech segment.
[0091] For the specific implementation process, please refer to step S201, which will not be repeated here.
[0092] Step S402: For each sub-speech segment, obtain at least a first multimodal feature determined based on the sub-speech segment and its transcribed text, and at least a second multimodal feature determined based on the source language speech and source language text of the term.
[0093] Optionally, the sub-speech segment can be encoded using the encoding module of a large model to obtain a first speech encoding feature; the transcribed text of the sub-speech segment can also be encoded using the encoding module of the large model to obtain a first text encoding feature; the first speech encoding feature and the first text encoding feature can then be weighted and fused (e.g., weighted summation, or weighted concatenation) to obtain a first multimodal feature. Similarly, the source language speech of the term can be encoded using the encoding module of the large model to obtain a second speech encoding feature; the source language text of the term can also be encoded using the encoding module of the large model to obtain a second text encoding feature; the second speech encoding feature and the second text encoding feature can then be weighted and fused (e.g., weighted summation, or weighted concatenation) to obtain a second multimodal feature.
[0094] Optionally, if a source language speech segment to be translated is obtained along with an associated video segment, a keyframe can be extracted from the sub-video segment corresponding to the sub-speech segment, and a first multimodal feature determined based on the sub-speech segment, its transcribed text, and the keyframe, and a second multimodal feature determined at least based on the source language speech and source language text of the term and the related image of the term.
[0095] The keyframe can be encoded using the large model's encoding module to obtain the first image encoding feature. The first speech encoding feature, the first text encoding feature, and the first image encoding feature are then weighted and fused to obtain the first multimodal feature. Similarly, the related images of the term can be encoded using the large model's encoding module to obtain the second image encoding feature. The second speech encoding feature, the second text encoding feature, and the second image encoding feature are then weighted and fused to obtain the second multimodal feature.
[0096] The weight of the first speech coding feature is greater than the weight of the first text coding feature, and the weight of the first text coding feature is greater than the weight of the first image coding feature; the weight of the second speech coding feature is greater than the weight of the second text coding feature, and the weight of the second text coding feature is greater than the weight of the second image coding feature.
[0097] Step S403: Calculate the similarity between the first multimodal feature and the second multimodal term feature, and determine the term with the highest similarity to the first multimodal feature and greater than the fifth threshold as the target term related to the source language speech segment.
[0098] If there is no term with the highest similarity to the first multimodal feature and greater than the fifth threshold, it is determined that there is no target term related to the source language speech segment.
[0099] In an optional embodiment, the flowchart for translating source language speech segments based on the modal information of the target term and the pre-configured weights of different modalities is shown below. Figure 5 As shown, it may include:
[0100] Step S501: Encode the source language speech segment to obtain the encoded feature sequence.
[0101] The source language speech segments can be encoded using the encoding module of a large model to obtain encoded feature sequences.
[0102] Optionally, multimodal information of source language speech segments and target terms can be added to the prompt word template to obtain prompt words. These prompt words are then input into the large model to obtain the encoded feature sequence obtained by the encoding module. The prompt word template also includes task instructions that instruct the large model to encode the multimodal information of the target terms and translate the source language speech segments into target language text.
[0103] Step S502: Encode each modal information of the target term to obtain the encoding features of each modality of the target term.
[0104] The modal information of the target term can be encoded separately through the encoding module of the large model.
[0105] When the information of the target term's text modality includes both the source language text and the target language text, the source language text and the target language text can be encoded as a single text.
[0106] Optionally, the text modality information for each term in the multimodal terminology database may also include, but is not limited to, at least one of the following: the domain to which the term belongs, the degree of matching between the term and that domain, the definition of the term in that domain, example sentences, and other source language information. When encoding the text modality information, the source language text of the target term, the target language text, and at least one of the above-mentioned information can be encoded as a single text.
[0107] The large model can be used to determine the matching degree between a term and any given domain. The following prompts can be used to guide the large model in determining the matching degree between a term and any given domain: if a term appears only in any given domain, then the matching degree between this term and that domain is 100%; if it appears in the domain in the vast majority of cases, then the matching degree between this term and that domain is 80%.
[0108] Step S503: Weighted fusion of the encoding features of each modality of the target term to obtain the multimodal features of the target term.
[0109] As an example, the weight of the speech modality can be 0.5, the weight of the text modality can be 0.3, and the weight of the image modality can be 0.2. If there are only two modalities, the speech modality can have a weight of 0.6 and the text modality can have a weight of 0.4.
[0110] Step S504: Decode the encoded feature sequence based on the multimodal features of the target term to obtain the target language text fragment.
[0111] The encoded feature sequence and the multimodal features of the target term can be input into the decoding module of the large model. The decoding module decodes the encoded feature sequence based on the multimodal features of the target term to obtain the target language text fragment.
[0112] In an optional embodiment, one way to translate source language speech segments based on the modal information of the target terms and the pre-configured weights of different modalities can be:
[0113] The source language speech segment is updated by replacing the sub-speech segments related to the target term with the source language speech of the target term. Based on the modal information of the target term and the pre-configured weights of different modalities, the updated source language speech segment is translated to obtain the target language text segment.
[0114] By replacing sub-speech segments related to the target term in the source language speech segment with the source language speech of the target term, the translation process can focus more on the translation of technical terms, thereby further improving the accuracy of technical term translation.
[0115] The aforementioned large model can be trained in the following way:
[0116] A large model is pre-trained based on multiple tasks to obtain a pre-trained large model; these multiple tasks include, but are not limited to: text translation task, terminology speech recognition task, speech-image matching task, and text-image matching task.
[0117] Text translation is a task that translates source language text into target language text.
[0118] The term speech recognition task refers to the process of recognizing speech as text.
[0119] The speech-image matching task is to find an image in an image library that matches the input speech.
[0120] The text-image matching task is to find an image in an image library that matches the input text.
[0121] The training methods for each task can use existing methods, which will not be elaborated here.
[0122] Fine-tuning a pre-trained large model based on a speech translation task yields a well-trained large model. The fine-tuning process may include:
[0123] The large model is updated by taking source language speech samples and multimodal information of related terms as input, and taking the target language text output by the large model as close as possible to the target language text label of the source language speech samples.
[0124] The specific process of translating source language speech samples based on a large model can be found in the aforementioned embodiments, and will not be repeated here.
[0125] In the target language text labels of the source language speech samples, a preset label is added before the term, indicating that the term is followed by the preset label.
[0126] By adding preset labels before terms, large models can focus on the multimodal information of target terms when predicting preset labels during the inference process, thus enabling more accurate term translation.
[0127] Corresponding to the method embodiments, this application also provides an electronic device. (See reference...) Figure 6 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic device in these embodiments can be a terminal device (e.g., an in-vehicle infotainment system, a large-screen device, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0128] like Figure 6 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0129] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0130] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the speech translation methods provided in this application.
[0131] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the speech translation methods provided in this application.
[0132] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0134] In the above embodiments, the functionality can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product. Those skilled in the art can use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.
[0135] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0137] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A speech translation method, characterized in that, The method includes: Obtain the source language audio segment to be translated; Retrieve terms related to the source language speech segment from a multimodal terminology database; the multimodal information of each term in the multimodal terminology database includes: source language text, target language text, and source language speech; If a target term related to the source language speech segment is retrieved, the source language speech segment is translated based on the modal information of the target term and the pre-configured weights of different modalities to obtain a target language text segment; wherein the weight of the speech modality is greater than the weight of the non-speech modality.
2. The method according to claim 1, characterized in that, The step of retrieving terms related to the source language speech segment from the multimodal terminology database includes: At least based on the source language speech fragment and its transcribed text, retrieve terms related to the source language speech fragment from a multimodal terminology database.
3. The method according to claim 2, characterized in that, Retrieving terms related to the source language speech segment from a multimodal terminology database based on the source language speech segment and its transcribed text, including: For each term in the multimodal terminology database, using the length of the source language speech of the term as the target length, a sub-speech segment of the target length is determined within the source language speech segment; for each sub-speech segment, whether the term is related to the source language speech segment is determined using any of the following methods: Calculate the first similarity between the sub-speech segment and the source language speech of the term, and the second similarity between the transcribed text of the sub-speech segment and the source language text of the term. Weight the first similarity and the second similarity and fuse them to obtain the comprehensive similarity between the sub-speech segment and the term. The term with the largest comprehensive similarity to the sub-speech segment and greater than the threshold is identified as the target term related to the source language speech segment. Alternatively, calculate the first similarity between the sub-speech segment and the source language speech of the term, and determine the term with the largest first similarity to the sub-speech segment that is greater than a first threshold as the target term related to the source language speech segment; otherwise, calculate the second similarity between the transcribed text of the sub-speech segment and the source language text of the term, and determine the term with the largest second similarity to the sub-speech segment that is greater than a second threshold as the target term related to the source language speech segment; Alternatively, obtain a first multimodal feature determined at least based on the sub-speech segment and its transcribed text, and a second multimodal feature determined at least based on the source language speech and source language text of the term; calculate the similarity between the first multimodal feature and the second multimodal term feature; and determine the term with the highest similarity to the first multimodal feature and greater than a fifth threshold as the target term related to the source language speech segment.
4. The method according to claim 1, characterized in that, Based on the modal information of the target term and the pre-configured weights of different modalities, the source language speech segment is translated, including: The source language speech segment is encoded to obtain an encoded feature sequence; Encode each modal information of the target term to obtain the encoding features of each modality of the target term; The encoded features of each modality of the target term are weighted and fused to obtain the multimodal features of the target term; The encoded feature sequence is decoded based on the multimodal features of the target term to obtain the target language text fragment.
5. The method according to claim 1, characterized in that, The multimodal information for each term also includes the term's associated image; The text modality has a greater weight than the image modality.
6. The method according to claim 1, characterized in that, Based on the modal information of the target term and the pre-configured weights of different modalities, the source language speech segment is translated, including: The sub-speech segments related to the target term in the source language speech segment are replaced with the source language speech of the target term to obtain the updated source language speech segment; Based on the modal information of the target term and the pre-configured weights of different modalities, the updated source language speech segment is translated to obtain the target language text segment.
7. The method according to claim 1, characterized in that, The process of retrieving terms related to the source language speech segment from a multimodal terminology database and translating the source language speech segment is implemented through a large model, which is trained in the following manner: The large model is pre-trained based on multiple tasks to obtain a pre-trained large model; the multiple tasks include: text translation task, terminology speech recognition task, speech-image matching task, and text-image matching task. The pre-trained large model is fine-tuned based on the speech translation task to obtain a well-trained large model.
8. The method according to claim 7, characterized in that, Fine-tuning the pre-trained large model based on a speech translation task includes: The parameters of the large model are updated by taking the source language speech samples and the multimodal information of the related terms as input, and taking the target language text output by the large model as close as possible to the target language text label of the source language speech samples. In the target language text labels of the source language speech samples, a preset label is added before the term, indicating that the term is followed by the preset label.
9. An electronic device, characterized in that, The electronic device includes at least one processor and a memory connected to the processor; wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the speech translation method as described in any one of claims 1 to 8.
10. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the speech translation method as described in any one of claims 1 to 8.
11. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the speech translation method as described in any one of claims 1 to 8.