A multimodal speech translation method, system and terminal device

Abstract

The present invention proposes a multimodal speech translation method, system and terminal device, which belongs to the field of speech translation technology. The method includes: receiving a speech input sequence; translating the speech input sequence into an intermediate text sequence through a first modality; judging whether there is a predetermined text subsequence in the intermediate text sequence; if there is, translating the predetermined text subsequence into a target text sequence through a second modality, and replacing the predetermined text subsequence in the intermediate text sequence; and displaying the intermediate text sequence on the display interface. The system includes a speech receiving unit, a text modality translation unit, an intermediate result recognition unit, a digital modality translation unit, a picture modality translation unit and a dynamic display unit. The present invention also proposes a terminal device for implementing the method. The technical solution of the present invention can translate the speech sequence in a multimodal manner, and can highlight the target sequence of the speech sequence in a graphic manner, thereby improving the efficiency of speech communication.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech translation, and in particular relates to a multimodal speech translation method, system and terminal device, and a storage medium for implementing the method. Background Art

[0002] With the development of artificial intelligence technology, speech translation technology has achieved leapfrog development, from being able to translate simple sentences word by word at the beginning to being able to translate long sentences in real time. For example, most of the real-time speech translation engines in the prior art can already realize real-time recognition of continuous audio streams, real-time recognition and translation of speech input content, converting it into text information and returning the corresponding text stream.

[0003] For example, authorized Chinese invention patent CN113436606B discloses a method for translating original sound speech. The method sends the language pronunciation features as feature values ​​into the deep neural network DNN training and learning. After learning, the language feature model feature vector and the human voice feature model feature vector for reference of the translation and synthesis modules are obtained respectively. The speech synthesis simulation is performed through the synthetic speech module to emit a sound similar to the speaker's voice, so that the synthesized speech after translation is highly close to the characteristics of the speaker himself.

[0004] However, in practical applications, the inventors have found that in situations involving voice interaction and communication, not all voice interactions are effectively recorded. Since each person's expression habits are different, there are inevitably some meaningless voice sequences in the voice recording that are meaningless to the communication between the two parties. If these sequences are omitted, it will not affect the communication efficiency, so it is not necessary to translate the original text word for word; correspondingly, the two parties in the interaction may pay more attention to the key parts of the voice sequence, hoping to identify and highlight these key target sequences, so as to obtain better communication effects. However, the prior art does not take these issues into consideration. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a multimodal speech translation method, system and terminal device, and a storage medium for implementing the method.

[0006] In a first aspect of the present invention, a multimodal speech translation method is provided, wherein the method is applied to a terminal including a display interface and a speech receiving channel.

[0007] In terms of overall structure, the method comprises the following steps:

[0008] receiving a speech input sequence through the speech receiving channel;

[0009] Translating the speech input sequence into an intermediate text sequence through a first modality;

[0010] Determining whether the intermediate text sequence contains a predetermined text subsequence;

[0011] If it exists, the predetermined text subsequence is translated into a target text sequence through a second modality, and then the predetermined text subsequence in the intermediate text sequence is replaced;

[0012] The intermediate text sequence is displayed on the display interface.

[0013] wherein, determining whether the intermediate text sequence has a predetermined text subsequence;

[0014] If not, return to the step of receiving a speech input sequence through the speech receiving channel.

[0015] If so, translating the predetermined text subsequence into a target text sequence through a second modality, and displaying the target text sequence on the display interface;

[0016] The target text sequence is displayed in a manner different from the intermediate text sequence ...

[0017] As a further improvement, determining whether the intermediate text sequence has a predetermined text subsequence;

[0018] If so, determining whether the predetermined text subsequence meets the image conversion condition;

[0019] If it is consistent, translating the predetermined text subsequence into a picture sequence through a third modality, and displaying the picture sequence on the display interface;

[0020] The display area of ​​the picture sequence on the display interface is different from the display area of ​​the intermediate text sequence on the display interface.

[0021] When the method is implemented specifically by a computer program, the first modal translation is performed by a first process, the second modal translation is performed by a second process, and the third modal translation is performed by a third process;

[0022] The first process is a system resident process;

[0023] The display interface dynamically divides and merges multiple display areas based on the activation states of the second process and the third process.

[0024] As one of the specific application scenarios of the present invention, the predetermined text subsequence includes at least one of a digital text and a predetermined noun;

[0025] As a specific combined application scenario of the present invention, the predetermined text subsequence includes a chemical noun sequence;

[0026] Determining whether the predetermined text subsequence meets the image conversion condition specifically includes:

[0027] Determining whether the predetermined text subsequence contains digital text that can be converted into a digital picture displayed in a predetermined format;

[0028] For example, the chemical nouns may be converted into chemical formulas and combined with displayed pictures.

[0029] and / or;

[0030] It is determined whether the predetermined text subsequence contains a predetermined noun that can be converted into a corresponding picture display.

[0031] In a second aspect of the present invention, a multimodal speech translation system is proposed, the system comprising a speech receiving unit, a text modality translation unit, an intermediate result recognition unit, a digital modality translation unit, a picture modality translation unit and a dynamic display unit.

[0032] In terms of structural function, each subunit realizes the following functions:

[0033] The speech receiving unit is used to receive a speech input sequence;

[0034] The text modality translation unit translates the speech input sequence into an intermediate text sequence;

[0035] The intermediate result recognition unit is used to recognize whether the intermediate text sequence has a predetermined text subsequence, and activate the digital modality translation unit and / or the picture modality translation unit based on the recognition result;

[0036] The digital modality translation unit translates the digital text included in the predetermined text subsequence into a target text sequence;

[0037] The picture modality translation unit translates the target text sequence or the predetermined text subsequence into a picture sequence;

[0038] The dynamic display unit adjusts the number of display areas of the display interface of the multimodal speech translation system based on the activation status of the digital modality translation unit and the picture modality translation unit, and displays the intermediate text sequence, target text sequence, and picture sequence in different display areas in different display modes.

[0039] The predetermined text subsequence includes a chemical noun sequence;

[0040] The picture modality translation unit translates the predetermined text subsequence into a picture sequence, specifically including:

[0041] The picture modality translation unit converts the chemical nouns into chemical formulas and displays the combined pictures.

[0042] The target text sequence includes a digital text string;

[0043] The image modality translation unit translates the target text sequence into an image sequence, specifically including:

[0044] The image modality translation unit translates the digital text string into an image display sequence displayed in scientific notation, exponential notation, or N-base form.

[0045] The dynamic display unit is connected to the intermediate result recognition unit;

[0046] When the intermediate result recognition unit recognizes that the intermediate text sequence does not have a predetermined text subsequence, the initial setting state of the dynamic display unit is restored, and the initial setting state is a default state of the display interface of the system.

[0047] In a third aspect of the present invention, a terminal device is provided, which may be, for example, a data interaction device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program may be a data interaction program, and when the processor executes the computer program, the steps of the above-mentioned multimodal speech translation method are implemented.

[0048] In a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned multimodal speech translation method are implemented.

[0049] The technical solution of the present invention can translate the speech sequence in a multimodal manner, and can highlight the target sequence of the speech sequence in a graphic manner, thereby improving the efficiency of speech communication.

[0050] Specifically, the technical solution of the present invention, for a voice input sequence containing a predetermined text subsequence (digital text, predetermined nouns, chemical noun sequences), first translates the voice input sequence into an intermediate text sequence, identifies the predetermined text subsequences therein, and then performs text-to-image conversion to dynamically display voice text or picture sequences in different formats, so that both parties in communication can quickly grasp the key points of voice communication and improve communication efficiency.

[0051] It can also be seen in the subsequent further improvements of the embodiments that the present invention recognizes the audio stream as text in real time, is suitable for long sentence voice input, audio and video subtitles, conferences, speech subtitles on the same screen, etc., intelligent language processing, and can also perform intelligent error correction on the intermediate recognition results, display the intermediate text results in real time, and quickly identify the key audio streams; if it involves speech translation between different languages, summary translation can be achieved based on the key audio streams, further improving the efficiency of speech translation.

[0052] Further advantages of the present invention will be further reflected in detail in the specific embodiments section in conjunction with the drawings of the specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a main flow chart of a multimodal speech translation method according to an embodiment of the present invention;

[0055] Figure 2 is realized Figure 1 A schematic diagram of the internal architecture of a visualization terminal of the multimodal speech translation method;

[0056] Figure 3 is realized Figure 1 A schematic diagram of software data flow of the multimodal speech translation method;

[0057] Figure 4 is realized Figure 1 A schematic diagram of a unit module of a multimodal speech translation system of the multimodal speech translation method;

[0058] Figure 5-Figure 8 It is a schematic diagram of the technical effect of implementing speech translation in different scenarios using the technical solution of the present invention;

[0059] Fig. 9 is realized Figure 1 A schematic diagram of the structure of a terminal device of the method. DETAILED DESCRIPTION

[0060] The invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0061] Figure 1 It is a main flow chart of a multimodal speech translation method according to an embodiment of the present invention.

[0062] Figure 1 The method is applied to a terminal including a display interface and a voice receiving channel, and specifically includes the following steps:

[0063] receiving a speech input sequence through the speech receiving channel;

[0064] Translating the speech input sequence into an intermediate text sequence through a first modality;

[0065] Determining whether the intermediate text sequence contains a predetermined text subsequence;

[0066] If it exists, the predetermined text subsequence is translated into a target text sequence through a second modality, and then the predetermined text subsequence in the intermediate text sequence is replaced;

[0067] The intermediate text sequence is displayed on the display interface.

[0068] In order to better describe the implementation principle of the method and technical solution, Figure 2-Figure 3 This is explained from two aspects: hardware and software.

[0069] Figure 2 is realized Figure 1 A schematic diagram of the internal architecture of a visualization terminal of the multimodal speech translation method.

[0070] The visualization terminal device includes a voice receiving channel, a first modal unit, a recognition modal unit, a second modal unit and a display interface.

[0071] The voice receiving channel is used to receive a voice input sequence;

[0072] As an example, the speech input sequence may be a user's speech output captured in real time, or a real-time speech sequence transmitted online.

[0073] For example, in the online teaching of Chinese real-time teaching, the course voice sequence told by the teacher in real time is:

[0074] (1) “Hydrogen burns in oxygen to produce water. The reaction formula is: two hydrogen molecules + one oxygen molecule produce two water molecules”;

[0075] Of course, the online teaching can also be real-time teaching in English, and the course voice sequence narrated by the teacher in real time is:

[0076] (2) "Hydrogen burns in oxygen to generate water, the reaction formula is: two hydrogen molecules+one oxygen molecule to generate two watermolecules."

[0077] It can be understood that the prior art can directly translate the above-mentioned speech sequences into the above-mentioned text sequences (with Chinese-to-English translation or English-to-Chinese translation), thereby realizing text sequence output.

[0078] Translating the speech input sequence into an intermediate text sequence through a first modal unit;

[0079] The first modal unit is a speech-to-text modal unit, which is used to translate the speech sequence into a text sequence, for example, to obtain the aforementioned text sequence (1) or (2);

[0080] The recognition modality unit is used to determine whether the intermediate text sequence contains a predetermined text subsequence;

[0081] In various embodiments of the present invention, the predetermined text subsequence includes at least one of a digital text, a predetermined noun, and in particular a predetermined chemical noun;

[0082] Taking the above text sequence (1) or (2) as an example, there are predetermined nouns, especially predetermined chemical nouns (hydrogen, oxygen, combustion, generating water; two hydrogen molecules + one oxygen molecule generate two water molecules) in sequence (1);

[0083] At this time, the second modal unit is started, and after the predetermined text subsequence is translated into a target text sequence through the second modal unit, the predetermined text subsequence in the intermediate text sequence is replaced.

[0084] As an example, “hydrogen” can be translated into a target text sequence H through the second modal unit. 2 , "oxygen" is translated into the target text sequence O through the second modal unit 2 , "Hydrogen, oxygen, combustion, generate water; two hydrogen molecules + one oxygen molecule generate two water molecules" is translated into the target text sequence "2H 2 +O 2 =2H 2 O”.

[0085] Then, the intermediate text sequence is displayed on the display interface.

[0086] The above process can be found in Figure 3 , Figure 3 is realized Figure 1 A schematic diagram of the software data flow of the multimodal speech translation method.

[0087] exist Figure 3 In the embodiment, the intermediate text sequence is displayed on the display interface including a summary graphic display of the intermediate text sequence. The summary graphic display effect will be further introduced in the subsequent embodiments.

[0088] In the above method, it is determined whether the intermediate text sequence contains a predetermined text subsequence;

[0089] If not, return to the step of receiving a speech input sequence through the speech receiving channel.

[0090] If so, translating the predetermined text subsequence into a target text sequence through a second modality, and displaying the target text sequence on the display interface;

[0091] The target text sequence is displayed in a manner different from the intermediate text sequence ...

[0092] Preferably, the target text sequence is displayed on the display interface in the form of a picture, and the intermediate text sequence is displayed on the display interface in the form of a text.

[0093] As a further preferred embodiment, determining whether the intermediate text sequence has a predetermined text subsequence;

[0094] If so, determining whether the predetermined text subsequence meets the image conversion condition;

[0095] If it is consistent, translating the predetermined text subsequence into a picture sequence through a third modal unit, and displaying the picture sequence on the display interface;

[0096] The display area of ​​the picture sequence on the display interface is different from the display area of ​​the intermediate text sequence on the display interface.

[0097] Preferably, the picture sequence is displayed in a middle display area on the display interface, and the middle text sequence is displayed in a sidebar display area on the display interface.

[0098] According to the above introduction, the first modal unit is always in an active state, and the second modal unit is activated based on the recognition result of the recognition unit.

[0099] Therefore, in order to ensure the real-time performance and effectiveness of the process and save resources, in the above embodiment, the first modal translation is performed by the first process, the second modal translation is performed by the second process, and the third modal translation is performed by the third process;

[0100] The first process is a system resident process;

[0101] The display interface dynamically divides and merges multiple display areas based on the activation states of the second process and the third process.

[0102] In the above embodiment, the predetermined text subsequence includes at least one of a digital text and a predetermined noun;

[0103] Determining whether the predetermined text subsequence meets the image conversion condition specifically includes:

[0104] Determining whether the predetermined text subsequence contains digital text that can be converted into a digital picture displayed in a predetermined format;

[0105] and / or;

[0106] It is determined whether the predetermined text subsequence contains a predetermined noun that can be converted into a corresponding picture display.

[0107] The technical effect of this embodiment may be further described in the subsequent introduction.

[0108] exist Figure 1-Figure 3 Based on Figure 4 . Figure 4 is realized Figure 1 A schematic diagram of a unit module of a multimodal speech translation system of the multimodal speech translation method.

[0109] exist Figure 4 In the figure, a multimodal speech translation system is shown, which includes a speech receiving unit, a text modality translation unit, an intermediate result recognition unit, a digital modality translation unit, a picture modality translation unit and a dynamic display unit.

[0110] Specifically, in this embodiment,

[0111] The speech receiving unit is used to receive a speech input sequence;

[0112] The text modality translation unit translates the speech input sequence into an intermediate text sequence;

[0113] The intermediate result recognition unit is used to recognize whether the intermediate text sequence has a predetermined text subsequence, and activate the digital modality translation unit and / or the picture modality translation unit based on the recognition result;

[0114] The digital modality translation unit translates the digital text included in the predetermined text subsequence into a target text sequence;

[0115] The picture modality translation unit translates the target text sequence or the predetermined text subsequence into a picture sequence;

[0116] The dynamic display unit adjusts the number of display areas of the display interface of the multimodal speech translation system based on the activation status of the digital modality translation unit and the picture modality translation unit, and displays the intermediate text sequence, target text sequence, and picture sequence in different display areas in different display modes.

[0117] The predetermined text subsequence includes a chemical noun sequence;

[0118] The picture modality translation unit translates the predetermined text subsequence into a picture sequence, specifically including:

[0119] The picture modality translation unit converts the chemical nouns into chemical formulas and displays the combined pictures.

[0120] The target text sequence includes a digital text string;

[0121] The image modality translation unit translates the target text sequence into an image sequence, specifically including:

[0122] The image modality translation unit translates the digital text string into an image display sequence displayed in scientific notation, exponential notation, or N-base form.

[0123] The dynamic display unit is connected to the intermediate result recognition unit;

[0124] When the intermediate result recognition unit recognizes that the intermediate text sequence does not have a predetermined text subsequence, the initial setting state of the dynamic display unit is restored, and the initial setting state is a default state of the display interface of the system.

[0125] By default, there is only one text display area to save hardware resources.

[0126] Based on the above embodiments, Figure 5-Figure 8 The present invention further introduces further preferred embodiments of the present application.

[0127] exist Figure 5 In the process, the speech input sequence, the intermediate text sequence, the predetermined text subsequence, and the target text sequence are all in the same first language, and there is no cross-language translation process in the whole process.

[0128] exist Figure 5 In the speech input sequence, "the number of particles contained in one mole of any particle is Avogadro's constant, and its exact value is: 6.02214076 times 10 to the power of 23";

[0129] The speech-to-text engine of the prior art can realize the above-mentioned speech-to-text translation, that is, the text sequence is displayed as (the number of particles contained in one mole of any particles is Avogadro's constant, and its exact value is: 6.02214076 times 10 to the power of 23), which is called "intermediate text sequence" in the present invention.

[0130] However, the final display effect of the present invention is not the text of the prior art, but further determines whether there is a predetermined text subsequence in the intermediate text sequence; if so, the predetermined text subsequence is translated into a target text sequence through a second modality, and then the predetermined text subsequence in the intermediate text sequence is replaced; and the intermediate text sequence is displayed on the display interface.

[0131] Specific to Figure 5 In the example, "one mole, Avogadro's constant, and 6.02214076 times 10 to the power of 23" are all "predetermined text subsequences".

[0132] Obviously, the "predetermined text subsequence" here includes digital text and chemical nouns.

[0133] Figure 5 The final display effect is "The number of particles contained in 1 mol of any particle is Avogadro's constant N A , its exact value is: 6.02214076×10 23 . "

[0134] Obviously, Figure 5 The embodiment can display the key subsequence of the speech sequence in a predetermined format, so that both the learners can quickly grasp the key points.

[0135] Figure 6 The embodiment shows the situation where the speech input sequence, the intermediate text sequence, the predetermined text subsequence, and the target text sequence are in different languages, and the whole process involves a cross-language translation process.

[0136] That is, the speech input sequence and the intermediate text sequence are in the first language (English), and the predetermined text subsequence and the target text sequence are in the second language (Chinese).

[0137] It should be noted that Figure 6 In the embodiment, the speech sequence dictated by the narrator is in English, and the translated intermediate text is still in English instead of Chinese. The displayed predetermined text subsequence and target text sequence are both translated from the English text sequence instead of directly translating from the English speech sequence into Chinese. It can be understood that this translation method can better ensure that the expression of the predetermined text subsequence is retained intact in the original text, which is more in line with the invention purpose of this application.

[0138] Figure 7An example of a specific application of the present invention in the field of chemical reaction formula teaching is shown.

[0139] exist Figure 7 In the course, the teacher's spoken speech sequence is "Hydrogen burns in oxygen to produce water. The reaction formula is two hydrogen molecules plus one oxygen molecule to produce two water molecules";

[0140] The speech-to-text engine of the prior art can realize the above-mentioned speech-to-text translation, that is, the text sequence is displayed as (hydrogen burns in oxygen to generate water, and its reaction formula is two hydrogen molecules plus one oxygen molecule to generate two water molecules), which is called "intermediate text sequence" in the present invention.

[0141] However, the final display effect of the present invention is not the text of the prior art, but further determines whether there is a predetermined text subsequence in the intermediate text sequence; if so, the predetermined text subsequence is translated into a target text sequence through a second modality, and then the predetermined text subsequence in the intermediate text sequence is replaced; and the intermediate text sequence is displayed on the display interface.

[0142] Specific to Figure 7 In the example, "hydrogen, oxygen, combustion, generate, water, two hydrogen molecules plus one oxygen molecule generate two water molecules" are all "predetermined text subsequences".

[0143] Obviously, the "predetermined text subsequence" here includes digital text, chemical nouns and chemical reaction formulas.

[0144] Figure 7 The final display effect is "H 2 In O 2 Combustion generates H 2 0, the reaction formula is: 2H 2 +O 2 =2H 2 O”.

[0145] As a further improvement, Figure 7 In the method, it is determined whether the predetermined text subsequence contains digital text that can be converted into a digital picture displayed in a predetermined format; and / or it is determined whether the predetermined text subsequence contains a predetermined noun that can be converted into a corresponding picture display.

[0146] If included, the predetermined text subsequence is translated into a picture sequence through a third modality, and the picture sequence is displayed on the display interface; the display area of ​​the picture sequence on the display interface is different from the display area of ​​the intermediate text sequence on the display interface.

[0147] like Figure 7 As shown, the reaction formula can be: 2H2 +O 2 =2H 2 O" is indicated by a picture in the central display area.

[0148] exist Figure 7 In the process, the speech input sequence, the intermediate text sequence, the predetermined text subsequence, and the target text sequence are all in the same first language, and there is no cross-language translation process in the whole process.

[0149] Figure 8 The embodiment shows the situation where the speech input sequence, the intermediate text sequence, the predetermined text subsequence, and the target text sequence are in different languages, and the whole process involves a cross-language translation process.

[0150] That is, the speech input sequence and the intermediate text sequence are in the first language (English), and the predetermined text subsequence and the target text sequence are in the second language (Chinese).

[0151] At this point, as a further improvement, intelligent error correction can be performed on the intermediate recognition results, the intermediate text results can be displayed in real time, and the key audio streams can be quickly identified; if speech translation between different languages ​​is involved, summary translation can be achieved based on the key audio streams.

[0152] exist Figure 8 The output includes "H 2 In O 2 Combustion generates H 2 0, the reaction formula is: 2H 2 +O 2 =2H 2 O", and the summary translation result image is Figure 8 The final reaction formula is: 2H 2 +O 2 =2H 2 O” is displayed in the central display area through the picture.

[0153] therefore, Figure 6 or Figure 8 The technical solution can also be summarized as:

[0154] receiving a speech input sequence through the speech receiving channel;

[0155] Translating the speech input sequence into an intermediate text sequence through a first modality;

[0156] Determining whether the intermediate text sequence contains a predetermined text subsequence;

[0157] If it exists, the predetermined text subsequence is translated into a target text sequence through a second modality, and then the predetermined text subsequence in the intermediate text sequence is replaced;

[0158] Generate a picture-based display summary picture based on the intermediate text sequence,

[0159] The picture-based summary picture is displayed in the middle area of ​​the display interface.

[0160] It should be pointed out that Figure 1 The steps or methods and processes can be automatically implemented by computer program instructions. Fig. 9 , provides a terminal device, which can be a data interaction device, including a bus, a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing power secondary device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing power secondary device so that a series of operating steps are executed on the computer or other programmable power secondary device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable power secondary device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0163] The technical solution of the present invention can translate the speech sequence in a multimodal manner, and can highlight the target sequence of the speech sequence in a graphic manner, thereby improving the efficiency of speech communication.

[0164] Specifically, the technical solution of the present invention, for a voice input sequence containing a predetermined text subsequence (digital text, predetermined nouns, chemical noun sequences), first translates the voice input sequence into an intermediate text sequence, identifies the predetermined text subsequences therein, and then performs text-to-image conversion to dynamically display voice text or picture sequences in different formats, so that both parties in communication can quickly grasp the key points of voice communication and improve communication efficiency.

[0165] It can also be seen in the subsequent further improvements of the embodiments that the present invention recognizes the audio stream as text in real time, is suitable for long sentence voice input, audio and video subtitles, conferences, speech subtitles on the same screen, etc., intelligent language processing, and can also perform intelligent error correction on the intermediate recognition results, display the intermediate text results in real time, and quickly identify the key audio streams; if it involves voice translation between different languages, summary picture translation can be implemented and displayed based on the key audio stream, further improving the efficiency of voice translation.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

[0167] The present invention does not particularly specify the structure of some modules, which shall be subject to the contents recorded in the prior art. The prior art mentioned in the above background technology section of the present invention can be used as part of the present invention to understand the meaning of some technical features or parameters. The scope of protection of the present invention shall be subject to the contents actually recorded in the claims.

Claims

1. A multimodal speech translation method, the method being applied to a terminal including a display interface and a speech receiving channel, characterized in that: The method comprises the following steps: receiving a speech input sequence through the speech receiving channel; The speech input sequence is translated into an intermediate text sequence through a first modality; the first modality translation is performed by a first process; the speech input sequence and the intermediate text sequence are in a first language; Determine whether the intermediate text sequence contains a predetermined text subsequence; the predetermined text subsequence includes a chemical noun sequence; if so, translate the predetermined text subsequence into a target text sequence through a second modality, and then replace the predetermined text subsequence in the intermediate text sequence; and perform the second modality translation by a second process; The predetermined text subsequence and the target text sequence are in a second language; Generate a picture-based display summary picture based on the intermediate text sequence; Displaying the picture-based summary image in the middle area of ​​the display interface; The method further comprises: Determining whether the intermediate text sequence contains a predetermined text subsequence; If so, determining whether the predetermined text subsequence meets the image conversion condition; If the conditions are met, the chemical noun sequence included in the predetermined text subsequence is translated into a chemical formula combination display picture sequence through a third modality, and the picture sequence is displayed on the display interface; The third modal translation is performed by a third process; The display area of ​​the picture sequence on the display interface is different from the display area of ​​the intermediate text sequence on the display interface; The first process is a system resident process; The display interface dynamically divides and merges multiple display areas based on the activation states of the second process and the third process.

2. A multimodal speech translation method as claimed in claim 1, characterized in that: The method further comprises: Determining whether the intermediate text sequence contains a predetermined text subsequence; If not, return to the step of receiving a speech input sequence through the speech receiving channel.

3. A multimodal speech translation method as claimed in claim 1, characterized in that: The method further comprises: Determining whether the intermediate text sequence contains a predetermined text subsequence; If so, translating the predetermined text subsequence into a target text sequence through a second modality, and displaying the target text sequence on the display interface; The target text sequence is displayed in a manner different from the intermediate text sequence ...

4. A multimodal speech translation method as claimed in claim 1, characterized in that: The predetermined text subsequence includes at least one of a digital text and a predetermined noun; Determining whether the predetermined text subsequence meets the image conversion condition specifically includes: Determining whether the predetermined text subsequence contains digital text that can be converted into a digital picture displayed in a predetermined format; and / or; It is determined whether the predetermined text subsequence contains a predetermined noun that can be converted into a corresponding picture display.

5. A multimodal speech translation system for implementing the multimodal speech translation method according to any one of claims 1 to 4, the system comprising a speech receiving unit, a text modality translation unit, an intermediate result recognition unit, a digital modality translation unit, a picture modality translation unit and a dynamic display unit, characterized in that: The speech receiving unit is used to receive a speech input sequence; The text modality translation unit translates the speech input sequence into an intermediate text sequence; The intermediate result recognition unit is used to recognize whether the intermediate text sequence has a predetermined text subsequence, and activate the digital modality translation unit and / or the picture modality translation unit based on the recognition result; The digital modality translation unit translates the digital text included in the predetermined text subsequence into a target text sequence; The picture modality translation unit translates the target text sequence or the predetermined text subsequence into a picture sequence; The dynamic display unit adjusts the number of display areas of the display interface of the multimodal speech translation system based on the activation status of the digital modality translation unit and the picture modality translation unit, and displays the intermediate text sequence, target text sequence, and picture sequence in different display areas in different display modes.

6. A multimodal speech translation system as claimed in claim 5, characterized in that: The predetermined text subsequence includes a chemical noun sequence; The picture modality translation unit translates the predetermined text subsequence into a picture sequence, specifically including: The picture modality translation unit converts the chemical nouns into chemical formulas and displays the combined pictures.

7. A multimodal speech translation system as claimed in claim 5, characterized in that: The target text sequence includes a digital text string; The image modality translation unit translates the target text sequence into an image sequence, specifically including: The image modality translation unit translates the digital text string into an image display sequence displayed in scientific notation, exponential notation, or N-base form.

8. A multimodal speech translation system according to any one of claims 5 to 7, characterized in that: The dynamic display unit is connected to the intermediate result recognition unit; When the intermediate result recognition unit recognizes that the intermediate text sequence does not have a predetermined text subsequence, the initial setting state of the dynamic display unit is restored, and the initial setting state is a default state of the display interface of the system.

Images