Speech translation simultaneous transmission method and device, translation machine and storage medium

Abstract

The invention provides a voice translation and simultaneous transmission method and device, a translation machine and a storage medium, and the method comprises the steps: receiving to-be-recognized voice data, transmitting the voice data to a first server, obtaining a recognition text, and displaying the recognition text in real time; the first server performs text recognition on the voice data to obtain a recognition text and returns the recognition text in real time; the recognition text is sent to a second server, and a translation result is obtained and displayed in real time; the second server translates the recognition text to obtain a translation result and returns the translation result in real time; the translation result is sent to a third server, and a synthetic audio is obtained and broadcasted in real time; and the third server performs voice synthesis on the translation result to obtain a synthesized audio and returns the synthesized audio in real time. According to the method, the translation result is displayed in real time, the synthesized audio is broadcasted in real time, and the duration of waiting for the translation result and the synthesized audio by the user is shortened, so that the time consumed by the user is shortened, the user can see the translation result and hear the broadcast of the synthesized audio by lifting a key, and the use experience of the user is further improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, translator and storage medium for simultaneous speech translation. Background Art

[0002] With the rapid iteration and update of voice translation AI technology, intelligent translation products are emerging in an endless stream, bringing great convenience to people's cross-language and multi-scenario communication.

[0003] At present, the voice translation scenario based on the cascade of AI capabilities, that is, the translation scenario from voice to voice, is implemented by cascading the voice recognition, text translation, and voice synthesis AI capabilities. For example: user A speaks Chinese and the voice is recognized into Chinese text, then the Chinese text is translated into English text, and then the English text is synthesized into audio for broadcast through the voice AI capability, so that users A and B can communicate across languages.

[0004] However, the existing technical solution needs to wait until the user finishes speaking and obtains all the recognition results before translating and synthesizing. The user's perceived time for voice translation is the time from when the user lifts the button to when the corresponding audio of the translation result begins to be broadcast. Such perceived time will be affected by the length of the translation and synthesis time. In addition, when the audio of the user's speech is long, the translation time will increase. At the same time, due to the long synthesized text, the synthesis time will also increase. Based on the increase in time consumption at these two points, when the user lifts the button, that is, when the translator device finishes recording, it will take a visible waiting time to see the translation result and hear the corresponding synthesized audio broadcast, which greatly affects the user experience. Summary of the invention

[0005] The present invention provides a method and device for simultaneous speech translation, a translation machine and a storage medium, which are used to solve the problem in the prior art that in a speech translation scenario, it is necessary to wait for the user to finish speaking and obtain all recognition results before performing translation synthesis. The user's perceived time consumption of speech translation is the time consumption from the user lifting a key to hearing the start of the broadcast of the audio corresponding to the translation result. Such perceived time consumption will be affected by the length of the translation and synthesis time, thereby reducing the defect of the user's use experience.

[0006] The present invention provides a method for simultaneous speech translation, comprising the following steps: Receive voice data to be recognized, and send the voice data to the first server to obtain recognized text and display it in real time; the first server is used to perform text recognition on the voice data, obtain the recognized text and return it in real time; The recognized text is sent to a second server to obtain a translation result and display it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; The translation result is sent to a third server to obtain synthesized audio and broadcast it in real time; the third server is used to perform speech synthesis on the translation result to obtain the synthesized audio and return it in real time.

[0007] According to a method for simultaneous speech translation provided by the present invention, the recognized text is sent to a second server, and a translation result is obtained and displayed in real time, including: The recognized text is semantically segmented to obtain a plurality of segmented sub-sentences, and the plurality of segmented sub-sentences are sent to the second server in the form of a stream to obtain a plurality of translation results and display them in real time; the second server is used to translate the plurality of segmented sub-sentences, obtain the plurality of translation results and return them in real time; Accordingly, the translation result is sent to a third server to obtain a synthesized audio and broadcast it in real time, including: The multiple translation results are spliced ​​to obtain multiple spliced ​​translation results, and the multiple spliced ​​translation results are semantically segmented to obtain multiple target segmentation results, and the multiple target segmentation results are sent to the third server in the form of a stream to obtain multiple synthesized audios and broadcast them in real time; the third server is used to perform speech synthesis on the multiple target segmentation results to obtain the multiple synthesized audios and return them in real time.

[0008] According to a method for simultaneous speech translation provided by the present invention, the semantic segmentation of the multiple concatenated translation results to obtain multiple target segmentation results includes: Determine the speech synthesis duration corresponding to the multiple target segmentation results, and the broadcasting duration of the synthesized audio corresponding to the multiple target segmentation results; Based on the speech synthesis duration and the broadcasting duration, a splitting condition is determined, and based on the splitting condition, semantic segmentation is performed on the multiple splicing translation results to obtain the multiple target segmentation results.

[0009] According to a speech translation and simultaneous interpretation method provided by the present invention, the splitting condition is that the broadcasting duration of the synthesized audio corresponding to the first segmentation result among the multiple target segmentation results is greater than the speech synthesis duration of the second segmentation result among the multiple target segmentation results.

[0010] According to a method for simultaneous speech translation provided by the present invention, the semantic segmentation of the recognized text to obtain a plurality of segmented sub-sentences includes: Based on the semantic information of each participle in the recognized text and the context information of each participle, determining the association degree between each participle and an adjacent participle, wherein the association degree is used to characterize the degree of consistency of the semantics expressed by each participle and an adjacent participle; When the correlation between any participle and an adjacent participle is less than a preset threshold, the any participle is segmented from the adjacent participle to obtain the plurality of segmented sub-sentences.

[0011] According to a method for simultaneous speech translation provided by the present invention, the second server is specifically used for: The recognized text is translated based on the language information, and the translation result is obtained and returned in real time; the language information is carried by the voice data.

[0012] The present invention also provides a speech translation and simultaneous interpretation device, comprising the following units: A first real-time display unit is used to receive voice data to be recognized, and send the voice data to a first server to obtain recognized text and display it in real time; the first server is used to perform text recognition on the voice data, obtain the recognized text and return it in real time; A second real-time display unit is used to send the recognized text to a second server, obtain a translation result and display it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; The third real-time broadcast unit is used to send the translation result to the third server, obtain the synthesized audio and broadcast it in real time; the third server is used to perform speech synthesis on the translation result, obtain the synthesized audio and return it in real time.

[0013] The present invention also provides a translation machine, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned simultaneous speech translation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the simultaneous speech translation method as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the simultaneous speech translation method as described above is implemented.

[0016] The speech translation simultaneous interpretation method, device, translation machine and storage medium provided by the present invention receive speech data to be recognized, and send the speech data to a first server to obtain recognized text and display it in real time; the first server is used to perform text recognition on the speech data, obtain recognized text and return it in real time; then send the recognized text to a second server to obtain a translation result and display it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; finally, send the translation result to a third server to obtain synthesized audio and broadcast it in real time; the third server is used to perform speech synthesis on the translation result, obtain synthesized audio and return it in real time. The method reduces the time the user waits for the translation result and synthesized audio by displaying the translation result in real time and broadcasting the synthesized audio in real time, thereby reducing the user's sensory time consumption, and enables the user to see the translation result and hear the synthesized audio broadcast by lifting the button, thereby further improving the user's user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of key translation provided by the present invention.

[0019] Figure 2 This is the calling process of voice dictation translation in the prior art.

[0020] Figure 3 This is one of the flow charts of the speech translation and simultaneous interpretation method provided by the present invention.

[0021] Figure 4 This is the second flow chart of the speech translation and simultaneous interpretation method provided by the present invention.

[0022] Figure 5 It is a schematic diagram of data interaction between the translation machine client and the server provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the speech translation and simultaneous interpretation device provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the translation machine provided by the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the present invention more clear, the following will, in conjunction with the accompanying drawings in the present invention, clearly and completely describe the technical solutions in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the scope of protection of the present invention.

[0026] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category.

[0027] Figure 1 is a schematic diagram of key translation provided by the present invention, Figure 2 is the call process of speech dictation translation in the prior art, as Figure 1 、 Figure 2 shown. Currently, for the speech translation scenario implemented based on AI ability cascading, that is, the speech-to-speech translation scenario. Here, taking the Chinese-to-English speech translation scenario as an example, its implementation process is as follows: When the user presses a key, the user then emits a voice containing "Hello". This voice is translated into "Hello" through text. After the translation is completed, the obtained English text "Hello" is subjected to speech synthesis to convert the English text into a corresponding voice signal, and finally an English voice audio corresponding to "Hello" is generated and output.

[0028] The implementation method of the above speech-to-speech translation scenario is: cascading the AI capabilities of speech recognition, text translation, and speech synthesis. For example: User A speaks Chinese and is recognized as Chinese recognition text through speech recognition. Then all the recognition texts are aggregated. Secondly, the Chinese recognition text is translated into English text, and then the English text is synthesized into an audio through the speech AI ability for broadcasting, realizing cross-language communication between User A and B.

[0029] Based on the above problems, the present invention provides a method for simultaneous interpretation of speech translation, Figure 3 is one of the schematic flowcharts of the method for simultaneous interpretation of speech translation provided by the present invention, as Figure 1 shown. This method includes step 110, step 120, and step 130.

[0030] Step 110, receive the speech data to be recognized, and send the speech data to the first server to obtain the recognition text and display it in real time; the first server is used to perform text recognition on the speech data, obtain the recognition text, and return it in real time.

[0031] Specifically, the execution subject of the method can be a translation machine, that is, a client, or a server. The following is an explanation combining the two execution subjects.

[0032] When the executor of the method is a translation machine, the translation machine receives voice data to be recognized, and the voice data to be recognized is the voice data that needs to be subsequently translated and interpreted. The voice data can come from the user's direct oral statement, or can be obtained through a sound pickup device. Here, the sound pickup device can be a smart phone, a tablet computer, or a smart appliance, such as a stereo, a television, and an air conditioner. After the sound pickup device obtains the voice data through a microphone array, it can also amplify and reduce noise on the voice data. The embodiment of the present invention does not specifically limit this.

[0033] Specifically, a voice button may be provided on the translation machine. When the translation machine detects that the user presses the voice button, the translation machine may start recording. When the translation machine detects that the user releases the voice button, the translation machine may stop recording, thereby obtaining the voice data directly spoken by the user.

[0034] Here, the voice data to be recognized may be voice data in Chinese, English, Spanish, etc., which is not specifically limited in the embodiment of the present invention.

[0035] After receiving the voice data, the translator can send the voice data to the first server to obtain the recognized text and display it in real time. Here, the first server is used to perform text recognition on the voice data, obtain the recognized text and return it in real time.

[0036] Here, the first server may be a recognition engine. It is understandable that text recognition of speech data to obtain recognized text may be achieved by deploying a recognition engine with a large language model (LLM). The large language model may be a pre-trained language model such as the iFlytek Spark model, the XLNet model (extreme Multi-label Learning Network), or the ROBERTa model (Robustly Optimized BERT approach). The embodiment of the present invention does not specifically limit this.

[0037] It should be noted that the translation machine and the first server may implement data interaction via the GRPC (Google Remote Procedure Call) communication protocol, or via the REST (Representational State Transfer) communication protocol, etc., and the embodiment of the present invention does not specifically limit this.

[0038] When the execution subject of the method is a server, the server receives the voice data to be recognized, and then can send the voice data to the first server to obtain the recognized text and display it in real time.

[0039] It should be noted that data interaction between the server and the first server can be achieved through the GRPC communication protocol, or through the REST communication protocol, etc., and the embodiment of the present invention does not make specific limitations on this.

[0040] It is understandable that the voice data can be "The weather is so nice today, where shall we go to play?", and the recognized text can be "The weather is so nice today", and the real-time display of the recognized text, that is, displaying "The weather is so nice today" on the translation machine, shows that there is no need to aggregate all the recognition results for display, which greatly reduces the user's sensory time and eliminates the need for waiting time, thereby improving the user experience.

[0041] Step 120, sending the recognized text to a second server, obtaining a translation result and displaying it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time.

[0042] Specifically, after obtaining the recognized text, the translator can send the recognized text to the second server to obtain the translation result and display it in real time. Here, the second server is used to translate the recognized text, obtain the translation result and return it in real time.

[0043] Here, the second server can be a translation engine. It can be understood that translating the recognized text to obtain a translation result can be achieved by deploying a translation engine with a large language model. The large language model can be a pre-trained language model such as iFlytek Spark model, XLNet model, ROBERTa model, etc. The embodiment of the present invention does not make specific limitations on this.

[0044] Here, the second server may translate the recognized text based on the language information to obtain a translation result. It should be understood that the language information covers the source language type corresponding to the recognized text and the target language type corresponding to the translation result.

[0045] It should be noted that the translation machine and the second server may implement data interaction via the GRPC communication protocol, or via the REST communication protocol, etc., and this is not specifically limited in the embodiment of the present invention.

[0046] When the execution subject of the method is a server, the server can send the recognized text to the second server to obtain the translation result and display it in real time.

[0047] It should be noted that data interaction between the server and the second server can be achieved through the GRPC communication protocol, or through the REST communication protocol, etc., and the embodiment of the present invention does not specifically limit this.

[0048] It is understandable that real-time display of translation results, without the need to aggregate all translation results for display, greatly reduces user sensory time consumption, eliminates the need for waiting time, and improves the user experience.

[0049] Step 130, sending the translation result to a third server, obtaining synthesized audio and broadcasting it in real time; the third server is used to perform speech synthesis on the translation result, obtain the synthesized audio and return it in real time.

[0050] Specifically, after obtaining the translation result, the translation machine can send the translation result to the third server, obtain the synthesized audio and broadcast it in real time. Here, the third server is used to perform speech synthesis on the translation result, obtain the synthesized audio and return it in real time.

[0051] Taking into account that the translation results may have defects such as too long sentences and too long speech synthesis time corresponding to the translation results, the translation results can be spliced ​​and then semantically split based on the spliced ​​translation results to obtain more reasonable multiple target segmentation results, and then the target segmentation results are sent to the third server.

[0052] Here, the third server can be a synthesis engine. It can be understood that speech synthesis is performed on the translation result to obtain synthesized audio and can be achieved by a synthesis engine deployed with a large language model. The large language model can be a pre-trained language model such as iFlytek Spark model, XLNet model, ROBERTa model, etc. The embodiment of the present invention does not make specific limitations on this.

[0053] It should be noted that the translation machine and the third server may implement data interaction through the GRPC communication protocol, or through the REST communication protocol, etc., and this is not specifically limited in the embodiment of the present invention.

[0054] When the execution subject of the method is a server, the server can send the recognized text to a third server to obtain the translation result and display it in real time.

[0055] It should be noted that data interaction between the server and the third server can be achieved through the GRPC communication protocol, or through the REST communication protocol, etc., and the embodiment of the present invention does not make specific limitations on this.

[0056] It is understandable that real-time broadcasting of synthesized audio reduces the time users have to wait for the synthesized audio to be broadcasted, thereby reducing user sensory time consumption and enabling users to hear the synthesized audio start broadcasting as soon as they lift the button, thereby further improving the user experience.

[0057] It should be noted that the speech-to-speech translation scenario usually involves multiple speech interactions between user A and user B. Therefore, the multiple speech interactions correspond to repeated execution of steps 110 to 130 described above.

[0058] It is understandable that the first server can encapsulate the recognition text and then return the encapsulated structured data to the translator or server, so that the translator or server can obtain the recognition text by parsing. Similarly, the second server can encapsulate the translation result and then return the encapsulated structured data to the translator or server, so that the translator or server can obtain the translation result by parsing. Similarly, the third server can encapsulate the synthesized audio and then return the encapsulated structured data to the translator or server, so that the translator or server can obtain the synthesized audio by parsing.

[0059] It should be noted that, in the above process, the first server, the second server and the third server may be different servers or the same server, that is, the text recognition, translation and speech synthesis functions are concentrated in one server, and the embodiment of the present invention does not specifically limit this.

[0060] The method provided by the embodiment of the present invention receives voice data to be recognized, and sends the voice data to a first server to obtain recognized text and display it in real time; the first server is used to perform text recognition on the voice data, obtain recognized text and return it in real time; the recognized text is then sent to a second server to obtain a translation result and display it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; finally, the translation result is sent to a third server to obtain synthesized audio and broadcast it in real time; the third server is used to perform speech synthesis on the translation result, obtain synthesized audio and return it in real time. This method reduces the time the user waits for the translation result and synthesized audio by displaying the translation result in real time and broadcasting the synthesized audio in real time, thereby reducing the user's sensory time consumption, and enables the user to see the translation result and hear the synthesized audio broadcast by lifting the button, thereby further improving the user's experience.

[0061] Based on the above embodiment, step 120 includes: Step 121, semantically segmenting the recognized text to obtain a plurality of segmented clauses, and sending the plurality of segmented clauses to the second server in the form of a stream, obtaining a plurality of translation results and displaying them in real time; the second server is used to translate the plurality of segmented clauses, obtain the plurality of translation results and return them in real time; Accordingly, step 130 includes: Step 131, concatenate the multiple translation results to obtain multiple concatenated translation results, perform semantic segmentation on the multiple concatenated translation results to obtain multiple target segmentation results, send the multiple target segmentation results to the third server in the form of a stream, obtain multiple synthesized audios and broadcast them in real time; the third server is used to perform speech synthesis on the multiple target segmentation results, obtain the multiple synthesized audios and return them in real time.

[0062] Specifically, the recognized text is semantically segmented to obtain a plurality of segmented sub-sentences, and the plurality of segmented sub-sentences are sent to the second server in the form of a stream, and a plurality of translation results are obtained and displayed in real time. Here, the second server is used to translate the plurality of segmented sub-sentences, obtain a plurality of translation results and return them in real time, thereby realizing the progressive streaming of translation on the screen, allowing the user to quickly see the translation results.

[0063] Here, the association between each segmentation and adjacent segmentations can be determined based on the semantic information of each segmentation in the recognized text and the context information of each segmentation, and then the recognized text is segmented based on the association between each segmentation and adjacent segmentations to obtain multiple segmented subsentences.

[0064] It can be understood that based on the semantic information of each segmentation in the recognized text and the context information of each segmentation, the correlation between each segmentation and the adjacent segmentation is determined, and then based on the correlation between each segmentation and the adjacent segmentation, the recognized text is segmented to obtain multiple segmented sub-sentences, which is equivalent to semantically segmenting the recognized text based on the segmentation part of speech, segmentation structure, etc., and can also be combined with complete clause end punctuation in different languages, so as to achieve the purpose of cutting long sentences into short sentences according to semantics, thereby improving the accuracy and reliability of semantic segmentation.

[0065] It should be noted that the multiple segmentation clauses are sent to the second server in the form of a stream, that is, the multiple segmentation clauses are used as data blocks and the data blocks are divided into smaller blocks for transmission, without transmitting the entire data block at one time. This method can reduce the demand for bandwidth, especially in network transmission. At the same time, streaming transmission can process data step by step without loading the entire data set into memory, thereby saving the use of computing resources.

[0066] Since the data packets corresponding to the multiple segmentation clauses are divided into smaller blocks for transmission, data streams of any size can be processed without being restricted by a fixed size. This is particularly effective for processing real-time generated data or data streams of uncertain size. Through streaming transmission, the data packets corresponding to the multiple segmentation clauses can be continuously transmitted from the translator or server to the second server without interruption or pause.

[0067] Accordingly, if the translation results are too long after splicing, the speech synthesis time will increase accordingly, which may cause the translation machine to jam. Therefore, multiple translation results can be spliced ​​to obtain multiple spliced ​​translation results, and multiple spliced ​​translation results can be semantically segmented to obtain multiple target segmentation results, and the multiple target segmentation results can be sent to the third server in the form of a stream to obtain multiple synthesized audios and broadcast in real time. Here, the third server is used to perform speech synthesis on multiple target segmentation results, obtain multiple synthesized audios and return them in real time.

[0068] The method provided by the embodiment of the present invention takes into account that if the translation results are too long after being spliced, the time consumption for speech synthesis will increase accordingly. Multiple translation results are spliced ​​to obtain multiple spliced ​​translation results, and the multiple spliced ​​translation results are semantically segmented to obtain multiple target segmentation results. The multiple target segmentation results are sent to a third server in the form of a stream to obtain multiple synthesized audios and broadcast them in real time. Thus, the time the user waits for the audio broadcast is reduced, the user's sensory time consumption is reduced, and the user can hear the synthesized audio start broadcasting as soon as the button is lifted.

[0069] Based on the above embodiment, the semantic segmentation of the multiple concatenated translation results in step 131 is performed to obtain multiple target segmentation results, including: Step 1311, determining the speech synthesis duration corresponding to the multiple target segmentation results, and the broadcasting duration of the synthesized audio corresponding to the multiple target segmentation results; Step 1312: determine a splitting condition based on the speech synthesis duration and the broadcasting duration, and perform semantic segmentation on the multiple concatenated translation results based on the segmentation condition to obtain the multiple target segmentation results.

[0070] Specifically, considering that the splitting conditions for semantic segmentation of multiple spliced ​​translation results are related to the respective speech synthesis durations and the broadcasting durations of the synthesized audio, the speech synthesis durations corresponding to the multiple target segmentation results and the broadcasting durations of the synthesized audio corresponding to the multiple target segmentation results are first determined.

[0071] Then, based on the speech synthesis duration and the broadcasting duration, the splitting conditions are determined, and based on the splitting conditions, semantic segmentation is performed on the multiple spliced ​​translation results to obtain multiple target segmentation results.

[0072] Here, the splitting condition is that the broadcasting duration of the synthesized audio corresponding to the first segmentation result among the multiple target segmentation results is greater than the speech synthesis duration of the second segmentation result among the multiple target segmentation results.

[0073] It is understandable that the splitting condition is set in this way mainly to avoid the situation where the translator has finished broadcasting the previously returned synthesized audio segment while the next synthesized audio segment has not yet been returned in real time, which will cause the translator's broadcast audio to become stuck.

[0074] The method provided by the embodiment of the present invention determines the speech synthesis duration corresponding to multiple target segmentation results and the broadcasting duration of the synthesized audio corresponding to the multiple target segmentation results, and then determines the splitting conditions based on the speech synthesis duration and the broadcasting duration, and semantically segments the multiple spliced ​​translation results based on the splitting conditions to obtain multiple target segmentation results, thereby avoiding the situation where the translation machine has finished broadcasting the synthesized audio segment returned last time, while the next synthesized audio segment has not been returned in real time, which will cause the translation machine to broadcast audio jams.

[0075] Based on the above embodiment, the semantic segmentation of the recognized text in step 121 to obtain multiple segmentation sub-sentences includes: Step 1211, based on the semantic information of each participle in the recognized text and the context information of each participle, determining the association degree between each participle and an adjacent participle, wherein the association degree is used to represent the consistency degree of the semantics expressed by each participle and an adjacent participle; Step 1212: When the correlation between any participle and an adjacent participle is less than a preset threshold, the any participle is segmented from the adjacent participle to obtain the plurality of segmented sub-sentences.

[0076] Specifically, based on the semantic information of each segmentation in the recognized text and the context information of each segmentation, the correlation between each segmentation and adjacent segmentations is determined. Here, the correlation is used to characterize the degree of consistency between the semantics expressed by each segmentation and the adjacent segmentations.

[0077] Semantic information refers to the specific meaning or significance of each word in the recognized text. It is not limited to the literal meaning of the word, but also includes the meaning of the word in a specific context, culture, and field. Contextual information refers to the specific position of each word in the recognized text and other words or sentences around it.

[0078] Specifically, first, the recognized text needs to be segmented and POS tagged. Segmentation is to divide the continuous recognized text into independent words or phrases, while POS tagging is to assign a POS tag to each word (such as noun, verb, adjective, etc.). Next, the semantic information of each segmented word needs to be extracted. This can be achieved by using pre-trained word vector models such as word2vec, BERT (Bidirectional Encoder Representations from Transformers), etc. Then use the context embedding model to extract the context information corresponding to each segmented word. After extracting the semantic information and context information of the segmented word, various methods can be used to calculate the correlation between the segmented word and the adjacent segmented word. For example, the cosine similarity of the semantic information and the contextual information can be calculated as the correlation between each segmented word and the adjacent segmented word.

[0079] When the correlation between any participle and an adjacent participle is less than a preset threshold, any participle is segmented from the adjacent participle to obtain a plurality of segmented sub-sentences.

[0080] It can be understood that by splitting the segmentations with a correlation less than a preset threshold from the adjacent segmentations, i.e., the segmentations with low correlation, the ambiguity between the segmentations can be reduced, making the meaning of each segmentation clause clearer and more specific, thereby improving the accuracy and reliability of semantic segmentation.

[0081] Based on the above embodiment, the second server is specifically used for: The recognized text is translated based on the language information, and the translation result is obtained and returned in real time; the language information is carried by the voice data.

[0082] Specifically, the second server translates the recognized text based on the language information, obtains the translation result and returns it in real time. The language information is carried by the voice data. The language information covers the source language type corresponding to the recognized text and the target language type corresponding to the translation result.

[0083] It is understandable that when the voice data carries language information, the second server can immediately apply the large language model to translate the voice data from the source language type to the target language type. This avoids the need for additional language identification during the translation process, thereby improving the immediacy of the translation.

[0084] In addition, when the voice data carries language information, it means that users do not need to manually select the language or make additional settings. This not only simplifies the operation process, but also reduces translation errors caused by incorrect language selection, thereby improving the user experience.

[0085] Based on any of the above embodiments, the present invention provides a method for simultaneous speech translation. Figure 4 This is the second flow chart of the method for simultaneous speech translation provided by the present invention. Figure 5 It is a schematic diagram of data interaction between the translation machine client and the server provided by the present invention, such as Figure 4 , Figure 5 As shown, the method includes: In the first step, the translation machine client establishes a long connection with the AI ​​server based on the HTTP2 (Hypertext Transfer Protocol version 2) communication custom protocol, and establishes a session and data interaction based on the asynchronous bidirectional flow characteristics of HTTP2 protocol multiplexing and binary framing on one connection.

[0086] In the second step, the AI ​​server establishes sessions with the recognition engine, translation engine, and synthesis engine respectively. The AI ​​server exchanges data with the recognition engine, translation engine, and synthesis engine respectively through the GRPC communication protocol.

[0087] In the third step, the translator client sends the audio stream to the AI ​​server in real time based on the HTTP2 client stream.

[0088] In the fourth step, the AI ​​server receives the audio stream in real time and sends it to the recognition engine, which performs text recognition on the audio stream, obtains the recognized text and returns it to the AI ​​server. The AI ​​server then sends the recognized text to the translator client in real time based on the HTTP2 server stream, and the translator client displays the recognition results in real time. In addition, the audio stream sent by the translator client and the recognized text stream sent by the AI ​​server are both implemented in the same session on a TCP (Transmission Control Protocol Connection) connection.

[0089] In the fifth step, the AI ​​server obtains the recognized text in real time, and semantically segments the recognized text into complete clauses of reasonable length to obtain multiple segmented sub-sentences. The multiple segmented sub-sentences are streamed into the translation engine, which translates the multiple segmented sub-sentences to obtain multiple translation results. The translation engine sends the multiple translation results to the translation machine client in real time based on the HTTP2 server stream, and the translation machine client displays the translation results in real time.

[0090] In the sixth step, the AI ​​server will splice multiple translation results obtained in real time to obtain multiple splicing results, and then perform semantic segmentation on the multiple splicing results. Here, the semantic segmentation algorithm will split the sentence length according to the AI ​​synthesis time to obtain multiple target segmentation results, and stream the multiple target segmentation results into the synthesis engine to obtain a synthesized audio stream. This operation is mainly to avoid the translator broadcasting the previous returned synthesized audio segment, while the next synthesized audio segment has not been returned in real time, which will cause the translator to broadcast the audio to freeze.

[0091] Finally, the AI ​​server sends the synthesized audio stream to the translator client in real time based on the HTTP2 server stream. When the user lifts the button, the translator client has already aggregated the synthesized audio and can implement the "lift-the-key-and-play" function.

[0092] Depend on Figure 4 It can be seen that the translator client sends the audio stream to the AI ​​server in real time, and the AI ​​server sends the recognized text stream, translation result stream and synthesized audio stream to the translator client in the same session on a TCP connection. Figure 5 In the process, the translator client first sends Headers and multiple Data to the server, where Headers usually contain some metadata, such as request type, content type, authentication information, etc. Data is the actual content to be transmitted, which may be text, pictures, audio, etc. When the translator client finishes sending data, it will send Data with end flag. Data with end flag refers to data with an end mark (or terminator, end mark), which is an effective means to ensure data integrity and accuracy. It indicates the end position of the data by adding a special mark at the end of the data stream, thereby avoiding data loss or error. Of course, the server can send recognition text streams, translation result streams, and synthesized audio streams to the translator client, that is, the server sends Headers and multiple Data to the translator client to achieve data interaction.

[0093] In addition, when the translator client no longer sends voice data to the server, it will send RST_STREAM to the server. RST_STREAM is used to notify the abnormal termination of the stream. When one end wants to close a stream and does not want to continue sending or receiving data on the stream, it can send a RST_STREAM frame.

[0094] It should be noted that Figure 5 The client and server establish a session and data exchange based on a TCP connection based on the HTTP2 communication custom protocol.

[0095] It should be noted that the AI ​​server in the embodiment of the present invention and the object referred to by the server are the same.

[0096] It is understandable that the HTTP2 communication custom protocol allows multiple requests and responses to be made simultaneously on the same TCP connection. This multiplexing mechanism can significantly reduce latency and connection overhead. Since there is no need to establish a new connection for each request, network resources can be used more efficiently, especially in environments with high network latency or limited number of connections.

[0097] HTTP2 communication custom protocol can also improve transmission efficiency: by splitting HTTP protocol messages into smaller binary frames for transmission, HTTP2 makes parsing and processing more efficient. This approach reduces the resource waste and complexity caused by text parsing, thereby improving overall transmission efficiency.

[0098] Compared with the prior art, the method provided by the embodiment of the present invention greatly reduces the user's sensory time consumption. The longer the user's speaking audio is, the more obvious the superiority of the method in quickly displaying the translation result and the audio synthesis result on the screen is. The user can immediately see the translation result and hear the synthesized audio broadcast by lifting the button without waiting time, which greatly improves the user's experience.

[0099] The speech translation and interpretation apparatus provided by the present invention is described below. The speech translation and interpretation apparatus described below and the speech translation and interpretation method described above can be referred to each other.

[0100] Based on any of the above embodiments, the present invention provides a speech translation simultaneous interpretation device, Figure 6 Schematic diagram of the structure of the speech translation device provided by the present invention. Figure 6 As shown, the device comprises: The first real-time display unit 610 is used to receive the voice data to be recognized, and send the voice data to the first server to obtain the recognized text and display it in real time; the first server is used to perform text recognition on the voice data, obtain the recognized text and return it in real time; The second real-time display unit 620 is used to send the recognized text to the second server, obtain the translation result and display it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; The third real-time broadcast unit 630 is used to send the translation result to the third server, obtain the synthesized audio and broadcast it in real time; the third server is used to perform speech synthesis on the translation result, obtain the synthesized audio and return it in real time.

[0101] The device provided by the embodiment of the present invention receives voice data to be recognized, and sends the voice data to a first server to obtain recognized text and display it in real time; the first server is used to perform text recognition on the voice data, obtain recognized text and return it in real time; the recognized text is then sent to a second server to obtain a translation result and display it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; finally, the translation result is sent to a third server to obtain synthesized audio and broadcast it in real time; the third server is used to perform speech synthesis on the translation result, obtain synthesized audio and return it in real time. This method reduces the time the user waits for the translation result and synthesized audio by displaying the translation result in real time and broadcasting the synthesized audio in real time, thereby reducing the user's sensory time consumption, and enables the user to see the translation result and hear the synthesized audio broadcast by lifting the button, thereby further improving the user's experience.

[0102] Based on any of the above embodiments, the second real-time display unit 620 is specifically used for: The second real-time display sub-unit is used to semantically segment the recognized text to obtain a plurality of segmented sub-sentences, and send the plurality of segmented sub-sentences to the second server in the form of a stream to obtain a plurality of translation results and display them in real time; the second server is used to translate the plurality of segmented sub-sentences, obtain the plurality of translation results and return them in real time; Accordingly, the third real-time broadcast unit 630 is specifically used for: The real-time broadcast subunit is used to splice the multiple translation results to obtain multiple spliced ​​translation results, and semantically segment the multiple spliced ​​translation results to obtain multiple target segmentation results, and send the multiple target segmentation results to the third server in the form of a stream to obtain multiple synthesized audios and broadcast them in real time; the third server is used to perform speech synthesis on the multiple target segmentation results to obtain the multiple synthesized audios and return them in real time.

[0103] Based on any of the above embodiments, the real-time broadcast subunit is specifically used for: Determine the speech synthesis duration corresponding to the multiple target segmentation results, and the broadcasting duration of the synthesized audio corresponding to the multiple target segmentation results; Based on the speech synthesis duration and the broadcasting duration, a splitting condition is determined, and based on the splitting condition, semantic segmentation is performed on the multiple splicing translation results to obtain the multiple target segmentation results.

[0104] Based on any of the above embodiments, the splitting condition is that the broadcasting duration of the synthesized audio corresponding to the first segmentation result among the multiple target segmentation results is greater than the speech synthesis duration of the second segmentation result among the multiple target segmentation results.

[0105] Based on any of the above embodiments, the second real-time display subunit is specifically used to: Based on the semantic information of each participle in the recognized text and the context information of each participle, determining the association degree between each participle and an adjacent participle, wherein the association degree is used to characterize the degree of consistency of the semantics expressed by each participle and an adjacent participle; When the correlation between any participle and an adjacent participle is less than a preset threshold, the any participle is segmented from the adjacent participle to obtain the plurality of segmented sub-sentences.

[0106] Based on any of the foregoing embodiments, the second server is specifically configured to: The recognized text is translated based on the language information, and the translation result is obtained and returned in real time; the language information is carried by the voice data.

[0107] Figure 7 It is a structural schematic diagram of the translation machine provided by the present invention, such as Figure 7 As shown, the translation machine may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the speech translation method, which includes: receiving speech data to be recognized, and sending the speech data to a first server, obtaining a recognized text and displaying it in real time; the first server is used to perform text recognition on the speech data, obtain the recognized text and return it in real time; the recognized text is sent to a second server, and a translation result is obtained and displayed in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; the translation result is sent to a third server, and a synthesized audio is obtained and broadcasted in real time; the third server is used to perform speech synthesis on the translation result, obtain the synthesized audio and return it in real time.

[0108] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech translation and simultaneous interpretation methods provided by the above methods, which include: receiving speech data to be recognized, and sending the speech data to a first server to obtain recognized text and display it in real time; the first server is used to perform text recognition on the speech data, obtain the recognized text and return it in real time; the recognized text is sent to a second server to obtain a translation result and display it in real time; the second server is used to translate the recognized text, obtain the translation result and return it in real time; the translation result is sent to a third server to obtain synthesized audio and broadcast it in real time; the third server is used to perform speech synthesis on the translation result, obtain the synthesized audio and return it in real time.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech translation and simultaneous interpretation methods provided by the above-mentioned methods, the method comprising: receiving speech data to be recognized, and sending the speech data to a first server, obtaining recognized text and displaying it in real time; the first server is used to perform text recognition on the speech data, obtain the recognized text and return it in real time; the recognized text is sent to a second server, obtains a translation result and displays it in real time; the second server is used to translate the recognized text, obtains the translation result and returns it in real time; the translation result is sent to a third server, obtains synthesized audio and broadcasts it in real time; the third server is used to perform speech synthesis on the translation result, obtain the synthesized audio and return it in real time.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for simultaneous speech translation, characterized in that: include: Receiving voice data to be recognized, and sending the voice data to the first server, obtaining recognized text and displaying it in real time; The first server is used to perform text recognition on the voice data, obtain the recognized text and return it in real time; Sending the recognized text to a second server to obtain a translation result and display it in real time; The second server is used to translate the recognized text, obtain the translation result and return it in real time; The translation result is sent to a third server to obtain a synthesized audio and broadcast it in real time; The third server is used to perform speech synthesis on the translation result to obtain the synthesized audio and return it in real time.

2. The method for simultaneous speech translation according to claim 1, characterized in that: The step of sending the recognized text to the second server, obtaining the translation result and displaying it in real time includes: The recognized text is semantically segmented to obtain a plurality of segmented sub-sentences, and the plurality of segmented sub-sentences are sent to the second server in the form of a stream to obtain a plurality of translation results and display them in real time; the second server is used to translate the plurality of segmented sub-sentences, obtain the plurality of translation results and return them in real time; Accordingly, the translation result is sent to a third server to obtain a synthesized audio and broadcast it in real time, including: The multiple translation results are spliced ​​to obtain multiple spliced ​​translation results, and the multiple spliced ​​translation results are semantically segmented to obtain multiple target segmentation results, and the multiple target segmentation results are sent to the third server in the form of a stream to obtain multiple synthesized audios and broadcast them in real time; the third server is used to perform speech synthesis on the multiple target segmentation results to obtain the multiple synthesized audios and return them in real time.

3. The method for simultaneous speech translation according to claim 2, characterized in that: The semantic segmentation of the multiple concatenated translation results is performed to obtain multiple target segmentation results, including: Determine the speech synthesis duration corresponding to the multiple target segmentation results, and the broadcasting duration of the synthesized audio corresponding to the multiple target segmentation results; Based on the speech synthesis duration and the broadcasting duration, a splitting condition is determined, and based on the splitting condition, semantic segmentation is performed on the multiple splicing translation results to obtain the multiple target segmentation results.

4. The method for simultaneous speech translation according to claim 3, characterized in that: The splitting condition is that the broadcasting duration of the synthesized audio corresponding to the first segmentation result among the multiple target segmentation results is greater than the speech synthesis duration of the second segmentation result among the multiple target segmentation results.

5. The method for simultaneous speech translation according to claim 2, characterized in that: The semantic segmentation of the recognized text to obtain a plurality of segmentation sub-sentences includes: Based on the semantic information of each participle in the recognized text and the context information of each participle, determining the association degree between each participle and an adjacent participle, wherein the association degree is used to characterize the degree of consistency of the semantics expressed by each participle and an adjacent participle; When the correlation between any participle and an adjacent participle is less than a preset threshold, the any participle is segmented from the adjacent participle to obtain the plurality of segmented sub-sentences.

6. The method for simultaneous speech translation according to any one of claims 1 to 5, characterized in that: The second server is specifically used for: The recognized text is translated based on the language information, and the translation result is obtained and returned in real time; the language information is carried by the voice data.

7. A speech translation and simultaneous interpretation device, characterized in that: include: A first real-time display unit is used to receive voice data to be recognized, and send the voice data to a first server to obtain recognized text and display it in real time; The first server is used to perform text recognition on the voice data, obtain the recognized text and return it in real time; A second real-time display unit, used to send the recognized text to a second server, obtain a translation result and display it in real time; The second server is used to translate the recognized text, obtain the translation result and return it in real time; A third real-time broadcasting unit, used for sending the translation result to a third server, obtaining a synthesized audio and broadcasting it in real time; The third server is used to perform speech synthesis on the translation result to obtain the synthesized audio and return it in real time.

8. A translation machine, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech translation and simultaneous interpretation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for simultaneous speech translation as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for simultaneous speech translation as claimed in any one of claims 1 to 6 is implemented.

Images