Intelligent real-time language synchronous translation system and terminal thereof

Abstract

The invention relates to an intelligent real-time language synchronous translation system and a terminal thereof. The system comprises a voice segmentation module used for carrying out semantic analysis on target voice to be translated and segmenting the target voice according to obtained semantic information to obtain voice units; the scene recognition module is used for analyzing the voice unit to obtain scene information; the syntactic rhythm analysis module is used for converting the voice unit into a text, performing syntactic analysis, and analyzing the voice unit in combination with the information and the semantic information to obtain voice rhythm information and statement structure information; the semantic optimization module is used for correcting the semantic information based on the statement structure information, the scene information and the voice rhythm information to obtain optimized semantic information; and the speech translation module is used for translating the optimized semantic information based on the target language to obtain a target language translation. According to the system, through multi-module cooperation, voice semantics can be accurately mastered, different scenes are fitted, and the accuracy, smoothness and practicability of translation are effectively improved.

Description

Technical Field

[0001] The invention belongs to the field of computer and communication technology, and in particular relates to an intelligent real-time language synchronous translation system and a terminal thereof. Background Art

[0002] As the globalization process continues to advance, communication and cooperation between countries have become more frequent, and the demand for cross-language and cross-cultural communication is growing. Language translation plays a vital role in many fields such as international trade, diplomacy, tourism, education, and medical care. It is not only a tool to build bridges between different languages ​​and cultures, but also the key to achieving barrier-free information flow and promoting understanding and cooperation.

[0003] However, traditional real-time language translation technologies usually rely on simple speech recognition and translation models. These methods usually first convert audio signals into text through speech recognition technology, and then use translation models to convert text into the target language. Although this method can achieve basic translation functions, the accuracy of translation is often affected when faced with complex language structures, accent changes or unclear contexts. It also lacks the ability to flexibly adapt to different voice scenarios and cannot make intelligent dynamic adjustments, resulting in poor real-time and fluency of translation. Summary of the invention

[0004] Based on this, it is necessary to address the above technical problems. The purpose of the present invention is to provide an efficient and intelligent real-time language synchronous translation system to improve the accuracy and fluency of translation and meet the language translation needs in different scenarios.

[0005] In a first aspect, the present application provides an intelligent real-time language synchronous translation system, the system comprising: The speech segmentation module is used to perform semantic analysis on the target speech to be translated through the knowledge graph model to obtain semantic information, and to segment the target speech based on the semantic information to obtain speech units; The scene recognition module is used to analyze the speech unit, determine the speech scene of the current conversation, and obtain scene information; The syntactic rhythm analysis module is used to convert the speech unit into speech text, and perform syntactic analysis on the speech text to obtain sentence structure information, and analyze the speech unit in combination with sentence structure information and semantic information according to speech signal processing technology to obtain speech rhythm information; A semantic optimization module is used to modify the semantic information based on sentence structure information, scene information and speech rhythm information to obtain optimized semantic information. The modification operations include semantic adjustment, semantic screening and semantic reinforcement. The speech translation module is used to input the optimized semantic information into the language translation model for translation based on the target language to obtain the target language translation.

[0006] In one embodiment, the speech segmentation module includes: The speech segmentation subunit is used to: Performing a frame operation on the target speech to obtain speech frames; Feature extraction is performed based on each speech frame to obtain acoustic features, which include MFCC coefficients; The MFCC coefficients are used for phoneme recognition to obtain a phoneme sequence, and the phoneme sequence is semantically constrained based on semantic information to obtain an optimized phoneme sequence; The cost matrix is ​​constructed according to the dynamic programming algorithm, and the optimized phoneme sequence is segmented to obtain speech units.

[0007] In one embodiment, the cost matrix is ​​calculated as follows: ; in, is the cost matrix, is the phoneme recognition accuracy function, is the semantic score function, is the acoustic feature continuity function of the target speech, , and are the weight coefficients of the corresponding functions, Represents an optimized phoneme sequence.

[0008] In one embodiment, the calculation formula of the phoneme recognition accuracy function is: ; in, is the phoneme recognition accuracy function, represents the probability that the kth factor is correctly identified.

[0009] In one embodiment, the syntactic rhythm analysis module includes: The information extraction subunit is used to: The endpoint detection algorithm based on energy threshold and zero-crossing rate analyzes the pause interval in the speech unit and obtains the pause duration; Analyze the pitch changes of speech units according to the autocorrelation function method to obtain the pitch change trajectory; Calculate the speech rate information of the speech unit based on the syllable segmentation algorithm; The dynamic analysis subunit is used to analyze the pause duration, pitch change trajectory, speech rate information, sentence structure information and semantic information based on the dynamic programming algorithm to obtain speech rhythm information.

[0010] In one embodiment, the speech segmentation module further includes: Semantic extraction subunit, used to: Based on named entity recognition technology, perform entity recognition on speech text to obtain key entities; The key entities are identified and matched through the knowledge graph model, and the relationships between the key entities are extracted to obtain relationship information; Perform semantic reasoning on key entities and relationship information through the knowledge graph model to obtain extended semantic information; The extended semantic information is combined with the attention mechanism to evaluate its importance and obtain optimized extended semantic information; The optimized extended semantic information, key entity and relationship information are integrated to obtain semantic information.

[0011] In one embodiment, the system further comprises a translation evaluation module for: Evaluate the target language translation according to the source language text of the target speech to obtain an evaluation result; When the evaluation result is uncertain, correction instructions are sent to the user, which include displaying the problematic translation and providing correction options.

[0012] In a second aspect, the present application also provides an intelligent real-time language synchronous translation method, the method comprising: The target speech to be translated is semantically analyzed through the knowledge graph model to obtain semantic information, and the target speech is segmented based on the semantic information to obtain speech units; Analyze the speech unit, determine the speech scene of the current conversation, and obtain scene information; Convert the speech unit into speech text, perform syntactic analysis on the speech text to obtain sentence structure information, and analyze the speech unit in combination with sentence structure information and semantic information based on speech signal processing technology to obtain speech rhythm information; Based on the sentence structure information, scene information and speech rhythm information, the semantic information is corrected to obtain optimized semantic information. The correction operation includes semantic adjustment, semantic screening and semantic reinforcement. Based on the target language, the optimized semantic information is input into the language translation model for translation to obtain the target language translation.

[0013] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the system in any one of the first aspects when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the system of any one of the first aspects when the computer program is executed by a processor.

[0015] The above-mentioned intelligent real-time language synchronous translation system, method, computer device and storage medium use the knowledge graph model to perform semantic analysis on the target speech to be translated through the speech segmentation module, extract semantic information, and segment the target speech based on these semantic information to obtain speech units. This module can ensure that the speech signal is scientifically decomposed and provide clear speech unit data for subsequent processing. Based on the speech unit, the scene recognition module is used to perform context analysis in combination with keywords and background noise in the speech, and the specific speech scene of the current dialogue is judged to generate scene information, which provides important context support for accurate understanding of semantics. Secondly, the syntactic rhythm analysis module can convert the speech unit into speech text. On the one hand, syntactic analysis can be performed to clarify the sentence structure information. On the other hand, based on the speech signal processing technology, the speech unit can be further analyzed in combination with sentence structure information and semantic information to extract speech rhythm information. This information helps the translation system better understand the pauses, stress and tone in the speech signal, and ensure that the translation result conforms to the grammar and rhythm of the target speech.

[0016] The semantic optimization module can perform semantic adjustment, semantic screening, and semantic reinforcement on semantic information based on sentence structure information, scene information, and voice rhythm information, ensuring that the optimized semantic information is more in line with the actual context and expression requirements, thereby improving the accuracy and fluency of translation. Finally, the system uses the voice translation module to guide the target language, input the optimized semantic information into the language translation model for translation, and can generate high-quality target language translations to meet the user's needs for real-time language synchronization translation.

[0017] Compared with traditional speech translation systems, this system uses the knowledge graph model for speech segmentation, which achieves a more accurate division of speech units from a semantic perspective and improves the accuracy of basic processing. Through the collaborative work of multiple modules, the system optimizes semantics by comprehensively considering multi-dimensional information such as scenes, syntax, and speech rhythm, effectively solving the problem of semantic ambiguity, improving the accuracy and depth of semantic understanding, and thus significantly improving the quality and fluency of translation, providing users with more reliable and efficient translation services for real-time language communication in different scenarios, and enhancing the intelligence level and practicality of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of the structure of an intelligent real-time language synchronous translation system provided by an exemplary embodiment of the present invention; Figure 2 A flow chart of an intelligent real-time language synchronous translation method provided for an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] In one embodiment, Figure 1 As shown, an intelligent real-time language synchronous translation system 100 is provided. This embodiment is illustrated by applying the system to a terminal. It can be understood that the system can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the system includes: The speech segmentation module 101 is used to perform semantic analysis on the target speech to be translated through the knowledge graph model to obtain semantic information, and to segment the target speech based on the semantic information to obtain speech units.

[0022] Specifically, the knowledge graph model contains a large amount of language knowledge, entity relationships, semantic associations and other information. By matching and reasoning the content in the target speech with this knowledge, the semantic information contained in the target speech can be obtained, such as determining the exact reference of a word in a sentence, understanding the logical association between words, etc., providing sufficient semantic basis for subsequent speech segmentation operations. The target speech can be segmented based on semantic information to obtain speech units. A speech unit refers to a relatively independent and complete speech segment in terms of semantics, which can be a complete sentence, phrase, or a phrase with specific semantics. Compared with traditional speech segmentation methods that often rely on acoustic features, such as physical properties such as energy and frequency of speech to divide speech segments, dividing speech units based on semantic information can better follow the logic and expression habits of the speech itself, and can avoid the problem of cutting off semantically coherent content or mixing different semantic contents that should be separated, which may occur when segmenting based solely on acoustic features, thereby improving the accuracy and effectiveness of the entire translation system and laying a good foundation for subsequent accurate translation.

[0023] The scene recognition module 102 is used to analyze the speech unit, determine the speech scene of the current dialogue, and obtain scene information.

[0024] In the scene recognition module 102, the speech unit can be analyzed from multiple angles, such as extracting keywords for recognition. There are often specific high-frequency words in different scenes, such as "menu", "order", "waiter" and other words may appear in the restaurant scene. And the background noise characteristics of the speech are recognized. For example, the office scene may be accompanied by keyboard tapping, meeting discussion, etc., and the outdoor scene will have background sounds such as vehicle driving and wind. In addition, the scene recognition module 102 can also be combined with other information, such as visual scene information recognized by the camera image, or location information determined by the positioning system, etc. for fusion recognition, and finally determine the specific speech scene of the current speech, and output the scene information. This scene information is very important for the subsequent accurate understanding of semantics, the selection of appropriate translation expressions, etc., because the same sentence may have different meanings and appropriate expressions in different scenes. This information helps the subsequent semantic optimization module to perform more accurate semantic screening and adjustment of ambiguous words and sentences, so that the translation results are more in line with the actual dialogue situation, improve the accuracy and practicality of the entire translation, make the translation more in line with the language expression habits in the corresponding scene, and better assist users in cross-language communication.

[0025] The syntactic rhythm analysis module 103 is used to convert the speech unit into speech text, and perform syntactic analysis on the speech text to obtain sentence structure information, and analyze the speech unit according to the speech signal processing technology in combination with the sentence structure information and semantic information to obtain speech rhythm information.

[0026] Specifically, the syntactic rhythm analysis module 103 can analyze and process the acoustic features (such as the frequency, energy, duration, etc. of the speech) in the speech unit through speech recognition technology, identify it and convert it into corresponding text content, so that it can be analyzed more conveniently and deeply in text form later. By performing syntactic analysis on the speech text through grammatical rules, the clear hierarchical relationship and grammatical structure between the various parts of the sentence can be sorted out, thereby obtaining sentence structure information. This sentence structure information plays a key role in accurately understanding the semantics of the sentence, grasping the semantic focus, and subsequently processing in combination with other information. And based on the speech signal processing technology combined with the sentence structure information, the speech unit is analyzed, for example, the speech rhythm information can be obtained by analyzing the acoustic features such as the pause duration, pitch change, and speech speed change in the speech unit signal. By combining semantic information in this process, the analysis of speech rhythm can be more in line with the actual meaning and grammatical logic of the language. For example, after determining the key components such as subject, predicate and object based on the sentence structure, we can analyze how the rhythmic characteristics of the speech parts corresponding to these components reflect the semantic content such as sentence emphasis and modification relationships. Finally, the speech rhythm information of the entire speech unit can be accurately extracted. These rhythmic information have an important impact on the subsequent semantic optimization and the shaping of the natural fluency of the translation, and can help the system generate translation results that are more in line with language expression habits and better reflect the speaker's intentions.

[0027] The semantic optimization module 104 is used to perform correction operations on the semantic information based on the sentence structure information, scene information and speech rhythm information to obtain optimized semantic information. The correction operations include semantic adjustment, semantic screening and semantic reinforcement.

[0028] Specifically, the semantic optimization module 104 can perform comprehensive and targeted correction operations on the original semantic information through the sentence structure information, scene information and voice rhythm information obtained by the above modules. Indicatively, the grammatical relationship of each component in the sentence and the influence of the collocation rules between words on the semantics can be considered through the sentence structure information, and the semantic information can be semantically adjusted to exclude other semantic possibilities that do not conform to grammatical logic, so that the semantics of each word in the sentence can be accurate, thereby ensuring the accuracy of the semantic understanding of the entire sentence. Since the same word often has different meanings in different scenarios, in order to make the semantics more in line with the actual dialogue situation, the semantic information can be semantically screened by combining the scene information to avoid the semantic ambiguity caused by the polysemy of a word, so that the entire semantic information is closely aligned with the specific dialogue scene, thereby enhancing the rationality and accuracy of semantic understanding. With the help of voice rhythm information, it can be comprehensively considered that the speaker will emphasize certain content through some characteristic changes in the voice (such as pitch increase, slow speech speed, etc.) during the expression process, and these emphasized parts often have a more important position in semantics, so as to strengthen the semantics, make the core semantic focus of the entire sentence clearer, and enable the optimized semantic information to better reflect the speaker's true expression intention.

[0029] The speech translation module 105 is used to input the optimized semantic information into the language translation model for translation based on the target language to obtain a target language translation.

[0030] The speech translation module 105 can be by the optimized semantic information obtained after the previous multiple modules are processed layer by layer, according to the target language set by the user, input into the corresponding language translation model to carry out translation operation, thereby generate the final target language translation. This language translation model is based on a large amount of bilingual or multilingual parallel corpora such as corresponding sentences, phrases and other contents between the source language and the target language for training, learning these language patterns, vocabulary correspondences, grammatical structure conversions and other knowledge, and then building up the mapping ability from one language to another language. After the optimized semantic information is input into the language translation model, this model can be through a series of complicated calculations and conversion processes, and finally output the target language translation. This process realizes the important function of helping users to communicate across languages ​​in real time and accurately.

[0031] The system first performs semantic analysis on the target speech to be translated through the speech segmentation module 101 to obtain semantic information, and then segments the target speech based on the information to obtain speech units. This process makes the division of speech units more in line with the inherent logic and semantic structure of the language, effectively avoids the semantic fragmentation problem that may be caused by the traditional speech segmentation method, and provides a reliable guarantee for the efficient operation of subsequent modules. And through the scene recognition module 102, it can analyze the keyword information in the speech unit, keenly capture the background noise and other features, judge the speech scene of the current dialogue, and obtain scene information. This step enables the system to flexibly adjust the semantic understanding strategy according to different scene environments, effectively respond to various complex and changeable communication scenes, ensure that the translation results are highly consistent with the actual context, avoid semantic deviations or translation errors caused by unclear scenes, and further improve the intelligence level and user experience of the entire translation system.

[0032] Secondly, in the syntactic rhythm analysis module 103, the system can convert the speech unit into speech text, and perform syntactic analysis on it, extract sentence structure information, and clarify the grammatical relationship and hierarchical structure of each component in the sentence. And based on the speech signal processing technology, the module can analyze the speech unit in combination with sentence structure information and semantic information to obtain speech rhythm information. These information help to accurately grasp the semantic focus, logical level and emotional intention of the speaker of the sentence, etc., which can enable the system to perceive the inner emotion and expression intention of the language more delicately, further optimize the quality of semantic interpretation and translation, and improve the natural fluency and readability of the translation. In the semantic optimization module 104, the system can perform semantic adjustment, semantic screening and semantic reinforcement on the semantic information based on sentence structure information, scene information and speech rhythm information, effectively solving the common semantic ambiguity and fuzziness problems in the language, so as to obtain optimized semantic information that is more in line with the target speech expression habits. Finally, the optimized semantic information can be input into the language translation model for translation according to the target language through the speech translation module 105, and the target language translation can be obtained, which meets the user's real-time and accurate language synchronization translation needs. This module combines all the above processing results and realizes high-quality, real-time speech translation through deep learning models.

[0033] Through the collaborative work of the above five modules, the system can parse voice information in different scenarios with high accuracy, optimize semantic expression efficiently, and quickly and accurately convert it into the target language, providing users with smooth, natural and accurate translation services. Compared with traditional language translation systems, the system significantly improves the intelligence level of voice processing and the accuracy of translation through voice segmentation, scene judgment, grammatical analysis, semantic optimization and language translation, effectively reduces translation errors caused by inaccurate semantic understanding, improper scene adaptation and other problems, enhances the practicality and reliability of the system in different language communication scenarios, and provides strong technical support for promoting cross-language communication and cooperation on a global scale.

[0034] In an exemplary embodiment, the speech segmentation module includes: The speech segmentation subunit is used to: Performing a frame operation on the target speech to obtain speech frames; Feature extraction is performed based on each speech frame to obtain acoustic features, which include MFCC coefficients; The MFCC coefficients are used for phoneme recognition to obtain a phoneme sequence, and the phoneme sequence is semantically constrained based on semantic information to obtain an optimized phoneme sequence; The cost matrix is ​​constructed according to the dynamic programming algorithm, and the optimized phoneme sequence is segmented to obtain speech units.

[0035] Specifically, in the process of speech processing, in order to facilitate subsequent analysis and processing, it is first necessary to perform a frame operation on the continuous target speech, and obtain speech frames by dividing it into shorter time segments. Schematically, the target speech can be framed by determining the frame length and frame shift. The frame length can be selected to be around 20-30 milliseconds. The frame shift is generally smaller than the frame length, such as 10-15 milliseconds, which can represent the overlapping part between two adjacent frames. By determining the appropriate frame shift, it can be ensured that the target speech signal does not lose too much information during the framing process, so that the subsequent processing can capture the speech features more comprehensively.

[0036] Since different speech contents have different acoustic performances, these differences can be quantified by extracting acoustic features from speech frames, such as MFCC coefficients (Mel-Frequency Cepstral Coefficients). MFCC coefficients can simulate the human ear's perception of speech and characterize the acoustic characteristics of speech frames from multiple dimensions, such as the energy proportion of different frequency bands, spectrum shape and other information. After obtaining the MFCC coefficients, they can be used as input to identify the phoneme sequence through a pre-trained phoneme recognition model. Schematically, a phoneme is the smallest speech unit in speech that can distinguish meaning. The phoneme recognition model can be built based on machine learning, for example, using a hidden Markov model combined with a deep neural network. Through training, the relationship between the MFCC coefficients and the corresponding phonemes in a large number of speech samples can be learned. When the MFCC coefficients of a new speech frame are input, the corresponding phoneme sequence is output, that is, it is determined which phonemes are composed of this speech in sequence.

[0037] Since the phoneme sequence obtained by relying on acoustic features may not be completely accurate or conform to the semantic logic of the language, the initially recognized phoneme sequence can be adjusted by introducing semantic information, excluding unreasonable phoneme combinations, and selecting a phoneme arrangement that is more in line with semantics, so as to obtain an optimized phoneme sequence, so that the information at the phoneme level is more consistent with the actual content of the language to be expressed, and to prepare for the subsequent more accurate segmentation of speech units. Based on the obtained optimized phoneme sequence, the cost of segmenting the optimized phoneme sequence into different speech units can be measured by comprehensively considering multiple factors, such as the accuracy of phoneme recognition, semantic rationality, and continuity of acoustic features of speech signals, etc., setting a suitable quantization function for each factor, and assigning weight coefficients according to the importance of different factors in the segmentation decision, so as to construct a cost matrix. Based on the dynamic programming algorithm, the path with the minimum cumulative cost, that is, the optimal speech unit segmentation method, can be found through the cost matrix. According to this path, the optimized phoneme sequence can be segmented to obtain speech units that conform to the actual semantics and acoustic characteristics of speech. These speech units lay a solid foundation for realizing high-quality intelligent real-time language synchronous translation services.

[0038] In an exemplary embodiment, the cost matrix is ​​calculated as follows: ; in, is the cost matrix, is the phoneme recognition accuracy function, is the semantic score function, is the acoustic feature continuity function of the target speech, , and are the weight coefficients of the corresponding functions, Represents an optimized phoneme sequence.

[0039] The above cost matrix formula is constructed by comprehensively considering factors such as phoneme recognition accuracy, semantic rationality, and continuity of acoustic features of speech signals, setting appropriate quantization functions for each factor, and assigning weight coefficients according to their importance in segmentation decisions. This process helps to find the path with the minimum cumulative cost based on the dynamic programming algorithm, that is, the optimal speech unit segmentation method.

[0040] In an exemplary embodiment, the calculation formula of the phoneme recognition accuracy function is: ; in, is the phoneme recognition accuracy function, represents the probability that the kth factor is correctly identified.

[0041] The above phoneme recognition accuracy function formula is determined by calculating the average probability of correct recognition of factors from i to j. Specifically, the correct recognition probability of each factor from i to j is first calculated. Sum and divide by the number of factors , and obtain the average correct recognition probability.

[0042] In an exemplary embodiment, the syntactic rhythm analysis module includes: The information extraction subunit is used to: The endpoint detection algorithm based on energy threshold and zero-crossing rate analyzes the pause interval in the speech unit and obtains the pause duration; Analyze the pitch changes of speech units according to the autocorrelation function method to obtain the pitch change trajectory; Calculate the speech rate information of the speech unit based on the syllable segmentation algorithm; The dynamic analysis subunit is used to analyze the pause duration, pitch change trajectory, speech rate information, sentence structure information and semantic information based on the dynamic programming algorithm to obtain speech rhythm information.

[0043] Specifically, a speech signal has a certain amount of energy when it is uttered, and the energy is lower during the pause. An energy threshold can be set, and when the energy of the speech signal is lower than this threshold, it is considered to be in a pause state. And the zero-crossing rate can represent the number of times the speech signal crosses the zero axis per unit time. In the speech signal, voiced sounds usually have a lower zero-crossing rate, while unvoiced sounds and pauses have a higher zero-crossing rate. In the information extraction subunit, the endpoint detection algorithm combining the energy threshold and the zero-crossing rate can be used to more accurately detect the endpoints of the speech, that is, determine the positions of the beginning and end of the speech, and then find the pause interval, and calculate the time length of the pause interval to obtain the pause duration. The autocorrelation function can measure the similarity between the signal and itself under different time delays, and in speech signal analysis, the pitch is related to the fundamental frequency of the speech signal. Through the autocorrelation function, the periodic characteristics of the speech signal can be found, and the pitch can be determined. In the information extraction subunit, the autocorrelation function method can be applied to the speech unit frame by frame to obtain the pitch value of each time point or time period, and these pitch values ​​are connected in time order to obtain the pitch change trajectory.

[0044] In schematic form, a syllable is one of the basic units of speech, and speech rate can be measured by the number of syllables per unit time. The syllable segmentation algorithm can determine the boundaries of syllables based on the acoustic features of speech, such as energy and frequency, and then segment the syllables. After the segmentation is completed, the speech rate can be calculated by counting the number of syllables within a certain period of time and combining the length of this period of time. The dynamic programming algorithm can comprehensively consider the pause duration, pitch change trajectory, speech rate information, sentence structure information and semantic information to construct a state space containing this information, and find the optimal combination through iterative calculation to determine the speech rhythm and obtain speech rhythm information. The speech rhythm information may include the distribution of stress and rhythm patterns in the sentence, which helps to understand the semantic focus and expression style of the sentence, making the generated or recognized speech more natural and fluent.

[0045] In an exemplary embodiment, the speech segmentation module further includes: Semantic extraction subunit, used to: Based on named entity recognition technology, perform entity recognition on speech text to obtain key entities; The key entities are identified and matched through the knowledge graph model, and the relationships between the key entities are extracted to obtain relationship information; Perform semantic reasoning on key entities and relationship information through the knowledge graph model to obtain extended semantic information; The extended semantic information is combined with the attention mechanism to evaluate its importance and obtain optimized extended semantic information; The optimized extended semantic information, key entity and relationship information are integrated to obtain semantic information.

[0046] Specifically, named entity recognition can identify entities with specific meanings from speech text, such as names of people, places, names of organizations, time, date, currency, etc., providing a basis for further semantic analysis and knowledge extraction. The knowledge graph is a graph-based data structure that can use entities as nodes and the relationships between entities as edges. It can then identify and match the identified key entities, determine whether these entities have corresponding nodes in the pre-built knowledge graph, and analyze the connections between these key entities to obtain relationship information. This relationship information can reveal the hidden semantic structure in the text and help understand the logic and meaning of the entire text, such as causal relationships, belonging relationships, etc. By using the knowledge graph model and performing semantic reasoning based on the existing key entities and relationship information, it is possible to mine the semantic content that is not directly expressed but implicit in the speech text and obtain extended semantic information. This extended semantic information can help fill the semantic gaps that may exist in the text, enabling the system to process and analyze the speech content from a more comprehensive perspective, and provide richer semantic materials for subsequent language processing tasks. And through the attention mechanism, the importance of each element in the extended semantic information can be evaluated, and some extended semantic content that may be less relevant or unimportant can be removed, making the subsequent semantic fusion and processing more efficient and accurate, and obtaining extended semantic information. Finally, by fusing the optimized extended semantic information with the original key entity and relationship information, a complete semantic information with rich connotations can be formed.

[0047] In an exemplary embodiment, the system further includes a translation evaluation module for: Evaluate the target language translation according to the source language text of the target speech to obtain an evaluation result; When the evaluation result is uncertain, correction instructions are sent to the user, which include displaying the problematic translation and providing correction options.

[0048] Indicatively, the translation evaluation module can compare the source language text and the target language translation of the target speech through a variety of methods. For example, a statistically based evaluation method can be used to calculate the similarity between the translation and the high-quality reference translation using a large-scale bilingual parallel corpus, that is, the translation quality is evaluated by calculating statistical indicators such as the probability of vocabulary matching and the coherence of phrases. When the evaluation result is uncertain, that is, the translation may have problems such as inaccurate semantics, grammatical errors, or inappropriate vocabulary, the system will send a correction instruction to the user. This correction instruction will first display the problematic translation, so that the user can intuitively see where the problem has occurred, and can provide multiple correction options to assist the user in modifying the translation based on the system's own language knowledge and possible correction directions. This translation evaluation module can evaluate the translation based on the source language text and feedback the results, and send instructions containing problematic translations and correction options to the user when the results are uncertain, which helps to ensure translation quality, assist users in modification and promote system optimization, and enhance user experience and trust.

[0049] Based on the same inventive concept, Figure 2 As shown, the embodiment of the present application also provides an intelligent real-time language synchronous translation method, the method comprising: S201: performing semantic analysis on the target speech to be translated by using a knowledge graph model to obtain semantic information, and segmenting the target speech based on the semantic information to obtain speech units; S202: Analyze the speech unit, determine the speech scene of the current dialogue, and obtain scene information; S203: converting the speech unit into speech text, and performing syntactic analysis on the speech text to obtain sentence structure information, and analyzing the speech unit according to speech signal processing technology in combination with the sentence structure information and semantic information to obtain speech rhythm information; S204: Based on the sentence structure information, the scene information and the speech rhythm information, a correction operation is performed on the semantic information to obtain optimized semantic information, where the correction operation includes semantic adjustment, semantic screening and semantic reinforcement; S205: Inputting the optimized semantic information into a language translation model for translation based on the target language to obtain a target language translation.

[0050] In the above-mentioned intelligent real-time language synchronous translation method, the target speech to be translated is firstly semantically analyzed through the knowledge graph model to obtain semantic information, and the target speech is segmented based on the semantic information to obtain speech units. This process can effectively extract the key information in the speech and convert it into speech units, providing a semantic basis for subsequent translation processing. By comprehensively considering various key elements in the speech unit, such as keywords, background noise characteristics, etc., further analysis is performed to determine the speech scene in which the current dialogue is located, and scene information is obtained. This information helps to determine the context behind the speech, thereby improving the accuracy and context adaptability of the translation. Secondly, the speech unit will be converted into speech text for syntactic analysis to obtain sentence structure information. Based on speech signal processing technology, the system can analyze the speech unit through sentence structure information and semantic information, and further extract speech rhythm information. This speech rhythm information can reflect the characteristics of speaking speed, pauses, etc., and provide auxiliary support for the tone and expression of translation. By integrating this information, it is possible to better grasp the rhythm and semantics of the speech, more delicately perceive the inner emotions and expression intentions of the language, and further optimize the quality of semantic interpretation and translation.

[0051] Finally, the semantic information can be modified by combining sentence structure information, scene information and voice rhythm information to obtain optimized semantic information. This process makes the semantics more precise, clear, coherent and logical, and can more accurately reflect the speaker's true intentions. It effectively solves common semantic ambiguity and vagueness in language, and provides a solid and reliable semantic foundation for subsequent high-quality translation, thereby ensuring the accuracy and coherence of the translation content. Based on the target language, the optimized semantic information is input into the language translation model for translation to obtain the target language translation. Through this series of steps, the method can achieve efficient and accurate real-time language synchronous translation, greatly improve the fluency and practicality of the translation, and ensure that the translation results are consistent with the context.

[0052] Furthermore, the target speech is segmented based on semantic information to obtain speech units, including: Performing a frame operation on the target speech to obtain speech frames; Feature extraction is performed based on each speech frame to obtain acoustic features, which include MFCC coefficients; The MFCC coefficients are used for phoneme recognition to obtain a phoneme sequence, and the phoneme sequence is semantically constrained based on semantic information to obtain an optimized phoneme sequence; The cost matrix is ​​constructed according to the dynamic programming algorithm, and the optimized phoneme sequence is segmented to obtain speech units.

[0053] Furthermore, the calculation formula of the cost matrix is ​​as follows: ; in, is the cost matrix, is the phoneme recognition accuracy function, is the semantic score function, is the acoustic feature continuity function of the target speech, , and are the weight coefficients of the corresponding functions, Represents an optimized phoneme sequence.

[0054] Furthermore, the calculation formula of the phoneme recognition accuracy function is: ; in, is the phoneme recognition accuracy function, represents the probability that the kth factor is correctly identified.

[0055] Furthermore, according to the speech signal processing technology, the speech unit is analyzed in combination with the sentence structure information and the semantic information to obtain the speech rhythm information, including: The endpoint detection algorithm based on energy threshold and zero-crossing rate analyzes the pause interval in the speech unit and obtains the pause duration; Analyze the pitch changes of speech units according to the autocorrelation function method to obtain the pitch change trajectory; Calculate the speech rate information of the speech unit based on the syllable segmentation algorithm; Based on the dynamic programming algorithm, the pause duration, pitch change trajectory, speaking speed information, sentence structure information and semantic information are analyzed to obtain the speech rhythm information.

[0056] Furthermore, the target speech to be translated is semantically analyzed through the knowledge graph model to obtain semantic information, including: Based on named entity recognition technology, perform entity recognition on speech text to obtain key entities; The key entities are identified and matched through the knowledge graph model, and the relationships between the key entities are extracted to obtain relationship information; Perform semantic reasoning on key entities and relationship information through the knowledge graph model to obtain extended semantic information; The extended semantic information is combined with the attention mechanism to evaluate its importance and obtain optimized extended semantic information; The optimized extended semantic information, key entity and relationship information are integrated to obtain semantic information.

[0057] Furthermore, the method further comprises: Evaluate the target language translation according to the source language text of the target speech to obtain an evaluation result; When the evaluation result is uncertain, correction instructions are sent to the user, which include displaying the problematic translation and providing correction options.

[0058] In an exemplary embodiment, the present invention further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of an intelligent real-time language synchronous translation system of the present application are implemented. A multi-core processor is preferred to improve the parallel processing capability of the system. Memory: Provides sufficient temporary storage space to support the operation of the program and the processing of data. The memory capacity should be large enough to accommodate a large amount of supply information and computing tasks.

[0059] In an exemplary embodiment, the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of an intelligent real-time language synchronous translation system of the present application are implemented. The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk, etc. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM).

[0060] The above-mentioned embodiments only express several implementation methods of the embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the embodiments of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. An intelligent real-time language synchronous translation system, characterized in that: The system comprises: A speech segmentation module is used to perform semantic analysis on the target speech to be translated through a knowledge graph model to obtain semantic information, and to segment the target speech based on the semantic information to obtain speech units; A scene recognition module is used to analyze the speech unit, determine the speech scene of the current conversation, and obtain scene information; A syntactic rhythm analysis module, used for converting the speech unit into speech text, performing syntactic analysis on the speech text to obtain sentence structure information, and analyzing the speech unit in combination with the sentence structure information and the semantic information according to speech signal processing technology to obtain speech rhythm information; A semantic optimization module, used for performing a correction operation on the semantic information based on the sentence structure information, the scene information and the speech rhythm information to obtain optimized semantic information, wherein the correction operation includes semantic adjustment, semantic screening and semantic reinforcement; The speech translation module is used to input the optimized semantic information into a language translation model for translation based on the target language to obtain a target language translation.

2. The system according to claim 1, characterized in that The speech segmentation module comprises: The speech segmentation subunit is used to: Performing a frame operation on the target speech to obtain speech frames; Performing feature extraction based on each of the speech frames to obtain acoustic features, wherein the acoustic features include MFCC coefficients; Performing phoneme recognition on the MFCC coefficients to obtain a phoneme sequence, and performing semantic constraints on the phoneme sequence based on the semantic information to obtain an optimized phoneme sequence; A cost matrix is ​​constructed according to a dynamic programming algorithm, and the optimized phoneme sequence is segmented to obtain the speech unit.

3. The system according to claim 2, characterized in that The calculation formula of the cost matrix is ​​as follows: ； in, is the cost matrix, is the phoneme recognition accuracy function, is the semantic score function, is the acoustic feature continuity function of the target speech, , and are the weight coefficients of the corresponding functions, represents the optimized phoneme sequence.

4. The system according to claim 3, characterized in that The calculation formula of the phoneme recognition accuracy function is: ； in, is the phoneme recognition accuracy function, represents the probability that the kth factor is correctly identified.

5. The system according to claim 1, characterized in that The syntactic rhythm analysis module includes: The information extraction subunit is used to: Analyze the pause interval in the speech unit based on the endpoint detection algorithm of energy threshold and zero-crossing rate to obtain the pause duration; Analyzing the pitch change of the speech unit according to the autocorrelation function method to obtain a pitch change trajectory; Calculating the speech rate information of the speech unit based on the syllable segmentation algorithm; The dynamic analysis subunit is used to analyze the pause duration, the pitch change trajectory, the speech rate information, the sentence structure information and the semantic information based on a dynamic programming algorithm to obtain the speech rhythm information.

6. The system according to claim 1, characterized in that The speech segmentation module also includes: Semantic extraction subunit, used to: Based on named entity recognition technology, entity recognition is performed on the speech text to obtain key entities; The key entities are identified and matched by the knowledge graph model, and the relationships between the key entities are extracted to obtain relationship information; Perform semantic reasoning on the key entities and the relationship information through the knowledge graph model to obtain extended semantic information; The extended semantic information is combined with an attention mechanism to perform importance evaluation to obtain optimized extended semantic information; The optimized and extended semantic information, the key entity and the relationship information are integrated to obtain the semantic information.

7. The system according to claim 1, characterized in that The system further comprises a translation evaluation module, which is used to: Evaluate the target language translation according to the source language text of the target speech to obtain an evaluation result; When the evaluation result indicates that there is uncertainty, a correction instruction is sent to the user, wherein the correction instruction includes displaying the problematic translation and providing correction options.

8. An intelligent real-time language synchronous translation method, characterized in that: The method comprises: Performing semantic analysis on the target speech to be translated by using a knowledge graph model to obtain semantic information, and segmenting the target speech based on the semantic information to obtain speech units; Analyze the speech unit, determine the speech scene of the current dialogue, and obtain scene information; Converting the speech unit into speech text, performing syntactic analysis on the speech text to obtain sentence structure information, and analyzing the speech unit in combination with the sentence structure information and the semantic information according to speech signal processing technology to obtain speech rhythm information; Based on the sentence structure information, the scene information and the speech rhythm information, a correction operation is performed on the semantic information to obtain optimized semantic information, wherein the correction operation includes semantic adjustment, semantic screening and semantic reinforcement; The optimized semantic information is input into a language translation model for translation based on the target language to obtain a target language translation.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the system according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the system according to any one of claims 1 to 7 are implemented.

Images