Method and equipment for realizing double-screen different-display real-time translation machine

Abstract

The invention provides a double-screen different-display real-time translation machine implementation method and device, and the method comprises the following steps: 1, obtaining and starting a double-screen different-display real-time translation machine, and initializing an SOC SSD2386 chip, two touch screens, a plurality of microphones distributed near the two touch screens and a storage unit of the double-screen different-display real-time translation machine; by adopting a double-screen different display technology, the target language text information can be respectively and clearly displayed on the touch screen of the opposite user, so that the situation of image disorder caused by mixed display of multiple languages on the same screen is avoided, the information reading speed and accuracy of the user are greatly improved, and the communication fluency and efficiency are ensured; according to the method, the target language text information is individually set according to the display preference of the user, the font size, the color and the like are adjusted, and only the translation result required by the opposite user is displayed, so that the interference of irrelevant information on the user is effectively avoided, the user experience is improved, and the smooth communication is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of translation machines, and in particular to a method and device for realizing a dual-screen different-display real-time translation machine. Background Art

[0002] In the current translation equipment market, various types of translation equipment emerge in an endless stream, and they play their respective roles in different application scenarios. Among them, some translation equipment is presented in the form of a mobile phone screen, which is convenient for individuals to use when traveling or working, and can help users to communicate in basic ways in different language environments. Some translation equipment is designed to be small and exquisite, easy to carry, and its smaller screen size is suitable for users to query the translation of words and simple sentences at any time, and has important applications in learning assistance. However, in the complex scenario of face-to-face communication between multilingual users, traditional translation equipment has obvious shortcomings; On the one hand, existing technologies can usually only display conversation information in all languages ​​on a single screen, completely ignoring the significant differences in language habits and visual perception among different user groups. Mixing multiple languages ​​on the same screen will make the picture look messy, which seriously affects the user's ability to read information quickly and accurately, thereby reducing the fluency and efficiency of communication; On the other hand, different user groups have their own unique language preferences. In actual use, some users are only interested in content in a specific language and may be resistant to text in other languages ​​that appear on the screen. Such additional information may even distract their attention during communication and hinder the smooth progress of communication. To this end, a method and device for realizing a dual-screen real-time translation machine are proposed. Summary of the invention

[0003] In view of this, the embodiments of the present invention hope to provide a method and device for implementing a dual-screen real-time translation machine to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial choice.

[0004] In order to solve the above technical problems, a technical solution adopted by the present application is: a method for implementing a dual-screen real-time translation machine, comprising the following steps: Step 1: Obtain and start the dual-screen real-time translator, and initialize the SOCSSD2386 chip, two touch screens, a plurality of microphones distributed near the two touch screens, and a storage unit of the dual-screen real-time translator; Step 2: Based on microphones distributed near the two screens, the original language voice signals of the two users are collected respectively, and the collected original language voice signals are pre-processed using a noise reduction algorithm; Step 3: The pre-processed original language voice signal is transmitted to the SOC SSD2386 chip, and the original language voice signal is analyzed using the built-in long short-term memory network model to identify the user's language type; Step 4: input the original language speech signal after the language type is identified into the translation module based on the deep learning neural network architecture, convert the original language speech signal into the target language text information, and use the speech synthesis technology to convert the target language text information into the target language speech signal; Step 5: Display the target language text information on the touch screen of the other user, and perform personalized display settings on the target language text information according to the user's display preference. At the same time, play the converted target language voice signal through the corresponding speaker; Step 6: Collect user feedback on translation results and display and playback effects, and adjust the parameters of the language recognition model, translation module and speech synthesis technology based on the user feedback.

[0005] As a further preferred embodiment of the present technical solution, in step 3, the method for identifying the language type of the user comprises the following steps: Step 301: extracting features from the original language speech signal to extract key features; Step 302: Input the extracted key features into the long short-term memory network model, and perform in-depth analysis and pattern matching on the key features through multi-layer neuron processing according to the differences in acoustic and semantic features of different languages; Step 303: Identify the user's language type through a training process of continuous iteration and optimization.

[0006] As a further preferred embodiment of the present technical solution, in step 4, the method for converting the original language speech signal into the target language text information comprises the following steps: Step 401: input the original language speech signal after the language type is identified into the translation module, and use the tokenization technology to perform lexical analysis on the text corresponding to the original language speech signal to generate tokenized original language text information; Step 402: Perform syntactic analysis on the tokenized original language text information, construct a syntactic tree or dependency graph, and determine the grammatical structure and dependency relationship of the original language text information through a syntactic analysis algorithm; Step 403: Map the semantic information of the original language to the target language according to the grammatical structure and dependency relationship of the original language text information, and generate a result of semantic mapping and conversion; Step 404: Generate text information in the target language according to the results of semantic mapping and conversion and the grammatical rules of the target language.

[0007] As a further preferred embodiment of the present technical solution, in step 4, the method of converting the target language text information into the target language speech signal using speech synthesis technology comprises the following steps: Step 4011, obtaining the generated target language text information and performing preprocessing; Step 4012: extracting acoustic features from the preprocessed target language text information using an acoustic feature extraction algorithm; Step 4013: using a vocoder to convert the extracted acoustic features into a target language speech signal; Step 4014: Post-process the generated target language speech signal using audio processing technology.

[0008] As a further preferred embodiment of the technical solution, in step 2, when the original language voice signals of the two users are respectively collected based on the microphones distributed near the two screens, a multi-channel audio collection technology is adopted, and an independent gain control module is set for each microphone channel to dynamically adjust the gain according to the frequency characteristics and loudness of the voices of different users; When the noise reduction algorithm is used to pre-process the collected original language speech signal, a noise reduction method combining adaptive filtering and spectral subtraction is adopted; In the process of collecting the original language voice signals of the two users, the collected voice signals are time-stamped and source-stamped in real time.

[0009] As a further preferred embodiment of the present technical solution, in step five, when the target language text information is displayed on the touch screen of the other party's user, a font enlargement and reduction function is provided for the displayed text information, and a text scrolling display function is supported.

[0010] As a further preferred embodiment of the present technical solution, in step six, when adjusting the parameters of the language recognition model, the translation module and the speech synthesis technology according to the user feedback information, a phased adjustment strategy is adopted.

[0011] To solve the above technical problems, another technical solution adopted by the present application is: a dual-screen real-time translation machine, including a chip module, an audio acquisition module, a language recognition module, a translation module, a speech synthesis module, a display module and a feedback processing module; The chip module is used to coordinate and control the operation of the entire translation machine, perform data operations and task scheduling, including receiving and processing audio data, driving the long short-term memory network model to perform language recognition, and interacting with the translation module and the display module; The audio acquisition module is used to collect the original language voice signals of two users using a microphone, dynamically adjust the gain according to the frequency characteristics and loudness of the voices of different users with the help of multi-channel audio acquisition technology and an independent gain control module, and perform real-time time tagging and source tagging on the collected voice signals; The language recognition module is used to receive the pre-processed original language voice signal, and identify the user's language type by performing feature extraction, deep analysis and pattern matching operations on the original language voice signal; The translation module is used to convert the original language speech signal after the language type is identified into the target language text information, and generate the target language text information according to the grammatical rules of the target language; The speech synthesis module is used to convert the target language text information into the target language speech signal; The display module is used to display the target language text information on the touch screen of the other party user; The feedback processing module is used to collect user feedback information on translation results and display and playback effects, and to adjust the relevant parameters of the language recognition model, translation module and speech synthesis technology using a phased adjustment strategy based on the user feedback information, so as to continuously optimize the system.

[0012] As a further preferred embodiment of the present technical solution, the chip module adopts a parallel processing mechanism when performing data calculations and task scheduling; the microphone in the audio acquisition module is an array microphone having multiple sound pickup units in different directions, and the multiple sound pickup units automatically adjust the sound pickup range and direction according to different usage scenarios.

[0013] As a further preferred embodiment of the present technical solution, the dual-screen real-time translation machine is equipped with multiple connection interfaces, including at least one USB interface, a Type-C interface and a headphone interface, the USB interface is used to connect an external storage device for data transmission and backup; the Type-C interface is used for data transmission and charging; the headphone interface is used to connect headphones, and use headphones to listen to the translated voice.

[0014] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution: 1. The present invention adopts dual-screen different display technology to clearly display the target language text information on the touch screen of the other user, avoiding the cluttered screen caused by multiple languages ​​mixed on the same screen, greatly improving the speed and accuracy of users reading information, and ensuring the fluency and efficiency of communication; for example, in cross-language business negotiations or international communication activities, both users can focus on the translation results on their own screens without being distracted by other language information, making the communication process smoother and more efficient; 2. The present invention personalizes the target language text information according to the user's display preferences, such as adjusting the font size, color, background and layout, and at the same time only displays the translation results required by the other user, effectively avoiding the interference of irrelevant information on the user; for groups with specific language preferences such as Indian users, it can accurately meet their needs of only paying attention to specific languages ​​(such as English), improve user experience, and effectively promote the smooth communication; for example, in international tourism scenarios, when tourists from different countries use this translation machine to communicate, they can set the display mode according to their own habits, focus on understanding the other party's intentions, and enhance the communication effect; 3. The present invention conducts in-depth analysis of speech signals through a long short-term memory network model, combines multi-feature extraction (such as acoustics, prosody, contextual semantic features, etc.) and complex translation processes (including tokenization, syntactic analysis, semantic mapping and grammatical rule generation, etc.), and can accurately identify multiple languages ​​and achieve high-quality translation, providing reliable guarantees for cross-language communication; in international academic conferences or cultural exchange activities, it can accurately convey professional terms and expressions with rich cultural connotations, and promote effective communication between people from different fields and cultural backgrounds.

[0015] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of a process for implementing a dual-screen real-time translator according to the present invention; Figure 2 A schematic diagram of a flow chart of a method for identifying a user's language type according to the present invention; Figure 3 A schematic diagram of a process for converting an original language speech signal into a target language text information method of the present invention; Figure 4 A schematic diagram of a flow chart of a method for converting target language text information into target language speech signals according to the present invention; Figure 5 The present invention is a functional module schematic diagram of a dual-screen real-time translation machine. DETAILED DESCRIPTION

[0018] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0019] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0020] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0022] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0023] Figure 1 1 is a flowchart of a method for implementing a dual-screen real-time translator according to an embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present application is not limited to the method of the present invention. Figure 1 The process sequence shown is limited. Figure 1-Figure 4 As shown: A method for implementing a dual-screen real-time translation machine includes the following steps: Step 1: Obtain and start the dual-screen real-time translator, and initialize the SOCSSD2386 chip, two touch screens, a plurality of microphones distributed near the two touch screens, and a storage unit of the dual-screen real-time translator; Specifically, first, you need to obtain and start the dual-screen real-time translator. This step is the basis of the entire translation process and ensures that the translator is in working condition. Next, initialize the key components of the dual-screen real-time translator, including: SOC SSD2386 chip: This is the core processing unit of the translator, responsible for coordinating and controlling the operation of the entire translator, performing data operations and task scheduling, initializing the chip, ensuring that it can work normally, and laying the foundation for subsequent data processing and task execution; Two touch screens: Dual screens with different displays are one of the features of this translator. The two touch screens are used by two users respectively. The touch screens are initialized to ensure that they can display information normally and provide users with a clear visual experience. At the same time, the mechanism of the adjustable display direction and angle of the touch screen is initialized, allowing users to flexibly adjust the screen direction within a certain range (for example, the horizontal direction can be rotated from 0° to 180°, and the vertical direction can be adjusted from -30° to 30°) to adapt to different usage scenarios and user positions. Multiple microphones distributed near the two touch screens: These microphones are used to collect the original language speech signals of the two users, initialize the microphones, ensure that they can accurately capture the sound, and provide high-quality voice data for subsequent language recognition; Storage unit: The storage unit is used to store various data during the operation of the translator, such as language recognition models, translation results, etc. Initialize the storage unit to ensure that it can read and write data normally, providing guarantee for the stable operation of the translator; By initializing these key components, the dual-screen real-time translation machine is ready to enter the next working stage, which is to collect and process the user's original language voice signal.

[0024] Step 2: Based on microphones distributed near the two screens, the original language voice signals of the two users are collected respectively, and the collected original language voice signals are pre-processed using a noise reduction algorithm; Specifically, in order to achieve more accurate voice collection, this step adopts multi-channel audio collection technology. Each microphone channel is equipped with an independent gain control module, so that the gain can be dynamically adjusted according to the frequency characteristics and loudness of different users' voices to ensure that the collected voice signals are clear and accurate. At the same time, in the process of collecting voice signals, the collected voice signals will also be time-stamped and source-stamped in real time. This will help the subsequent processing and analysis of voice signals to ensure that the voice content of different users can be accurately distinguished. After collecting the voice signals, the noise reduction algorithm is used to pre-process the collected original language voice signals. This step adopts a noise reduction method that combines adaptive filtering with spectral subtraction. First, adaptive filtering technology is used to estimate and remove background noise that is not related to the voice signal. Then, spectral subtraction further processes the signal by subtracting the estimated value of the noise spectrum from the spectrum of the noisy voice signal to reduce the residual noise components. This method can significantly improve the clarity and intelligibility of voice signals, especially when used in noisy environments, further improving the clarity and quality of voice signals.

[0025] Step 3: The pre-processed original language voice signal is transmitted to the SOC SSD2386 chip, and the original language voice signal is analyzed using the built-in long short-term memory network model to identify the user's language type; Specifically, first, the preprocessed original language speech signal is transmitted to the SOCSSD2386 chip through an appropriate interface or bus. The preprocessing process usually includes steps such as noise reduction and filtering to improve the quality and clarity of the speech signal; Then, the SOC SSD2386 chip has a built-in long short-term memory (LSTM) network model, which is a special recurrent neural network (RNN) that is particularly suitable for processing and predicting time series data, such as speech signals. It can deeply analyze speech signals by capturing long-term dependencies in time series. Next, the LSTM model extracts features and performs pattern matching on the input voice signal to identify the user's language type. This process is based on the model's analysis of the differences in acoustic and semantic features of different languages. The model uses multiple layers of neurons to perform in-depth analysis and pattern matching on key features in the voice signal, and ultimately determines the user's language type. Among them, the application of the long short-term memory network model enables the translator to accurately recognize multiple languages, thereby providing accurate language information for subsequent translation and display steps. The implementation of this step depends on the powerful computing power of the SOC SSD2386 chip and the accuracy of the built-in model. Through this step, the dual-screen real-time translator can accurately identify the user's language type, providing a basis for subsequent dual-screen display and real-time translation.

[0026] Step 4: input the original language speech signal after the language type is identified into the translation module based on the deep learning neural network architecture, convert the original language speech signal into target language text information, and use speech synthesis technology to convert the target language text information into the target language speech signal; Specifically, first, the original language speech signal after the language type is identified in step 3 is input into the translation module. At this time, the signal has been pre-processed and its language type has been accurately identified; Then, tokenization technology is used to perform lexical analysis on the text corresponding to the original language speech signal to generate tokenized original language text information. This step helps to break down the text into units that are easier to handle, such as words, phrases, etc. Next, the tokenized original language text information is subjected to syntactic analysis to construct a syntactic tree or dependency graph. Through the syntactic analysis algorithm, the grammatical structure and dependency relationship of the original language text information can be determined, which is the basis for understanding the semantics of the text. Subsequently, the semantic information of the original language is mapped to the target language according to the grammatical structure and dependency relationship of the original language text information. This step generates the result of semantic mapping and conversion, which is the core of text translation. Finally, based on the results of semantic mapping and conversion, text information in the target language is generated according to the grammatical rules of the target language; The specific method of converting the speech signal into the target language using speech synthesis technology is as follows: First, the generated target language text information is obtained and preprocessed for subsequent acoustic feature extraction; Then, the acoustic feature extraction algorithm is used to extract the acoustic features of the preprocessed target language text information. These features include pitch, intensity, timbre, etc., which are the basis of speech synthesis. Next, the extracted acoustic features are converted into a target language speech signal using a vocoder, wherein a vocoder is an algorithm or device that can convert text features into speech signals; Finally, the generated target language speech signal is post-processed using audio processing technology to improve the naturalness and clarity of the speech.

[0027] Step 5: Display the target language text information on the touch screen of the other user, and perform personalized display settings on the target language text information according to the user's display preference. At the same time, play the converted target language voice signal through the corresponding speaker; Specifically, once the translation module completes the conversion from the original language to the target language, the text information in the target language will be immediately sent to the other party's touch screen. The dual-screen display technology ensures that each user can see the translation results for themselves on their own screen, thereby avoiding information confusion. Considering that users' display preferences may vary, the translation machine will adjust the display mode of text information according to the user's personalized settings, which may include adjustments to font size, color, background, and layout to ensure that users can read the translation results comfortably. At the same time, the translation machine will also play the converted target language voice signal through the built-in speaker, so that users can not only read the translation results, but also confirm the accuracy of the translation by hearing. In order to ensure the clarity and naturalness of the voice playback, the translation machine will optimize the sound quality of the voice signal, which may include removing noise, adjusting the volume and speech speed, etc., to provide the best listening experience. During the entire translation and display process, the translator will ensure the real-time nature of all steps, which means that the entire process from voice signal acquisition, preprocessing, translation to final display and playback will be completed in a very short time to ensure smooth conversations between users; at the same time, dual-screen display technology also requires the translator to be able to coordinate the display content on the two screens, which includes ensuring that the information on the two screens is updated synchronously, and dynamically adjusting the display content based on the user's interactive behavior (such as clicking, sliding, etc.).

[0028] Step 6: Collect user feedback on translation results and display and playback effects, and adjust parameters of the language recognition model, translation module, and speech synthesis technology according to user feedback; Specifically, users can evaluate the accuracy, fluency and naturalness of the translation results through feedback buttons on the touch screen or voice commands. These evaluations can be qualitative (such as "good", "average", "poor") or quantitative (such as a scoring system); users can also provide feedback on the clarity of the display interface, the rationality of the layout, and the volume, speaking speed and sound quality of the voice playback. These feedbacks help the system understand the user's visual and auditory preferences; the collected user feedback information will be organized into a database for subsequent analysis and processing, and the user feedback will be quantitatively analyzed using statistical analysis methods to identify common problems or improvement points in the translation results and display playback effects; Based on user feedback, adjust the parameters of the language recognition model to improve the accuracy and robustness of recognition, which may include adjusting the structure of the model, adding training data or optimizing the algorithm; fine-tune the parameters of the translation module to improve the accuracy and fluency of the translation results, which may involve adjusting the weights of the translation model, optimizing the translation strategy or introducing new translation rules; based on user feedback on the speech playback effect, adjust the parameters of the speech synthesis technology to improve the naturalness and clarity of the speech, which may include adjusting the parameters of the speech synthesizer, optimizing the speech library or introducing new speech synthesis algorithms; apply the adjusted parameters and models to the translation system, and regularly update the system to ensure that the translation system can continue to maintain its optimal state.

[0029] In one embodiment, specifically, in step 3, the method for identifying the language type of the user includes the following steps: Step 301: extracting features from the original language speech signal to extract key features; Specifically, a method based on the combination of Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC) is used for feature extraction. For the collected original language speech signal, it is first divided into frames of fixed duration (such as 20-30 milliseconds) to capture the characteristic changes of the speech signal in a short period of time. Then, the MFCC parameters are calculated for each frame signal, and 12-13 MFCC coefficients are usually extracted. These coefficients can effectively characterize the spectral characteristics of the speech and have a good ability to distinguish the pronunciation characteristics of vowels and consonants in different languages ​​and the changes in tone. At the same time, the LPC coefficients are calculated, generally taking 10-12 orders. The LPC coefficients can reflect the vocal tract model characteristics of the speech signal and further supplement the acoustic feature information of the speech. Through this combination, the extracted key features not only include basic information such as the frequency and amplitude of the speech, but also capture characteristics such as the vocal tract resonance during the speech production process, providing a rich information basis for subsequent language recognition. In addition to acoustic features, the prosodic features of speech are also extracted. Prosodic features include pitch contour, intensity change, speaking speed and rhythm. The pitch contour is extracted by analyzing the fundamental frequency change of the speech signal. For example, the autocorrelation method or the Hilbert transform-based method is used to estimate the fundamental frequency to capture the differences in intonation between different languages. For intensity change, the energy envelope of the speech signal is calculated to observe its fluctuations in the sentence. Different languages ​​have different intensity change patterns when emphasizing key words or expressing different semantics. The speaking speed and rhythm are obtained by counting the number of syllables per unit time, pause intervals and other information. These prosodic features play an important role in distinguishing language types and identifying the sentence structure and semantic expression of different languages. To further extract the contextual semantic features of the speech, the n-gram model (such as 2-gram or 3-gram) in natural language processing technology is used to analyze the text corresponding to the speech signal. The consecutive n words are taken as a unit, and their occurrence frequency and co-occurrence relationship in different language texts are counted to construct a semantic feature vector. For example, in English, the co-occurrence frequency of "the" and "book" is relatively high, while in French, the co-occurrence frequency of "le" and "livre" may be relatively high. In this way, the vocabulary collocation and semantic combination habits of different languages ​​can be captured, providing more in-depth semantic information support for language recognition.

[0030] Step 302: Input the extracted key features into the long short-term memory network model, and perform in-depth analysis and pattern matching on the key features through multi-layer neuron processing according to the differences in acoustic and semantic features of different languages; Specifically, the extracted key features are input into the long short-term memory network (LSTM) model. The input layer of the model first normalizes the feature vector so that its value range is within a suitable interval (such as [-1,1]) to improve the stability and convergence speed of model training. The hidden layer of the model consists of multiple LSTM units, usually 2-3 hidden layers, each containing 128-256 LSTM units. Each LSTM unit can effectively process the time series characteristics of speech signals and learn the long-term dependencies between different languages ​​in acoustic and semantic features through its unique gating mechanism (input gate, forget gate and output gate). During the training process, the model is optimized using stochastic gradient descent (SGD) or its improved algorithms (such as Adagrad, Adadelta, etc.). The weight and bias parameters of the model are adjusted according to the training samples of different languages, so that the model can accurately distinguish the characteristic patterns of different languages. For example, for the speech characteristics of English and French, the model gradually masters the pronunciation characteristics of English plosives in terms of acoustic characteristics and the semantic association with the preceding and following phonemes through learning, and distinguishes the unique acoustic and semantic characteristics of nasalized vowels in French. In this way, in the processing process of multi-layer neurons, the recognition ability of different language features is continuously enhanced, and the accuracy of pattern matching is improved. The attention mechanism is used to enhance the model's focus on key features. Based on the LSTM model, an attention layer is introduced to assign different weights to different features according to their importance in language recognition. For those features that play a key role in distinguishing language types (such as tone changes unique to certain languages, unique consonant combinations, etc.), higher attention weights are given, so that the model can focus more on these important features during the analysis and matching process, further improving the accuracy and efficiency of recognition.

[0031] Step 303: Identify the user's language type through a training process of continuous iteration and optimization; Specifically, first, we built a large-scale multilingual speech dataset that covers the world's major languages ​​(such as English, Chinese, French, Spanish, Arabic, etc.). Each language contains a wealth of speech samples, including different accents, speaking speeds, contexts, etc. For example, the English dataset contains speech data with multiple accents, such as British English, American English, Australian English, and samples in different contexts, such as daily conversations, speeches, and news broadcasts, to ensure that the model can adapt to various actual usage scenarios. Then, the model is trained and evaluated using the cross-validation method. The data set is divided into a training set, a validation set, and a test set, usually in a ratio of 7:2:1. In each training iteration, the model is trained using the training set, and the performance of the model is monitored using the validation set. The model's hyperparameters (such as the learning rate, the number of hidden layer nodes, etc.) are adjusted based on indicators such as the loss function value and accuracy on the validation set. After multiple iterations of training (usually 100-200 iterations), the performance of the model on the validation set reaches a stable and high level. Finally, regularly test the trained model using the test set to evaluate the model's generalization ability on unseen data. If the model's accuracy on the test set decreases or overfitting occurs, further adjust the model structure or increase the amount of training data to optimize the model. For example, you can add regularization terms (such as L1 or L2 regularization) to prevent overfitting, or collect more edge case speech data (such as speech with severe accents or noise interference) to enrich the training set, continuously improve the model's robustness and recognition accuracy, and ultimately achieve accurate recognition of the user's language type.

[0032] In one embodiment, specifically, in step 4, the method of converting the original language speech signal into the target language text information includes the following steps: Step 401: input the original language speech signal after the language type is identified into the translation module, and use the tokenization technology to perform lexical analysis on the text corresponding to the original language speech signal to generate tokenized original language text information; Specifically, when the original language speech signal after identifying the language type is input into the translation module, first, a natural language processing toolkit (such as NLTK or SpaCy) is used to tokenize the text corresponding to the original language speech signal; for different languages, the tokenization rules will be different; for example, for English, the text will be split into words and symbols according to spaces and punctuation marks, while dealing with abbreviations and special characters, such as "don't" will be tokenized as ["do","n't"], and for hyphenated words, such as "well-known" will be split into ["well","-","known"]; for Chinese, it is divided according to words and characters, while considering special cases of Chinese word segmentation, such as dealing with out-of-vocabulary words, ambiguous segmentation, etc., and using statistical and rule-based methods, such as the maximum matching method, hidden Markov model, etc., to tokenize the Chinese sentence "What he said is very interesting." as ["他","说","的","话","很","有意思","。"].

[0033] When performing lexical analysis, according to the lexical rules of the original language, a part-of-speech tag is assigned to each token; for English, the part-of-speech of the token may include nouns (NN), verbs (VB), adjectives (JJ), adverbs (RB), prepositions (IN), etc. For example, "running" will be tagged as a verb (VBG), and "quickly" will be tagged as an adverb (RB); for Chinese, a similar part-of-speech tagging will also be performed, such as "跑步" being tagged as a verb (VV), and "美丽" being tagged as an adjective (JJ); at the same time, special lexical phenomena in different languages are processed, such as the plural form in English (books->book+s), verb tense changes (ran->run+ed), and the collocation of quantifiers in Chinese (the "个" in "一个" is a quantifier), etc., to accurately identify the grammatical function and semantic role of each token in the original language.

[0034] Step 402: Perform syntactic analysis on the tokenized original language text information, construct a syntactic tree or dependency graph, and determine the grammatical structure and dependency relationship of the original language text information through a syntactic analysis algorithm; Specifically, a syntactic analysis algorithm based on statistics and rules, such as a dependency syntactic analysis algorithm or a phrase structure syntactic analysis algorithm, is used to perform syntactic analysis on the tokenized original language text information; for dependency syntactic analysis, a dependency graph is constructed, and the structure of the original language sentence is determined by analyzing the dependency relationship between words, such as subject-predicate relationship, modification relationship, object relationship, etc.; for example, in the sentence "The cat chased the mouse.", it is determined that "cat" is the subject of "chased", "mouse" is the object of "chased", and "the" modifies "cat" and "mouse" respectively; for the Chinese sentence "I like to eat apples.", it can be determined that "I" is the subject, "like" is the predicate, "eat apples" is part of the predicate, and "apple" is the object of "eat"; In the process of constructing the syntactic tree, for complex sentences, the hierarchical structure and component nesting of the sentence are considered; for example, for the English sentence "The man who is wearing a hat is my father.", a syntactic tree containing attributive clauses will be constructed to clarify the relationship of "who is wearing a hat" as an attributive clause modifying "man"; for the Chinese sentence "The book I bought yesterday is very interesting.", "I bought it yesterday" will be identified as the attributive part modifying "book", and this structural relationship will be clearly displayed through the syntactic tree or dependency graph, providing accurate structural information for subsequent semantic understanding and translation.

[0035] Step 403: Map the semantic information of the original language to the target language according to the grammatical structure and dependency relationship of the original language text information, and generate a result of semantic mapping and conversion; Specifically, a semantic mapping model based on deep learning is used. The model is trained on a large-scale parallel corpus, which contains a large number of sentence pairs in the source language and the target language and their semantic annotation information. The semantic information of the source language is mapped to the target language according to the grammatical structure and dependency relationship of the source language. For some polysemous words, their specific semantics are determined according to the context and dependency relationship. For example, the English word "bank" can mean "bank" or "river bank" in different contexts. Its semantics in a specific sentence can be clarified through the syntactic structure and dependency relationship. Special expressions in the source language, such as idioms, metaphors, culture-specific words, etc., are processed specially; for example, "kick the bucket" in English cannot be directly translated word for word. Through a semantic mapping model, according to its cultural connotation, it is converted into the corresponding expression in the target language, such as "pass away" in Chinese. For semantic asymmetries between different languages, such as "I am looking forward to" in English may need to be converted to "我期待着" in Chinese, considering semantic expansion, contraction, or transfer to ensure the accuracy and naturalness of semantic mapping.

[0036] Step 404: Generate the text information in the target language according to the results of semantic mapping and conversion, in accordance with the grammar rules of the target language. Generate the text information in the target language according to the results of semantic mapping and conversion, in accordance with the grammar rules of the target language. For the word order and sentence order of the target language, follow the basic grammar rules of its language. For example, English usually has the order of subject - predicate - object, while Japanese may have the order of subject - object - predicate. According to the grammar rules and morphological characteristics of the target language, organize the semantic units after semantic mapping into complete sentences. Correctly handle the morphological changes of the target language, such as verb conjugation and noun declension; when generating sentences in the target language, consider the language characteristics of the target language. For example, verbs in Spanish have multiple conjugation forms according to person, tense, mood, etc., and accurate conjugation is performed according to semantic information and grammatical requirements when generating sentences; for nouns in French, there are different endings according to gender and number, and correct selection and change are made during the generation process. Adjust the fluency of the generated sentences in the target language, and use a language model (such as an n - gram - based language model) for evaluation and optimization to avoid generating sentences that are rigid or do not conform to the expression habits of the target language; for example, use a 3 - gram model to check whether the generated sentences conform to the lexical collocations and phrase structures of the target language, and adjust and polish the less natural expressions to ensure that the generated text information in the target language is natural and fluent, conforming to the language habits and expression norms of the target language.

[0037] In one embodiment, specifically, in step four, the method of converting the text information in the target language into a speech signal in the target language using speech synthesis technology includes the following steps: Step 4011: Obtain the generated text information in the target language and perform pre - processing. Specifically, first, the generated target language text information is normalized. Different languages ​​have different normalization rules. For example, for English, numbers are converted to English words (such as "2025" is converted to "twothousand and twenty-five"), abbreviations are expanded (such as "Mr." is converted to "Mister"), and special characters and symbols are processed uniformly to ensure the consistency of the text. For Chinese, traditional Chinese is converted to simplified Chinese, and some special symbols and punctuation usage habits are processed (such as converting full-width punctuation to half-width punctuation). Perform word segmentation on the text. For languages ​​like Chinese, use a word segmentation algorithm to split continuous text into words or phrases. You can use a dictionary-based word segmentation method combined with a statistical model (such as a hidden Markov model, maximum entropy model, etc.) to improve the accuracy of word segmentation. For languages ​​like English, perform simple word segmentation based on spaces and punctuation marks, and handle some special cases, such as hyphenated words and abbreviations. Mark the text prosody to indicate the stress position and intonation information in the sentence; mark different intonation patterns for different sentence types such as declarative sentences, interrogative sentences, and exclamatory sentences; for example, in English, declarative sentences generally use a falling tone, while interrogative sentences use a rising tone, and this information is indicated by adding special marks or modifying the text representation (such as adding accent marks or marking intonation curves); for Chinese, determine stress and intonation based on semantics and grammatical structures, such as adding accent marks to the emphasized parts.

[0038] Step 4012: extracting acoustic features from the preprocessed target language text information using an acoustic feature extraction algorithm; Specifically, deep learning techniques, such as neural network architecture, are used to extract acoustic features from the preprocessed target language text information; an acoustic model is constructed, which may be a sequence-to-sequence model based on deep learning, such as an acoustic model based on the Transformer architecture, and the preprocessed text is used as input, and the text is converted into a vector representation through an embedding layer and input into the model; During the training process, the model learns the mapping relationship between text and acoustic features. The extracted acoustic features include fundamental frequency (F0), amplitude spectrum, formant frequency, duration, etc. For example, for each syllable or phoneme, the model predicts its corresponding fundamental frequency curve and duration based on its position in the sentence, stress mark and other information to reflect the rhythmic characteristics of different languages. For the four tones of Chinese, the model predicts different fundamental frequency contours based on the tone marks. For vowels and consonants in English, different formant frequencies are extracted based on different pronunciation characteristics. Taking into account the phenomenon of co-articulation of language, when extracting acoustic features, the model will adjust the acoustic features of the current phoneme according to the influence of the previous and next phonemes or words; for example, in English, the pronunciation of a vowel may change due to the influence of the previous and next consonants. The model will comprehensively consider the contextual information to produce more accurate acoustic features and make the generated speech more natural.

[0039] Step 4013: using a vocoder to convert the extracted acoustic features into a target language speech signal; Specifically, you can choose WaveNet vocoder or other advanced vocoder technologies; WaveNet vocoder is based on deep learning, trained with a large amount of speech data, and takes acoustic features as input; For the extracted acoustic features such as fundamental frequency and amplitude spectrum, the WaveNet vocoder gradually generates the original speech waveform by generating probability distribution; for example, based on the input acoustic features, it predicts the probability distribution of each sampling point, and then samples from the distribution to obtain the final speech signal. It can generate high-quality speech waveforms that can well simulate the details and naturalness of speech; For the speech characteristics of different languages, the vocoder can generate speech signals that conform to the speech characteristics of each language by adjusting model parameters or using sub-models trained for different languages. For example, for different tones in Chinese, the vocoder will generate speech waveforms with corresponding tone contours based on the acoustic characteristics. For the connected reading and weak reading phenomena in English, the corresponding waveforms are generated based on the acoustic characteristics to ensure the fluency of the speech.

[0040] Step 4014: post-processing the generated target language speech signal using audio processing technology; Specifically, first, a noise reduction algorithm, such as Wiener filtering, spectral subtraction, etc., is used to perform noise reduction processing on the generated target language speech signal; for environmental noise and noise introduced during the model generation process, the noise spectrum is estimated by analyzing the spectral characteristics of the noise, and then the noise spectrum is subtracted from the spectrum of the speech signal to obtain a purer speech signal; for example, if low-frequency background noise is detected in the speech signal, the Wiener filter can be used to remove the noise according to the frequency and amplitude of the noise to improve the clarity of the speech; Then, the amplitude of the speech signal is scaled according to the user's preference or the system's default setting; for example, the user can adjust the volume slider on the system interface and adjust the volume within the range of -30dB to 30dB, and the system will scale the amplitude of the speech signal linearly or nonlinearly according to the user's setting; Next, the playback speed of the speech is changed by modifying the duration information of the speech signal; for fast-speed playback, the duration of each syllable or phoneme can be shortened, and for slow-speed playback, the duration is extended; at the same time, in order to ensure the naturalness of the speech, when adjusting the speech speed, the rhythmic characteristics will be adjusted according to the characteristics of the language to avoid unnatural phenomena caused by changes in speech speed; Finally, the equalizer is used to balance the speech signal and adjust the energy of different frequency components to improve the sound quality of the speech. For some languages ​​where the low-frequency energy is weak or the high-frequency energy is too strong, the frequency response curve is adjusted by the equalizer to make the energy distribution of the speech in each frequency band more balanced, thereby improving the fullness and clarity of the sound. In addition, some sound effects, such as reverberation effects, can be added to make the generated speech sound more natural and vivid; for simulations in large rooms or outdoor environments, add appropriate reverberation effects, and adjust the time and intensity of reverberation according to different usage scenarios to enhance the user's auditory experience.

[0041] In one embodiment, specifically, in step 2, based on the microphones distributed near the two screens, when the original language voice signals of the two users are collected respectively, a multi-channel audio collection technology is adopted, and an independent gain control module is set for each microphone channel to dynamically adjust the gain according to the frequency characteristics and loudness of the voices of different users; wherein, the translation machine is equipped with multiple microphones, which are distributed around the two touch screens to form a multi-channel audio collection system; each microphone is responsible for collecting the sound signals in its vicinity to ensure that the range in which the two users may make sounds can be fully covered; for example, in the horizontal direction, the microphones are arranged at a certain distance to effectively capture voices from different angles; in the vertical direction, a reasonable layout is also made to avoid omissions in sound collection due to differences in the height of the users' heads; the multi-channel audio collection system adopts a synchronous clock mechanism to ensure that the audio data collected by each microphone is synchronized in time, which is crucial for the subsequent accurate analysis and processing of voice signals from different users; for example, through a precise clock synchronization circuit, all microphones start and Stop collecting to ensure that the collected audio data can be accurately aligned on the time axis so as to effectively process and distinguish the voices of different users in the future; wherein, an independent gain control module is set for each microphone channel, which can monitor the frequency characteristics and loudness of the input audio signal in real time; first, the input audio signal is subjected to spectrum analysis through algorithms such as fast Fourier transform (FFT) to obtain its frequency component information; then, the strength and main frequency distribution of the current audio signal are determined according to the preset frequency range and loudness threshold; when it is detected that the user's voice is weak or in the low frequency band, the gain control module automatically increases the gain to improve the signal strength; when the sound is too strong or there are too many high-frequency components, which may cause distortion, the gain is appropriately reduced; for example, for whispering or low-frequency voice, the gain can be linearly increased within a certain range (such as 10-30dB); for shouting or high-frequency sharp voice, the gain can be reduced by 5-20dB to ensure that the collected voice signal is within the appropriate dynamic range, which not only ensures that the signal is clear and discernible, but also avoids signal distortion or noise amplification caused by improper gain.

[0042] When the collected original language speech signal is preprocessed by using the noise reduction algorithm, a noise reduction method combining adaptive filtering and spectral subtraction is adopted; among them, the adaptive filtering algorithm is based on the minimum mean square error (LMS) or recursive least squares (RLS) criterion, and the coefficients of the filter are continuously adjusted according to the input speech signal and the reference noise signal; during the initialization stage of the translator or the idle period during operation, the system will collect environmental noise as a reference signal to establish a noise model; when a mixed signal containing speech and noise is collected, the adaptive filter dynamically adjusts the filter parameters according to the previously established noise model to make the filter output as close to the noise signal as possible, and then subtracts the filter output from the original input signal, thereby effectively removing background noise components similar to the noise model; for example, in a noisy conference room environment, for the persistent air conditioning noise, equipment humming, etc., the adaptive The filter can quickly learn its frequency characteristics and intensity changes, adjust the filter parameters in real time, and effectively remove these noises from the speech signal; among them, the spectral subtraction method first performs a short-time Fourier transform (STFT) on the collected noisy speech signal to obtain its spectral representation; then, by estimating the spectrum of the noise, usually the average spectrum is calculated in the silent segment of the speech signal or the period when the noise is relatively stable to obtain the noise spectrum estimate; the estimated value of the noise spectrum is subtracted from the spectrum of the noisy speech signal to obtain the speech spectrum after preliminary noise reduction; in order to avoid problems such as "music noise" caused by spectral subtraction, some post-processing techniques are also used, such as spectral smoothing, over-subtraction factor adjustment, etc.; for example, after subtracting the noise spectrum, the obtained spectrum is smoothed by median filtering to reduce the spectrum fluctuation caused by inaccurate noise estimation, so as to make the denoised speech signal more natural.

[0043] In the process of collecting the original language voice signals of two users, the collected voice signals are time-stamped and source-stamped in real time. Among them, the time stamp uses high-precision time stamp technology to mark the collected voice signals in real time. The accuracy of the time stamp can reach milliseconds or even higher, ensuring that the start time and duration of each voice segment can be accurately recorded. For example, when the microphone detects the starting edge of the voice signal, the system immediately records the current time information, and as the voice signal is continuously collected, the time stamp is continuously updated until the voice signal ends. The time stamp information is closely bound to the audio data, and the integrity of the time information is always maintained during the subsequent data transmission, storage and processing. This is of great significance for analyzing the timing relationship of voice signals, synchronizing data collected by multiple microphones, and realizing delay control of real-time translation. For example, when processing a scene where two users speak alternately, the time stamp can accurately determine the order and time interval of each user's speech, providing accurate time clues for the language recognition and translation modules. Assign a unique identifier to each microphone as a source tag; during the audio data collection process, record the identifier together with the corresponding voice signal; when the voice signal passes through the subsequent processing link, it always carries its source tag information; in this way, in a multi-microphone collection environment, it is possible to clearly distinguish which microphone collected each voice signal, thereby accurately determining the source user of the voice signal; for example, in subsequent voice processing and analysis, if a voice segment is found to have problems or requires further processing, the corresponding microphone and user can be quickly located based on the source tag, facilitating targeted processing and optimization.

[0044] In one embodiment, specifically, in step 5, when the target language text information is displayed on the touch screen of the other party's user, a font enlargement and reduction function is provided for the displayed text information, and a text scrolling display function is supported; wherein the font enlargement and reduction function allows the user to adjust the font size of the text information displayed on the touch screen according to personal reading habits or vision conditions; The specific implementation method is as follows: Through the touch screen programming software, find the font icon in the property settings, then adjust the font size and confirm; in this way, the user can directly see the change of font size on the touch screen; Provide users with shortcut key combinations, such as "Ctrl" + "+" to enlarge the font, and "Ctrl" + "-" to reduce the font, so that users can quickly adjust the font size; Set up a slider or button on the touch screen, and the user can gradually adjust the font size by sliding or clicking; The text scrolling display function allows you to scroll to view the complete text content when the text information displayed on the touch screen exceeds the screen range; The specific implementation method is as follows: Set a certain scrolling speed to make the text information scroll automatically on the screen; users can adjust the scrolling speed through settings; Provide scroll bars or touch-swipe areas where users can manually scroll text by sliding their fingers. When the text scrolls to the edge of the screen, the system can automatically detect and stop scrolling to prevent the text from exceeding the screen range.

[0045] In one embodiment, specifically, in step 6, when adjusting the parameters of the language recognition model, the translation module, and the speech synthesis technology according to the user feedback information, a phased adjustment strategy is adopted; wherein the phased adjustment strategy is a systematic and orderly method, which aims to gradually optimize the relevant parameters of the language recognition model, the translation module, and the speech synthesis technology; the strategy divides the entire process into multiple stages, each stage has clear goals and steps to ensure the effectiveness and accuracy of the parameter adjustment; The specific steps for adjusting the strategy in stages are as follows: Phase 1: Data collection and preprocessing Collect user feedback on language recognition, translation, and speech synthesis through user feedback channels (such as online surveys, user reviews, customer service feedback, etc.); clean, classify, and organize the collected user feedback data for subsequent analysis; Phase 2: Preliminary parameter adjustments Identify the parameters that need to be adjusted based on the common problems reported by users; set the initial parameter values ​​of the language recognition model, translation module and speech synthesis technology based on the analysis results; use the initially adjusted parameters to test language recognition, translation and speech synthesis in the test environment and record the test results; Phase 3: Parameter Optimization and Verification Based on the test results, fine-tune the parameters to improve the accuracy and naturalness of language recognition, translation, and speech synthesis; use the optimized parameters to test again to verify the effect of parameter adjustment; if the test results are not ideal, return to the second stage, continue to adjust the parameters and retest until a satisfactory effect is achieved; Phase 4: Deployment and Implementation Deploy the optimized parameters to the production environment to ensure that the language recognition model, translation module, and speech synthesis technology can perform well in actual applications; establish a monitoring mechanism to track the system performance after parameter adjustment in real time so as to promptly identify and resolve problems.

[0046] In summary, an implementation method of a dual-screen real-time translation machine provided by an embodiment of the present invention first initializes key components, then collects and preprocesses two user voice signals, and then uses specific models and technologies to complete language recognition, text translation and speech synthesis, then displays information on the touch screen and plays voice, and finally adjusts relevant parameters in stages based on user feedback. It effectively solves the problems of information display confusion and inability to meet user language preferences in traditional translation devices in face-to-face communication scenarios among multilingual users, and realizes efficient, accurate and personalized real-time translation functions.

[0047] Figure 5 Schematic diagram of the functional modules of a dual-screen real-time translator according to an embodiment of the present application. Figure 5 As shown, a dual-screen real-time translation machine includes a chip module, an audio acquisition module, a language recognition module, a translation module, a speech synthesis module, a display module and a feedback processing module; The chip module is used to coordinate and control the operation of the entire translation machine, perform data operations and task scheduling, including receiving and processing audio data, driving the long short-term memory network model for language recognition, and interacting with the translation module and display module; The audio acquisition module is used to collect the original language voice signals of two users using a microphone. With the help of multi-channel audio acquisition technology and an independent gain control module, the gain is dynamically adjusted according to the frequency characteristics and loudness of the voices of different users, and the collected voice signals are time-stamped and source-stamped in real time; The language recognition module is used to receive the pre-processed original language voice signal, and identify the user's language type by performing feature extraction, in-depth analysis and pattern matching operations on the original language voice signal; A translation module is used to convert the original language speech signal after the language type is identified into the target language text information, and generate the target language text information according to the grammatical rules of the target language; A speech synthesis module, used to convert target language text information into target language speech signals; A display module, used to display the target language text information on the touch screen of the other party's user; The feedback processing module is used to collect user feedback on translation results and display playback effects, and to adjust the relevant parameters of the language recognition model, translation module and speech synthesis technology in a phased adjustment strategy based on user feedback information to continuously optimize the system.

[0048] In one embodiment, specifically, the chip module adopts a parallel processing mechanism when performing data calculations and task scheduling; the microphone in the audio acquisition module is an array microphone with multiple sound pickup units in different directions, and the multiple sound pickup units automatically adjust the sound pickup range and direction according to different usage scenarios.

[0049] In one embodiment, specifically, the dual-screen real-time translation machine is equipped with multiple connection interfaces, including at least one USB interface, one Type-C interface and one headphone interface. The USB interface is used to connect an external storage device for data transmission and backup; the Type-C interface is used for data transmission and charging; the headphone interface is used to connect headphones, and use headphones to listen to the translated voice.

[0050] In summary, the embodiment of the present invention provides a dual-screen different-display real-time translation machine, which coordinates and controls the operation of each module through the chip module. The audio acquisition module collects and processes voice signals, the language recognition module identifies the language type, the translation module performs text translation, the speech synthesis module generates voice signals, the display module displays the translation results, and the feedback processing module collects feedback and optimizes the system. The chip module adopts a parallel processing mechanism, the array microphone of the audio acquisition module can automatically adjust the direction of the pickup range, and is also equipped with a variety of connection interfaces to facilitate data transmission, charging, and headphone use, etc., so as to achieve efficient dual-screen different-display real-time translation function and continuously improve performance.

[0051] For other details about the technical solutions for implementing each module in the dual-screen real-time translator in the above embodiment, please refer to the description of the implementation method of the dual-screen real-time translator in the above embodiment, which will not be repeated here.

[0052] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0053] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0054] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0055] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0056] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0057] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0058] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0059] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for implementing a dual-screen real-time translation machine, characterized in that: The following steps are involved: Step 1: Obtain and start the dual-screen real-time translator, and initialize the SOC SSD2386 chip, two touch screens, multiple microphones distributed near the two touch screens, and a storage unit of the dual-screen real-time translator; Step 2: Based on microphones distributed near the two screens, the original language voice signals of the two users are collected respectively, and the collected original language voice signals are pre-processed using a noise reduction algorithm; Step 3: The pre-processed original language voice signal is transmitted to the SOC SSD2386 chip, and the original language voice signal is analyzed using the built-in long short-term memory network model to identify the user's language type; Step 4: input the original language speech signal after the language type is identified into the translation module based on the deep learning neural network architecture, convert the original language speech signal into target language text information, and use speech synthesis technology to convert the target language text information into the target language speech signal; Step 5: Display the target language text information on the touch screen of the other user, and perform personalized display settings on the target language text information according to the user's display preference. At the same time, play the converted target language voice signal through the corresponding speaker; Step 6: Collect user feedback on translation results and display and playback effects, and adjust the parameters of the language recognition model, translation module and speech synthesis technology based on the user feedback.

2. A method for implementing a dual-screen real-time translation machine according to claim 1, characterized in that: In step three, the method for identifying the user's language type includes the following steps: Step 301: extracting features from the original language speech signal to extract key features; Step 302: Input the extracted key features into the long short-term memory network model, and perform in-depth analysis and pattern matching on the key features through multi-layer neuron processing according to the differences in acoustic and semantic features of different languages; Step 303: Identify the user's language type through a training process of continuous iteration and optimization.

3. The method for implementing a dual-screen real-time translation machine according to claim 1, characterized in that: In step 4, the method for converting the original language speech signal into the target language text information comprises the following steps: Step 401: input the original language speech signal after the language type is identified into the translation module, and use the tokenization technology to perform lexical analysis on the text corresponding to the original language speech signal to generate tokenized original language text information; Step 402: Perform syntactic analysis on the tokenized original language text information, construct a syntactic tree or dependency graph, and determine the grammatical structure and dependency relationship of the original language text information through a syntactic analysis algorithm; Step 403: Map the semantic information of the original language to the target language according to the grammatical structure and dependency relationship of the original language text information, and generate a result of semantic mapping and conversion; Step 404: Generate text information in the target language according to the results of semantic mapping and conversion and the grammatical rules of the target language.

4. A method for implementing a dual-screen real-time translation machine according to claim 3, characterized in that: In step 4, the method of converting the target language text information into the target language speech signal by using speech synthesis technology comprises the following steps: Step 4011, obtaining the generated target language text information and performing preprocessing; Step 4012: extracting acoustic features from the preprocessed target language text information using an acoustic feature extraction algorithm; Step 4013: using a vocoder to convert the extracted acoustic features into a target language speech signal; Step 4014: Post-process the generated target language speech signal using audio processing technology.

5. The method for implementing a dual-screen real-time translation machine according to claim 1, characterized in that: In step 2, when the original language voice signals of the two users are collected respectively based on the microphones distributed near the two screens, a multi-channel audio collection technology is adopted, and an independent gain control module is set for each microphone channel to dynamically adjust the gain according to the frequency characteristics and loudness of the voices of different users; When the noise reduction algorithm is used to pre-process the collected original language speech signal, a noise reduction method combining adaptive filtering and spectral subtraction is adopted; In the process of collecting the original language voice signals of the two users, the collected voice signals are time-stamped and source-stamped in real time.

6. The method for implementing a dual-screen real-time translation machine according to claim 1, characterized in that: In step five, when the target language text information is displayed on the touch screen of the other party's user, a font enlargement and reduction function is provided for the displayed text information, and a text scrolling display function is supported.

7. The method for implementing a dual-screen real-time translation machine according to claim 1, characterized in that: In step six, when adjusting the parameters of the language recognition model, the translation module and the speech synthesis technology according to the user feedback information, a phased adjustment strategy is adopted.

8. A dual-screen real-time translator with different displays, applied to a dual-screen real-time translator with different displays according to any one of claims 1 to 7, characterized in that: It includes chip module, audio acquisition module, language recognition module, translation module, speech synthesis module, display module and feedback processing module; The chip module is used to coordinate and control the operation of the entire translation machine, perform data operations and task scheduling, including receiving and processing audio data, driving the long short-term memory network model to perform language recognition, and interacting with the translation module and the display module; The audio acquisition module is used to collect the original language voice signals of two users using a microphone, dynamically adjust the gain according to the frequency characteristics and loudness of the voices of different users with the help of multi-channel audio acquisition technology and an independent gain control module, and perform real-time time tagging and source tagging on the collected voice signals; The language recognition module is used to receive the pre-processed original language voice signal, and identify the user's language type by performing feature extraction, deep analysis and pattern matching operations on the original language voice signal; The translation module is used to convert the original language speech signal after the language type is identified into the target language text information, and generate the target language text information according to the grammatical rules of the target language; The speech synthesis module is used to convert the target language text information into the target language speech signal; The display module is used to display the target language text information on the touch screen of the other party user; The feedback processing module is used to collect user feedback information on translation results and display and playback effects, and to adjust the relevant parameters of the language recognition model, translation module and speech synthesis technology using a phased adjustment strategy based on the user feedback information, so as to continuously optimize the system.

9. A dual-screen real-time translation machine according to claim 8, characterized in that: The chip module adopts a parallel processing mechanism when performing data calculations and task scheduling; the microphone in the audio acquisition module is an array microphone with multiple sound pickup units in different directions, and the multiple sound pickup units automatically adjust the sound pickup range and direction according to different usage scenarios.

10. The dual-screen real-time translation machine according to claim 8, characterized in that: The dual-screen real-time translation machine is equipped with multiple connection interfaces, including at least one USB interface, a Type-C interface and a headphone interface. The USB interface is used to connect an external storage device for data transmission and backup; the Type-C interface is used for data transmission and charging; the headphone interface is used to connect headphones and use headphones to listen to the translated voice.

Images