Method for carrying out simultaneous transmission call translation by using earphone

Abstract

The invention discloses a method for carrying out simultaneous transmission call translation by using an earphone, relates to the field of call translation, and solves the problem that in the process of simultaneous transmission call translation, if only the paraphrase of original voice is rigidly transmitted in a target language, the connotation and context of an original text cannot be fully captured and expressed, and the interpretation of the original text cannot be fully captured and expressed. The problems of serious information distortion and connotation loss are often caused; by simulating the mood of a speaker, the quality of simultaneous transmission call translation can be remarkably improved, the emotion expression is enhanced, the information accuracy is improved, the communication effect is enhanced, and the user experience is improved; in the aspect of technology implementation, the speech synthesis and sentiment analysis technology can provide powerful support, so that tone simulation is more natural and real; according to the method, voice data of different accent, different speakers, different background noise and different speech speeds are collected, a more robust and accurate voice recognition model can be trained, the method is suitable for diversified voice input, and the generalization ability of the method in different scenes is improved.

Description

Technical Field

[0001] The invention relates to the field of conversation translation, and in particular to a method for performing simultaneous conversation translation using earphones. Background Art

[0002] In today's globalized world, cross-border communication is becoming more and more frequent. However, language differences often become a barrier to communication. Although traditional translation methods, such as manual translation and machine translation, have solved this problem to a certain extent, they still have problems such as low efficiency and high cost. With the development of artificial intelligence technology, especially the rapid progress of speech recognition and natural language processing technology, the demand for real-time simultaneous interpretation is increasing. In addition, the emergence of mixed reality technology has made it possible to provide a more intuitive and interactive translation experience. Therefore, how to achieve efficient, low-cost, multilingual, and multimodal real-time simultaneous interpretation has become an important research topic.

[0003] The application with publication number CN117979246A relates to the field of call translation, and specifically to a method and system for completing two-way call translation by one person wearing a wireless headset. Either party of the call wears a wireless headset, and the language spoken by the party wearing the wireless headset is translated into the language spoken by the other party, and the translated voice is transmitted to the other party; at the same time, the language spoken by the other party is translated into the language spoken by the party wearing the wireless headset, and the obtained translated voice is played through the wireless headset.

[0004] In the process of simultaneous interpretation, if the interpretation of the original speech is simply conveyed in the target language in a rigid manner, without fully capturing and expressing the emotions and connotations of the original text, this will often lead to serious distortion of information and loss of connotation. For native speakers of the target language, they only receive the surface content of the text, but fail to deeply understand the multi-level information behind it, such as the culture, emotions and context. This simple text conversion can easily lead to cognitive bias, making their understanding far from the actual information content being conveyed, or even going in the opposite direction. Therefore, when conducting simultaneous interpretation, we must focus on the comprehensive understanding and precise expression of the original text to ensure accurate transmission of information and effective communication. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method for simultaneous interpretation of conversations using headphones, which solves the problem that in the process of simultaneous interpretation of conversations, if the interpretation of the original speech is only rigidly conveyed in the target language, but the emotion and connotation of the original text are not fully captured and expressed, this often leads to serious distortion of information and loss of connotation.

[0006] To achieve the above object, the present invention is implemented by the following technical solution: a method for simultaneous interpretation of a call using headphones, comprising the following steps:

[0007] S1. Search the target language’s speech samples online, including recordings and corresponding speech text transcriptions, and the speech samples cover different accents, speakers, speaking speeds, and background noises to establish a speech dataset; obtain user speech data and perform preprocessing;

[0008] S2. Establish a speech recognition model based on a lightweight convolutional neural network. By extracting local features of the target language, using the CTC algorithm, obtain the user's voice data, realize language recognition, and generate a type of signal to send to the translation component; establish a user identity database;

[0009] S3, through the translation component, analyzing a type of signal, determining the translation object language, including text translation and emotional translation, generating text translation data and emotional translation data respectively; confirming the user, making personalized adjustments to the text translation and emotional translation according to the user tag, generating a translation voice signal through speech synthesis, and sending it to the target terminal;

[0010] S4. Optimize model operations, including collecting feedback data, in-depth exploration of sentiment translation, and in-depth sentiment exploration of early sentiment translation and good performance cases based on the user identity database;

[0011] S5, Personalized customization, through the analysis of usage data, divide the communication field according to the translation results, and match the relevant users, automatically record the common vocabulary, phrases and expressions of users in this field, and give priority to this information in the text translation of step S3, and share the learning results with other users through cloud services to form a continuously evolving translation community.

[0012] As a further solution of the present invention, in step S1, the process of establishing the speech data set includes the following steps: searching the speech text of the target language online, where the target language is determined according to market demand and application scenarios; classifying the speech text according to the spoken language, and establishing corresponding accent folders, and under each accent folder, classifying it again according to the speaker, and establishing a speaker folder;

[0013] The speaker folder contains recordings of the speaker speaking at different speeds under different background noises and the corresponding recording text transcriptions to generate a speech dataset.

[0014] As a further solution of the present invention, in step S1, obtaining user voice data and performing preprocessing includes the following steps:

[0015] The user's voice signal is collected through the microphone of the headset, and the collected voice signal is pre-processed, including noise reduction, gain control and spectrum analysis, and converted into a digital signal.

[0016] As a further solution of the present invention, in step S2, establishing a speech recognition model includes the following steps:

[0017] Receive a speech data set, convert the speech data into a spectrogram using a Mel-spectrogram, extract features of the target language, and use it as the input of the speech recognition model; perform data enhancement on the speech data, including adding noise, speed change, and pitch change operations, and perform segmentation processing on the enhanced speech data to generate a training set, a validation set, and a test set in a ratio of 7:1:2;

[0018] Use the training set data to train the model and adjust the model parameters, including convolution kernel weights, convolution layer bias terms, weight matrices, and fully connected layer bias terms, as well as Adam optimizer and SGD optimizer.

[0019] Use validation set data to adjust model parameters to prevent overfitting;

[0020] Use the test set data to evaluate the model performance and calculate the accuracy;

[0021] The accuracy is calculated as:

[0022]

[0023] Define the preset standard. If the accuracy is less than the preset standard, which is 90% here, the model is considered to have overfitting or underfitting problems.

[0024] Observe the loss curves, accuracy curves, and word error rate curves of the training set and validation set. If the loss and accuracy of the training set gradually decrease, and the loss and accuracy of the validation set first decrease and then increase, it is determined to be overfitting; if the loss and accuracy of the training set and validation set are both high and there is no obvious downward or upward trend, it is determined to be underfitting;

[0025] To address overfitting, increase the diversity of training data by adding noise, speed change, and pitch change techniques to generate more training set samples and improve the generalization ability of the model;

[0026] To address underfitting, increase the model complexity by increasing the number of model layers, neurons, and convolution kernels to improve the model's learning ability.

[0027] The calculation formula of the convolution kernel weight is:

[0028]

[0029] W new is the updated weight, W old is the weight before updating, the initial value is 0, η is the learning rate, the value is 0.001; is the gradient of the loss function with respect to the weight, calculated by the back propagation algorithm, and its value range is [-1,1];

[0030] The update formula of the convolutional layer bias term is:

[0031]

[0032] b new is the updated convolutional layer bias term, b old is the convolutional layer bias item before updating, and its initial value is 0;

[0033] The calculation formula of the weight matrix is:

[0034]

[0035] N new is the updated weight, N old is the weight before updating, the initial value is 0, η is the learning rate, here the value is 0.001; is the gradient of the loss function with respect to the weight, calculated by the back propagation algorithm, and its value range is [-1,1];

[0036] The update formula of the fully connected layer bias term is:

[0037]

[0038] a new is the updated convolutional layer bias term, a old is the convolutional layer bias item before updating, and its initial value is 0;

[0039] The update formula of Adam optimizer is:

[0040]

[0041]

[0042]

[0043]

[0044]

[0045] m t and v t are the exponentially weighted average of the gradient and the exponentially weighted average of the squared gradient, β1 and β2 are the decay rates, and ε is the smoothing term;

[0046] The calculation formula of the SGD optimizer is:

[0047]

[0048] D new is the updated weight, D old is the weight before updating, the initial value is 0, η is the learning rate, the value is 0.001; It is the gradient of the loss function with respect to the weight, calculated by the back-propagation algorithm, and its value range is [-1, 1].

[0049] As a further solution of the present invention, in step S2, establishing a user identity database and identifying and processing voice data include the following steps:

[0050] Through the speech recognition model, the user's speech data is input to extract acoustic features, including fundamental frequency, formant, spectrum, power spectrum density, cepstrum, Mel-frequency cepstrum coefficients, linear prediction cepstrum coefficients, pitch, intensity, duration, zero-crossing rate and short-time energy; based on the extracted features, it is compared with the user registration information in the identity database. If the comparison is successful, the user information, including personal usage areas and common vocabulary, is called to perform speech data recognition processing; if the comparison fails, the speech data is directly recognized and processed, and a new account is registered in the identity database to store the corresponding feature information;

[0051] The recognition and processing operation of the voice data is to convert the format according to the language category, generate a type of information, and send it to the translation component.

[0052] As a further solution of the present invention, in step S3:

[0053] Perform word segmentation on the text, cut it into words or phrases, remove punctuation and stop words in the text, and convert the text into the same format, such as lowercase; identify fixed phrases in the text and convert them into corresponding phrases in the target language; identify slang and informal expressions in the text and convert them into corresponding slang in the target language; analyze the components within the sentence, such as subject, predicate and object, and analyze the logical relationship between sentences, such as transition, cause and effect, progression and parallelism; generate text translation content in the target language;

[0054] Analyze the internal components and external logical relationships of the sentence to ensure the accuracy and fluency of the translated text while ensuring the grammatical correctness of the translated text.

[0055] As a further embodiment of the present invention:

[0056] Based on the speech text, the acoustic features of the speech, such as pitch, volume and speaking speed, are extracted. The emotion in the speech is identified through the emotional model, and the first-level emotional information is generated and annotated. The annotation content is to identify the intensity of the emotion, such as strong, medium and mild. Through speech synthesis, speech information with specific emotions is generated to complete the mapping of the emotional connotation of the original speech.

[0057] As a further solution of the present invention, the establishment of the emotion model includes the following steps:

[0058] Extract features from speech samples, including acoustic features of speech, such as pitch, volume, and speaking speed, and text features, including sentiment words and sentiment polarity; manually perform sentiment annotation on speech and text data, mark sentiment categories, train sentiment classifiers to identify sentiment categories in speech and text, and train sentiment generators to generate speech and text with specific emotions;

[0059] The performance of the model is evaluated, and the accuracy, recall rate and F1 score of the large emotional model are calculated. By comparing with the preset values, if any performance indicator is lower than the preset value, the voice sample is optimized, that is, text data enhancement is performed, including synonym replacement, random insertion, random deletion and random exchange, and voice data enhancement is performed, including noise addition, speed change and tone change. The operation of establishing the large voice model is repeated for the optimized voice sample until the performance indicators are greater than or equal to the preset values.

[0060] As a further solution of the present invention, collecting feedback data and further exploring emotional translation include the following steps:

[0061] In the user voice data, unrepeated recordings and texts are marked and recorded as feedback data; sentiment analysis tools, such as NLP models, are used to segment the feedback data into sentiment categories, such as segmenting "happiness" into "excitement", "satisfaction" or "pleasure", etc., to generate secondary sentiment information and add annotations to identify the intensity of the sentiment, such as strong, medium and mild;

[0062] The secondary sentiment information is made to cover the primary sentiment information in the sentiment translation step in S3.

[0063] Beneficial Effects

[0064] By simulating the speaker's tone, the present invention can significantly improve the quality of simultaneous interpretation, enhance emotional expression, improve information accuracy, enhance communication effects, and improve user experience; in terms of technical implementation, speech synthesis and emotion analysis technology can provide strong support to make tone simulation more natural and realistic.

[0065] By collecting speech data with different accents, different speakers, different background noises and speaking speeds, the present invention can train a more robust and accurate speech recognition model to adapt to diverse speech inputs; diverse speech data sets help the model learn a wider range of speech features, thereby improving its generalization ability in different scenarios and reducing its dependence on specific speakers or environments.

[0066] By selecting terms and expressions that are more suitable for specific fields according to the user's personal usage field, the present invention can provide more personalized speech recognition services, improve recognition accuracy, and continuously accumulate user data. The model can continue to learn and optimize, improve recognition performance, and enhance the user experience.

[0067] Through sentiment segmentation, the present invention can more accurately capture and express the sentiment in the original text, reducing the ambiguity and inaccuracy of sentiment translation; through the intensity of standard sentiment, the intensity of sentiment can be accurately conveyed, making the translation more delicate and realistic; at the same time, by using feedback data to train the model, the sentiment translation ability of the model can be continuously optimized, making it more adaptable to diverse emotional expressions.

[0068] The present invention provides more personalized translation services by matching relevant users, giving priority to the user's commonly used vocabulary, phrases and expressions in the communication field, significantly improving the accuracy of translation, user experience and system robustness; sharing learning results with other users through cloud services to form a continuously evolving translation community, promotes data sharing and collaboration, and has important commercial value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Embodiment 1

[0072] See also Figure 1 The present application provides a method for simultaneous interpretation of a call using headphones, comprising the following steps:

[0073] S1. Search the target language’s speech samples online, including recordings and corresponding speech text transcriptions, and the speech samples cover different accents, speakers, speaking speeds, and background noises to establish a speech dataset; obtain user speech data and perform preprocessing;

[0074] The process of establishing a speech dataset includes the following steps:

[0075] Search the target language’s speech texts online. The target language is determined based on market demand and application scenarios. Classify the speech texts according to the spoken language and create corresponding accent folders. Under each accent folder, classify them again according to the speaker and create speaker folders.

[0076] The speaker folder contains recordings of the speaker speaking at different speeds under different background noises and the corresponding recording text transcriptions to generate a speech dataset;

[0077] By collecting speech data with different accents, different speakers, different background noises and speaking speeds, a more robust and accurate speech recognition model can be trained to adapt to diverse speech inputs; diverse speech data sets help the model learn a wider range of speech features, thereby improving its generalization ability in different scenarios and reducing its dependence on specific speakers or environments;

[0078] Acquiring user voice data and preprocessing it includes the following steps:

[0079] The user's voice signal is collected through the microphone of the headset, and the collected voice signal is pre-processed, including noise reduction, gain control and spectrum analysis, and converted into a digital signal.

[0080] S2. Establish a speech recognition model based on a lightweight convolutional neural network. By extracting local features of the target language, using the CTC algorithm, obtain the user's voice data, realize language recognition, and generate a type of signal to send to the translation component; establish a user identity database;

[0081] A: Building a speech recognition model includes the following steps:

[0082] Receive a speech data set, convert the speech data into a spectrogram using a Mel-spectrogram, extract features of the target language, and use it as the input of the speech recognition model; perform data enhancement on the speech data, including adding noise, speed change, and pitch change operations, and perform segmentation processing on the enhanced speech data to generate a training set, a validation set, and a test set in a ratio of 7:1:2;

[0083] Use the training set data to train the model and adjust the model parameters, including convolution kernel weights, convolution layer bias terms, weight matrices, and fully connected layer bias terms, as well as Adam optimizer and SGD optimizer.

[0084] Use validation set data to adjust model parameters to prevent overfitting;

[0085] Use the test set data to evaluate the model performance and calculate the accuracy;

[0086] The accuracy is calculated as:

[0087]

[0088] Define the preset standard. If the accuracy is less than the preset standard, which is 90% here, the model is considered to have overfitting or underfitting problems.

[0089] Observe the loss curves, accuracy curves, and word error rate curves of the training set and validation set. If the loss and accuracy of the training set gradually decrease, and the loss and accuracy of the validation set first decrease and then increase, it is determined to be overfitting; if the loss and accuracy of the training set and validation set are both high and there is no obvious downward or upward trend, it is determined to be underfitting;

[0090] To address overfitting, increase the diversity of training data by adding noise, speed change, and pitch change techniques to generate more training set samples and improve the generalization ability of the model;

[0091] To address underfitting, increase the model complexity by increasing the number of layers, neurons, and convolution kernels in the model to improve the model's learning ability.

[0092] B: Through the speech recognition model, the user's speech data is input to extract acoustic features, including fundamental frequency, formant, spectrum, power spectrum density, cepstrum, Mel-frequency cepstrum coefficients, linear prediction cepstrum coefficients, pitch, intensity, duration, zero-crossing rate and short-time energy; based on the extracted features, it is compared with the user registration information in the identity database. If the comparison is successful, the user information, including personal usage areas and common vocabulary, is called to perform speech data recognition processing; if the comparison fails, the speech data is directly recognized and processed, and a new account is registered in the identity database to store the corresponding feature information;

[0093] According to the user's personal use field, we can choose terms and expressions that are more suitable for specific fields, provide more personalized speech recognition services, improve recognition accuracy, continuously accumulate user data, and continuously learn and optimize the model to improve recognition performance and enhance the user experience;

[0094] The recognition and processing operation of the speech data is to convert the format according to the language category, generate a type of information, and send it to the translation component;

[0095] Example: For users in the medical field, generate a vocabulary containing medical terms to improve the professionalism of the translation;

[0096] The calculation formula of the convolution kernel weight is:

[0097]

[0098] W new is the updated weight, W oldis the weight before updating, the initial value is 0, η is the learning rate, the value is 0.001; is the gradient of the loss function with respect to the weight, calculated by the back propagation algorithm, and its value range is [-1,1];

[0099] Example: Weight update:

[0100] Assume W old is 0.01, is 0.5, then W new =0.01-0.001*0.5=0.0095;

[0101] The update formula of the convolutional layer bias term is:

[0102]

[0103] b new is the updated convolutional layer bias term, b old is the convolutional layer bias item before updating, and its initial value is 0;

[0104] Example: Bias update: Assumption b old is 0, is 0.3, then b new =0-0.001*0.3=-0.0003;

[0105] The calculation formula of the weight matrix is:

[0106]

[0107] N new is the updated weight, N old is the weight before updating, the initial value is 0, η is the learning rate, here the value is 0.001; is the gradient of the loss function with respect to the weight, calculated by the back propagation algorithm, and its value range is [-1,1];

[0108] Example: Weight matrix update: Assume N old is 0.01, is 0.4, then N new =0.01-0.001*0.4=0.006;

[0109] The update formula of the fully connected layer bias term is:

[0110]

[0111] a new is the updated convolutional layer bias term, a oldis the convolutional layer bias item before updating, and its initial value is 0;

[0112] Example: Fully connected layer bias update: Assume a old is 0.03, is 0.2, then a new =0.03-0.001*0.2=0.0298;

[0113] The update formula of Adam optimizer is:

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] m t and v t are the exponentially weighted average of the gradient and the exponentially weighted average of the squared gradient, β1 and β2 are the decay rates, and ε is the smoothing term;

[0120] Example: β1 is 0.9, β2 is 0.999, ε is 1e-8, m t-1 and v t-1 The value of is 0. is 0.3, then m t =0+(1-0.9)*0.3=0.03, v t =0+(1-0.999)*0.3=0.0003; continue to substitute: but

[0121] The calculation formula of the SGD optimizer is:

[0122]

[0123] D new is the updated weight, D old is the weight before updating, the initial value is 0, η is the learning rate, the value is 0.001; is the gradient of the loss function with respect to the weight, calculated by the back propagation algorithm, and its value range is [-1,1];

[0124] Example: D old is 0, η is 0.001, is -0.5, then Dnew =0-0.001*(-0.5)=0.0005;

[0125] S3, through the translation component, analyzing a type of signal, determining the translation object language, including text translation and emotional translation, generating text translation data and emotional translation data respectively; confirming the user, making personalized adjustments to the text translation and emotional translation according to the user tag, generating a translation voice signal through speech synthesis, and sending it to the target terminal;

[0126] Text translation refers to the textualization of the original voice data content, including phrase connections and language slang connections, and includes the following steps:

[0127] Perform word segmentation on the text, cut it into words or phrases, remove punctuation and stop words in the text, and convert the text into the same format, such as lowercase; identify fixed phrases in the text and convert them into corresponding phrases in the target language; identify slang and informal expressions in the text and convert them into corresponding slang in the target language; analyze the components within the sentence, such as subject, predicate and object, and analyze the logical relationship between sentences, such as transition, cause and effect, progression and parallelism; generate text translation content in the target language;

[0128] Analyze the internal components and external logical relationships of the sentence to ensure the accuracy and fluency of the translated text while ensuring the grammatical correctness of the translated text;

[0129] Emotional translation refers to extracting features from the original voice data, simulating the user's tone, performing natural language processing, and establishing an emotional model to simulate the connotation, rhythm, intonation, and tone of the original voice. Emotional translation includes the following steps:

[0130] Based on the speech text, the acoustic features of the speech, such as pitch, volume and speaking speed, are extracted. The emotion in the speech is identified through the emotion model, and the first-level emotion information is generated and annotated. The annotation content is to identify the intensity of the emotion, such as strong, medium and mild. Through speech synthesis, speech information with specific emotions is generated to complete the mapping of the original speech connotation emotion.

[0131] The establishment of the emotion model includes the following steps:

[0132] Extract features from speech samples, including acoustic features of speech, such as pitch, volume, and speaking speed, and text features, including sentiment words and sentiment polarity; manually perform sentiment annotation on speech and text data, mark sentiment categories, train sentiment classifiers to identify sentiment categories in speech and text, and train sentiment generators to generate speech and text with specific emotions;

[0133] Evaluate the performance of the model, calculate the accuracy, recall rate and F1 score of the big emotion model, and compare with the preset values. If any performance indicator is lower than the preset value, optimize the speech sample, that is, perform text data enhancement, including synonym replacement, random insertion, random deletion and random exchange, and speech data enhancement, including noise addition, speed change and tone change. Repeat the operation of establishing the big speech model for the optimized speech sample until the performance indicators are greater than or equal to the preset value.

[0134] By simulating the speaker's tone, the quality of simultaneous interpretation can be significantly improved, emotional expression can be enhanced, information accuracy can be improved, communication effectiveness can be enhanced, and user experience can be improved. In terms of technical implementation, speech synthesis and sentiment analysis technology can provide strong support to make tone simulation more natural and realistic.

[0135] Embodiment 2

[0136] S4. Optimize model operations, including collecting feedback data, in-depth exploration of sentiment translation, and in-depth sentiment exploration of early sentiment translation and good performance cases based on the user identity database;

[0137] In the user voice data, unrepeated recordings and texts are marked and recorded as feedback data; sentiment analysis tools, such as NLP models, are used to segment the feedback data into sentiment categories, such as segmenting "happiness" into "excitement", "satisfaction" or "pleasure", etc., to generate secondary sentiment information and add annotations to identify the intensity of the sentiment, such as strong, medium and mild; the secondary sentiment information is made to cover the primary sentiment information in the sentiment translation step in S3 to optimize the sentiment translation step, and these data are used for model training to improve the sentiment translation capability of the model;

[0138] Through sentiment segmentation, we can more accurately capture and express the emotions in the original text, reducing the ambiguity and inaccuracy of sentiment translation; through the intensity of standard emotions, we can accurately convey the intensity of emotions, making the translation more delicate and realistic; at the same time, by using feedback data to train the model, we can continuously optimize the model's sentiment translation capabilities, making it more adaptable to diverse emotional expressions.

[0139] S5. Personalization: By analyzing usage data, we divide the communication fields according to the translation results and match the relevant users. We automatically record the common words, phrases and expressions of users in this field and give priority to this information in future translations. At the same time, we share the learning results with other users through cloud services to form a continuously evolving translation community.

[0140] By matching relevant users, more personalized translation services are provided, giving priority to the user's commonly used vocabulary, phrases and expressions in the communication field, significantly improving translation accuracy, user experience and system robustness; sharing learning results with other users through cloud services to form a continuously evolving translation community, which promotes data sharing and collaboration and has important commercial value and application prospects.

[0141] Embodiment 3

[0142] The specific implementation process of this embodiment includes the entire implementation process of the above two groups of embodiments.

[0143] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0144] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A method for simultaneous interpretation of a conversation using headphones, characterized in that: The following steps are involved: S1. Search the target language’s speech samples online, including recordings and corresponding speech text transcriptions, and the speech samples cover different accents, speakers, speaking speeds, and background noises to establish a speech dataset; obtain user speech data and perform preprocessing; S2. Establish a speech recognition model based on a lightweight convolutional neural network. By extracting local features of the target language, using the CTC algorithm, obtain the user's voice data, realize language recognition, and generate a type of signal to send to the translation component; establish a user identity database; S3, analyzing a type of signal through a translation component, determining the translation object language, including text translation and sentiment translation, and generating text translation data and sentiment translation data respectively; Confirm the user, make personalized adjustments to the text translation and emotional translation based on the user tag, generate the translated voice signal through speech synthesis, and send it to the target terminal; S4. Optimize model operations, including collecting feedback data, in-depth exploration of sentiment translation, and in-depth sentiment exploration of early sentiment translation and good performance cases based on the user identity database; S5, Personalized customization, through the analysis of usage data, divide the communication field according to the translation results, and match the relevant users, automatically record the common vocabulary, phrases and expressions of users in this field, and give priority to this information in the text translation of step S3, and share the learning results with other users through cloud services to form a continuously evolving translation community.

2. The method for simultaneous interpretation using headphones according to claim 1, characterized in that: In step S1, the process of establishing the speech data set includes the following steps: Search the target language’s speech texts online. The target language is determined based on market demand and application scenarios. Classify the speech texts according to the spoken language and create corresponding accent folders. Under each accent folder, classify them again according to the speaker and create speaker folders. The speaker folder contains recordings of the speaker speaking at different speeds under different background noises and the corresponding recording text transcriptions to generate a speech dataset.

3. The method for simultaneous interpretation using headphones according to claim 1, characterized in that: In step S1, the user's voice data is obtained and pre-processed, including the following steps: The user's voice signal is collected through the microphone of the headset, and the collected voice signal is pre-processed, including noise reduction, gain control and spectrum analysis, and converted into a digital signal.

4. The method for simultaneous interpretation using headphones according to claim 1, characterized in that: In step S2, establishing a speech recognition model includes the following steps: Receive a speech data set, convert the speech data into a spectrogram using a Mel-spectrogram, extract features of the target language, and use it as the input of the speech recognition model; perform data enhancement on the speech data, including adding noise, speed change, and pitch change operations, and perform segmentation processing on the enhanced speech data to generate a training set, a validation set, and a test set in a ratio of 7:1:2; Use the training set data to train the model and adjust the model parameters, including convolution kernel weights, convolution layer bias terms, weight matrices, and fully connected layer bias terms, as well as Adam optimizer and SGD optimizer. Use validation set data to adjust model parameters to prevent overfitting; Use the test set data to evaluate the model performance and calculate the accuracy; The accuracy is calculated as: Define the preset standard. If the accuracy is less than the preset standard, which is 90% here, the model is considered to have overfitting or underfitting problems. Observe the loss curves, accuracy curves, and word error rate curves of the training set and validation set. If the loss and accuracy of the training set gradually decrease, and the loss and accuracy of the validation set first decrease and then increase, it is determined to be overfitting; if the loss and accuracy of the training set and validation set are both high and there is no obvious downward or upward trend, it is determined to be underfitting; To address overfitting, increase the diversity of training data by adding noise, speed change, and pitch change techniques to generate more training set samples and improve the generalization ability of the model; To address underfitting, increase the model complexity by increasing the number of model layers, neurons, and convolution kernels to improve the model's learning ability. The calculation formula of the convolution kernel weight is: W new is the updated weight, W old is the weight before updating, the initial value is 0, η is the learning rate, the value is 0.001; is the gradient of the loss function with respect to the weight, calculated by the back propagation algorithm, and its value range is [-1,1]; The update formula of the convolutional layer bias term is: b new is the updated convolutional layer bias term, b old is the convolutional layer bias item before updating, and its initial value is 0; The calculation formula of the weight matrix is: N new is the updated weight, N old is the weight before updating, the initial value is 0, η is the learning rate, here the value is 0.001; is the gradient of the loss function with respect to the weight, calculated by the back propagation algorithm, and its value range is [-1,1]; The update formula of the fully connected layer bias term is: a new is the updated convolutional layer bias term, a old is the convolutional layer bias item before updating, and its initial value is 0; The update formula of Adam optimizer is: m t and v t are the exponentially weighted average of the gradient and the exponentially weighted average of the squared gradient, β1 and β2 are the decay rates, and ε is the smoothing term; The calculation formula of the SGD optimizer is: D new is the updated weight, D old is the weight before updating, the initial value is 0, η is the learning rate, the value is 0.001; It is the gradient of the loss function with respect to the weight, calculated by the back-propagation algorithm, and its value range is [-1, 1].

5. The method for simultaneous interpretation using headphones according to claim 1, characterized in that: In step S2, establishing a user identity database and processing the voice data recognition includes the following steps: Through the speech recognition model, the user's speech data is input to extract acoustic features, including fundamental frequency, formant, spectrum, power spectrum density, cepstrum, Mel-frequency cepstrum coefficients, linear prediction cepstrum coefficients, pitch, intensity, duration, zero-crossing rate and short-time energy; based on the extracted features, it is compared with the user registration information in the identity database. If the comparison is successful, the user information, including personal usage areas and common vocabulary, is called to perform speech data recognition processing; if the comparison fails, the speech data is directly recognized and processed, and a new account is registered in the identity database to store the corresponding feature information; The recognition and processing operation of the voice data is to convert the format according to the language category, generate a type of information, and send it to the translation component.

6. The method for simultaneous interpretation using headphones according to claim 1, characterized in that: In step S3, text translation includes the following steps: Perform word segmentation on the text, cut it into words or phrases, remove punctuation marks and stop words in the text, and convert the text into the same format; Identify fixed phrases in the text and convert them into corresponding phrases in the target language; Identify slang and informal expressions in the text and convert them into corresponding slang in the target language; Analyze the components within a sentence, analyze the logical relationship between sentences, and generate text translation content in the target language.

7. The method for simultaneous interpretation using headphones according to claim 6, characterized in that: In step S3, the emotion translation includes the following steps: Based on the speech text, the acoustic features of the speech are extracted, and the emotions in the speech are identified through the emotional model. The first-level emotional information is generated and annotated. The annotated content is the intensity of the identified emotion. Through speech synthesis, speech information with specific emotions is generated to complete the mapping of the emotional connotation of the original speech.

8. The method for simultaneous interpretation using headphones according to claim 7, characterized in that: The establishment of the emotion model includes the following steps: Extract features from speech samples, including acoustic features of speech, such as pitch, volume, and speaking speed, and text features, including sentiment words and sentiment polarity; manually perform sentiment annotation on speech and text data, mark sentiment categories, train sentiment classifiers to identify sentiment categories in speech and text, and train sentiment generators to generate speech and text with specific emotions; The performance of the model is evaluated, and the accuracy, recall rate and F1 score of the large emotional model are calculated. By comparing with the preset values, if any performance indicator is lower than the preset value, the voice sample is optimized, that is, text data enhancement is performed, including synonym replacement, random insertion, random deletion and random exchange, and voice data enhancement is performed, including noise addition, speed change and tone change. The operation of establishing the large voice model is repeated for the optimized voice sample until the performance indicators are greater than or equal to the preset values.

9. The method for simultaneous interpretation using headphones according to claim 1, characterized in that: In step S4, feedback data is collected and the in-depth exploration of sentiment translation includes the following steps: Among the user voice data, the unrepeated recordings and texts are marked and recorded as feedback data; the sentiment analysis tool is used to segment the feedback data into sentiment categories, generate secondary sentiment information, and add annotations, the annotation content of which is to identify the intensity of the sentiment; The secondary sentiment information is made to cover the primary sentiment information in the sentiment translation step in S3.

Images