Cross-language voice communication system integrating AI voice cloning and real-time translation

Abstract

The invention discloses a cross-language voice communication system fusing AI voice cloning and real-time translation, and relates to the technical field of voice processing, and the system comprises the steps: obtaining a first input voice of a first client, and carrying out the language type recognition to obtain a first input voice language type; when the language type of the first input voice is different from the language type identifier of the second client, performing semantic recognition to obtain a character recognition result of the first input voice; translating the first input voice character recognition result according to the second client language type identifier to obtain a corresponding voice character recognition result; obtaining physical feature information of the first input voice; and performing voice cloning according to the physical feature information and the character recognition result of the second input voice to obtain the second input voice, and sharing the second input voice to the second client. According to the invention, the technical problems of lack of personalized voice feature reservation and insufficient real-time performance in cross-language communication in the prior art are solved, and the technical effect of cross-language and personalized real-time voice communication is realized.

Description

Technical Field

[0001] The present invention relates to the field of voice processing technology, and in particular to a cross-language voice call system integrating AI voice cloning and real-time translation. Background Art

[0002] With the rapid growth of global communication and cross-language communication needs, cross-language voice call technology has been widely used in business, social, education and other fields. However, as users' requirements for communication quality and naturalness continue to increase, traditional cross-language voice call technology faces increasing challenges. Traditional systems use simple text translation and speech synthesis methods, which lack the retention of voice personality characteristics, resulting in a lack of personalized voice feature retention and insufficient real-time performance in cross-language calls. Summary of the invention

[0003] The present application provides a cross-language voice call system integrating AI voice cloning and real-time translation, which is used to solve the technical problems of the prior art in lack of personalized voice feature retention and insufficient real-time performance in cross-language calls.

[0004] In view of the above problems, the present application provides a cross-language voice call system that integrates AI voice cloning and real-time translation.

[0005] In a first aspect of the present application, a cross-language voice call system integrating AI voice cloning and real-time translation is provided, the system comprising:

[0006] A voice acquisition module, the voice acquisition module is used to obtain a first input voice of the first client when the first client and the second client are talking; a language recognition module, the language recognition module performs language type recognition on the first input voice to obtain the language type of the first input voice; a semantic recognition module, the semantic recognition module performs semantic recognition on the first input voice when the language type of the first input voice is different from the language type identifier of the second client to obtain a first input voice text recognition result; a translation module, the translation module translates the first input voice text recognition result according to the language type identifier of the second client to obtain a second input voice text recognition result; a physical feature information acquisition module, the physical feature information acquisition module obtains physical feature information of the first input voice; a voice cloning module, the voice cloning module performs voice cloning according to the physical feature information and the second input voice text recognition result to obtain a second input voice; and a voice sharing module, the voice sharing module shares the second input voice to the second client.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] In this application, when a first client and a second client are talking, a first input voice of the first client is obtained; the language type of the first input voice is identified to obtain the language type of the first input voice; when the language type of the first input voice is different from the language type identifier of the second client, semantic recognition is performed on the first input voice to obtain the first input voice text recognition result; the first input voice text recognition result is translated according to the language type identifier of the second client to obtain the second input voice text recognition result; the physical feature information of the first input voice is obtained; voice cloning is performed according to the physical feature information and the second input voice text recognition result to obtain the second input voice; and the second input voice is shared to the second client. The present invention solves the technical problems of the prior art in lack of personalized voice feature retention and insufficient real-time performance in cross-language calls. Through the steps of language recognition, semantic recognition, translation, physical feature extraction and voice cloning, the voice content of the first client is converted into voice that conforms to the language and sound characteristics of the second client, and then transmitted to the second client in real time through the network protocol, thereby achieving the technical effect of cross-language and personalized real-time voice calls. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of the structure of a cross-language voice call system integrating AI voice cloning and real-time translation provided in an embodiment of the present application;

[0011] Figure 2 This is a schematic diagram of the structure of an exemplary electronic device of the present application.

[0012] Explanation of the reference numerals: speech acquisition module 11, language recognition module 12, semantic recognition module 13, translation module 14, physical feature information acquisition module 15, speech cloning module 16, speech sharing module 17, bus 300, receiver 301, processor 302, transmitter 303, memory 304, bus interface 305. DETAILED DESCRIPTION

[0013] This application provides a cross-language voice call system that integrates AI voice cloning and real-time translation, aiming to solve the technical problems of the prior art in cross-language calls that lack personalized voice feature retention and insufficient real-time performance. Through the steps of language recognition, semantic recognition, translation, physical feature extraction and voice cloning, the voice content of the first client is converted into voice that conforms to the language and sound characteristics of the second client, and then transmitted to the second client in real time through the network protocol, thereby achieving the technical effect of cross-language and personalized real-time voice calls.

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0015] It should be noted that any variations of the terms "include" and "have" are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0016] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides a cross-language voice call system integrating AI voice cloning and real-time translation, the system comprising:

[0017] The voice acquisition module 11 is used to obtain a first input voice of the first client when the first client and the second client are talking.

[0018] In an embodiment of the present application, firstly, the authorization of the first client user is requested before the voice acquisition begins, and the user needs to explicitly agree that the system collects and processes his voice information. After obtaining the user's authorization, the voice acquisition program is started to establish a real-time voice connection with the first client. The voice acquisition module receives the user's voice data in real time through the microphone of the device, and performs basic preprocessing based on the set audio format, sampling rate and other parameters. The voice acquisition module uses streaming processing to convert the user's voice input into a continuous audio data stream. Through this process, the first input voice of the first client is obtained.

[0019] The language recognition module 12 performs language type recognition on the first input speech to obtain the language type of the first input speech.

[0020] In the embodiment of the present application, the language type identifier of the first client is first obtained, and the single-channel classification accuracy of the first language type is matched according to the identifier, so as to configure a suitable first language type recognition model. Then, the single-channel classification accuracy of the second language type is matched according to the language type identifier of the second client, and the second language type recognition model is configured.

[0021] After the two language models are configured, the model with more single channels in the first language type recognition model and the second language type recognition model is extracted, and the model is used to perform language type recognition on the first input speech, and finally the language type of the first input speech is obtained.

[0022] Furthermore, in the system provided in the embodiment of the application, the language recognition module 12 is also used for:

[0023] Obtain a first client language type identifier; match the first language type single-channel classification accuracy according to the first client language type identifier, and configure a first language type recognition model; match the second language type single-channel classification accuracy according to the second client language type identifier, and configure a second language type recognition model; extract the model with more single channels of the first language type recognition model and the second language type recognition model to perform speech type recognition on the first input speech, and obtain the first input speech language type.

[0024] In the embodiment of the present application, the language type identifier of the first client is first obtained from the user information database, which is the language preference information uploaded by the user when registering, to ensure that the system can recognize the user's commonly used language.

[0025] Next, according to the language type identifier of the first client, the corresponding historical single-channel classification record data is extracted, and the percentage of accurate classifications in the classification records is counted, which is the first language type single-channel classification accuracy. If the classification accuracy is less than or equal to the preset accuracy threshold, multiple language classification single channels are retrieved and connected in parallel to generate the first language type recognition model.

[0026] At the same time, for the language type identifier of the second client, configure the second language type recognition model. This process is similar to the configuration of the first language type recognition model. According to the language type identifier of the second client, obtain and count its historical single-channel classification data, and calculate the second language type single-channel classification accuracy. If the accuracy is lower than the threshold, call multiple single channels for parallel connection to generate a second language type recognition model suitable for the second client.

[0027] After the first and second language type recognition models are configured, the single channel numbers of the two are compared, and the model with more single channels is selected for actual language recognition. This model has higher recognition ability and accuracy. The model receives the first input speech and performs language type recognition on it, and finally outputs the language type of the first input speech.

[0028] Furthermore, the system provided in the application embodiment is also used for:

[0029] When the language type of the first input voice is identical to the language type identifier of the second client, the first input voice is directly shared with the second client.

[0030] In an embodiment of the present application, the language type of the first input voice is compared with the language type identifier of the second client. If the language types are the same, for example, both are English or French, it means that the two languages ​​are consistent, and steps such as translation are skipped, and the first input voice is shared to the second client.

[0031] Furthermore, in the system provided in the embodiment of the application, the language recognition module 12 is also used for:

[0032] According to the first client language type identifier, historical single-channel classification record data is counted; the proportion of accurate classification of the historical single-channel classification record data is counted, and is set as the first language type single-channel classification accuracy; when the first language type single-channel classification accuracy is less than or equal to the classification accuracy threshold, N language classification single channels are retrieved for parallel connection to generate the first language type recognition model, N ≥ [(1-first language type single-channel classification accuracy) * single-channel integration quantity threshold], where N is an integer.

[0033] In the embodiment of the present application, the user's commonly used language type is first confirmed by reading the first client language type identifier in the user information database. The identifier is specified during user registration, such as "English" or "Spanish". This process is implemented through an API or data query method to ensure that the system can identify the user's language preference.

[0034] Then, according to the language type identification obtained, enter the historical single-channel classification record database and extract the single-channel classification record data related to the language. These data come from the classification information accumulated in the historical training tasks of the language type, and record the results of the single-channel classification tasks for the language at different time points. Each record includes whether the single-channel classifier successfully recognizes the language sample. Single-channel classification means that the system uses only one classifier to handle speech classification tasks. These single-channel classifiers use different algorithms in historical training, such as support vector machines (SVM), convolutional neural networks (CNN), long short-term memory networks (LSTM), etc.

[0035] Then, from the extracted single-channel classification record data, the percentage of accurate classifications in historical tasks under this language type is counted, that is, the ratio of the number of successful classifications to the total number of classifications in the single-channel classification task of this language type. When calculating, the single-channel classification accuracy of the first language type is obtained by "successful classification times / total classification times", which indicates the recognition effect of the model on this language in the single-channel task.

[0036] The accuracy is then judged by a threshold to determine whether the model needs to be further optimized. If the single-channel classification accuracy of the first language type is lower than or equal to the set classification accuracy threshold, such as 90%, the model is further optimized to ensure that the recognition accuracy meets the requirements.

[0037] When the classification accuracy is not up to standard, multiple language classification single channels are called for parallel connection to generate an optimized integrated model. This method is implemented through multi-channel integration technology, where multiple single-channel classifiers process one input in parallel to improve the overall classification accuracy. Specifically, the required number of single channels N is calculated, and the number of single channels N is calculated by the formula N≥[(1-first language type single channel classification accuracy)*single channel integration number threshold]. The "single channel integration number threshold" in the formula is set by technical experts. Through this step, the number of channels that need to be called is determined. Among them, the number of single channels N is the smallest positive integer greater than or equal to [(1-first language type single channel classification accuracy)*single channel integration number threshold]. In order to select the optimal single-channel classifier, the single channels with the highest recognition accuracy in the top N among all single-channel classifiers are preferentially selected to ensure that the best performing single-channel classifier is used in the integration process.

[0038] Finally, the first language type recognition model is configured through parallel connection and ensemble learning. The generated first language type recognition model is composed of multiple single-channel classifiers in parallel, and each single-channel classifier independently performs language recognition on the input speech sample in the model. The final language type determination is achieved through majority voting (also known as ensemble voting). Specifically, when the first input speech is passed into the model, each single-channel classifier will give a recognition result, and the language type that appears most frequently in all single-channel classifiers is counted as the final result output.

[0039] The semantic recognition module 13 performs semantic recognition on the first input speech when the first input speech language type is different from the second client language type identifier to obtain a first input speech text recognition result.

[0040] In an embodiment of the present application, when the language type of the first input voice is different from the language type identifier of the second client, semantic recognition is performed on the first input voice. Specifically, when the language type of the first input voice is different from the language type identifier of the second client, semantic recognition is performed through an automatic speech recognition (ASR) model to obtain a text recognition result of the first input voice. In this process, the ASR model is responsible for converting voice data into text. The ASR model is pre-trained, using a convolutional neural network algorithm during the training process, and is trained with a large amount of speech-text pairing data. These data include voice samples in various languages ​​and accents, with corresponding text transcriptions to ensure that the model can learn the mapping relationship between different voice features and text. Through this supervised learning, the ASR model can recognize the semantics of the input voice and generate corresponding text.

[0041] The first input speech is converted into text through recognition of the ASR model to obtain a text recognition result of the first input speech.

[0042] The translation module 14 translates the first input voice and text recognition result according to the second client language type identifier to obtain a second input voice and text recognition result.

[0043] In an embodiment of the present application, the second client language type identifier is first read, such as "French" or "Spanish". Then the text recognition result of the first input speech is input into the neural machine translation model (NMT) for translation. The NMT model is used to process the translation task from the source language to the target language. It is trained with bilingual parallel corpus data during pre-training. These corpora contain paired source and target language texts to help the model learn how to convert between different languages. During the training process, the model uses a supervised learning algorithm to improve the quality and accuracy of the translation by gradually adjusting the parameters.

[0044] When performing translation, the NMT model first processes the source language text through an encoder. The encoder converts the source language text into semantic vector representations, which represent the grammatical and semantic information of the source text. Next, the semantic vectors generated by the encoder are passed to the decoder, which gradually decodes the semantic vectors into the target language text according to the target language type identifier of the second client. When generating the target text, the decoder ensures that the translation content is both accurate and in line with the language habits of the target language based on the vocabulary and grammar rules of the target language. Finally, the second input speech and text recognition result is generated through the above process.

[0045] A physical feature information obtaining module 15, wherein the physical feature information obtaining module 15 obtains the physical feature information of the first input speech.

[0046] In an embodiment of the present application, the first input speech is first subjected to frequency analysis, and the time domain signal is converted into a frequency domain signal using a fast Fourier transform (FFT) to extract the pitch feature. The pitch is determined by the fundamental frequency of the speech, reflecting the high and low changes in the tone of the speech. Through FFT, the various frequency components in the speech signal are decomposed and the position of the fundamental frequency is identified. In speech processing, the speech signal is divided into multiple short time segments, and FFT calculation is performed on each segment to locate the peak of the main frequency. The change in the frequency peak reflects the dynamics of the speaker's pitch. Next, the short-time Fourier transform (STFT) is used to extract the timbre features. STFT divides the speech signal into multiple short time segments, performs Fourier transform on each segment separately, and generates the spectrum information at each moment. By analyzing these spectral features, the unique energy distribution pattern of the timbre is identified, so that it can restore the speaker's voice texture during speech cloning.

[0047] Then the overall energy of the speech signal is calculated to extract the sound intensity feature. Sound intensity represents the loudness of speech and is a manifestation of the strength of sound. The sound intensity value is obtained by calculating the signal energy. The specific method is to divide the speech signal into multiple short time segments and calculate the average energy of the signal in each segment. The higher the energy in the segment, the greater the sound intensity, which reflects the loudness of the sound.

[0048] Finally, the duration feature is extracted by measuring the overall duration of the speech signal. The duration indicates the duration of speech and reveals the speaker's speaking speed and rhythm. The start and end times of the speech signal are marked to determine the total duration of the speech, thereby capturing the speaker's natural speaking rhythm.

[0049] Through the above steps, the physical feature information of the first input speech is obtained, including pitch, timbre, intensity and duration.

[0050] The voice cloning module 16 performs voice cloning according to the physical feature information and the second input voice and text recognition result to obtain the second input voice.

[0051] In the embodiment of the present application, firstly, the adapted voice cloning model is activated according to the language type identifier of the second client. Then, the physical feature information is converted into a physical feature multi-element array. This array contains the characteristics of the first input voice, such as timbre, pitch, intensity and length. Then, the physical feature multi-element array is input into the voice cloning model together with the text recognition result of the second input voice. The voice cloning model generates voice content using the text recognition result, adjusts the sound quality characteristics of the voice according to the physical feature array, and outputs the second input voice.

[0052] Furthermore, in the system provided by the embodiment of the application, the voice cloning module 16 is also used for:

[0053] According to the second client language type identifier, a voice cloning model is activated; according to the physical feature information, a physical feature multi-element array is constructed; and the physical feature multi-element array and the second input voice and text recognition result are input into the voice cloning model to obtain the second input voice.

[0054] In the embodiment of the present application, the corresponding voice cloning model is first activated according to the language type identifier of the second client. Each voice cloning model is specially trained to adapt to the voice characteristics of different target languages.

[0055] Next, a physical feature multi-element array is constructed based on the previously extracted physical feature information, including timbre, pitch, intensity, and duration. This multi-element array contains key parameters that describe the speaker's voice quality characteristics. Specifically, timbre data provides a unique sound texture for the cloned voice, pitch data ensures the accuracy of the pitch, intensity reflects the loudness characteristics of the voice, and duration determines the rhythm and speed of the voice. Together, these features constitute the speaker's unique voice characteristics, ensuring that the personalized style of the original voice is preserved in the voice cloning.

[0056] The physical feature multi-element array is then input into the activated speech cloning model together with the second input speech text recognition result. The speech cloning model generates speech content from the input text and adjusts the sound quality characteristics of the speech according to the physical feature multi-element array. The model generates speech content that conforms to the target language based on the text recognition results, while the physical feature multi-element array gives the generated speech the sound quality characteristics of the original speaker, making the cloned speech close to the original speech in timbre, pitch and rhythm.

[0057] Finally, after the above steps, a second input voice is generated. This voice is not only faithful to the text recognition result of the second input voice in terms of content, but also retains the personalized sound quality characteristics of the first input voice, achieving a natural, coherent voice cloning effect with the style of the original speaker.

[0058] Furthermore, in the system provided by the embodiment of the application, the voice cloning module 16 is also used for:

[0059] A semantic feature extraction model and a speech physical feature extraction model are obtained; first text data are collected according to a preset language type identifier; a first physical feature value assignment multi-array is randomly configured according to a speech physical feature value interval; a full convolutional neural network is unsupervisedly trained with the first text data and the first physical feature value assignment multi-array to obtain a first cloned speech; feature extraction is performed on the first cloned speech according to the semantic feature extraction model to obtain a first cloned text data; feature extraction is performed on the first cloned speech according to the speech physical feature extraction model to obtain a first cloned physical feature multi-array; when a first deviation between the first text data and the first cloned text data is less than or equal to a first deviation threshold, and a second deviation between the first physical feature value assignment multi-array and the first cloned physical feature multi-array is less than or equal to a second deviation threshold, it is considered that the training loss meets the convergence requirement; when at least 98% of the training losses meet the convergence requirement after Q consecutive trainings, the full convolutional neural network is set as the speech cloning model, Q≥10000.

[0060] In an embodiment of the present application, a semantic feature extraction model and a speech physical feature extraction model are first loaded. The semantic feature extraction model adopts a Transformer-based natural language processing model. These models are pre-trained with large-scale text data and can accurately capture the semantic information in the speech. Through the semantic feature extraction model, the generated speech semantic content can be compared with the original text to ensure that the speech is consistent with the original content in expression. The speech physical feature extraction model includes a timbre feature extraction model, a pitch feature extraction model, a sound intensity feature extraction model, and a sound length feature extraction model.

[0061] Then, the first text data of the corresponding language is collected according to the preset language type identifier. This data is extracted from a public bilingual corpus or a preset text database, and the corpus contains a large amount of text data. At the same time, a first physical feature value assignment multivariate array is randomly generated according to the standard value range of timbre, pitch, intensity and length. The array contains the initial value of the sound quality feature, so as to provide a preliminary sound quality benchmark for the model in the early stage of training.

[0062] The first text data and the first physical feature value multi-array are then used as input data and input into the Fully Convolutional Network (FCN) for unsupervised training. During the unsupervised training process, the FCN gradually learns the mapping relationship between text and sound quality through the convolution layer to generate an output speech signal. At this point, the model generates a preliminary speech output, namely the first cloned speech, as the initial version of the training speech.

[0063] After the generated first cloned speech is processed by the semantic feature extraction model, the semantic information is extracted to generate the first cloned text data. The first text data and the first cloned text data are respectively encoded to obtain a first encoding result and a second encoding result. The difference between the two encoding results is calculated by the Hamming distance and is set as the first deviation. At the same time, the first cloned speech is input into the speech physical feature extraction model to extract the timbre, pitch, intensity and length features to generate a first cloned physical feature multi-array. The array is then compared with the original first physical feature assignment multi-array to calculate the physical feature difference between the two. The Euclidean distance method is used to calculate, and the second deviation is obtained by calculation.

[0064] During the training process, if the semantic deviation (first deviation) is less than or equal to the first deviation threshold, and the physical feature deviation (second deviation) is less than or equal to the second deviation threshold, it is considered that the training of the current model has reached the convergence requirement. Among them, the first deviation threshold and the second deviation threshold are pre-set by technical experts. To ensure the stability of model convergence, when at least 98% of the training losses meet the convergence requirements in Q consecutive trainings, the fully convolutional neural network model is finally set as a voice cloning model to meet the deployment requirements.

[0065] Through the above steps, a voice cloning model is obtained.

[0066] Furthermore, the system provided in the application embodiment is also used for:

[0067] The speech physical feature extraction model includes a timbre feature extraction model, a pitch feature extraction model, a sound intensity feature extraction model and a sound length feature extraction model.

[0068] In an embodiment of the present application, the speech physical feature extraction model includes a timbre feature extraction model, a pitch feature extraction model, a sound intensity feature extraction model and a sound length feature extraction model.

[0069] The timbre feature extraction model is used to analyze the timbre information of speech. To extract the timbre information, the timbre feature extraction model uses a convolutional neural network to process the spectrogram of the speech signal. Specifically, the speech signal is first converted into a mel-spectrogram through mel-spectrogram analysis. This spectrum representation method is more in line with the human ear's perception of frequency. The converted spectrogram is input into the CNN model for multi-layer convolution processing. The CNN model applies filters in different convolution layers to extract the frequency and energy distribution characteristics of the audio, thereby capturing the core characteristics of the timbre.

[0070] The pitch feature extraction model is used to extract the pitch information of speech. The pitch feature extraction model uses the short-time Fourier transform (STFT) technology for analysis. Specifically, the speech signal is first divided into multiple short time segments (for example, 20 to 30 milliseconds). STFT is performed on each time segment and converted into a spectrogram. In each spectrum segment, the position of the frequency peak is detected, and this peak is the fundamental frequency of the segment. The fundamental frequency information records the speaker's pitch changes, ensuring that the original pitch characteristics can be faithfully reproduced when cloning speech.

[0071] The sound intensity feature extraction model is used to measure the loudness characteristics of speech. Sound intensity reflects the strength of speech and is a key feature of emotional expression. The sound intensity feature extraction model measures the energy level of speech signals through the signal energy calculation method. Specifically, the speech signal is first divided into short time segments (usually 10 to 20 milliseconds), and then the energy of each segment is calculated, and the sum of the squares is taken as the energy value of the segment. By averaging the energy of each segment and recording its changes over time, the dynamic expression of sound intensity is obtained. These sound intensity data help the model reproduce the emotional expression of the original speech during speech cloning, such as the softness or high pitch of the tone, so that the generated speech can accurately convey the speaker's tone in terms of volume.

[0072] The sound length feature extraction model is used to calculate the sound length. Sound length is the duration of speech, which affects the speaking speed and rhythm, and is an important factor in speech fluency. The sound length feature extraction model uses a time calculation method to measure the sound length, and determines the overall duration by calculating the start and end time of the speech signal. The non-silent segments of the speech signal are detected, and the start and end points of the speech segments are marked to calculate the total duration of the speech.

[0073] Furthermore, the system provided in the application embodiment is also used for:

[0074] Encode the first text data to obtain a first encoding result, encode the first cloned text data to obtain a second encoding result, calculate the Hamming distance between the first encoding result and the second encoding result, and set it as the first deviation; perform Euclidean distance analysis on the first physical feature assignment multi-array and the first cloned physical feature multi-array to obtain the second deviation.

[0075] In an embodiment of the present application, first, the text content of the original speech, that is, the first text data, is encoded. In order to convert the text content into a numerical representation, a word vector model, such as Word2Vec or BERT, is used. These word vector models can convert each word or character into a numerical vector by pre-training a large number of corpora to capture the semantic relationship between words. In this way, the first text data is converted into a first encoding result, and this numerical vector represents the semantic content of the original speech.

[0076] Next, the text content generated by the cloned speech, i.e., the first cloned text data, is encoded using the same encoding method to generate a second encoding result, which represents the semantic content of the cloned speech. In order to measure the semantic difference between the two, the Hamming distance between the first encoding result and the second encoding result is calculated. The Hamming distance is a classic calculation method for comparing the difference between two equal-length codes. Specifically, the two codes are aligned, each bit is compared bit by bit, and the number of different bits is counted. For example, if a bit is different in the two codes, the difference in the bit is counted in the Hamming distance. The total number of all different bits is the value of the Hamming distance. The smaller the Hamming distance, the closer the cloned speech is to the original speech in terms of semantic content. Through this process, the Hamming distance between the first encoding result and the second encoding result is obtained and defined as the first deviation.

[0077] In the physical feature consistency assessment, the physical features of the original and cloned voices are extracted by relying on the speech physical feature extraction model. Specifically, the timbre, pitch, intensity and length features are extracted from the original voice, and the first physical feature assignment multi-element array is constructed through these features. This step uses the timbre feature extraction model, the pitch feature extraction model, the intensity feature extraction model and the length feature extraction model. Through this process, the first physical feature assignment multi-element array is obtained.

[0078] Then, the corresponding physical features are extracted from the cloned voice through the voice physical feature extraction model, and the first cloned physical feature multi-element array is constructed, whose structure is consistent with the original physical feature array. This array represents the sound quality features of the cloned voice and is ready to be compared with the original feature array.

[0079] To evaluate the similarity of two physical feature arrays, the Euclidean distance between them is calculated, the values ​​of each corresponding position in the two multivariate arrays are compared one by one, the square of the difference between each pair of values ​​is calculated, all square values ​​are added, and the square root of the sum is taken to obtain the second deviation. For example, the physical feature array contains four dimensions: timbre, pitch, intensity, and length. The difference of the corresponding values ​​of each dimension is calculated, the square of the difference is added, and the square root is taken. The smaller the final Euclidean distance value, the higher the similarity of the two physical feature vectors. Through the above process, the second deviation is obtained.

[0080] The voice sharing module 17 shares the second input voice with the second client.

[0081] In the embodiment of the present application, the voice sharing module is responsible for sending the generated second input voice to the second client. The voice sharing module uses an appropriate network transmission protocol, such as HTTP or RTP, to transmit the second input voice to the second client to achieve real-time translation of cross-language voice calls.

[0082] In the embodiments of the present application, in summary, the embodiments of the present application have at least the following technical effects:

[0083] In this application, when a first client and a second client are talking, a first input voice of the first client is obtained; the language type of the first input voice is identified to obtain the language type of the first input voice; when the language type of the first input voice is different from the language type identifier of the second client, semantic recognition is performed on the first input voice to obtain the first input voice text recognition result; the first input voice text recognition result is translated according to the language type identifier of the second client to obtain the second input voice text recognition result; the physical feature information of the first input voice is obtained; voice cloning is performed according to the physical feature information and the second input voice text recognition result to obtain the second input voice; and the second input voice is shared to the second client. The present invention solves the technical problems of the prior art in lack of personalized voice feature retention and insufficient real-time performance in cross-language calls. Through the steps of language recognition, semantic recognition, translation, physical feature extraction and voice cloning, the voice content of the first client is converted into voice that conforms to the language and sound characteristics of the second client, and then transmitted to the second client in real time through the network protocol, thereby achieving the technical effect of cross-language and personalized real-time voice calls.

[0084] Embodiment 2. Based on the inventive concept of a cross-language voice call system integrating AI voice cloning and real-time translation in the aforementioned embodiment, the present application also provides an electronic device, comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the system described in any one of the aforementioned embodiments 1.

[0085] Figure 2 This is a schematic diagram of the structure of an exemplary electronic device of this application. Figure 2In the embodiment, the bus architecture is represented by bus 300, which may include any number of interconnected buses and bridges, and bus 300 connects various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, namely a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 302 is responsible for managing bus 300 and general processing, while memory 304 may be used to store data used by processor 302 when performing operations.

[0086] Embodiment 3, based on the cross-language voice call system integrating AI voice cloning and real-time translation in the above embodiment, with the same inventive concept, the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the system described in any one of the above embodiment 1 when executed.

[0087] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0088] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0089] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A cross-language voice call system integrating AI voice cloning and real-time translation, characterized by: include: A voice acquisition module, the voice acquisition module is used to obtain a first input voice of the first client when the first client and the second client are talking; a language recognition module, wherein the language recognition module performs language type recognition on the first input speech to obtain a language type of the first input speech; A semantic recognition module, wherein when the language type of the first input voice is different from the second client language type identifier, the semantic recognition module performs semantic recognition on the first input voice to obtain a first input voice text recognition result; A translation module, wherein the translation module translates the first input voice and text recognition result according to the second client language type identifier to obtain a second input voice and text recognition result; A physical feature information acquisition module, wherein the physical feature information acquisition module acquires physical feature information of the first input voice; A voice cloning module, which performs voice cloning according to the physical feature information and the second input voice and text recognition result to obtain a second input voice; A voice sharing module is used to share the second input voice with the second client.

2. The system according to claim 1, characterized in that Performing language type recognition on the first input speech to obtain the language type of the first input speech includes: Obtaining a first client language type identifier; According to the first client language type identifier, matching the first language type single-channel classification accuracy rate, configuring a first language type recognition model; According to the second client language type identifier, matching the second language type single-channel classification accuracy rate, configuring a second language type recognition model; A model with more single channels of the first language type recognition model and the second language type recognition model is extracted to perform speech type recognition on the first input speech to obtain the language type of the first input speech.

3. The system according to claim 1, characterized in that Also includes: When the language type of the first input voice is identical to the language type identifier of the second client, the first input voice is directly shared with the second client.

4. The system according to claim 2, characterized in that According to the first client language type identifier, matching the first language type single-channel classification accuracy, configuring the first language type recognition model includes: According to the first client language type identifier, historical single-channel classification record data is counted; Counting the percentage of accurate classification of the historical single-channel classification record data, and setting it as the first language type single-channel classification accuracy rate; When the classification accuracy of the first language type single channel is less than or equal to the classification accuracy threshold, N language classification single channels are retrieved for parallel connection to generate the first language type recognition model, N ≥ [(1-first language type single channel classification accuracy) * single channel integration number threshold], where N is an integer.

5. The system according to claim 1, wherein: Performing voice cloning according to the physical feature information and the second input voice and text recognition result to obtain the second input voice includes: activating a voice cloning model according to the second client language type identifier; Constructing a physical feature multivariate array according to the physical feature information; The physical feature multi-element array and the second input voice and text recognition result are input into the voice cloning model to obtain the second input voice.

6. The system according to claim 5, characterized in that The voice cloning model training step comprises: Obtaining a semantic feature extraction model and a speech physical feature extraction model; Collecting first text data according to a preset language type identifier; Randomly configure a first physical feature value assignment multi-element array according to a speech physical feature value interval; Performing unsupervised training on a fully convolutional neural network using the first text data and the first physical feature value assignment multi-element array to obtain a first cloned voice; Performing feature extraction on the first cloned speech according to the semantic feature extraction model to obtain first cloned text data; Extracting features from the first cloned speech according to the speech physical feature extraction model to obtain a first cloned physical feature multi-element array; When a first deviation between the first text data and the first cloned text data is less than or equal to a first deviation threshold, and a second deviation between the first physical feature value multi-array and the first cloned physical feature multi-array is less than or equal to a second deviation threshold, the training loss is deemed to meet the convergence requirement; When at least 98% of the training losses meet the convergence requirement after Q consecutive trainings, the fully convolutional neural network is set as the voice cloning model, and Q≥10000.

7. The system according to claim 6, characterized in that The speech physical feature extraction model includes a timbre feature extraction model, a pitch feature extraction model, a sound intensity feature extraction model and a sound length feature extraction model.

8. The system according to claim 6, characterized in that include: Encode the first text data to obtain a first encoding result, encode the first cloned text data to obtain a second encoding result, calculate the Hamming distance between the first encoding result and the second encoding result, and set it as the first deviation; Perform Euclidean distance analysis on the first physical feature value assignment multi-array and the first clone physical feature multi-array to obtain the second deviation.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor is used to implement the cross-language voice call system integrating AI voice cloning and real-time translation as described in any one of claims 1 to 8 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a cross-language voice call system integrating AI voice cloning and real-time translation as described in any one of claims 1 to 8 is implemented.

Images