6409results about "Speech synthesis" patented technology

The invention relates to dialect speech synthesis, in particular to a semi-supervised dialect emotion speech synthesis system based on hybrid experts, which comprises a text analysis module for preprocessing an input dialect text and generating a text representation vector through feature extraction and feature fusion; the mixed expert module is used for acquiring dialect acoustic features, rhythm features, emotional features and general acoustic features; the dynamic routing module is used for realizing intelligent cooperation among experts through a task-aware soft routing algorithm; the semi-supervised learning module is used for training supervised learning by using the marked dialect emotion voice data and training self-supervised learning by using the unmarked dialect emotion voice data; the acoustic parameter generation module is used for integrating the output of each expert to generate a complete acoustic parameter set; the neural vocoder is used for converting the acoustic parameter set into final dialect emotion voice; the method can effectively overcome the defect that the dialect emotional voice is difficult to accurately synthesize under the condition of lack of sample resources.

Systems, methods, and computer program products are disclosed for selectively employing a processing mode based on a continuous gesture. Selectively employing a processing mode may include detecting an existence of a mode-selection gesture by an individual. Upon detecting the existence of the mode-selection gesture, a first mode for processing facial micromovements of the individual may be continuously implemented while continuously detecting the mode-selection gesture. Following the continuous detection of the mode-selection gesture, a cessation of the mode-selection gesture may be detected. In response, implementation of the first mode may cease, and implementation of a second mode different from the first mode may be initiated for processing facial micromovements of the individual.

Systems, methods, and computer program products are disclosed for generating a common record based on differing source inputs. Generating a common record based on differing source inputs may include receiving via at least one sensor first non-audible signals indicative of verbalization of an individual; interpreting first words of the individual at least in part using the first non-audible signals received from the at least one sensor; receiving second signals generated by a source other than the individual; performing speech recognition on the second signals to interpret second words from the source; and using the first words and the second words to generate a record.

The invention discloses a robot anthropomorphic interaction method based on multi-modal emotion recognition and customized portrait generation. The method comprises the following steps: S1, dynamically fusing multi-modal emotions; the method comprises the following steps: S1, synchronously acquiring voice, visual and text signals through a multi-source heterogeneous sensor, capturing a user voice stream by a high-fidelity microphone array, and extracting acoustic characteristics such as intonation and speed, S2, performing cross-modal reasoning; s3, synchronously generating contents; step S4: style migration; step S5, anthropomorphic voice and expression generation; according to the method, man-machine interaction emotion is analyzed and generated by utilizing a large language model and multi-modal information fusion, the singleness of interaction emotion and the deficiency of emotional sharing ability are avoided, a strong emotion interaction characteristic is achieved, the image of the robot is obtained through a generative technology and can be migrated to any image, the limitation that a specific image is independently made is broken through, and the interaction effect of the robot is improved. The advantage that one robot can be suitable for different scenes is achieved.

Techniques for using a model to generate a response to a user input, where the response is associated with a personality determined to be relevant to the user input, are described. The system receives a user input and context data associated with the user input. Using the user input data and / or the context data, the system determines a personality (e.g., including a personality type and / or personality characteristics) relevant to the user input. The system generates a prompt instructing a model to generate a response to the user input that corresponds to the personality. The model processes the prompt to generate a response to the user input that corresponds to the personality. In some embodiments, the model generates a request for another component of the system to generate information responsive to the user input. The model may transform the responsive information into the personality-associated response.

The present disclosure provides an earphone comprising a housing assembly, a speaker, and an ear hook assembly. The housing assembly includes a first accommodating cavity and a connecting portion that is provided with a socket extending along a first direction. The first accommodating cavity and a first channel and a second channel included in the socket are sequentially connected. A first abutment surface is disposed at transition between the first and second channels. The first abutment surface is inclined to the first direction. The ear hook assembly includes a first connecting end that includes a plug portion. The plug portion is inserted into the socket and includes a second abutment surface and an annular surface. The second abutment surface is inclined to the first direction and faces the first abutment surface. The annular surface extends from an inner edge of the second abutment surface toward the first abutment surface.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of medical health, financial science and technology and the like, and discloses a voice generation method which comprises the following steps: constructing a multi-language voice synthesis model, obtaining plain text data and paired voice text data, and constructing an expansion vocabulary; updating a language perception embedding layer and model parameters, and converting an input text into a mark sequence; and the encoder extracts context semantic features, extracts pronunciation rule features, and the decoder fuses the features to generate an acoustic feature sequence, and converts the acoustic feature sequence into target voice data. According to the invention, the multi-language speech synthesis model is combined with the language perception embedding layer, so that the speech generation capability of a low-resource language is improved; the text conversion accuracy is improved by expanding the vocabulary, the target language learning ability is enhanced by unsupervised training, the low data environment adaptability is optimized by supervised training, and the speech naturalness and fluency are improved by feature fusion.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of medical health, financial science and technology, cultural transmission and the like, and discloses a voice generation method based on multi-modal fusion, which comprises the following steps: collecting audio data to extract timbre features, and training a field feature timbre generation model; analyzing text semantic recognition emotion information, adjusting speech synthesis parameters, combining personalized information to construct a parameter mapping table, fusing to generate a synthesis control parameter sequence, aligning the synthesis control parameter sequence with character labels, visual elements and background music data, driving a domain feature timbre generation model, and generating synthesis data of synchronous speech, text, vision and music. Domain timbres are generated through timbre feature training, speech expression is optimized by combining semantic analysis and emotion recognition, user requirements are matched based on personalized information, and time alignment is performed by fusing text, vision and music data, so that the synthesized speech has domain features, emotion adaptability and personalization, and the speech quality is improved. And the voice immersion and the information transmission capability are improved.

The invention belongs to the technical field of speech recognition. The invention provides a low-resource language self-adaptive speech recognition method based on AdLoRA Plus. The low-resource language self-adaptive speech recognition method comprises the following steps of: obtaining a speech According to the embodiment of the invention, the AdLoRA Plus fine tuning technology is introduced, so that the speech recognition precision of the Whisper model is remarkably improved in a low-resource language environment. Through a low-rank adapter layer and dynamic learning rate adjustment, efficient parameter updating is realized under limited training data. According to the method, a strategy of freezing basic parameters is adopted, only an AdLoRA Plus adapter layer is trained, calculation overhead and training time are remarkably reduced, pre-training knowledge is kept, and overfitting is prevented. The method has wide adaptability and expansibility, and is ensured to be outstanding in practical application. The voice recognition precision, the reasoning efficiency and the training efficiency are effectively improved, multiple challenges in application in small languages and specific fields in the prior art are solved, and the method has important practical application value.

Disclosed in the present disclosure is a rich-media document auxiliary generation apparatus. The apparatus comprises a material extraction module, a theme sorting module, a semantic retrieval module, a structured data text generation module, an illustration recommendation module and a video composition module. The present disclosure uses intelligent means to assist a user to efficiently generate a high-quality rich-media composite document, thereby quickly and accurately describing a theme event in an all-round way.

According to an aspect of the present invention, there is provided an electrical audio signal processing system and device, comprising: a computer graphics processing and selective visual display system with a screen; an eye tracking device; a processor; one or more computer memory devices; wherein the processor is arranged for operations comprising: measuring the user's eye movements to ascertain the specific word on which the user is fixated, by the eye tracking device; modifying the display at the user's current fixation point; applying a delay between the presentation of successive graphic elements based on the user's calculated rate to accommodate the user's required time; and presenting elements to the user at a rate based upon the user's required time.

A neural end-to-end system is provided for the face and voice preserving translation of videos. The system is a pipeline of multiple models that produces a video of the original speaker speaking in the target language with modified lip movement to match the target speech, while preserving emphases and prosody of the original speech, and voice characteristics of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by the translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases in the target sentence. The resulting synthetic speech is then converted back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a generative model generates frames of adapted lip movements which are combined with the audio to produce the final output. The disclosure further describes several use-cases and configurations that apply these techniques to video conferencing, dubbing, low-bandwidth transmission, speech enhancement and assistive technology for the hearing impaired.

A computer-implemented method for operating an artificial intelligence voice agent system includes receiving voice input through communication channels; analyzing converted text through natural language processing (NLP) pipelines implementing intent recognition and sentiment analysis detecting emotional cues using a multimodal large language model (LLM); generating response content using machine learning models trained on domain-specific corpora; converting generated responses to synthetic speech through text-to-speech (TTS) engines; integrating with a customer relationship management (CRM) platforms or an enterprise resource planning (ERP) database; and implementing continuous learning by updating language understanding models using conversation logs, voice recognition parameters based on user feedback, and response generation patterns. One implementation is a computer-implemented system and method that operates a suite of intelligent interactive devices and platforms including an artificial intelligence voice agent, enhanced communication platforms, an intimacy companion system, and augmented / virtual reality eyeglasses. Further, one implementation includes AR / VR eyeglasses that project visual content onto interchangeable lenses or directly onto the user's retina via laser-based retinal projection, provide prescription adjustments, incorporate ear-mounted sensors for monitoring physiological parameters like heart rate, oxygen saturation, and blood pressure, and utilize wireless data transmission, onboard environmental sensing, and remote calibration, all designed to offer dynamically adaptive, secure, and context-aware interactions across communication, personal assistance, health monitoring, and immersive augmented or virtual reality environments.

Systems and methods are provided for machine learning models configured as zero-shot personalized text-to-speech models which comprise a feature extractor, a speaker encoder, and a text-to-speech module. The feature extractor is configured to extract acoustic features and prosodic features from new target reference speech associated with the new target speaker. The speaker encoder is configured to generate a speaker embedding corresponding to the new target speaker based on the acoustic features extracted from the new target reference speech. The text-to-speech module is configured to generate the personalized voice corresponding for the new target speaker based on the speaker embedding and the prosodic features extracted from the new target reference speech without applying the text-to-speech module on new labeled training data associated with the new target speaker.

Provided in the embodiments of the present application are a video generation method and interaction method based on a digital human, and a device, a storage medium and a program product. In the embodiments of the present application, text-to-speech processing is performed on the basis of voice features of a user and an emotion label, speech-to-expression processing is performed on the basis of a mapping relationship between the voice features of the user and expression coefficients, and a digital human model is rendered on the basis of speech signals and the expression coefficients, so as to obtain video data of the digital human model. Thus, voice features of a user are accurately simulated, so as to ensure that a speech output of a digital human sounds natural and is also highly personalized, thereby realizing personalized driving of the digital human, and improving the realism of the digital human in terms of voice and dynamic images. Thus, the user experience is improved, and the interactivity of the digital human and the authenticity and immersion are enhanced.

A cyber security system that protects against cyber threats including a synthetic clone of a voice of a speaker can include several components. A deep learning model is trained to analyze an audio file and produce one or more embeddings of the audio file. One or more AI classifiers are trained to analyze the one or more embeddings of the audio file from the deep learning model to determine whether it is likely that the voice of the speaker engaging with a user is real or the synthetic clone of the voice of the speaker. The voice clone detection bot can be resident on a computing device of the user and can integrate with different sources of audio data on the computing device of the user in order to collect the audio file containing an attempt to synthetically clone the voice of the speaker protected by the cyber security system.

An apparatus for end-to-end text-to-speech synthesis is provided. The apparatus comprises input interface circuitry configured to receive first input data indicative of a phoneme and second input data indicative of a first target duration for the phoneme. The apparatus further comprises processing circuitry configured to, using a trained machine-learning model, map the phoneme to a state using an encoder sub-model of the trained machine-learning model, estimate a second target duration for the phoneme based on the state and determine an attention weight based on the first target duration and the second target duration. The processing circuitry is further configured to map the state to audio data based on the attention weight using a decoder sub-model of the trained machine-learning model, wherein the audio data are indicative of an audio waveform representing speech.

Systems, methods, and computer program products are disclosed for establishing nonvocalized communications. Establishing nonvocalized communications may include detecting a directional gesture made by a wearer of a first wearable device configured to determine facial skin micromovements of the wearer, wherein the directional gesture identifies a second device in proximity to the first wearable device. Based on the directional gesture, a wireless communication channel for enabling a nonvocalized communication between the first wearable device and the second device is selected. Thereafter, specific facial skin micromovements of the wearer of the first wearable device are determined. The specific facial skin micromovements indicate words to be communicated in an absence of perceptible vocalization by the wearer of the first wearable device. Then, the words are transmitted via the wireless communication channel from the first wearable device to the second device for presentation via the second device.

Methods, systems, and computer programs are presented for audio-visual speech generation with multi-modal latent diffusion models. One method includes encoding raw audio signals and video frames into respective latent spaces using audio and visual autoencoders. A text transcript is processed into phoneme sequences using a text transcript processor. The audio and visual latent spaces are conditioned using the text transcript and a conditioning variable. Joint distributions of the visual and audio latent spaces, text transcript, and conditioning variable are learned using a multi-modal latent diffusion model. The model adds noise to the latent audio-visual representations and predicts the noise through denoising neural networks. An inverted diffusion process is utilized to generate diverse speech content and speaker characteristics, resulting in realistic audio-visual speech. The technology presented provides a novel approach to conditional speech generation with potential applications in speech synthesis, voice conversion, and speech recognition.

An acoustic output device is provided. The acoustic output device includes a housing; a bone conduction vibrator configured to generate bone conduction sound waves and transmit the bone conduction sound waves to a cochlea through the housing to produce sound; an air conduction vibrator configured to generate air conduction sound waves, the air conduction sound waves being transmitted to an ear through sound guiding holes in the housing; and a processing module configured to provide the bone conduction vibrator and the air conduction vibrator with a first audio signal and a second audio signal, respectively, the first audio signal and the second audio signal having a frequency division point, the first audio signal and the second audio signal including components with frequencies above and below the frequency division point, respectively. The frequency division point is not less than 600 Hz.

The invention relates to the technical field of artificial intelligence, and discloses a method for scoring question and answer pairs based on a reward mechanism to realize large model fine tuning, comprising the following steps: S1, screening initial question and answer pairs from historical chatting records of an artificial customer service; s2, scoring the initial question and answer pairs based on a reward mechanism; s3, screening high-score question and answer pairs and updating the business question and answer pair knowledge base; s4, performing fine tuning training on the large model based on the updated business question and answer pair knowledge base; s5, executing an intelligent customer service question and answer task by using the fine-tuned large model, the S1 comprising: performing data cleaning and preprocessing on historical chat records, and eliminating invalid question and answer pairs; using a semantic model to calculate the semantic similarity between the questions and the answers, and screening question and answer pairs with complete semantics; according to the invention, through the reward mechanism-based question and answer pair scoring method and dynamic knowledge base updating, high business adaptability and accurate question and answer ability of a large model are realized, and the service efficiency and individuation level of intelligent customer service are improved.

The invention provides a video generation control method and a computer readable storage medium, and is applied to the technical field of computers. According to the method, the structural defect of a split script is solved through a large split model, the consistency of split image features is ensured through image generation based on reference image feature constraint, audio and picture synchronization is realized in combination with emotional speech synthesis and dynamic mouth shape alignment, the video fidelity is improved through a split video cue word guiding multi-subject generation algorithm, and the user experience is improved. Therefore, the technical problems of missing emotion logic, poor consistency of split image features, difficulty in audio synchronization, insufficient controllability of video generation and the like in film and television creation in the existing AIGC technology are effectively solved, and the efficiency and the quality of video content production are improved.

The invention relates to the technical field of speech synthesis, can be applied to business scenes such as financial science and technology, medical health and the like, and discloses a speech generation method, device, equipment and medium. And inputting a text speech language model to generate an intermediate code in combination with a speech code extracted based on a codebook generation mode, extracting a speaker feature vector in the prompt speech, decoding the intermediate code and the speaker feature vector by a generative adversarial decoder, and outputting a target speech. According to the method, fine-grained text representation is established by fusing character semantics and pinyin pronunciation information, and voice cloning is completed through unified voice language modeling and an adversarial generation mechanism in combination with codebook-driven acoustic coding and speaker personality characteristics, so that the naturalness, similarity and pronunciation accuracy of generated voices are improved.

The invention relates to the technical field of voice translation, in particular to an intelligent voice automatic translation system based on AI recognition. The system comprises a speech recognition module, a language translation module, an emotion analysis module and a speech synthesis module. The speech recognition module converts the speech signals into the text sequence, the text sequence is adjusted and optimized by using the context perception capability of the language model, and the language translation module translates the text sequence into the translated text, so that the accuracy of speech recognition is ensured, and the real meaning of a speaker is accurately expressed; meanwhile, the sentiment analysis module is used for carrying out sentiment analysis on a speaker, parameters of synthesized voice are automatically adjusted according to a mapping rule base between sentiment and voice parameters in the voice synthesis module, and a final translation result is broadcasted by taking a final translation as a text and voice; therefore, the voice meaning of the speaker can be expressed more vividly.

The invention relates to the technical field of voice processing, can be applied to business scenes in the fields of medical health, financial science and technology, barrier-free service and the like, and discloses a text-to-voice method based on knowledge distillation, which comprises the following steps: carrying out standardization processing on an input text to generate a standard text sequence; the lightweight text encoder encodes the standard text sequence to generate a text implicit vector; the non-autoregressive acoustic feature prediction module maps the text implicit vector into a student acoustic feature sequence, and calculates alignment loss through knowledge distillation; and performing structured pruning and parameter quantization based on alignment loss, generating an acoustic feature sequence by the optimized model, and converting the acoustic feature sequence into a voice waveform by a vocoder. According to the method, through knowledge distillation, pruning optimization and parameter quantification, the reasoning speed and the cross-equipment adaptability are improved while the model size and the calculation requirement are reduced, so that the TTS system can realize efficient, low-delay and low-power-consumption voice generation in a resource-constrained environment.

The invention discloses an accurate international communication digital human real-time dialogue method fused with a multi-modal technology. The method comprises the following steps: S1, constructing a digital human image and tone; s2, propagation content generation and problem guidance; s3, semantic analysis and intention clarification based on the real-time voice dialogue; s4, geographic preference modeling and path planning; s5, cross-context propagation content generated based on retrieval enhancement is generated; s6, visually displaying the propagation content; and S7, carrying out digital human-driven multi-language propagation content real-time output and feedback closed-loop optimization. Through accurate utterance expression analysis, accurate international propagation problem recommendation is provided, cross-context propagation content generation based on semantic understanding is realized, digital people with voice features and visual images are constructed, real-time dialogue interaction of users is realized, and user experience is improved. The system can carry out geographic modeling according to the region where the accurate problem is located, language preference and propagation object culture characteristics, and differential propagation path planning is achieved.

The invention discloses a system (100) for generating conversational content using a generative artificial intelligence (AI), said system (100) comprising: a user (101), an administrator (102), an application programming interface (API) server (103), a generative artificial intelligence (AI) server (104), a plurality of databases (105), a generative artificial intelligence (AI) processor (106), an audio generate processor (107), a text-to-speech processor / service provider (108), a video generation service (109), a video generation processor (110), and a memory communicatively coupled to the processor, wherein the memory stores processors instructions, which, on execution, causes the processor to generate at least one of conversational script, audio, video, or combination thereof. The system (100) allows users to create and customize various aspects of conversational content, including characters / personas / speakers, groups (of personas / characters / speakers), tones, content types, topics, conversation formats, and tone.

The present application provides training and speech generation methods and apparatuses for a speech generation model, an electronic device, a computer-readable storage medium, and a computer program product. The method comprises: obtaining a first speech generation model; obtaining sample data of a plurality of modalities; on the basis of a prompt image sequence and speech text, respectively calling a plurality of encoders to perform encoding, so as to obtain a multi-modal encoding vector sequence; on the basis of the multi-modal encoding vector sequence, calling a decoder to perform decoding, so as to obtain decoded text; determining a probability distribution for the decoded text and the sample data of the plurality of modalities, and determining a target loss on the basis of the probability distribution; and on the basis of the target loss, updating parameters of the decoder and at least one of the encoders, wherein the updated decoder and the plurality of updated encoders are configured to form a second speech generation model, and the second speech generation model is used to generate target speech text corresponding to a prompt image sequence of a target object.

The invention provides a text user sentiment analysis method and system based on multiple modes and AI, and relates to the technical field of text analysis, and the method comprises the steps: obtaining and preprocessing text, image and audio information of a user, and extracting semantic, visual and acoustic feature vectors; multi-modal features are fused through a cross-modal attention mechanism; utilizing a graph neural network to construct a user emotion social graph to calculate emotion propagation intensity; and finally, obtaining an analysis result containing emotion category and intensity through an emotion classifier. The emotion state of the user can be comprehensively captured, the emotion analysis accuracy is improved, and complex emotion expression is effectively recognized.

An automated conversational multi-player gaming platform is described. A machine-learning model is trained to use a natural language processor and a speech synthesizer as an automated dealer communicating with players of a game of chance. The trained machine-learning model receives requests from people to become players of a game of chance, outputs private game content associated with the game of chance to the players' private user interfaces, and outputs public game content associated with the game of chance to the players' public user interface. The trained machine-learning model receives selections of game playing options and wager values associated with the game of chance from the players via their private user interfaces, and outputs information identifying one player as a winner of the game of chance and a credit of the winner's wager value to the players' public user interface and / or the winner's private user interface.