Voice adaptive completion system based on multi-modal knowledge graph

Abstract

The invention discloses a voice adaptive completion system based on a multi-modal knowledge graph. The system comprises a data receiver, a data analyzer and a data inference device. The data receiver preprocesses received audio and video data and outputs the audio and video data to the data analyzer; the data analyzer analyzes the voice and the image to extract waveform time sequence features and lip track features, and a phoneme sequence is obtained through multi-mode joint representation; and the data inference device carries out domain session modeling and candidate text prediction according to historical texts, text inference is carried out in combination with a phoneme sequence, statements with semantics are obtained, and complemented voice is synthesized according to waveform features. According to the invention, through a phoneme reasoning model, phoneme recognition is carried out when the voice modality is lost, the domain session modeling is carried out on the historical text generated by the existing voice according to the semantic relationship between the entities in the multi-modal knowledge graph, so that reasoning is carried out to generate the text with semantic, the voice is synthesized in combination with the waveform characteristics of the user voice, and the complemented audio is formed.

Description

technical field [0001] The present invention relates to a technology in the field of speech processing, in particular to a multimodal knowledge map-based speech adaptive completion system for mobile terminals. Background technique [0002] Real-time audio and video technology is mostly used in real-time video chat, video conferencing, distance education, smart home, etc., but in actual use, due to the out-of-order or packet loss of data packets during network transmission, the resulting call jitter will cause a significant drop in call quality. The receiving end creates audio and video data through the packet loss repair system to fill the audio gap caused by packet loss or network delay. Audio data completion still faces the following difficulties in the process of mobile audio and video communication: First, audio generation is dominated by deep learning methods, and the opacity of the reasoning process leads to low interpretability of such methods, so it is difficult to t...

AI Technical Summary

Problems solved by technology

[0002] Real-time audio and video technology is mostly used in real-time video chat, video conferencing, distance education, smart home, etc., but in actual use, due to the out-of-order or packet loss of data packets during network transmission, the resulting call jitter will cause a significant drop in call quality. The receiving end creates audio and video data through the packet loss repair system to fill the audio gap caused by packet loss or network delay

Audio data completion still faces the following difficulties in the process of mobile audio and video communication: First, audio generation is dominated by deep learning methods, and the opacity of the reasoning process leads to low interpretability of such methods, so it is difficult to target Second, the current technology mainly uses single-modal data as the basis for model reasoning, ignoring the ability of the mobile terminal to perceive multiple modal information, resulting in incomplete perception of data and information by the system. cognitive limitations

Images