System and method for realizing cross-language communication based on multimodal assistance

Abstract

The present invention proposes a cross-language communication system and method based on multi-modal assistance. The method utilizes the foreground interaction module, data management module and semantic association module in the cross-language communication system to analyze conversation content and use natural language processing tools It can automatically extract the central topic and keywords in the conversation, and the semantic association module can automatically search for relevant pictures and video clips based on the detected central topic and keyword information, and provide them to both parties in an appropriate way, so as to promote mutual understanding and communication. Here, the pictures and videos used as aids in understanding can be automatically picked up from the Internet by searching, or directly obtained from a pre-marked multimedia library. Finally, the system generates a multimodal conversation summary based on the text chat information of the two parties in the conversation and the corresponding picture and video content.

Description

technical field [0001] The invention belongs to the fields of multimedia analysis and network communication, and relates to a method for realizing cross-language communication based on multi-mode assistance. Background technique [0002] With the rapid development of communication technology and Internet technology, a network instant messaging system completely different from traditional communication methods such as mail, telephone, and telegram has emerged, such as MSN and QQ. Traditional emails and telegrams mainly use text, and telephone calls mainly use voice, while instant messaging can not only use text and voice, but also can assist rich multimedia methods such as video and pictures. Through the instant messaging system, people across oceans can realize face-to-face real-time conversation. The whole earth has become a veritable global village. [0003] Language issues remain an insurmountable barrier in instant messaging for interlocutors who speak different langua...

AI Technical Summary

Problems solved by technology

[0008] The purpose of the present invention is to solve the technical defects of slow processing speed and difficult to use in the prior art, and to assist people who use different languages ​​to communicate online smoothly through multi-modal information

Images