A tutoring question-answering method and device based on multimodal interaction

By collecting and encoding multimodal data, inputting it into a large model of fine-tuning data in the education field, the teaching reply content is generated, which solves the problem of insufficient interactiveness of traditional tutoring Q&A systems and improves interactivity and teaching effect.

CN118503496BActive Publication Date: 2024-11-08DMAI (GUANGZHOU) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410931652.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-08
Estimated Expiration
2044-07-12

AI Technical Summary

Technical Problem

The traditional tutoring Q&A system is limited to text interaction and cannot make full use of multimodal information such as video and audio, limiting the richness and efficiency of the interaction.

Method used

By collecting user video, audio and active input data, these multimodal data are encoded and processed, feature vectors are generated, and inputted into a pre-trained multimodal model that has been fine-tuned by education data, obtain the execution strategy, and generate Targeted teaching replies.

Benefits of technology

Make full use of multimodal information to improve the interactivity and teaching effect of the tutoring Q&A system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118503496B_ABST
    Figure CN118503496B_ABST
Patent Text Reader

Abstract

The present invention discloses a tutoring question-and-answer method and device based on multimodal interaction, comprising: firstly, by collecting the user's video, audio and active input data, encoding and processing the multimodal data, and generating corresponding feature vectors. These vectors are input into a multimodal large model that is pre-trained and fine-tuned with education field data to obtain an execution strategy for the current problem. Finally, targeted teaching reply content is generated based on the strategy. Such a design makes full use of multimodal information and improves the interactivity and teaching effect of the tutoring question-and-answer system.
Need to check novelty before this filing date? Find Prior Art