Multimodal knowledge graph processing method, system, electronic device and storage medium

Abstract

The present invention relates to the field of ideological and political education, and specifically to a multimodal knowledge graph processing method, system, electronic device and storage medium. The invention obtains a target detection network, a behavior detection network, a speech recognition network, a sound model network and constructs a knowledge graph based on training of different modal data such as image, video, voice and text, so that the course courseware content can be automatically enriched by scanning the teacher's course framework and the class situation of students can be collected through a feedback module to continuously optimize the knowledge search process, so that the generated course courseware is more popular with students. At the same time, the present invention can automatically answer questions raised by students through knowledge search, and the question raising and answering are not limited to text, which can provide students with a more immersive experience and enhance students' interest in learning.

Description

Technical Field

[0001] The present invention relates to the field of ideological and political education, and specifically to a multimodal knowledge graph processing method, system, electronic device and storage medium. Background Art

[0002] Ideological and political education is of great significance in cultivating students' all-round development, improving the quality of education and teaching, and serving the national development strategy. However, it also faces some problems and challenges in actual teaching, mainly manifested in the single teaching content and methods, students' resistance, limited course time, teachers' inability to explain in depth, and students' difficulty in forming cognition. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a multimodal knowledge graph processing method, system, electronic device and storage medium to solve the problem of single teaching course content, while saving teachers' course courseware production time, allowing teachers to have more time to explore ideological and political education practice activities.

[0004] To achieve the above object, the present invention provides the following technical solution: a multimodal knowledge graph processing method, comprising:

[0005] S1. Based on knowledge data of different modalities, a course knowledge graph is established through knowledge extraction, knowledge fusion, and knowledge processing; based on image data in knowledge data of different modalities, a target detection neural network is trained through image processing and data annotation; based on video data in knowledge data of different modalities, a behavior detection neural network is trained through data processing and data annotation; based on voice data in knowledge data of different modalities, a voice recognition model is trained after data preprocessing and feature extraction, and a typical character voice model is trained at the same time; among which, the detection category of the trained target detection neural network and the character of the voice simulated by the voice model can be associated with the knowledge entity in the course knowledge graph, and the voice recognition model can recognize human voice and output text information, so that the connection between voice input and knowledge graph can be established through text;

[0006] S2. The teacher creates a basic framework for the course. The method reads the course framework in sequence, wherein the text data uses a natural language processing method to extract knowledge points and then completes the key information based on the course knowledge graph; the image data uses a trained target detection neural network to obtain image information after target detection and further completes the knowledge based on the course knowledge graph; the voice data obtains text information based on voice recognition and further extracts knowledge points through natural language processing, and finally completes the key information based on the course knowledge graph to enrich the teacher's course courseware;

[0007] S3. Students can ask questions, and the multimodal knowledge graph processing method performs knowledge search based on the course knowledge graph to answer the questions;

[0008] S4. Collect the amount of questions reported by students in class and mark the amount of questions collected as a; collect the action features of students in class, detect them based on the behavior detection neural network and analyze the detection results for concentration, marking them as b; record the test scores of students and mark them as c; evaluate the knowledge used to complete the teaching course knowledge through the formula: Y=k1a+k2b+k3c, where k1, k2, and k3 represent weights, set a threshold, and knowledge greater than the threshold means that it is popular with students. This part of knowledge is marked and feedback is given to increase the weight of this knowledge and optimize the results of knowledge search.

[0009] Preferably, the different modal knowledge data include at least: educational and teaching resources, network data resources, and images, videos, voices, and text data of special resources collected on-site by the school.

[0010] Preferably, the teacher's course framework voice data includes: course audio materials and the teacher's oral course framework.

[0011] Preferably, the target categories detected by the target detection neural network include at least: typical people and typical events.

[0012] Preferably, the behavior detection includes at least: nodding and raising the head.

[0013] Preferably, knowledge completion sets the depth of knowledge search to control the degree of course divergence.

[0014] Furthermore, the methods for S3 students to ask questions include at least: voice, text, and graphic methods.

[0015] Furthermore, the question answering methods include at least: voice answering, text answering, picture answering, multimedia audio and video image assisted answering and a mixed answering method of the above methods.

[0016] Furthermore, when answering questions by voice, when the answer to knowledge reasoning involves a specific person, the sound model uses the corresponding person's voice to answer the question, enhancing the rendering power of the answer and the student's sense of involvement. When the answer to knowledge reasoning involves a specific event, the sound data of the corresponding event in the knowledge data is used as background music when answering the question.

[0017] Preferably, knowledge is supplemented based on the teacher's basic course framework and a course mind map is generated.

[0018] At the same time, the present invention also provides a multimodal knowledge graph processing system, which includes: a knowledge module, a scanning module, a voice module, a display module, a reasoning module, a feedback module and a storage module. The knowledge module completes the knowledge search function based on the course knowledge graph. The scanning module completes the scanning work of the basic framework of the teacher's course. The voice module includes voice recognition and sound model, completes the voice recognition when asking questions, the voice data recognition in the teacher's course framework, and the character sound simulation output function when answering questions. The display module completes the course display and course mind map display after knowledge completion. The reasoning module completes the knowledge completion reasoning of the basic framework of the teacher's course and the knowledge reasoning function when asking questions. The feedback module receives the student feedback on the amount of questions, the student's concentration in the course, and the student's test scores, and evaluates the knowledge used for course knowledge completion and feeds back to the knowledge module. The storage module completes the storage of all intermediate data and result data, wherein the result data at least includes the teacher's course framework, the course content after knowledge completion, and the course mind map.

[0019] Furthermore, the present invention provides an electronic device, including a voice receiver and player, a memory, and a processor. The voice receiver and player receives voice data in students' questions or teachers' course frameworks and searches for answers and plays them when voice answers are given. The memory stores the above-mentioned system code and data, and the processor completes the data processing function described in the processing method.

[0020] Furthermore, the present invention provides a storage medium, which is a computer-readable storage medium for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the multimodal knowledge graph processing method described above.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The present invention obtains different algorithm models based on multiple modal data. Through the combined application of algorithm models, it can be based on the basic course framework conceived by the teacher, through model analysis, to obtain course framework information and enrich the course content based on the course knowledge graph. Since the knowledge graph contains multiple modal data such as images, voice, video, text, etc., the ultimately enriched course content will not be limited to text, and can present students with various forms of content such as pictures, texts, audio and video in class, thereby increasing the attractiveness of the course. At the same time, since the course content is automatically enriched based on the knowledge graph recommendation system, it can effectively save teachers' courseware production time. In particular, the course framework can receive teachers' audio input, thereby significantly saving courseware production time, allowing teachers to have more time to explore educational practice activities.

[0023] 2. Whether before, during or after class, the present invention can answer students' questions in a timely manner and provide rich answer modes. When students receive voice answering mode and their questions involve specific people, the present method and system will imitate the voice of the corresponding person to answer. When students' questions involve specific events, the present invention will automatically match the event music when answering, giving students an immersive experience and deepening their impression of the course.

[0024] 3. The present invention collects the amount of questions students ask, their concentration in class, and their test scores, and feeds them back into the knowledge graph, continuously optimizing the results of knowledge search, making the content of course knowledge supplement more and more popular with students. Through continuous iterative optimization, the course content searched and completed by the knowledge graph recommendation system is ultimately more and more popular with students, thereby increasing students' head-raising and nodding rates in class. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A logical schematic diagram of the method of the present invention;

[0026] Figure 2 It is a schematic diagram of the system structure of the present invention;

[0027] Figure 3 This is a schematic diagram of constructing the knowledge graph of the present invention;

[0028] Figure 4 A schematic diagram of a mind map generation framework for the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Embodiment 1:

[0031] like Figure 1-Figure 4As shown, the present invention provides a method, system, device and medium. When applied, first, several network models are trained according to the collected data, wherein the data comes from images, videos, voices, and text data of educational and teaching resources, network data resources, and special resources collected on-site in schools, such as relevant teaching resources of the Education Bureau, CNKI database data, Baidu Encyclopedia data, data collected by students, teaching activity data, etc. The data are screened to screen out data resources related to ideological and political education and further classify the data according to images, videos, voices, and texts. Different data are labeled with corresponding keyword labels. If the image is a person or a typical event, the image is associated with the corresponding person or event and annotated to train a target detection neural network model. Specific actions in the video are captured and annotated (such as nodding, raising the head, lowering the head, raising the hand, etc.), and a behavior detection neural network model is trained. The voice uses the text representation of the content as a label to train a voice recognition neural network. The specific character voice uses the character name as a label to train a sound model. Finally, all data, including images, voices, videos, and texts, are represented by labels to represent the content theme, and the course knowledge graph required by this course is trained. The training process is as follows Figure 3As shown, since the knowledge graph is mainly represented in the form of triples (entity-relationship-attribute), the content is generally text. In order to associate different modal knowledge, related images, voices, videos, etc. can be associated as attributes of entities. Therefore, when a question related to a certain person is to be queried, image, voice, and video information can be searched through the attributes of the person, thereby enriching the knowledge derivation results, making the content of knowledge completion have multiple forms, and enhancing the attractiveness of the course. Based on this, the preparation process of the method of the present invention is completed. When teachers use the system and method to make courseware, they only need to give a basic framework. For example, to establish a correct world view, teachers can give the framework as follows: 1. The concept of world view; 2. Classification of world view; 3. How to correctly establish a world view? 4. Give some examples of wrong world views; 5. Give some positive examples of world views; 6. Give the description of the world view of the person in the picture (give a picture). Based on this, the scanning module of the present invention starts to work and processes the text of the framework, such as processing "worldview concept" into two keywords "worldview" and "concept", and then searches for the "worldview" entity in the knowledge map through the keyword, and then finds the corresponding worldview definition in the attribute or relationship to give the answer. When there is a video, picture, or voice explanation of the worldview, the result of the knowledge search will also give the corresponding video, picture, or voice explanation, thereby completing the knowledge of the teacher framework. When a picture or voice is given in the framework, the scanning module obtains the person or event information represented by the picture through target detection or obtains the text content represented by the voice through voice recognition, and replaces the person or event with the picture (voice The method is to replace the voice file with the voice content text) and further cooperate with the text to perform knowledge search, finally enrich the course framework and generate course courseware, which is stored in the storage module and displayed in the display module when used. At the same time, the present invention further generates a mind map based on the knowledge, and the basic logic of the mind map is maintained as the teacher framework, and the keywords of the knowledge completion content corresponding to the teacher framework are supplemented as the subordinate content of the mind map. For example, the first-level title is "world view", the first of the second-level titles is "concept", and the third-level content after knowledge completion is "world view is a person's fundamental view of the whole world", and the second content of the third-level content is "world view is not only a problem of cognition, but also includes firm beliefs and positive actions", such as Figure 4 As shown, the mind map is saved in the storage module. When students need to review after class, they can download it from the storage module.

[0032] In order to further reduce the time for teachers to produce courseware, teachers can directly use voice input. When teachers provide a recording of the courseware framework, the scanning module scans the text content corresponding to the voice through voice recognition, and then uses the text mode to process and search for knowledge based on natural language processing methods. In this way, teachers can save time on courseware production, so that they can have more time for more valuable things, such as scientific research, practical exploration, etc.

[0033] Embodiment 2:

[0034] The present invention provides a question-and-answer function, and students can ask questions in multiple modes, such as directly inputting text to ask questions (such as what is the world view?), or giving a picture and asking questions (such as who is this? What are his propositions?), or directly inputting questions orally (the input oral language is actually saved as a recording file and sent to the service backend). After receiving the student's question, the scanning module of the present invention scans the student's question. The scanning mode is the same as the course courseware. The scanning result also searches for knowledge based on the course knowledge map to obtain the answer to the question, and finally presents the answer to the student. At the same time, in order to increase students' interest in ideological and political learning, the answer presentation method can be a voice playback mode, a text mode, a picture mode, a multimedia form or a combination of forms, such as giving the text content of the answer while reading it out. When the knowledge associated with the answer search is a picture, voice or video, the relevant pictures, voice and video are presented to the student together, so that the student can obtain more content and feel more attractive.

[0035] Furthermore, in order to make students feel more involved, when students' questions involve a certain character, the voice module of the present invention automatically matches the voice model of the character and uses the voice of the character to answer the question when answering the question. When students' questions involve a certain event, the typical representative music of the event is used as background music when answering the question. Of course, the two can be combined, that is, when both characters and events are involved, the character voice and background music coexist, which makes students more immersed. For the character voice part, when the real voice data of the character cannot be obtained, the character voice in classic movies and TV series is used to represent it, and the music corresponding to the event is also matched with the background music in classic movies and TV series.

[0036] Embodiment three:

[0037] The present invention provides a feedback module, which collects the amount of questions a, the concentration b and the test scores c of students' classroom feedback, and provides feedback on the knowledge in the knowledge graph used in the course through the formula Y=k1a+k2b+k3c, where k iRepresents weights. When more emphasis is placed on student concentration and test scores, the weights can be set to k1=0.2, k2=0.4, k3=0.4 respectively. It is further explained that the problems fed back by students in class are the students' negative opinions on the course. The number of questions in each class is taken and divided by the cardinality (such as half of the number of students), and recorded as a. The students' classroom concentration is detected in real time in the classroom through a behavior detection neural network. The number of students who raise their heads, nod, and raise their hands when answering questions is continuously detected, and the head-raising rate, nodding rate, hand-raising rate, etc. are calculated. The final classroom student concentration can be obtained by taking a single data of the head-raising rate, or by weighted summing up the head-raising rate, nodding rate, hand-raising rate and other data according to a certain proportion of weights, and recorded as b. The final student test score is the average score of the students in the class divided by the total score to obtain the feedback amount corresponding to the student test score, recorded as c. Since the data are normalized separately, the final calculated Y value is a decimal. The decimal weights the knowledge used in this course and indicates the students' acceptance (popularity) of the current knowledge. When it is continuously iterated, it will be able to continuously confirm which knowledge students are more popular with, and continuously improve the quality of knowledge search when teachers complete the course framework. Initially, the weights of all knowledge are the same (set to 1). At this time, knowledge completion is performed based on the search depth (depth controls the degree of divergence). It should be clear that the knowledge completion in the initial situation is also correct knowledge, but there is no information about the popularity of students.

[0038] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal knowledge graph processing method, characterized in that: S1. Based on different modal knowledge data, establish the course knowledge graph through knowledge extraction, knowledge fusion and knowledge processing; based on the image data in different modal knowledge data, train the target detection neural network through image processing and data annotation; Based on the video data in different modal knowledge data, after data processing and data labeling, the behavior detection neural network is trained; Based on the speech data in different modal knowledge data, after data preprocessing and feature extraction, the speech recognition model is trained, and the typical character sound model is trained at the same time; the detection category of the trained target detection neural network and the character of the sound simulated by the sound model can be associated with the knowledge entities in the course knowledge graph. The speech recognition model can recognize human voices and output text information, which can realize the connection between speech input and knowledge graph through text; S2. The teacher creates a basic framework for the course. The method reads the course framework in sequence, wherein the text data uses a natural language processing method to extract knowledge points and then completes the key information based on the course knowledge graph; the image data uses a trained target detection neural network to obtain image information after target detection and further completes the knowledge based on the course knowledge graph; the voice data obtains text information based on voice recognition and further extracts knowledge points through natural language processing, and finally completes the key information based on the course knowledge graph to enrich the teacher's course courseware; S3. Students can ask questions, and the multimodal knowledge graph processing method performs knowledge search based on the course knowledge graph to answer the questions; S4. Collect the amount of questions that students report in class and mark the amount of questions collected as a; collect the action features of students in class, detect them based on the behavior detection neural network, and analyze the detection results for concentration, marking them as b; record the test scores of students and mark them as c; evaluate the knowledge used to complete the teaching course knowledge through the formula: Y=k1a+k2b+k3c, where k1, k2, and k3 represent weights, set a threshold, and knowledge greater than the threshold is popular with students. Mark and feedback the knowledge greater than the threshold, increase the weight of the knowledge greater than the threshold, and optimize the results of knowledge search; The different modal knowledge data include at least: images, videos, voices, and text data of educational and teaching resources, network data resources, and special resources collected on-site by the school; the voice data of the teacher's course framework include: course audio materials, and the teacher's oral course framework; the target categories detected by the target detection neural network include at least: typical characters and typical events; behavior detection includes at least: nodding and looking up; knowledge completion sets the depth of knowledge search to control the degree of course divergence; S3 students' question-asking methods include at least: voice, text, and graphic methods; question answering methods include at least: voice answering, text answering, picture answering, multimedia audio-visual image assisted answering, and mixed answering methods of the above methods; when answering by voice, when the answer to the knowledge reasoning involves a specific person, the sound model uses the corresponding person's voice to answer the question, enhancing the rendering power of the answer and the student's sense of substitution. When the answer to the knowledge reasoning involves a specific event, the sound data of the corresponding event in the knowledge data is used as background music to answer the question, giving students an immersive experience.

2. The multimodal knowledge graph processing method according to claim 1, characterized in that: Based on the teacher's basic course framework, knowledge is completed and a course mind map is generated.

3. A multimodal knowledge graph recommendation processing system, the system implementing the multimodal knowledge graph processing method according to any one of claims 1 to 2, the system comprising: A knowledge module, a scanning module, a voice module, a display module, a reasoning module, a feedback module and a storage module, characterized in that: the knowledge module completes the knowledge search function based on the course knowledge map, the scanning module completes the scanning work of the basic framework of the teacher's course, the voice module includes voice recognition and sound model, completes the voice recognition when asking questions, the voice data recognition in the teacher's course framework and the character sound simulation output function when answering questions, the display module completes the course display and the course mind map display after knowledge completion, the reasoning module completes the knowledge completion reasoning of the basic framework of the teacher's course and the knowledge reasoning function when asking questions, the feedback module receives the student feedback on the amount of questions, the student's concentration in the course and the student's test scores, and evaluates the knowledge used for course knowledge completion and feeds back to the knowledge module, the storage module completes the storage of all intermediate data and result data, wherein the result data at least includes the teacher's course framework, the course content after knowledge completion, and the course mind map.

4. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 2.

5. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the multimodal knowledge graph processing method described in any one of claims 1-2.

Images