Teaching recommendation method, system and equipment based on multi-modal fusion and storage medium

Abstract

The invention provides a teaching recommendation method, system and device based on multi-modal fusion and a storage medium, which can be applied to video teaching in the fields of finance and medical treatment, and the method comprises the following steps: obtaining teaching behavior data of a lecturer in a teaching video; synchronously acquiring learning interaction data and physiological signal data of each student during teaching; establishing time correlation mapping among the teaching behavior data, the learning interaction data and the physiological signal data, and detecting abnormal features in the teaching behavior data; judging whether the proportion of synchronously triggering learning interaction in the student group reaches a preset proportion or not when the abnormal feature appears, and generating an intervention instruction when the proportion reaches the preset proportion; performing real-time adjustment on teaching behaviors and teaching resource recommendation strategies of a lecturer based on the intervention instruction; according to the invention, teaching abnormal features can be detected in real time and group feedback can be quantified, so that teaching behaviors and resource recommendation can be optimized, and the teaching efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the field of teaching recommendation technology, and specifically relates to a teaching recommendation method, system, device and storage medium based on multimodal fusion. Background Art

[0002] In professional teaching scenarios in the medical and financial fields, accurately evaluating teaching effectiveness and optimizing the teaching process in real time are crucial to improving students' learning quality. In addition, the teaching process often involves complex knowledge systems and practical teaching, such as explaining surgical procedures in medical courses and risk model analysis in financial courses, which makes teaching more difficult.

[0003] Existing teaching evaluation systems mainly rely on static data such as instructor qualification certification, students' post-class ratings, and course tags for teaching evaluation and resource recommendations. For example, in surgical operation training, courses are recommended only based on the instructor's professional title level and the students' final assessment scores. In financial trading training, learning resources are matched based on the proportion of historical users' five-star reviews. This makes it impossible to conduct multi-dimensional observations of instructors' teaching behaviors and students' learning status.

[0004] This single-dimensional evaluation method, on the one hand, cannot capture the dynamic behavioral information of the instructor during the teaching process. For example, in a medical course, the instructor speaks too fast when explaining the core surgical steps, resulting in a gap in students' understanding, or in a finance course, the teaching gestures are not synchronized with the courseware content, affecting the efficiency of knowledge transfer. It is difficult to identify abnormal characteristics that may appear in the teaching process through analysis; on the other hand, it is also impossible to observe the dynamic feedback behavior of students during the teaching process, resulting in the students' high-frequency pause behavior or attention fluctuation signals when explaining complex knowledge points being ignored, affecting learning efficiency. Summary of the Invention

[0005] This application provides a teaching recommendation method, system, device and storage medium based on multimodal fusion, which can detect abnormal teaching characteristics in real time and quantify group feedback, so as to optimize teaching behavior and resource recommendations and improve teaching efficiency.

[0006] In order to solve the above technical problems, in a first aspect, the present application provides a teaching recommendation method based on multimodal fusion, comprising the following steps:

[0007] Obtain the teaching behavior data of the lecturer in the teaching video;

[0008] Synchronously collect each student's learning interaction data and physiological signal data during the lecture;

[0009] Establishing a time correlation mapping between the teaching behavior data, the learning interaction data, and the physiological signal data, and detecting abnormal features in the teaching behavior data;

[0010] determining whether a proportion of the learning interactions triggered simultaneously by the student group reaches a preset proportion when the abnormal feature occurs, and generating an intervention instruction when the preset proportion is reached;

[0011] adjusting the teaching behavior of the instructor and the teaching resource recommendation strategy in real time based on the intervention instruction.

[0012] As a further improvement of the present application, the teaching behavior data at least includes the explanation speed, the explanation tone, the explanation gesture, and the courseware flipping action of the instructor in the teaching video;

[0013] The learning interaction data at least includes the video pause behavior, the video review behavior, and the interactive information sending behavior of each student;

[0014] The physiological signal data at least includes the positive physiological signal and the negative physiological signal of each student.

[0015] As a further improvement of the present application, the detection of the abnormal feature in the teaching behavior data comprises:

[0016] When the explanation speed of the instructor is accelerated and the explanation tone is raised during teaching, the current teaching behavior is marked as an abnormal feature;

[0017] and / or, when there is a time delay error between the explanation gesture and the courseware flipping action of the instructor, the current teaching behavior is marked as an abnormal feature.

[0018] As a further improvement of the present application, the detection of the abnormal feature in the teaching behavior data further comprises:

[0019] obtaining interactive information in the learning interaction data; wherein the interactive information includes negative interactive information and positive interactive information;

[0020] constructing a contradiction index based on the positive interactive information and the negative physiological signal;

[0021] When the contradiction index is lower than a preset index threshold, the teaching behavior corresponding to the current positive interactive information is marked as an abnormal feature.

[0022] As a further improvement of the present application, the determination of whether the proportion of the learning interactions triggered simultaneously by the student group reaches a preset proportion when the abnormal feature occurs, and the generation of an intervention instruction when the preset proportion is reached, comprises:

[0023] When the abnormal feature occurs, it is determined whether the proportion of the students in the student group triggering the video pause behavior and the video review behavior simultaneously reaches a preset proportion, and the intervention instruction is generated when the preset proportion is reached.

[0024] As a further improvement of the present application, the intervention instruction is used to adjust the teaching behavior and teaching resource recommendation strategy of the instructor in real time, including:

[0025] Based on the intervention instruction, the instructor is reminded to adjust the teaching behavior, and supplementary materials are pushed to each student within a preset time.

[0026] As a further improvement of the present application, the time correlation mapping is established between the teaching behavior data, the learning interaction data and the physiological signal data, including:

[0027] The Bi-LSTM model is used to establish the time correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data.

[0028] In a second aspect, the present application provides a teaching recommendation system based on multi-modal fusion, which comprises:

[0029] An acquisition unit is configured to acquire teaching behavior data of an instructor in a teaching video.

[0030] A collection unit is configured to synchronously collect learning interaction data and physiological signal data of each student during teaching.

[0031] A detection unit is configured to establish a time correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data, and detect abnormal features in the teaching behavior data.

[0032] A generation unit is configured to determine whether the proportion of synchronous learning interaction in a student group reaches a preset proportion when the abnormal features appear, and generate an intervention instruction when the preset proportion is reached.

[0033] An adjustment unit is configured to adjust the teaching behavior and teaching resource recommendation strategy of the instructor in real time based on the intervention instruction.

[0034] In a third aspect, the present application provides a computer device, which comprises a processor and a memory coupled with the processor, and the memory stores a computing program, and the computing program is executed by the processor to make the processor execute the steps of the teaching recommendation method based on multi-modal fusion in any one of the above.

[0035] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the teaching recommendation method based on multi-modal fusion in any one of the above.

[0036] Compared with the prior art, the teaching recommendation method, system, device and storage medium based on multi-modal fusion provided by the application obtain lecturer teaching behavior data, synchronously collect student learning interaction data and physiological signal data, obtain multi-dimensional data information from two aspects of lecturer teaching and student real-time feedback, establish time correlation mapping for the teaching behavior data, learning interaction data and physiological signal data, accurately capture the corresponding relationship of teaching behavior and student reaction on the time axis, when detecting an abnormal feature, judge whether the proportion of synchronous learning interaction triggered in the student group reaches a preset proportion, filter accidental behavior of individual students to avoid invalid intervention, generate an intervention instruction when the preset proportion is reached, and real-time adjust the teaching behavior of the lecturer and the teaching resource recommendation strategy based on the intervention instruction, so that the teaching process can be dynamically calibrated according to the actual reaction of the students, and the teaching efficiency and learning effect are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 The flowchart of the teaching recommendation method based on multi-modal fusion provided by the present application;

[0039] Figure 2 The structure diagram of teaching behavior data in the teaching recommendation method based on multi-modal fusion provided by the present application;

[0040] Figure 3 The structure diagram of learning interaction data in the teaching recommendation method based on multi-modal fusion provided by the present application;

[0041] Figure 4 The structure diagram of physiological signal data in the teaching recommendation method based on multi-modal fusion provided by the present application;

[0042] Figure 5 The flowchart of marking abnormal features in the teaching recommendation method based on multi-modal fusion provided by the present application;

[0043] Figure 6 The flowchart of constructing a contradiction index in the teaching recommendation method based on multi-modal fusion provided by the present application;

[0044] Figure 7 The structure diagram of abnormal features in the teaching recommendation method based on multi-modal fusion provided by the present application;

[0045] Figure 8 A teaching recommendation system structure schematic diagram based on multi-modal fusion is provided for the embodiments of the present application.

[0046] Figure 9 A computer device structure schematic diagram is provided for the embodiments of the present application.

[0047] Figure 10 A storage medium structure schematic diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, further detailed description of the embodiments of the present application will be given below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and are not intended to limit the embodiments of the present application.

[0049] In the description of the embodiments of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0050] In order to make the description of the present disclosure more detailed and complete, the following describes the embodiments of the embodiments of the present application; but this is not the only form of implementation or use of the specific embodiments of the present application. The embodiments include the features of multiple specific embodiments and the method steps and their order used to construct and operate these specific embodiments. However, other specific embodiments can also be used to achieve the same or equivalent functions and step sequences.

[0051] In the embodiments of the present application, "exemplary", "in some embodiments", "in another embodiment" and the like are used to mean by way of example, illustration or description. Any embodiment or design scheme described as "exemplary" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "exemplary" is intended to present the concept in a specific way.

[0052] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0053] In professional teaching scenarios in the medical and financial fields, accurately evaluating teaching effectiveness and optimizing the teaching process in real time are crucial to improving students' learning quality. In addition, the teaching process often involves complex knowledge systems and practical teaching, such as explaining surgical procedures in medical courses and risk model analysis in financial courses, which makes teaching more difficult.

[0054] Existing teaching evaluation systems mainly rely on static data such as instructor qualification certification, students' post-class ratings, and course tags for teaching evaluation and resource recommendations. For example, in surgical operation training, courses are recommended only based on the instructor's professional title level and the students' final assessment scores. In financial trading training, learning resources are matched based on the proportion of historical users' five-star reviews. This makes it impossible to conduct multi-dimensional observations of instructors' teaching behaviors and students' learning status.

[0055] This single-dimensional evaluation method, on the one hand, cannot capture the dynamic behavioral information of the instructor during the teaching process. For example, in a medical course, the instructor speaks too fast when explaining the core surgical steps, resulting in a gap in students' understanding, or in a finance course, the teaching gestures are not synchronized with the courseware content, affecting the efficiency of knowledge transfer. It is difficult to identify abnormal characteristics that may appear in the teaching process through analysis; on the other hand, it is also impossible to observe the dynamic feedback behavior of students during the teaching process, resulting in the students' high-frequency pause behavior or attention fluctuation signals when explaining complex knowledge points being ignored, affecting learning efficiency.

[0056] In view of this, please refer to Figures 1-10 The embodiments of the present application provide a teaching recommendation method, system, device and storage medium based on multimodal fusion, which can detect abnormal teaching characteristics in real time and quantify group feedback, so as to optimize teaching behavior and resource recommendations and improve teaching efficiency.

[0057] In the fields of medical and financial education, teaching content usually involves complex knowledge systems and professional practical skills, such as surgical operation procedures and pathological structure demonstrations in medical courses, and risk model deduction and market analysis in financial courses. These contents usually need to be presented through online courses and PPT switching, so that the instructor's voice explanation and operation demonstration process are presented to students in real time, so that students can understand their abstract concepts and master related skills. Next, we will use teaching videos in medical and financial fields as examples to illustrate the teaching recommendation method based on multimodal fusion provided by this application.

[0058] Referring to Figure 1 , the flowchart of the teaching recommendation method based on multi-modal fusion provided in the embodiment of the present application, the detection method comprises the following steps:

[0059] Step S1: obtaining teaching behavior data of a lecturer in a teaching video;

[0060] As an optional implementation, referring to Figure 2 , the structural diagram of the teaching behavior data in the teaching recommendation method based on multi-modal fusion provided in the embodiment of the present application, the above-mentioned teaching behavior data at least includes the explanation speed, the explanation tone, the explanation gesture and the courseware flipping action of the lecturer in the teaching video.

[0061] In the embodiment of the present application, taking the teaching video in medical and financial fields as an example, the teaching video of the lecturer needs to be obtained in real time, and further the teaching behavior data of the lecturer in the teaching video needs to be obtained.

[0062] It needs to be noted that the teaching video here is preferably the live teaching video of the lecturer, which is convenient for real-time detection of teaching abnormal characteristics and quantitative student group feedback, so that the lecturer can timely adjust the teaching behavior.

[0063] In an optional embodiment, since the explanation manner, action habit and the like of the lecturer will directly affect the understanding and absorption of complex knowledge by students, the present application preferably obtains the explanation speed, the explanation tone, the explanation gesture and the courseware flipping action of the lecturer in the teaching video.

[0064] It can be understood that the explanation speed and the explanation tone can effectively show the emotions of the lecturer, for example, when some important content is explained, the lecturer may speed up and the tone may rise, so as to remind the students to concentrate on listening, while the gesture action can assist the lecturer to explain knowledge, and the courseware flipping can reflect the teaching rhythm, so as to accurately capture the behavior characteristics of the lecturer in knowledge explanation, rhythm control, teaching aid cooperation and the like, and avoid problems such as explanation stall, non-standard operation, content connection fault and the like in the teaching process.

[0065] Step S2: synchronously collecting learning interaction data and physiological signal data of each student during teaching;

[0066] As an optional implementation, referring to Figure 3 , the structural diagram of the learning interaction data in the teaching recommendation method based on multi-modal fusion provided in the embodiment of the present application, the above-mentioned learning interaction data at least includes the video pause behavior, the video review behavior and the interactive information sending behavior of each student;

[0067] Referring to Figure 4A structure diagram of physiological signal data in a teaching recommendation method based on multi-modal fusion provided in an embodiment of the present application, wherein the physiological signal data at least includes positive physiological signal and negative physiological signal of each student.

[0068] In the embodiment of the present application, learning interaction data actively generated by each student in the learning process and physiological signal data passively generated by the student in the learning process are synchronously collected in the teaching process of the lecturer, so as to ensure that the learning interaction data, the physiological signal data and the teaching behavior data of the lecturer are aligned in the time stamp, thereby accurately identifying whether the learning interaction data and the physiological signal data of each student are triggered in real time due to a certain teaching behavior.

[0069] For example, when the lecturer explains the steps of a complex operation in a medical course, the students frequently pause, which may indicate that the operation details of the current teaching are not effectively absorbed by the students, reflecting that the students have understanding jamming in the process of a specific teaching node; if the student repeatedly reviews the process of a specific teaching node, it indicates that the student needs to strengthen the understanding of the teaching part; if the student sends an interactive information sending behavior in the process of a specific teaching node, such as sending a "please repeat the explanation" or "not understood" scroll, it directly feedbacks that the student does not effectively absorb the current teaching knowledge.

[0070] In an optional embodiment, the student can collect physiological signal data in the process of listening to the course in real time by wearing a smart device, such as HRV (Heart Rate Variability) data and body temperature data; wherein the HRV data refers to the time variance between successive heart rate periods, and by analyzing the fluctuation of the heart rate, the stress of the student in the learning process can be reflected and quantified.

[0071] Specifically, the high-frequency component in the HRV data is related to the parasympathetic nervous activity, representing a relaxed and recovered state, and the low-frequency component reflects the balance between the sympathetic and parasympathetic nerves, and is related to stress and alertness. Therefore, in the embodiment of the present application, if the high-frequency component in the HRV data of the student increases in the process of listening to the course, it indicates that the student is currently relaxed and focused in mind and body, which is beneficial to information processing and memory consolidation, and is manifested as positive physiological signal in the physiological signal data; when the low-frequency component in the HRV data increases, it indicates that the student may have stress, anxiety or cognitive overload, which is manifested as negative physiological signal in the physiological signal data.

[0072] For example, when the lecturer's gestures are not synchronized with the courseware in a medical course, the student may synchronously appear negative physiological signal and high-frequency pause; when the lecturer's speaking speed is accelerated and the tone is raised, the student may synchronously appear negative physiological signal and frequent interactive information sending behavior.

[0073] In the embodiment of the present application, the video pause behavior, the video review behavior and the interactive information sending behavior generated by each student in the learning process are synchronously collected, and the positive physiological signals and the negative physiological signals generated by the student in the learning process are synchronously collected.

[0074] For example, when the gestures of the lecturer in a certain medical course are not synchronized with the courseware, the students may synchronously appear the negative physiological signals and the high-frequency pause, and when the explanation speed of the lecturer is accelerated and the explanation tone of the lecturer is raised, the students may synchronously appear the negative physiological signals and the frequent interactive information sending behavior, so as to subsequently establish the correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data based on the unified time axis.

[0075] Step S3: establishing the time correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data, and detecting the abnormal features in the teaching behavior data;

[0076] As an optional implementation, please refer to Figure 5 The flowchart for marking the abnormal features in the teaching recommendation method based on the multi-modal fusion provided in the embodiment of the present application, and the time correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data is established, including:

[0077] Step S30: establishing the time correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data based on the Bi-LSTM model.

[0078] In the embodiment of the present application, the unified time stamp is preferably added to the explanation speed, the explanation tone, the explanation gesture and the courseware flipping action of the lecturer, the video pause behavior, the video review behavior and the interactive information sending behavior of each student, and the positive physiological signals and the negative physiological signals of each student by the Bi-LSTM (Bidirectional Long Short-Term Memory) model, so as to capture the time sequence dependence relationship of each data.

[0079] For example, after the explanation speed of a certain lecturer is changed, the learning interaction data and the physiological signal data corresponding to the teaching behavior of each student may appear, thereby providing the basis for subsequent abnormal feature detection, and avoiding the misjudgment phenomenon caused by the time dislocation.

[0080] As an optional implementation, the detection of the abnormal features in the teaching behavior data includes:

[0081] Step S31: when the explanation speed of the lecturer is accelerated and the explanation tone of the lecturer is raised during the teaching, the current teaching behavior is marked as an abnormal feature; and / or,

[0082] When there is a time delay error between the lecturer's explanation gesture and the courseware flipping action, the current teaching behavior is marked as an abnormal feature.

[0083] In the embodiment of the application, the lecturer may speak too fast and the tone of the lecturer may be raised when explaining knowledge. For example, when the lecturer is explaining a core knowledge point, the lecturer's speech speed is detected to be too fast (more than 190 words per minute) and the tone is raised (more than 200 HZ). The students may not be able to understand and absorb the information in time, resulting in a decline in teaching quality. Therefore, the current teaching behavior needs to be marked as an abnormal feature of "explanation stall" to avoid the students missing the understanding of the knowledge point.

[0084] In an optional embodiment, there may also be a time delay error between the explanation gesture and the courseware flipping action. For example, in a financial course, the lecturer explains the chart in the PPT by gesture, but the corresponding courseware is not switched synchronously, so that the time delay error between the explanation gesture and the courseware flipping action exceeds the preset 1.2 seconds, resulting in the students being unable to understand the current explanation of the lecturer or missing the key courseware information. Therefore, the current teaching behavior needs to be marked as an abnormal feature of "incoordinated courseware switching".

[0085] Additionally, there may be a situation that the lecturer's courseware flipping interval is too short (for example, less than the preset 5S time) and there is no corresponding gesture to emphasize, resulting in too fast switching of the teaching content and the students being unable to understand in time. Similarly, the teaching behavior can be marked as an abnormal feature of "fast teaching rhythm".

[0086] Of course, other abnormal features that may occur in the teaching process in addition to the above-mentioned "explanation stall", "incoordinated courseware switching" and "fast teaching rhythm" can also be monitored. The specific detection standard of the abnormal features is not limited too much in the application, and a person skilled in the art can adjust it according to the actual situation.

[0087] As an optional implementation, please refer to Figure 6 The flowchart for constructing the contradiction index in the teaching recommendation method based on multi-modal fusion provided in the embodiment of the application, the above-mentioned detection of the abnormal features in the teaching behavior data further comprises:

[0088] Step S32: obtaining interaction information in the learning interaction data; wherein the interaction information comprises negative interaction information and positive interaction information;

[0089] Step S33: constructing a contradiction index based on the positive interaction information and the negative physiological signal;

[0090] Step S34: When the contradiction index is lower than a preset index threshold, the teaching behavior corresponding to the current positive interaction information is marked as an abnormal feature.

[0091] In the embodiments of the present application, the physiological signal data can be further supplemented by interaction information. For example, the interaction information in the learning interaction data can be obtained, and the interaction information can be classified by combining an NLP (Natural Language Processing Model) to avoid the phenomenon that a student shows positivity but is actually confused.

[0092] For example, the text features in the interaction information can be extracted by the NLP model, and the extracted text features can be classified into negative interaction information and positive interaction information. For example, the interaction information such as "can't understand", "too fast", and "can't keep up" is classified as negative interaction information, and the interaction information such as "understood", "clear", and "understood" is classified as positive interaction information.

[0093] Of course, a preset interaction information library can also be provided to classify common interaction information into negative interaction information and positive interaction information in advance, and the interaction information fed back by the student during the course can be directly classified. The present application does not make too much description on the detailed steps of how to classify the extracted text features into negative interaction information and positive interaction information by the NLP model.

[0094] When the positive interaction information and the negative physiological signal are obtained, a contradiction index C is constructed by combining the positive interaction information and the negative physiological signal, wherein C = number of positive interaction information / number of negative physiological signal; when the contradiction index C is lower than a preset index threshold (for example, lower than 0.3), the teaching behavior corresponding to the current positive interaction information is marked as an abnormal feature of "contradictory behavior".

[0095] In this way, the phenomenon that a student shows positivity but the actual learning effect is poor can be found in time. For example, in a medical course, students send positive pop-up windows to maintain the classroom atmosphere when the lecturer demonstrates the operation, but the physiological signal shows confusion; or a large amount of positive interaction information such as "the teacher really carefully" appears in the pop-up window, but the HRV data shows that the heart rate of 70% of the students continues to rise, avoiding relying only on positive interaction information for feedback and ignoring the real learning status of the students.

[0096] Step S4: When the abnormal feature appears, whether the proportion of the students in the student group who trigger the learning interaction at the same time reaches a preset proportion is determined, and an intervention instruction is generated when the preset proportion is reached.

[0097] As an optional implementation, the determination of whether the proportion of the students in the student group who trigger the learning interaction at the same time reaches a preset proportion when the abnormal feature appears and the generation of the intervention instruction when the preset proportion is reached include:

[0098] When the abnormal feature appears, it is judged whether the proportion of the video pause behavior and the video review behavior triggered synchronously in the student group reaches a preset proportion, and the intervention instruction is generated when the preset proportion is reached.

[0099] In the embodiments of the present application, when the abnormal feature appears in the teaching behavior data, the behavior of triggering learning interaction in the student group is monitored, and the video pause behavior and the video review behavior are monitored in the student group. The proportion of triggering the above-mentioned video pause behavior and video review behavior in the student group is counted, and when the group triggering proportion exceeds the preset proportion (such as more than 30%), it is indicated that the abnormal feature has universality, and the intervention instruction is generated; when the group triggering proportion does not reach the preset proportion, it is indicated that the abnormal feature is the behavior of individual students, and it is not necessary to intervene temporarily.

[0100] It should be noted that, since the time correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data has been established in step S3, it can be judged whether the above-mentioned video pause behavior and video review behavior are concentrated within a short time (such as within 10S) after the abnormal feature appears, avoiding the video pause behavior and video review behavior in the non-related period being included in the statistics, so that the accidental phenomenon of individual students pausing the video due to temporary phone call does not interfere with the learning situation of the student group.

[0101] It can be understood that, the present application forms a double verification through the teaching abnormal feature and the behavior response of the student group, avoids the influence of accidental phenomenon of individual students on the student group, and improves the reliability of the intervention instruction.

[0102] Taking a medical course as an example, if only 5% of the students pause the operation step explanation after the abnormal feature appears in the process of explaining a certain operation video, it may be that the content is difficult for beginners and belongs to a reasonable range, and if 35% of the students synchronously pause the behavior, it indicates that there is a universal problem in the current explanation method, and the intervention instruction needs to be generated immediately to remind the lecturer to intervene in time.

[0103] Step S5: Real-time adjustment of the teaching behavior and the teaching resource recommendation strategy of the lecturer based on the intervention instruction.

[0104] As an optional implementation manner, the real-time adjustment of the teaching behavior and the teaching resource recommendation strategy of the lecturer based on the intervention instruction includes:

[0105] Based on the intervention instruction, the lecturer is reminded to adjust the explanation speed, the explanation tone, the explanation gesture or the courseware page turning action, and supplementary materials are pushed to each student within a preset time.

[0106] In the embodiments of the present application, when the intervention instruction is generated, the lecturer will be reminded to adjust the current teaching behavior in time, and the supplementary materials will be pushed to each student within a preset time (such as 5S).

[0107] For example, refer to Figure 7 In the teaching recommendation method based on multi-modal fusion provided by the embodiments of the present application, when the abnormal feature of "stall explanation" is detected, the lecturer can be reminded to adjust the explanation speed and tone, such as adjusting the speed to within 190 words / minute and lowering the tone to 200HZ; when the abnormal feature of "incoordinated courseware switching" is detected, the lecturer can be reminded to adjust the time delay between the explanation gesture and the courseware flipping action; when the abnormal feature of "too fast teaching rhythm" is detected, the lecturer can be reminded to adjust the speed of teaching content switching, so that the lecturer can be reminded in time to adjust the specific teaching behavior through the intervention instruction.

[0108] Further, when the intervention is needed, the system will also dynamically generate and push the supplementary materials to each student within a preset time, such as the knowledge point graphic analysis, animation demonstration, and case expansion that are adapted to the current teaching node, so as to consolidate or strengthen the understanding of the current knowledge of the students, and avoid the students from giving up learning due to information overload.

[0109] In an optional embodiment, the present application will also evaluate the examination results of the students every month, record the examination results and progress of the students before and after learning, and quantify the actual teaching quality of each lecturer, so as to adjust the recommendation priority of the lecturer on the teaching platform.

[0110] Preferably, the present application recommends the courses of the lecturers with good examination result improvement effect, and preferentially recommends and displays the lecturers with good examination result improvement effect, so as to optimize the resource allocation of the lecturers according to the actual teaching ability of each lecturer, and encourage the lecturers to improve the teaching quality.

[0111] It can be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0112] The application provides a teaching recommendation method based on multi-modal fusion, which acquires teaching behavior data of a lecturer, synchronously collects learning interaction data and physiological signal data of students, acquires multi-dimensional data information from real-time feedback of the lecturer and the students, establishes time correlation mapping for the teaching behavior data, the learning interaction data and the physiological signal data, accurately captures the corresponding relationship of the teaching behavior and the student reaction on a time axis, judges whether the proportion of synchronous triggering of learning interaction in a student group reaches a preset proportion when an abnormal feature is detected, filters accidental behavior of individual students to avoid invalid intervention, generates an intervention instruction when the preset proportion is reached, and adjusts the teaching behavior of the lecturer and a teaching resource recommendation strategy in real time based on the intervention instruction, so that the teaching process can be dynamically calibrated according to the actual reaction of the students, and the teaching efficiency and the learning effect are effectively improved.

[0113] Based on the above-mentioned teaching recommendation method based on multi-modal fusion, the application provides a teaching recommendation system based on multi-modal fusion, please refer to Figure 8 The application provides a teaching recommendation system based on multi-modal fusion, please refer to

[0114] The acquisition unit is used for acquiring teaching behavior data of a lecturer in a teaching video.

[0115] The acquisition unit is used for acquiring teaching behavior data of a lecturer in a teaching video.

[0116] The detection unit is used for establishing time correlation mapping among the teaching behavior data, the learning interaction data and the physiological signal data, and detecting abnormal features in the teaching behavior data.

[0117] The generation unit is used for judging whether the proportion of synchronous triggering of learning interaction in a student group reaches a preset proportion when the abnormal features appear, and generating an intervention instruction when the preset proportion is reached.

[0118] The adjustment unit is used for adjusting the teaching behavior of the lecturer and a teaching resource recommendation strategy in real time based on the intervention instruction.

[0119] As an optional implementation, the above-mentioned teaching behavior data at least includes explanation speed, explanation tone, explanation gesture and courseware flipping action of the lecturer in the teaching video.

[0120] The learning interaction data at least includes video pause behavior, video review behavior and interactive information sending behavior of the students.

[0121] The physiological signal data at least includes positive physiological signal and negative physiological signal of the students.

[0122] As an optional implementation, the detecting of the abnormal feature in the teaching behavior data comprises:

[0123] When the explanation speed of the lecturer is accelerated and the explanation tone is raised during the teaching, the current teaching behavior is marked as an abnormal feature.

[0124] When there is a time delay error between the explanation gesture of the lecturer and the course flipping action, the current teaching behavior is marked as an abnormal feature.

[0125] As an optional implementation, the detecting of the abnormal feature in the teaching behavior data further comprises:

[0126] Obtaining interaction information in the learning interaction data; wherein, the interaction information comprises negative interaction information and positive interaction information;

[0127] Constructing a contradiction index based on the positive interaction information and the negative physiological signal;

[0128] When the contradiction index is lower than a preset index threshold, the teaching behavior corresponding to the current positive interaction information is marked as an abnormal feature.

[0129] As an optional implementation, when the abnormal feature occurs, the present application judges whether the proportion of the simultaneous triggering of learning interaction in the student group reaches a preset proportion, and generates an intervention instruction when the preset proportion is reached, comprising:

[0130] When the abnormal feature occurs, it is judged whether the proportion of the simultaneous triggering of the video pause behavior and the video review behavior in the student group reaches a preset proportion, and the intervention instruction is generated when the preset proportion is reached.

[0131] As an optional implementation, the present application adjusts the teaching behavior and teaching resource recommendation strategy of the lecturer in real time based on the intervention instruction, comprising:

[0132] Based on the intervention instruction, the lecturer is reminded to adjust the teaching behavior, and supplementary materials are pushed to each student within a preset time.

[0133] As an optional implementation, the present application establishes a time correlation mapping between the teaching behavior data, the learning interaction data and the physiological signal data, comprising:

[0134] Based on the Bi-LSTM model, a time correlation mapping is established between the teaching behavior data, the learning interaction data and the physiological signal data.

[0135] Other details of the implementation of the technical solutions of the units in the teaching recommendation system based on multi-modal fusion provided in the above embodiments can be found in the description of the teaching recommendation method based on multi-modal fusion in the above embodiments, which will not be repeated here.

[0136] It should be noted that each of the embodiments in the present specification adopts a progressive manner for description, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0137] Please refer to Figure 9 The computer device 90 provided in the embodiments of the present application includes a processor 91 and a memory 92 coupled with the processor 91.

[0138] The memory 92 stores a computing program, and the computer program is executed by the processor 91 to make the processor 91 execute the steps of the teaching recommendation method based on multi-modal fusion in the above embodiments.

[0139] The processor 91 can also be referred to as a CPU (Central Processing Unit). The processor 91 can be an integrated circuit chip with a processing capability. The processor 91 can also be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor.

[0140] Please refer to Figure 10The computer readable storage medium of the embodiment of the present application stores a computer program 100, the computer program 100 is executed by a processor to implement the artificial intelligence-based actuarial sub-method in the above embodiment, wherein the computer program 100 can be stored in the above storage medium in the form of a software product, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, or a computer, a server, a mobile phone, a tablet computer, etc. The server can be a stand-alone server, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0141] In several embodiments provided in the present application, it should be understood that the disclosed terminal, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0142] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a software functional unit. The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

[0143] The above embodiments are only exemplary embodiments adopted for illustrating the principles of the embodiments of the present application, and the embodiments of the present application are not limited thereto. Various modifications and improvements can be made by those of ordinary skill in the art without departing from the spirit and principle of the embodiments of the present application, and these modifications and improvements are also considered to be within the scope of protection of the embodiments of the present application.

Claims

1. A teaching recommendation method based on multimodal fusion, characterized in that: The following steps are involved: Obtain the teaching behavior data of the lecturer in the teaching video; Synchronously collect each student's learning interaction data and physiological signal data during the lecture; Establishing a time correlation mapping between the teaching behavior data, the learning interaction data, and the physiological signal data, and detecting abnormal features in the teaching behavior data; Determine whether the proportion of students in the group who have synchronously triggered learning interactions has reached a preset proportion when the abnormal characteristics occur, and generate intervention instructions when the preset proportion is reached; The lecturer's teaching behavior and teaching resource recommendation strategy are adjusted in real time based on the intervention instructions.

2. The teaching recommendation method based on multimodal fusion according to claim 1, characterized in that: The teaching behavior data includes at least the lecturer's speaking speed, tone, gestures, and page-turning actions in the teaching video; The learning interaction data includes at least each student's video pausing behavior, video replaying behavior, and interactive message sending behavior; The physiological signal data at least includes positive physiological signals and negative physiological signals of each student.

3. The teaching recommendation method based on multimodal fusion according to claim 2, characterized in that: The detecting abnormal features in the teaching behavior data includes: When the lecturer's speaking speed increases and the pitch rises during teaching, the current teaching behavior is marked as abnormal; And / or, when there is a time delay error between the lecturer's explanation gesture and the courseware page turning action, the current teaching behavior is marked as an abnormal feature.

4. The teaching recommendation method based on multimodal fusion according to claim 2, characterized in that: The detecting of abnormal features in the teaching behavior data further includes: Acquiring interactive information from the learning interactive data; wherein the interactive information includes negative interactive information and positive interactive information; constructing a conflict index based on the positive interaction information and the negative physiological signal; When the contradiction index is lower than a preset index threshold, the teaching behavior corresponding to the current positive interaction information is marked as an abnormal feature.

5. The teaching recommendation method based on multimodal fusion according to claim 2, characterized in that: The determining of whether the proportion of synchronously triggered learning interactions in the student group reaches a preset proportion when the abnormal feature occurs, and generating an intervention instruction when the preset proportion is reached, includes: When the abnormal feature appears, it is determined whether the proportion of the student group that simultaneously triggers the video pause behavior and the video replay behavior reaches a preset proportion, and the intervention instruction is generated when the preset proportion is reached.

6. The teaching recommendation method based on multimodal fusion according to claim 2, characterized in that: The real-time adjustment of the lecturer's teaching behavior and teaching resource recommendation strategy based on the intervention instruction includes: Based on the intervention instructions, the instructor is reminded to adjust the teaching behavior and push supplementary materials to each student within a preset time.

7. The teaching recommendation method based on multimodal fusion according to claim 1, characterized in that: The step of establishing a time correlation mapping between the teaching behavior data, the learning interaction data, and the physiological signal data includes: A time correlation mapping is established between the teaching behavior data, the learning interaction data and the physiological signal data based on the Bi-LSTM model.

8. A teaching recommendation system based on multimodal fusion, characterized in that: include: An acquisition unit, used to acquire the teaching behavior data of the lecturer in the teaching video; The acquisition unit is used to synchronously collect the learning interaction data and physiological signal data of each student during the lecture; a detection unit, configured to establish a time correlation mapping between the teaching behavior data, the learning interaction data, and the physiological signal data, and detect abnormal features in the teaching behavior data; a generating unit, configured to determine whether the proportion of synchronously triggered learning interactions in a group of learners reaches a preset proportion when the abnormal feature occurs, and to generate an intervention instruction when the preset proportion is reached; An adjustment unit is used to adjust the lecturer's teaching behavior and teaching resource recommendation strategy in real time based on the intervention instruction.

9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein a computing program is stored in the memory. When the computer program is executed by the processor, the processor performs the steps of the teaching recommendation method based on multimodal fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the teaching recommendation method based on multimodal fusion according to any one of claims 1 to 7.

Images