A classroom interaction analysis method and system based on multimodal fusion
Through multimodal fusion technology, combining voiceprints and human posture characteristics, a classroom interactive behavior coding system is built, which solves the problems of complex and inefficient manual analysis, and achieves more accurate and efficient classroom interactive behavior recognition and analysis.
Patent Information
- Application Number
- CN202210539253.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-05-18
AI Technical Summary
The existing classroom interactive analysis methods rely on manual methods, and there are problems such as subjective factors that have a large impact, complex operation, single data modality, and low efficiency.
A classroom interaction analysis method based on multimodal fusion is adopted, through voiceprint feature extraction and human posture estimation, an interactive behavior coding system is constructed in combination with educational theory, and a classroom interactive behavior analysis is performed by fusing voiceprint and action features.
Reduce the analytical burden of teachers, improve the speed and accuracy of analysis, provide more comprehensive and detailed classroom interactive behavior recognition, and help teachers understand teaching situations in a timely manner.
Smart Images

Figure CN114970701B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of educational informatization technology, and more specifically, relates to a classroom interaction analysis method and system based on multimodal fusion. Background Art
[0002] Voiceprints are similar to biometric information such as fingerprints, facial features, walking posture, pupils and irises, and can be used to identify a person's physiological attributes. Since each person's vocal organs have innate differences, and speaking habits vary due to differences in acquired learning and life, different voices have their own unique characteristics. Voiceprint Recognition (VPR), also known as Speaker Recognition (SR), is a biometric recognition technology that collects human voices through sound recording equipment and then processes and analyzes them with computers to identify the speaker's identity. This technology integrates knowledge from multiple disciplines, including signal processing, pattern recognition, natural language processing and expression, and is widely used in finance, public safety, smart security, smart education and other fields.
[0003] Posture estimation is an algorithm that uses computer vision recognition, wearable sensors and other methods to locate human joints and thus determine human posture. By calculating the coordinates of each joint point of the human body, the human skeleton features can be used to track, identify, and classify people. At present, posture estimation algorithms based on deep learning are divided into two categories according to the number of detection targets: single-person posture estimation and multi-person posture estimation. Among them, multi-person posture estimation is more suitable for real scenes and is also a research hotspot of major research institutions.
[0004] According to the principle, multi-person pose estimation can be divided into two different estimation methods: top-down (Two-Step) and bottom-up (Part-Base). The top-down detection order is to first detect the relative position of each person in the image, and then detect each person's body joints to determine the pose; while bottom-up means first detecting all body joints in the environment, and then connecting the joints according to the body structure. The above two recognition methods have their own advantages and disadvantages: the top-down detection accuracy is higher, but it is highly dependent on the detection quality of the human body bounding box, that is, the positioning of the detection box must be accurate; while the bottom-up method does not require such precise positioning to complete most of the pose estimation, but because the link of the joint point connection is later than the target detection, when two or more different people are very close in the image, the connection of the joint point will be ambiguous, and even misjudgment or omission will occur. In the classroom teaching video, the people in the classroom are dense and the distribution is relatively chaotic, and the relative positions of the people change frequently. The judgment of the main behavior in the classroom requires accurate human recognition results, and the bottom-up method is difficult to distinguish the occluded or overlapping human postures. Therefore, the top-down detection method is used to perform high-precision classroom human posture estimation.
[0005] Modality is a specific way in which things are observed or perceived. For example, the content of a class can be represented by the voice signal between teachers and students, and sound is a modality; it can be represented by the picture in the video, and the image is a modality; it can also be represented by the text content of the conversation between teachers and students, and text is a modality. Multi-modal fusion is also called multi-sensor information fusion or multi-source information fusion. Generally, according to the different fusion levels, fusion strategies can be divided into three categories: data layer fusion, feature layer fusion, and decision layer fusion. Data between the same thing and different modalities are generally multi-source heterogeneous data. Because the expression forms or characteristics of different modalities are different, the angles of expressing things will also be different. There will be information complementarity between data. Data from multiple sources are collected and summarized, and then the correlation between the data is used for joint representation, so that multi-modal analysis is more comprehensive than single-modal analysis.
[0006] Since the rise of classroom video analysis technology in the 1980s, studying the teaching process through video analysis has become a hot topic in educational informatization. In order to adapt to the analysis of classroom teaching behavior in an information-based teaching environment, coding systems with information technology characteristics have been continuously proposed. Typical classroom interaction analysis methods include the Flanders Interaction Analysis System (FIAS) and ST classroom teaching analysis. In addition, methods such as the Information Technology-based Interaction Analysis System (ITIAS), the improved Flanders Interaction Analysis System i-FIAS (improved FIAS), and the Verbal Interaction Classification System (VICS) provide certain support for classroom interaction behavior analysis and classroom effect evaluation.
[0007] The FIAS system uses manual observation and coding and has high requirements for observers. The classroom behavior coding process is complicated, the workload is large, it is mainly based on language analysis, and the coding system is imperfect. The i-FIAS system is an improvement on the FIAS system, but it focuses on language interaction in the classroom teaching process and lacks recording and analysis of details of teacher and student behavior. The ITIAS system is relatively one-sided in its classification of teachers' language, and the coding at the technical level is too simple. It only stays at the level of observing teachers or students using technology, ignoring the rich connotations of technology use.
[0008] In summary, in traditional classroom behavior analysis methods, most of the analysis process is completed manually, and automated methods have not yet been used. There are also problems such as single data modality, complex operation, and low efficiency. Summary of the invention
[0009] In view of the defects of the prior art, the purpose of the present invention is to provide a classroom interaction analysis method and system based on multimodal fusion, aiming to solve the problem that the results of manual feedback analysis of classroom videos are greatly affected by subjective factors and the process operation is complicated.
[0010] To achieve the above objectives, in a first aspect, the present invention provides a classroom interaction analysis method based on multimodal fusion, comprising the following steps:
[0011] Identify the classroom teaching videos to be analyzed;
[0012] Extracting voiceprint features from the classroom teaching video, and determining the speaker identity and speaking time corresponding to each voiceprint feature through a speaker recognition clustering algorithm;
[0013] Performing target detection on the classroom video to detect a human body; and performing posture estimation on the detected human body to determine key motion features of the human body; the key motion features include: standing posture, hand posture, and face facing posture;
[0014] The voiceprint features and key human movement features are integrated, and combined with a classroom interaction behavior coding system constructed in advance according to relevant educational theories and the integrated semantic features of teacher-student interaction behaviors, the classroom interaction behaviors of the classroom teaching video are analyzed; the specific reference principles for constructing the classroom interaction behavior coding system include: within a certain time period, after a student is detected speaking, if no one is detected standing, it is considered that the student is sitting to answer questions or give a speech; if a person is detected standing and there is no hand movement, it is considered that the student is standing to answer questions or give a speech; if a person is detected standing and there is hand movement, it is considered that the student is making gestures to explain the content; if a person is detected standing and raises his arm and points to the side, it is considered that the student is pointing to the display content or operating the electronic whiteboard; if a person is detected standing and raises his arm to the side facing the camera If a person points his hand to the side, it is considered that the student is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is on the podium listening to students answering questions or giving speeches. Within a certain period of time, after the teacher is detected speaking, if no person is detected standing, it is considered that the teacher is sitting and lecturing or asking questions. If one person is detected standing and there is no hand movement, it is considered that the teacher is standing and lecturing or asking questions. If one person is detected standing and there is hand movement, it is considered that the teacher is gesturing to explain the content. If one person is detected standing and facing the camera, raising his arm and pointing his hand to the side, it is considered that the teacher is pointing to the display content and explaining or writing on the blackboard. If one person is detected standing and facing the side of the camera, raising his arm and pointing his hand to the side, it is considered that the teacher is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is commenting on or instructing the students who are giving speeches.
[0015] In an optional example, the posture estimation of the detected human body is performed to determine the key motion features of the human body, specifically:
[0016] Select multiple key nodes of the human body, the key nodes include: head key nodes and body key nodes; the head key nodes include right ear, left ear, right eye, left eye and nose, the body key nodes include: neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee and left ankle;
[0017] The line between the right hip and the neck is taken as the first vector, the line between the right hip and the right knee is taken as the second vector, and the angle between the first vector and the second vector is taken as the angle between the torso and the right hip bone;
[0018] The line between the left hip and the neck is taken as the third vector, the line between the left hip and the left knee is taken as the fourth vector, and the angle between the third vector and the fourth vector is taken as the angle between the torso and the left hip bone;
[0019] When the angle between the trunk and the right hip bone and the angle between the trunk and the left hip bone are both in the range of 160°-180°, the human body is considered to be standing; otherwise, the human body is considered to be sitting.
[0020] In an optional example, the line between the right shoulder and the left shoulder is taken as the fifth vector, the line between the right shoulder and the right elbow is taken as the sixth vector, and the angle between the fifth vector and the sixth vector is taken as the right shoulder elbow angle;
[0021] The line between the left shoulder and the right shoulder is taken as the seventh vector, the line between the left shoulder and the left elbow is taken as the eighth vector, and the angle between the seventh vector and the eighth vector is taken as the left shoulder elbow angle;
[0022] The line between the right elbow and the right shoulder is taken as the ninth vector, the line between the right elbow and the right wrist is taken as the tenth vector, and the angle between the ninth vector and the tenth vector is taken as the right elbow-wrist angle;
[0023] The line between the left elbow and the left shoulder is taken as the eleventh vector, the line between the left elbow and the left wrist is taken as the twelfth vector, and the angle between the eleventh vector and the twelfth vector is taken as the left elbow-wrist angle;
[0024] When the shoulder-elbow angle on one side ranges between 120°-165°, the arm on that side is considered raised;
[0025] When the elbow-wrist angle on one side ranges from 65° to 110°, the elbow on that side is considered to be bent;
[0026] When one arm is raised and the elbow on that side is bent, the human body is considered to have a hand-raising gesture;
[0027] When one arm is raised and the elbow on that side is not bent, the person is considered to have raised his arm and pointed to the side;
[0028] When both arms are not raised, if both elbows are bent, the body is considered to be gesturing;
[0029] When both arms are not raised, if only one elbow is bent, the person is considered to have raised hand gesture;
[0030] When both arms are not raised and both elbows are not bent, it is considered that the body is not moving.
[0031] In an optional example, the eye vector is determined according to the line connecting the left eye and the right eye, and the slope of the perpendicular bisector of the eye vector is calculated; when the absolute value of the slope of the perpendicular bisector is between [1.6, +∞), or the slope cannot be calculated, it is considered that the human body is facing the front of the camera lens;
[0032] When the absolute value of the slope of the perpendicular bisector is between [0, 1.6), the human body is considered to be facing the side of the lens; or when only one of the two nodes of the left ear and the right ear of the human body is detected, or only one of the two nodes of the left eye and the right eye is detected, it is determined that the key nodes required to calculate the facial orientation are missing, and the human body is considered to be facing the side of the lens.
[0033] In an optional example, the specific reference principles for constructing the classroom interactive behavior coding system also include: if no speaker is detected and no key human body movement features are detected within a certain time period, it is considered that the classroom is silent or chaotic during this period.
[0034] In a second aspect, the present invention provides a classroom interaction analysis system based on multimodal fusion, comprising:
[0035] A teaching video determination unit, used to determine the classroom teaching video to be analyzed;
[0036] A voiceprint feature extraction unit, used to extract voiceprint features in the classroom teaching video, and determine the speaker identity and speaking time corresponding to each voiceprint feature through a speaker recognition clustering algorithm;
[0037] The motion feature detection unit is used to perform target detection on the classroom video to detect a human body; and to perform posture estimation on the detected human body to determine key motion features of the human body; the key motion features include: standing posture, hand posture and face facing posture;
[0038] The classroom interaction analysis unit is used to fuse the voiceprint features and key human movement features, and analyze the classroom interaction behavior of the classroom teaching video in combination with a classroom interaction behavior coding system constructed in advance according to relevant educational theories and the integrated semantic features of teacher-student interaction behaviors; the specific reference principles for constructing the classroom interaction behavior coding system include: within a certain time period, after a student is detected speaking, if no one is detected standing, it is considered that the student is sitting to answer questions or give a speech; if a person is detected standing and there is no hand movement, it is considered that the student is standing to answer questions or give a speech; if a person is detected standing and there is hand movement, it is considered that the student is making gestures to explain the content; if a person is detected standing and raises his arm and points to the side, it is considered that the student is pointing to the display content or operating the electronic whiteboard; if a person is detected standing and facing the side of the camera If a student raises his arm and points to the side, it is considered that the student is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is on the podium listening to students answering questions or giving speeches. Within a certain period of time, after the teacher is detected speaking, if no one is detected standing, it is considered that the teacher is sitting and lecturing or asking questions. If one person is detected standing and there is no hand movement, it is considered that the teacher is standing and lecturing or asking questions. If one person is detected standing and there is hand movement, it is considered that the teacher is gesturing to explain the content. If one person is detected standing and facing the camera and raising his arm and pointing to the side, it is considered that the teacher is pointing to the display content and explaining or writing on the blackboard. If one person is detected standing and facing the side of the camera and raising his arm and pointing to the side, it is considered that the teacher is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is commenting on or instructing the students who are giving speeches.
[0039] In an optional example, the motion feature detection unit selects multiple human key nodes, the key nodes include: head key nodes and body key nodes; the head key nodes include right ear, left ear, right eye, left eye and nose, and the body key nodes include: neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee and left ankle; the line between the right hip and the neck is used as the first vector, the line between the right hip and the right knee is used as the second vector, and the angle between the first vector and the second vector is used as the angle between the trunk and the right hip bone; the line between the left hip and the neck is used as the third vector, the line between the left hip and the left knee is used as the fourth vector, and the angle between the third vector and the fourth vector is used as the angle between the trunk and the left hip bone; when the angle between the trunk and the right hip bone and the angle between the trunk and the left hip bone are both in the range of 160°-180°, the human body is considered to be standing; otherwise, the human body is considered to be sitting.
[0040] In an optional example, the motion feature detection unit uses the line between the right shoulder and the left shoulder as the fifth vector, the line between the right shoulder and the right elbow as the sixth vector, and the angle between the fifth vector and the sixth vector as the right shoulder-elbow angle; uses the line between the left shoulder and the right shoulder as the seventh vector, the line between the left shoulder and the left elbow as the eighth vector, and the angle between the seventh vector and the eighth vector as the left shoulder-elbow angle; uses the line between the right elbow and the right shoulder as the ninth vector, the line between the right elbow and the right wrist as the tenth vector, and the angle between the ninth vector and the tenth vector as the right elbow-wrist angle; uses the line between the left elbow and the left shoulder as the eleventh vector, the line between the left elbow and the left wrist as the twelfth vector, and the eleventh vector and the left wrist as the twelfth vector. The angle between the twelfth vector is taken as the left elbow-wrist angle; when the shoulder-elbow angle of one side is between 120°-165°, the arm on that side is considered to be raised; when the elbow-wrist angle of one side is between 65°-110°, the elbow on that side is considered to be bent; when one arm is raised and the elbow on that side is bent, it is considered that the human body has raised his hand; when one arm is raised and the elbow on that side is not bent, it is considered that the human body has raised his arm and pointed to the side; when both arms are not raised, if both elbows are bent, it is considered that the human body is making a gesture; when both arms are not raised, if only one elbow is bent, it is considered that the human body has raised his hand; when both arms are not raised, if both elbows are not bent, it is considered that the human body has no movement.
[0041] In an optional example, the motion feature detection unit determines the eye vector based on the line connecting the left eye and the right eye, and calculates the slope of the perpendicular bisector of the eye vector; when the absolute value of the slope of the perpendicular bisector is between [1.6, +∞), or the slope cannot be calculated, it is considered that the human body is facing the front of the camera lens; when the absolute value of the slope of the perpendicular bisector is between [0, 1.6), it is considered that the human body is facing the side of the camera lens; or when only one of the two nodes of the left ear and the right ear of the human body is detected, or only one of the two nodes of the left eye and the right eye is detected, it is determined that the key node required for calculating the facial orientation is missing, and it is considered that the human body is facing the side of the camera lens.
[0042] In an optional example, the classroom interaction analysis unit constructs the classroom interaction behavior coding system with specific reference principles, which also include: within a certain time period, if no speaker is detected and no key human body movement features are detected, the classroom is considered to be silent or chaotic during this period.
[0043] In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the prior art:
[0044] The present invention provides a classroom interaction analysis method and system based on multimodal fusion, and provides an intelligent recognition method based on multimodal feature fusion for the problem of intelligent recognition and classification of classroom video interactive behaviors. This design can reduce the burden of teachers when conducting classroom feedback analysis after class, simplify operations using intelligent technology, speed up analysis, and help teachers obtain classroom teaching conditions in a timely manner.
[0045] The present invention provides a classroom interaction analysis method and system based on multimodal fusion, which takes image data and audio data into consideration at the same time. Fusion of the above two multimodal features of different dimensions can fully mine the information in the classroom in the process of describing classroom interactive behaviors, refine the classification of behaviors, and improve the accuracy of behavior recognition.
[0046] The present invention provides a classroom interaction analysis method and system based on multimodal fusion. In view of the characteristics of diversified classroom teaching modes and rich interactive behaviors, the present invention integrates and analyzes key information according to technologies such as voiceprint recognition and human posture estimation, proposes a classroom interaction analysis method and system based on multimodal fusion, and compiles an interactive behavior analysis system MFIAS based on multimodal fusion to solve practical problems in classroom teaching interaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of a classroom interaction analysis method based on multimodal fusion provided by an embodiment of the present invention;
[0048] Figure 2 It is a classroom interaction analysis framework diagram based on multimodal fusion provided by an embodiment of the present invention;
[0049] Figure 3 It is a flow chart of classroom teaching interactive behavior analysis provided by an embodiment of the present invention;
[0050] Figure 4 This is a diagram showing the effects of manual marking and algorithm recognition for classroom behavior recognition provided by an embodiment of the present invention.
[0051] Figure 5 It is an architecture diagram of a classroom interaction analysis system based on multimodal fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0053] In view of the defects and shortcomings of the existing technology, the present invention proposes a classroom interaction analysis method and system based on multimodal fusion, which aims to solve the problem that the results of teachers' manual feedback analysis of classroom videos are greatly affected by subjective factors and the process operation is complicated, as well as the problems existing in the above-mentioned classroom interaction analysis system. The present invention can help teachers analyze teaching situations such as teacher-student interaction and classroom structure more effectively, quickly and in a targeted manner, improve students' high-order thinking ability, and promote the occurrence of deep learning.
[0054] Figure 1 is a flow chart of a classroom interaction analysis method based on multimodal fusion provided by an embodiment of the present invention; Figure 1 As shown, the following steps are included:
[0055] S101, determining the classroom teaching video to be analyzed;
[0056] S102, extracting voiceprint features from the classroom teaching video, and determining the speaker identity and speaking time corresponding to each voiceprint feature through a speaker recognition clustering algorithm;
[0057] S103, performing target detection on the classroom video to detect a human body; and performing posture estimation on the detected human body to determine key motion features of the human body; the key motion features include: standing posture, hand posture, and face facing posture;
[0058] S104, integrating the voiceprint features and key human motion features, and combining with a classroom interaction behavior coding system constructed in advance according to relevant educational theories and the integrated semantic features of teacher-student interaction behaviors, analyzing the classroom interaction behaviors corresponding to the classroom teaching video; the specific reference principles for constructing the classroom interaction behavior coding system include: within a certain time period, after detecting that a student is speaking, if no one is detected standing, it is considered that the student is sitting to answer questions or give a speech; if a person is detected standing and there is no hand movement, it is considered that the student is standing to answer questions or give a speech; if a person is detected standing and there is hand movement, it is considered that the student is making gestures to explain the content; if a person is detected standing and raising his arm and pointing to the side, it is considered that the student is pointing to display content or operating an electronic whiteboard; if a person is detected standing and raising his arm to the side facing the camera, it is considered that the student is standing to answer questions or give a speech; if a person is detected standing and raising his arm to the side facing the camera, it is considered that the student is standing to explain the content; if a person is detected standing and raising his arm to the side facing the camera, it is considered that the student is pointing to display content or operating an electronic whiteboard; if a person is detected standing and raising his arm to the side facing the camera, it is considered that the student is standing to answer questions or give a speech ... If a person points his arm to the side, it is considered that the student is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is on the podium listening to students answering questions or giving speeches. Within a certain period of time, after the teacher is detected speaking, if no person is detected standing, it is considered that the teacher is sitting and lecturing or asking questions. If one person is detected standing and there is no hand movement, it is considered that the teacher is standing and lecturing or asking questions. If one person is detected standing and there is hand movement, it is considered that the teacher is gesturing to explain the content. If one person is detected standing and facing the camera with his arm raised and hand pointing to the side, it is considered that the teacher is pointing to the display content and explaining or writing on the blackboard. If one person is detected standing and facing the side of the camera with his arm raised and hand pointing to the side, it is considered that the teacher is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is commenting on or instructing the students who are giving speeches.
[0059] In a specific embodiment, the present invention provides a classroom interaction analysis method based on multimodal fusion, such as Figure 2 As shown, the method includes:
[0060] Use the microphone and high-definition camera in the classroom to obtain the classroom teaching video data to be identified;
[0061] Using voice activity detection and voiceprint recognition technology, extracting voiceprint features from the classroom teaching video data to be identified according to different speakers;
[0062] Use target detection and posture estimation technology to perform human action recognition on video data and extract key action features of the human body in the picture (hand posture, standing / sitting posture, human body orientation);
[0063] Based on classroom observations, we analyzed the similarities and differences between different behavioral categories and established the Interactive Behavior Coding Table (MFIAS) to uniformly describe all classroom interactive behaviors.
[0064] The obtained voiceprint features and action features are fused to generate video behavior feature description values arranged in time. A mapping relationship table is established based on the description values and the coding table to realize the transformation of multimodal features into interactive behaviors.
[0065] The final classroom teaching video interactive behavior sequence is obtained according to the mapping relationship table.
[0066] The classroom interaction analysis method and system based on multimodal fusion provided by the present invention solves the problems that the results of teachers' manual feedback analysis of classroom videos are greatly affected by subjective factors and the process operation is complicated. It explores and expands the research field of classroom teaching behavior analysis from a new perspective, and further provides support for better innovation of teaching activities and promoting students' active learning.
[0067] Figure 3 The flow chart of classroom teaching interactive behavior analysis provided by the example of the present invention includes the following specific steps:
[0068] Step 1: Obtain the classroom teaching video data to be trained. The present invention analyzes classroom interactive behaviors based on classroom teaching videos. Recorded videos have problems such as inconsistent formats and missing content, and the integrity and effectiveness of classroom video files are the basis for observing classroom interactive behaviors between teachers and students. Therefore, before analyzing classroom interactive behaviors, classroom teaching videos should be screened to ensure that the teaching video formats are consistent and the content is complete.
[0069] Step 2: Use voiceprint recognition technology to obtain the key frames of the teaching video data to be trained, and divide different classroom teaching activity events according to the voice signal and key frame features of each video.
[0070] Step 3: In step 2, according to the slicing habits of classroom behavior analysis coding, a 40-45 minute class audio is cut into multiple 3-second speech signals, the speech signal is framed and windowed, and the Mel-scale Frequency Cepstral Coefficients (MFCC) features are extracted and calculated. The speaker's identity is determined based on the speech signal clustering value.
[0071] Specifically, the speaker clustering recognition algorithm can distinguish speakers with different voiceprint features in a classroom video. This is because the entire processing process is to slice the video first, add it to a temporary voiceprint comparison library, and then compare the similarity of each voiceprint feature to determine the identity of different speakers.
[0072] Step 4: Use the target detection algorithm and multi-person human posture estimation algorithm to locate the key joints of the human body and determine the human body posture, including standing posture, hand posture, and human body facing posture.
[0073] Step 5: In step 4, due to the complex environment in the classroom and the frequent changes in the relative positions of people, it is necessary to first use the target detection algorithm to preliminarily extract the individual human bodies in the image, and then obtain the human body posture map corresponding to the original image through spatial transformation, optimization adjustment, and redundancy elimination.
[0074] Step 6: In step 4, select multiple key nodes of the human body. The key points of the head include the right ear, right eye (reye), left eye (leye) and nose (nose). The key points of the body are neck (neck), right shoulder (rsho), right elbow (relb), right wrist (rwri), left shoulder (lsho), left elbow (lelb), left wrist (lwri), right hip (rhip), right knee (rnke), right ankle (rank), left hip (lhip), left knee (lkne) and left ankle (lank).
[0075] Step 7: In step 6, further calculate the joint point vector group to determine the specific human body motion posture. A feature vector is composed of two adjacent human joint points. Let these two points be A(x 1 ,y 1 ) and B(x 2 ,y 2 ), then the limb structure vector of this part is expressed as
[0076] Step 8: Based on step 7, according to the relationship between human body structure and posture, the present invention establishes the following vector group angle according to the posture to be judged: θ neck-rsho-relb ,θ neck-lsho-lelb ,θ rsho-relb-rwri ,θ lsho-lelb-lwri Used for hand movement calculation, corresponding to the right shoulder angle, left shoulder angle, right elbow angle, and left elbow angle respectively; θ neck-rhip-rkne ,θ neck-lhip-lkne ,θ rhip-rkne-rank ,θ lhip-lkne-lank Used for leg movement calculations, corresponding to the right hip angle, left hip angle, left knee angle and right knee angle respectively.
[0077] Step 9: Extract standing posture features. Use the angle formula to calculate the angle between the bone vectors when standing. Specifically, the line between the right hip and the neck is used as the first vector, the line between the right hip and the right knee is used as the second vector, and the angle between the first vector and the second vector is used as the angle θ between the trunk and the right hip bone. neck-rhip-rkne ,; the line between the left hip and the neck is taken as the third vector, the line between the left hip and the left knee is taken as the fourth vector, and the angle between the third vector and the fourth vector is taken as the angle θ between the trunk and the left hip bone neck-lhip-lkneWhen a person is in a standing position, the angle θ between the torso and the hip bone is neck-rhip-rkne ,θ neck-lhip-lkne The range is 160°-180°. By determining the middle value M and then performing threshold range matching, the specific range that satisfies the standing action is determined.
[0078] Step 10: Extract standing posture features. Due to the uniqueness of classroom activities, the hand movements of speakers are generally classified into four categories: making gestures to explain, operating equipment, indicating content, and no specific movements. Similar to step 9, the hand movement graph is analyzed for posture angle rules. Since the human body is symmetrical, only one side of the movement angle is considered first. For movements such as raising hands, raising arms, and making gestures, the main thing is to judge the positional relationship between the arm joints and the torso, or the shoulder.
[0079] Specifically, the line between the right shoulder and the left shoulder is taken as the fifth vector, the line between the right shoulder and the right elbow is taken as the sixth vector, and the angle between the fifth vector and the sixth vector is taken as the right shoulder-elbow angle; the line between the left shoulder and the right shoulder is taken as the seventh vector, the line between the left shoulder and the left elbow is taken as the eighth vector, and the angle between the seventh vector and the eighth vector is taken as the left shoulder-elbow angle; the line between the right elbow and the right shoulder is taken as the ninth vector, the line between the right elbow and the right wrist is taken as the tenth vector, and the angle between the ninth vector and the tenth vector is taken as the right elbow-wrist angle; the line between the left elbow and the left shoulder is taken as the eleventh vector, the line between the left elbow and the left wrist is taken as the twelfth vector, and the angle between the eleventh vector and the twelfth vector is taken as the left elbow-wrist angle.
[0080] When the shoulder-elbow angle on one side is between 120°-165°, the arm on that side is considered to be raised; when the elbow-wrist angle on one side is between 65°-110°, the elbow on that side is considered to be bent.
[0081] As shown in Table 1, when one arm is raised and the elbow of that side is bent, it is considered that the human body has a hand-raising gesture; when one arm is raised and the elbow of that side is not bent, it is considered that the human body has raised the arm and pointed to the side; when both arms are not raised, if both elbows are bent, it is considered that the human body is making a gesture; when both arms are not raised, if only one elbow is bent, it is considered that the human body has a hand-raising gesture; when both arms are not raised, if both elbows are not bent, it is considered that the human body has no movement.
[0082] Table 1
[0083]
[0084] Step 11: Extraction of human face orientation features. The present invention introduces human face orientation as an additional judgment parameter to further refine the classification of teacher-student interaction on the electronic whiteboard. The overall orientation of the human body can be determined by calculating feature vectors such as the slope of the eyes and shoulders. Since the horizontal edge position of the camera lens in the classroom teaching video is parallel to the horizontal plane, it can be regarded as a normal vector in the horizontal direction. The orientation of the human body is identified and predicted by establishing a coordinate system.
[0085] Specifically, the eye vector is determined according to the line connecting the left eye and the right eye, and the slope of the perpendicular bisector of the eye vector is calculated; when the absolute value of the slope of the perpendicular bisector is between [1.6, +∞), or the slope cannot be calculated, it is considered that the human body is facing the front of the camera lens;
[0086] When the absolute value of the slope of the perpendicular bisector is between [0, 1.6), the human body is considered to be facing the side of the lens; or when only one of the two nodes of the left ear and the right ear of the human body is detected, or only one of the two nodes of the left eye and the right eye is detected, it is determined that the key nodes required to calculate the facial orientation are missing, and the human body is considered to be facing the side of the lens.
[0087] Step 12: Multimodal feature fusion, based on step 2-step 11, image feature data and voiceprint feature data are fused to achieve the transformation from feature extraction to behavior discrimination. The main subject of the present invention is classroom teaching interaction behavior, and the decision-making layer fusion method is used to fuse classroom multimodal data, and the Bayesian causal network is used as the algorithm basis for feature fusion.
[0088] Step 13: Based on step 12, the classroom interaction behavior coding system MFIAS is constructed according to relevant educational theories and the integrated semantic characteristics of teacher-student interaction behaviors, as shown in Table 2:
[0089] Table 2
[0090]
[0091]
[0092] Furthermore, based on the above-mentioned classroom interactive behavior coding system, a mapping comparison table of multimodal fusion behavior and classroom teaching interactive behavior is constructed according to the Bayesian causal network theory, as shown in Table 3.
[0093] Table 3
[0094]
[0095] It should be noted that the explanations of the parameters in Table 3 are as follows: Standing person detection: 0 means no standing person is detected; 1 means one standing person is detected; >1 means multiple standing persons are detected; Hand movement detection: 0 means no hand movement is detected; 1 means a raised hand gesture is detected; 2 means an arm is raised and the hand is pointed to the side; Facing detection: 0 means facing the direction of the camera, that is, facing the front; 1 means facing the side.
[0096] Step 14: Based on step 13, the results of classroom teaching interactive behavior recognition are analyzed. The present invention selects 28 videos for intelligent analysis, and also uses manual labeling to perform video analysis processing to compare the effects of the intelligent processing method and the manual method.
[0097] From the overall recognition analysis, we can see that the average accuracy of the 28 videos is about 84.6%, and the recognition effect is relatively good for most classes, such as Figure 4 As shown in the figure, the specific recognition effect of the "Virtual Reality and Interactive Technology" class. The reasons for the decline in the accuracy of classroom teaching interactive behavior analysis are:
[0098] From the perspective of voiceprint recognition, the classroom environment is noisy, and the environmental noise is sometimes similar to the characteristics of the recorded speaker, and the human voice information collected by the recording equipment will be disturbed by factors such as distance and obstructions. The combination of many factors reduces the accuracy of voiceprint recognition.
[0099] From the perspective of human posture recognition, teachers use touch screen bars to interact with whiteboards and instruct students' behaviors, and teaching activities include conversations and oral practice among multiple students. There will be many errors in the prediction results of action recognition in this part.
[0100] In summary, the classroom behavior intelligent recognition system proposed in the present invention can effectively shorten the analysis of classroom behavior and provide strong support for teachers to improve teaching quality, adjust teaching strategies and realize their own professional development after class.
[0101] Figure 5 is a diagram of the architecture of a classroom interaction analysis system based on multimodal fusion provided by an embodiment of the present invention. Figure 5 As shown, including:
[0102] A teaching video determination unit 510, used to determine the classroom teaching video to be analyzed;
[0103] A voiceprint feature extraction unit 520 is used to extract voiceprint features in the classroom teaching video, and determine the speaker identity and speaking time corresponding to each voiceprint feature through a speaker recognition clustering algorithm;
[0104] The motion feature detection unit 530 is used to perform target detection on the classroom video to detect a human body; and to perform posture estimation on the detected human body to determine key motion features of the human body; the key motion features include: standing posture, hand posture and face facing posture;
[0105] The classroom interaction analysis unit 540 is used to fuse the voiceprint features and the key human movement features, and analyze the classroom interaction behavior corresponding to the classroom teaching video in combination with the classroom interaction behavior coding system constructed in advance according to relevant educational theories and the integrated semantic features of the teacher-student interaction behavior; the specific reference principles for constructing the classroom interaction behavior coding system include: within a certain time period, after detecting that a student is speaking, if no one is detected standing, it is considered that the student is sitting to answer questions or give a speech; if a person is detected standing and there is no hand movement, it is considered that the student is standing to answer questions or give a speech; if a person is detected standing and there is hand movement, it is considered that the student is making gestures to explain the content; if a person is detected standing and raises his arm and points to the side, it is considered that the student is pointing to the display content or operating the electronic whiteboard; if a person is detected standing and facing the camera If a student raises his arm and points to the side, it is considered that the student is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is on the podium listening to students answering questions or giving speeches. Within a certain period of time, after the teacher is detected speaking, if no one is detected standing, it is considered that the teacher is sitting and lecturing or asking questions. If one person is detected standing and there is no hand movement, it is considered that the teacher is standing and lecturing or asking questions. If one person is detected standing and there is hand movement, it is considered that the teacher is gesturing to explain the content. If one person is detected standing and raises his arm and points to the side facing the camera, it is considered that the teacher is pointing to the display content and explaining or writing on the blackboard. If one person is detected standing and raises his arm and points to the side facing the camera, it is considered that the teacher is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is commenting on or instructing the students who are giving speeches.
[0106] Understandably, Figure 5 The detailed functional implementation of each unit in the method can be found in the introduction of the aforementioned method embodiment, which will not be repeated here.
[0107] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A classroom interaction analysis method based on multimodal fusion, characterized in that: The steps include: Identify the classroom teaching videos to be analyzed; Extracting voiceprint features from the classroom teaching video, and determining the speaker identity and speaking time corresponding to each voiceprint feature through a speaker recognition clustering algorithm; Performing target detection on the classroom video to detect human bodies; And perform posture estimation on the detected human body to determine the key action features of the human body; the key action features include: standing posture, hand posture and face facing posture; The voiceprint features and key human movement features are integrated, and combined with a classroom interaction behavior coding system constructed in advance according to relevant educational theories and the integrated semantic features of teacher-student interaction behaviors, the classroom interaction behaviors corresponding to the classroom teaching video are analyzed; the specific reference principles for constructing the classroom interaction behavior coding system include: within a certain time period, after a student is detected speaking, if no one is detected standing, it is considered that the student is sitting to answer questions or give a speech; if a person is detected standing and there is no hand movement, it is considered that the student is standing to answer questions or give a speech; if a person is detected standing and there is hand movement, it is considered that the student is making gestures to explain the content; if a person is detected standing and raises his arm and points to the side, it is considered that the student is pointing to the display content or operating the electronic whiteboard; if a person is detected standing and raises his arm to the side facing the camera If a person points his hand to the side, it is considered that the student is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is on the podium listening to students answering questions or giving speeches. Within a certain period of time, after the teacher is detected speaking, if no person is detected standing, it is considered that the teacher is sitting and lecturing or asking questions. If one person is detected standing and there is no hand movement, it is considered that the teacher is standing and lecturing or asking questions. If one person is detected standing and there is hand movement, it is considered that the teacher is gesturing to explain the content. If one person is detected standing and facing the camera, raising his arm and pointing his hand to the side, it is considered that the teacher is pointing to the display content and explaining or writing on the blackboard. If one person is detected standing and facing the side of the camera, raising his arm and pointing his hand to the side, it is considered that the teacher is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is commenting on or instructing the students who are giving speeches.
2. The method according to claim 1, characterized in that The posture estimation of the detected human body is performed to determine the key motion features of the human body, specifically: Select multiple key nodes of the human body, the key nodes include: head key nodes and body key nodes; the head key nodes include right ear, left ear, right eye, left eye and nose, the body key nodes include: neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee and left ankle; The line between the right hip and the neck is taken as the first vector, the line between the right hip and the right knee is taken as the second vector, and the angle between the first vector and the second vector is taken as the angle between the torso and the right hip bone; The line between the left hip and the neck is taken as the third vector, the line between the left hip and the left knee is taken as the fourth vector, and the angle between the third vector and the fourth vector is taken as the angle between the torso and the left hip bone; When the angle between the trunk and the right hip bone and the angle between the trunk and the left hip bone are both in the range of 160°-180°, the human body is considered to be standing; otherwise, the human body is considered to be sitting.
3. The method according to claim 2, characterized in that The line between the right shoulder and the left shoulder is taken as the fifth vector, the line between the right shoulder and the right elbow is taken as the sixth vector, and the angle between the fifth vector and the sixth vector is taken as the right shoulder-elbow angle; The line between the left shoulder and the right shoulder is taken as the seventh vector, the line between the left shoulder and the left elbow is taken as the eighth vector, and the angle between the seventh vector and the eighth vector is taken as the left shoulder elbow angle; The line between the right elbow and the right shoulder is taken as the ninth vector, the line between the right elbow and the right wrist is taken as the tenth vector, and the angle between the ninth vector and the tenth vector is taken as the right elbow-wrist angle; The line between the left elbow and the left shoulder is taken as the eleventh vector, the line between the left elbow and the left wrist is taken as the twelfth vector, and the angle between the eleventh vector and the twelfth vector is taken as the left elbow-wrist angle; When the shoulder-elbow angle on one side ranges between 120°-165°, the arm on that side is considered raised; When the elbow-wrist angle on one side ranges from 65° to 110°, the elbow on that side is considered to be bent; When one arm is raised and the elbow on that side is bent, the human body is considered to have a hand-raising gesture; When one arm is raised and the elbow on that side is not bent, the person is considered to have raised his arm and pointed to the side; When both arms are not raised, if both elbows are bent, the body is considered to be gesturing; When both arms are not raised, if only one elbow is bent, the person is considered to have raised hand gesture; When both arms are not raised and both elbows are not bent, it is considered that the body is not moving.
4. The method according to claim 2, characterized in that: The eye vector is determined based on the line connecting the left eye and the right eye, and the slope of the perpendicular bisector of the eye vector is calculated; when the absolute value of the perpendicular bisector slope is between [1.6, +∞), or the slope cannot be calculated, it is considered that the human body is facing the front of the camera; When the absolute value of the slope of the perpendicular bisector is between [0, 1.6), the human body is considered to be facing the side of the lens; or when only one of the two nodes of the left ear and the right ear of the human body is detected, or only one of the two nodes of the left eye and the right eye is detected, it is determined that the key nodes required to calculate the facial orientation are missing, and the human body is considered to be facing the side of the lens.
5. The method according to any one of claims 1 to 4, characterized in that: The specific reference principles for constructing the classroom interactive behavior coding system also include: if no speaker is detected and no key movement features of the human body are detected within a certain period of time, it is considered that the classroom is silent or chaotic during this period of time.
6. A classroom interaction analysis system based on multimodal fusion, characterized in that: include: A teaching video determination unit, used to determine the classroom teaching video to be analyzed; A voiceprint feature extraction unit, used to extract voiceprint features in the classroom teaching video, and determine the speaker identity and speaking time corresponding to each voiceprint feature through a speaker recognition clustering algorithm; The motion feature detection unit is used to perform target detection on the classroom video to detect a human body; and to perform posture estimation on the detected human body to determine key motion features of the human body; the key motion features include: standing posture, hand posture and face facing posture; The classroom interaction analysis unit is used to fuse the voiceprint features and key human movement features, and analyze the classroom interaction behavior corresponding to the classroom teaching video in combination with a classroom interaction behavior coding system constructed in advance according to relevant educational theories and the integrated semantic features of teacher-student interaction behaviors; the specific reference principles for constructing the classroom interaction behavior coding system include: within a certain time period, after a student is detected speaking, if no one is detected standing, it is considered that the student is sitting to answer questions or give a speech; if a person is detected standing and there is no hand movement, it is considered that the student is standing to answer questions or give a speech; if a person is detected standing and there is hand movement, it is considered that the student is making gestures to explain the content; if a person is detected standing and raises his arm and points to the side, it is considered that the student is pointing to the display content or operating the electronic whiteboard; if a person is detected standing and facing the side of the camera If a student raises his arm and points to the side, it is considered that the student is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is on the podium listening to students answering questions or giving speeches. Within a certain period of time, after the teacher is detected speaking, if no one is detected standing, it is considered that the teacher is sitting and lecturing or asking questions. If one person is detected standing and there is no hand movement, it is considered that the teacher is standing and lecturing or asking questions. If one person is detected standing and there is hand movement, it is considered that the teacher is gesturing to explain the content. If one person is detected standing and facing the camera and raising his arm and pointing to the side, it is considered that the teacher is pointing to the display content and explaining or writing on the blackboard. If one person is detected standing and facing the side of the camera and raising his arm and pointing to the side, it is considered that the teacher is operating the electronic whiteboard, explaining or writing on the blackboard. If multiple people are detected standing, it is considered that the teacher is commenting on or instructing the students who are giving speeches.
7. The system according to claim 6, characterized in that The motion feature detection unit selects multiple key nodes of the human body, and the key nodes include: head key nodes and body key nodes; the head key nodes include right ear, left ear, right eye, left eye and nose, and the body key nodes include: neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee and left ankle; the line between the right hip and the neck is used as the first vector, the line between the right hip and the right knee is used as the second vector, and the angle between the first vector and the second vector is used as the angle between the trunk and the right hip bone; the line between the left hip and the neck is used as the third vector, the line between the left hip and the left knee is used as the fourth vector, and the angle between the third vector and the fourth vector is used as the angle between the trunk and the left hip bone; when the angle between the trunk and the right hip bone and the angle between the trunk and the left hip bone are both in the range of 160°-180°, it is considered that the human body is standing; otherwise, it is considered that the human body is sitting.
8. The system according to claim 7, characterized in that The motion feature detection unit uses the line between the right shoulder and the left shoulder as the fifth vector, the line between the right shoulder and the right elbow as the sixth vector, and the angle between the fifth vector and the sixth vector as the right shoulder-elbow angle; uses the line between the left shoulder and the right shoulder as the seventh vector, the line between the left shoulder and the left elbow as the eighth vector, and the angle between the seventh vector and the eighth vector as the left shoulder-elbow angle; uses the line between the right elbow and the right shoulder as the ninth vector, the line between the right elbow and the right wrist as the tenth vector, and the angle between the ninth vector and the tenth vector as the right elbow-wrist angle; uses the line between the left elbow and the left shoulder as the eleventh vector, the line between the left elbow and the left wrist as the twelfth vector, and the angle between the eleventh vector and the twelfth vector the angle between the two is taken as the left elbow-wrist angle; when the shoulder-elbow angle of one side is between 120°-165°, the arm on that side is considered to be raised; when the elbow-wrist angle of one side is between 65°-110°, the elbow on that side is considered to be bent; when one arm is raised and the elbow on that side is bent, it is considered that the human body has raised his hand; when one arm is raised and the elbow on that side is not bent, it is considered that the human body has raised his arm and pointed to the side; when both arms are not raised, if both elbows are bent, it is considered that the human body is making a gesture; when both arms are not raised, if only one elbow is bent, it is considered that the human body has raised his hand; when both arms are not raised, if both elbows are not bent, it is considered that the human body has no movement.
9. The system according to claim 7, characterized in that The motion feature detection unit determines the eye vector according to the line connecting the left eye and the right eye, and calculates the slope of the perpendicular bisector of the eye vector; when the absolute value of the slope of the perpendicular bisector is between [1.6, +∞), or the slope cannot be calculated, it is considered that the human body is facing the front of the camera lens; when the absolute value of the slope of the perpendicular bisector is between [0, 1.6), it is considered that the human body is facing the side of the camera lens; or when only one node is detected between the two nodes of the left ear and the right ear of the human body, or only one node is detected between the two nodes of the left eye and the right eye, it is determined that the key node required for calculating the facial orientation is missing, and it is considered that the human body is facing the side of the camera lens.
10. The system according to any one of claims 6 to 9, characterized in that: The classroom interaction analysis unit also refers to the following principles when constructing the classroom interaction behavior coding system: if no speaker is detected and no key human body movement features are detected within a certain time period, it is considered that the classroom is silent or chaotic during this period.
Citation Information
Cited By
Method for identifying person speaking in video
CN116311445A
