Classroom teaching behavior analysis method and system based on semantic understanding

Abstract

The present invention discloses a classroom teaching behavior analysis method and system based on semantic understanding, which relates to the field of educational auxiliary technology, including configuring the audio weight under each teaching link according to the importance of audio data and the nature of the teaching link, and considering the nature of the teaching and the importance of audio data to standardize the audio weight under different teaching links, providing a reliable basis for subsequent teaching behavior analysis. The audio separation in the multi-audio mixed signal segment is performed through the processing parameters and ICA method of ICA, and the two text types of dual-object language text and single-object language text are identified through the timestamps between independent language texts and the semantic coherence index. The classroom teaching quality of classroom teaching is analyzed according to the dual-object language text, the single-object language text, the audio weight and other voice content, and the situation of classroom teaching behavior is accurately analyzed by analyzing other voice content, dialogue and monologue, thereby improving the adaptability and accuracy of classroom teaching behavior analysis.

Description

Technical Field

[0001] The present invention relates to the field of educational assistance technology, and in particular to a classroom teaching behavior analysis method and system based on semantic understanding. Background Art

[0002] With the continuous advancement of educational informatization, the analysis and evaluation of classroom teaching behavior has gradually become a research hotspot in the field of education. Traditional classroom teaching behavior analysis mainly relies on manual observation, recording and evaluation, which is not only time-consuming and laborious, but also greatly affected by the subjective factors of the observer, making it difficult to ensure the objectivity and accuracy of the evaluation. In recent years, with the rapid development of artificial intelligence and natural language processing technology, new ideas and methods have been provided for the analysis of classroom teaching behavior. As an important branch of artificial intelligence, natural language processing (NLP) can realize the automatic processing and understanding of text information, providing strong support for the analysis of speech behavior in classroom teaching. In the traditional teaching environment, the teacher's speech behavior is an important means of imparting knowledge, guiding students to think and stimulating students' interest in learning. However, due to the complexity and variability of the classroom environment, the speech behavior of teachers and students is often difficult to accurately capture and comprehensively analyze.

[0003] In the prior art, because the verbal behaviors of teachers and students are difficult to be accurately analyzed, the adaptability and accuracy of classroom teaching behavior analysis are low, which is not conducive to the adjustment of subsequent teaching strategies.

[0004] Therefore, how to improve the adaptability and accuracy of classroom teaching behavior analysis is a technical problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of low adaptability and poor accuracy of classroom teaching behavior analysis in the prior art, and to propose a classroom teaching behavior analysis method based on semantic understanding, which includes:

[0006] Obtain the class schedule of classroom teaching, analyze the class schedule of classroom teaching to determine the type of classroom teaching that exists, and process the teaching content of classroom teaching in a standardized process according to the type of classroom teaching, obtain the standardized process of classroom teaching under each type, and define each standardized process into multiple teaching links, and configure the audio weight of each teaching link according to the importance of audio data of the teaching link and the nature of the link;

[0007] Acquire audio data of classroom teaching, identify multi-audio mixed signal segments in the audio data of classroom teaching, analyze the mixed properties of the multi-audio mixed signal segments, determine ICA processing parameters based on the mixed properties, and perform audio separation in the multi-audio mixed signal segments by using the ICA processing parameters and ICA method to obtain multiple independent audio signals;

[0008] Through NLP technology, the independent audio signals are recognized and converted into text to obtain the teaching content text and other voice content of the classroom teaching, and the teaching content text of the classroom teaching is split into multiple independent language texts. The two types of text, dual-object language text and single-object language text, are identified through the timestamps and semantic coherence indicators between the independent language texts.

[0009] The dual-object language text, single-object language text and other voice content are synchronized to each teaching link, and the classroom teaching quality is analyzed according to the dual-object language text, single-object language text, audio proportion and other voice content. The multi-faceted teaching quality and the comprehensive quality of classroom teaching are visualized to help adjust or deploy teaching strategies.

[0010] In some embodiments of the present application, the audio weight of each teaching link is configured according to the importance of the audio data of the teaching link and the nature of the link, including:

[0011] Determine the number of data types of multi-source data and the reflection ratio of each type of data according to the nature of the teaching link, and determine the audio weight range of audio data in the multi-source data according to the number of data types of multi-source data and the reflection ratio of each type of data;

[0012] Acquire the collection records of multi-source data in the teaching link, analyze the audio quality of the audio data and the effective data volume in the audio data and the effective data volume of other multi-source data in the collection records of the multi-source data, determine the effective data volume ratio of the audio data by the effective data volume in the audio data and the effective data volume of other multi-source data, and determine the adjustment ratio in combination with the audio quality and the effective data volume ratio of the audio data;

[0013] The audio weight in each teaching link is configured according to the median and adjustment ratio within the audio weight range of the audio data in each teaching link.

[0014] In some embodiments of the present application, multiple audio mixed signal segments are identified in the audio data of classroom teaching, including:

[0015] Obtaining a complete audio data segment according to the audio data of the classroom teaching, and dividing the complete audio data segment into a plurality of audio sub-segments according to the distribution of valid data in the complete audio data segment;

[0016] The audio signal in the audio sub-segment is converted into a time domain representation to obtain a spectrogram. In the spectrogram, potential sound sources are identified by using an energy threshold, and are classified into a potential sound source set after statistics. A time-frequency mask is generated for each potential sound source in the potential sound source set. Clusters in the spectrogram are identified by relying on the time-frequency mask and a preset algorithm. The overlap of the time-frequency mask in each audio sub-segment is described according to the size of the overlapping area between multiple clusters, thereby identifying multi-audio mixed signal segments and single-audio mixed signal segments.

[0017] In some embodiments of the present application, the mixed properties of the multi-audio mixed signal segment are analyzed, and the ICA processing parameters are determined in combination with the mixed properties, including:

[0018] The mixed properties of the multi-audio mixed signal segment include the degree of sound source overlap and the relative signal-to-noise ratio. The processing parameters of ICA are a combination of the learning rate and the number of iterations.

[0019] The sound source overlap degree is obtained according to the size of the overlapping area between the multiple clumps, and the single audio mixed signal segment with the highest signal-to-noise ratio is found among the multiple single audio mixed signal segments, and the single audio mixed signal segment is used as the reference signal segment;

[0020] Dividing the multi-audio mixed signal segment into a plurality of partial signal segments, calculating the similarity between each partial signal segment and a reference signal segment, and determining the contribution power of each partial signal segment according to the similarity, thereby determining the signal power and noise power of the multi-audio mixed signal segment, and calculating the relative signal-to-noise ratio through the signal power and the noise power;

[0021] Generate processing levels based on the degree of sound source overlap and relative signal-to-noise ratio;

[0022]

[0023] Wherein, L is the processing level, α1 is the conversion coefficient of the sound source overlap degree, Q1 is the sound source overlap degree, α2 is the conversion coefficient of the relative signal-to-noise ratio, Q2 is the relative signal-to-noise ratio, and b is a preset constant;

[0024] The learning rate and number of iterations of ICA are set according to the processing level. Different processing levels correspond to different combinations of learning rate and number of iterations of ICA.

[0025] In some embodiments of the present application, the independent audio signal is subjected to speech recognition and text conversion by NLP technology to obtain the teaching content text and other speech content of the classroom teaching, including:

[0026] By converting and analyzing independent audio signals through NLP technology, we can obtain the teaching content text and other voice content of classroom teaching. The teaching content text of classroom teaching is a continuous long content text, and other voice content includes emotional features, speaking speed features, intonation features and non-voice features.

[0027] In some embodiments of the present application, the teaching content text of the classroom teaching is split into multiple independent language texts, including:

[0028] Performing initial splitting according to the timestamp of the teaching content text of the classroom teaching to obtain multiple segments of initial independent language text, distinguishing the main object of each initial independent language text, performing part-of-speech tagging on each initial independent language text, and judging whether each initial independent language text is semantically complete through the part-of-speech dependency tree and the main object;

[0029] For an initial independent language text with incomplete semantics, determine the missing parts of speech in the initial independent language text, so as to evaluate the degree of missing of the initial independent language text, and map a proximity according to the degree of missing, where the proximity is the length of time. Other initial independent language texts with incomplete semantics within the proximity of the context of the initial independent language text are classified into a set of texts to be divided, and the semantic association between each text in the set of texts to be divided and the initial independent language text is calculated respectively. Combined with the semantic association and the part-of-speech dependency tree, the texts in the set of texts to be divided are screened, and the screened texts and the initial independent language text are taken as independent language texts together, and the semantically complete initial independent language text is taken as an independent language text.

[0030] In some embodiments of the present application, two types of text, dual-object language text and single-object language text, are identified by using timestamps and semantic coherence indicators between independent language texts, including:

[0031] detecting whether there is a dialogue mark in the independent language texts according to the timestamps between the independent language texts, the subject objects of the independent language texts, and the interaction keywords between the independent language texts;

[0032] For an independent language text with a dialogue mark, the context of the independent language text is screened according to the timestamp, and the frequency of similar keyword occurrence, the spacing between similar keywords, the semantic relevance, and the degree of logical order between the independent language text in the screened context and the independent language text with the dialogue mark or between the independent language texts in the screened context are calculated, and the semantic coherence index is determined according to the frequency of similar keyword occurrence, the spacing between similar keywords, the semantic relevance, and the degree of logical order;

[0033]

[0034] Among them, G i，j is the semantic coherence index between the ith independent language text and the jth independent language text, τ1 is the weight corresponding to similar keywords, β1 and β2 are the conversion coefficients of similar keyword occurrence frequency and similar keyword spacing, respectively, W1 i，j 、W2i，j are the frequency of similar keywords and the distance between similar keywords between the i-th independent language text and the j-th independent language text, τ2 is the semantic weight of the independent language text, β3 is the conversion coefficient of semantic relevance, E1 i，j 、E2 i，j are the semantic association degree and logical order degree between the ith independent language text and the jth independent language text, respectively, and k is a preset constant;

[0035] According to the semantic coherence index, the independent language texts in the context after the independent language texts with dialogue marks are screened are traversed, and the independent language texts that meet the requirements of the semantic coherence index are taken as dual-object language texts;

[0036] For independent language texts without dialogue marks, matching is performed between the independent language texts according to preset matching conditions, thereby determining the single object language text.

[0037] In some embodiments of the present application, the classroom teaching quality of classroom teaching is analyzed according to the dual-object language text, the single-object language text, the audio proportion and other voice content, including:

[0038] The teaching behavior is evaluated according to the dual-object language text, single-object language text, and other voice content in each teaching link, and the teaching quality index of a single teaching link is obtained. The classroom teaching quality of the classroom teaching is determined by combining the audio proportion and the quality index of a single teaching link.

[0039] Correspondingly, the present application also provides a classroom teaching behavior analysis system based on semantic understanding, including:

[0040] A determination module is used to obtain a class schedule for classroom teaching, analyze the class schedule for classroom teaching to determine the type of classroom teaching that exists, and process the teaching content of classroom teaching in a standardized process according to the type of classroom teaching, obtain a standardized process for classroom teaching under each type, and define each standardized process into multiple teaching links, and configure the audio weight under each teaching link according to the importance of the audio data of the teaching link and the nature of the link;

[0041] A separation module is used to obtain audio data of classroom teaching, identify multi-audio mixed signal segments in the audio data of classroom teaching, analyze the mixed properties of the multi-audio mixed signal segments, determine ICA processing parameters based on the mixed properties, and perform audio separation in the multi-audio mixed signal segments through the ICA processing parameters and ICA method to obtain multiple independent audio signals;

[0042] The recognition module is used to perform speech recognition and text conversion on independent audio signals through NLP technology to obtain the teaching content text and other speech content of classroom teaching, and to split the teaching content text of classroom teaching into multiple independent language texts, and to identify two types of texts, dual-object language text and single-object language text, through the timestamps and semantic coherence indicators between the independent language texts;

[0043] The analysis module is used to synchronize dual-object language texts, single-object language texts and other voice contents to each teaching link, analyze the classroom teaching quality according to the dual-object language texts, single-object language texts, audio proportions and other voice contents, and visualize the multi-faceted teaching quality and the comprehensive quality of classroom teaching, so as to help adjust or deploy teaching strategies.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. Configure the audio weight of each teaching link according to the importance of audio data and the nature of the link. Consider the nature of teaching and the importance of audio data to standardize the audio weight of different teaching links, providing a reliable basis for subsequent teaching behavior analysis.

[0046] 2. The audio separation in the multi-audio mixed signal segment is performed through ICA processing parameters and ICA methods, which ensures the purity of the audio signal and can analyze each audio signal source independently. The two types of texts, dual-object language text and single-object language text, are identified through the timestamps between independent language texts and semantic coherence indicators, and the dialogue between teachers and students and the monologues of teachers or students are identified, so as to accurately capture the interactive behavior between teachers and students in the classroom.

[0047] 3. Analyze the teaching quality of classroom teaching according to dual-object language text, single-object language text, audio proportion and other voice content, and analyze other voice content, dialogues and monologues to accurately analyze the situation of classroom teaching behavior, which improves the adaptability and accuracy of classroom teaching behavior analysis and ensures the adaptation of subsequent teaching strategy adjustments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flow chart of the classroom teaching behavior analysis method based on semantic understanding proposed by the present invention;

[0049] Figure 2 This is a structural diagram of the classroom teaching behavior analysis system based on semantic understanding proposed by the present invention. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0051] Reference Figure 1 ,The classroom teaching behavior analysis method based on semantic understanding includes the following steps:

[0052] Step S101, obtain the class schedule for classroom teaching, analyze the class schedule for classroom teaching to determine the type of classroom teaching that exists, and process the teaching content of classroom teaching in a standardized process according to the type of classroom teaching to obtain a standardized process for classroom teaching of each type, and define each standardized process into multiple teaching links, and configure the audio weight of each teaching link according to the importance of the audio data of the teaching link and the nature of the link.

[0053] In this embodiment, all the types of classroom teaching in the school are determined according to the curriculum, and then the standardized process processing (defined as multiple teaching links) is performed for each type of classroom teaching. For example, the following multiple links are constructed into a standardized process;

[0054] Introduction: Through an engaging introduction, students’ interest and curiosity in learning are stimulated, laying the foundation for subsequent learning.

[0055] Teaching new knowledge: Based on the course content, adopt appropriate methods (such as lectures, demonstrations, case analysis, etc.) to impart new knowledge.

[0056] Practical operation / discussion: Provide students with opportunities to conduct practical operations, group discussions or case analysis to deepen their understanding and apply what they have learned.

[0057] Summary and Feedback: Summarize and review what has been learned and provide immediate feedback to help students consolidate their memory and correct misunderstandings.

[0058] Assign homework / extension learning: Assign homework or extension learning tasks to promote students' independent learning and further exploration.

[0059] In some embodiments of the present application, the audio weight of each teaching link is configured according to the importance of the audio data of the teaching link and the nature of the link, including:

[0060] Determine the number of data types of multi-source data and the reflection ratio of each type of data according to the nature of the teaching link, and determine the audio weight range of audio data in the multi-source data according to the number of data types of multi-source data and the reflection ratio of each type of data;

[0061] Acquire the collection records of multi-source data in the teaching link, analyze the audio quality of the audio data and the effective data volume in the audio data and the effective data volume of other multi-source data in the collection records of the multi-source data, determine the effective data volume ratio of the audio data by the effective data volume in the audio data and the effective data volume of other multi-source data, and determine the adjustment ratio in combination with the audio quality and the effective data volume ratio of the audio data;

[0062] The audio weight in each teaching link is configured according to the median and adjustment ratio within the audio weight range of the audio data in each teaching link.

[0063] In this embodiment, different teaching links have different natures, and the teaching behavior reflected by the audio data may also be limited. For example, the theoretical teaching link may rely more on audio data to convey knowledge, while the experimental operation link may focus more on students' actual operation and observation of experimental results. The number of data types of multi-source data corresponding to different teaching links and the reflection ratio of each type of data are different. Some links may require multi-source data such as audio and video to jointly reflect the teaching behavior. The reflection ratio of each type of data is the degree of response of each type of multi-source data to the link.

[0064] In this embodiment, audio quality is a combination of completeness and clarity, whether the audio data completely records the main content of the teaching link and the teacher's explanation, and whether the clarity of the audio is sufficient to support subsequent analysis and evaluation. The effective data volume in the audio data refers to the data in the audio data excluding invalid data such as blank audio and background noise, and the same is true for multi-source data. The adjustment ratio is determined by combining the audio quality and the proportion of effective data volume of the audio data. The adjustment ratio is determined by comprehensively considering the audio quality and the proportion of effective data volume of the audio data compared with other multi-source data, and the median within the audio weight range is adjusted according to the adjustment ratio.

[0065] Step S102, obtain audio data of classroom teaching, identify multi-audio mixed signal segments in the audio data of classroom teaching, analyze the mixing properties of the multi-audio mixed signal segments, determine ICA processing parameters based on the mixing properties, and use ICA processing parameters and ICA methods to perform audio separation in the multi-audio mixed signal segments to obtain multiple independent audio signals.

[0066] In this embodiment, multiple audio mixed signal segments are identified, and ICA processing parameters (both the number of iterations and the learning rate will affect the separation effect of the algorithm) are determined to perform audio separation of multiple sound sources. The ICA algorithm, or Independent Component Analysis, is a computational method for signal separation and extraction.

[0067] In some embodiments of the present application, multiple audio mixed signal segments are identified in the audio data of classroom teaching, including:

[0068] Obtaining a complete audio data segment according to the audio data of the classroom teaching, and dividing the complete audio data segment into a plurality of audio sub-segments according to the distribution of valid data in the complete audio data segment;

[0069] The audio signal in the audio sub-segment is converted into a time domain representation to obtain a spectrogram. In the spectrogram, potential sound sources are identified by using an energy threshold, and are classified into a potential sound source set after statistics. A time-frequency mask is generated for each potential sound source in the potential sound source set. Clusters in the spectrogram are identified by using the time-frequency mask and a preset algorithm. The overlap of the time-frequency mask in each audio sub-segment is described according to the size of the overlapping area between multiple clusters, thereby identifying multi-audio mixed signal segments and single-audio mixed signal segments.

[0070] In the present embodiment, in the previous audio-to-text process, the multi-audio mixed signal segment will affect the accuracy of the conversion, resulting in an increase in the subsequent analysis error. The multi-audio mixed signal segment refers to a plurality of sound sources making sounds together in the same period of time or a similar period of time. The audio signal is converted into a time-frequency representation (short-time Fourier transform (STFT)), and the spectrum obtained by STFT shows the energy distribution of the audio signal at different times and frequencies. An energy threshold is set to identify possible sound sources. When the energy of a certain area in the spectrum exceeds the threshold, it can be considered that there may be a sound source in the area.

[0071] In this embodiment, the time-frequency mask is a binary or continuous value representation used to represent the activity level of different sound sources in the audio signal in time and frequency. For a multi-source mixed signal, each sound source may have a corresponding time-frequency mask, which marks the active area of ​​the sound source on the time-frequency plane. Therefore, the "clusters" identified from the time-frequency diagram can be regarded as the basis for constructing the time-frequency mask, that is, each cluster corresponds to an active area in the time-frequency mask of its sound source. In audio signal processing, especially when analyzing the time-frequency diagram, "clusters" generally refer to continuous areas with similar characteristics in the time-frequency diagram. These areas may be formed due to the activity of a certain sound source, which is manifested as energy concentration within a specific time and frequency range. These clusters can be identified from the time-frequency diagram through clustering algorithms (such as K-means) or image processing techniques (such as edge detection, connected region marking, etc.). Each cluster may correspond to an activity pattern of a sound source at a specific time and frequency. The overlapping area between clusters refers to the area where two or more clusters intersect or approach each other in time and frequency on the time-frequency diagram. This overlap may indicate that two or more sound sources are active at similar frequencies during the same period of time, causing their signals to interfere with each other in the mixed signal.

[0072] In some embodiments of the present application, the mixed properties of the multi-audio mixed signal segment are analyzed, and the ICA processing parameters are determined in combination with the mixed properties, including:

[0073] The mixed properties of the multi-audio mixed signal segment include the degree of sound source overlap and the relative signal-to-noise ratio. The processing parameters of ICA are a combination of the learning rate and the number of iterations.

[0074] The sound source overlap degree is obtained according to the size of the overlapping area between the multiple clumps, and the single audio mixed signal segment with the highest signal-to-noise ratio is found among the multiple single audio mixed signal segments, and the single audio mixed signal segment is used as the reference signal segment;

[0075] Dividing the multi-audio mixed signal segment into a plurality of partial signal segments, calculating the similarity between each partial signal segment and a reference signal segment, and determining the contribution power of each partial signal segment according to the similarity, thereby determining the signal power and noise power of the multi-audio mixed signal segment, and calculating the relative signal-to-noise ratio through the signal power and the noise power;

[0076] Generate processing levels based on the degree of sound source overlap and relative signal-to-noise ratio;

[0077]

[0078] Wherein, L is the processing level, α1 is the conversion coefficient of the sound source overlap degree, Q1 is the sound source overlap degree, α2 is the conversion coefficient of the relative signal-to-noise ratio, Q2 is the relative signal-to-noise ratio, and b is a preset constant;

[0079] The learning rate and number of iterations of ICA are set according to the processing level. Different processing levels correspond to different combinations of learning rate and number of iterations of ICA.

[0080] In this embodiment, for each divided partial signal segment, its similarity with the reference signal segment is calculated. Similarity calculation can be performed using a variety of methods, such as cross-correlation function, Pearson correlation coefficient, etc. These methods can quantify the degree of similarity between two signal segments. For each partial signal segment, based on its similarity with the reference signal segment, the power contributed by the reference signal in the partial signal segment can be estimated. This usually requires some form of weighted processing of the partial signal segments, with the weights determined by the similarity. The power contributed by the reference signal in all partial signal segments is accumulated to obtain an overall power estimate of the reference signal in the entire mixed signal segment.

[0081] In this embodiment, the relative signal-to-noise ratio is calculated by using the signal power and the noise power. The ratio of the signal power to the noise power is the relative signal-to-noise ratio.

[0082] Step S103, perform speech recognition and text conversion on the independent audio signal through NLP technology to obtain the teaching content text and other speech content of the classroom teaching, and split the teaching content text of the classroom teaching into multiple independent language texts, and identify the two types of text, dual-object language text and single-object language text, through the timestamps and semantic coherence indicators between the independent language texts.

[0083] In this embodiment, in the process of applying natural language processing (NLP) technology to classroom teaching behavior analysis, in addition to converting audio data into text information for analysis, a variety of valuable information can also be extracted, which is crucial for in-depth understanding of the classroom environment, the interaction between teachers and students, and the teaching effect. Other voice content includes emotions, speech speed pauses, non-language sounds, etc. Dual-object language texts are conversations and dialogues between teachers and students, and single-object language texts are unilateral monologues of teachers or students, such as discussions between students, teachers' teaching of knowledge content, etc.

[0084] In some embodiments of the present application, the independent audio signal is subjected to speech recognition and text conversion by NLP technology to obtain the teaching content text and other speech content of the classroom teaching, including:

[0085] By converting and analyzing independent audio signals through NLP technology, we can obtain the teaching content text and other voice content of classroom teaching. The teaching content text of classroom teaching is a continuous long content text, and other voice content includes emotional features, speaking speed features, intonation features and non-voice features.

[0086] In this embodiment, the emotional features, speech speed features, intonation features and non-speech features include the following:

[0087] Sentiment Analysis:

[0088] By analyzing features such as intonation, speaking speed, and volume in audio data, NLP technology can infer the speaker's emotional state, such as positive, negative, neutral, etc. This is of great significance for evaluating teachers' teaching attitudes and students' learning interests and participation.

[0089] Speech speed and pause analysis:

[0090] The speed of speech and the frequency of pauses can reflect the speaker's thinking process, nervousness or the control of teaching rhythm. This is very helpful for analyzing the teacher's teaching style and the difficulty of students understanding the questions.

[0091] Non-verbal sound recognition:

[0092] In addition to verbal content, audio data may also contain non-verbal sounds such as coughing, laughing, and applause. These sounds also contain rich information, such as students' reactions and classroom atmosphere.

[0093] In some embodiments of the present application, the teaching content text of the classroom teaching is split into multiple independent language texts, including:

[0094] Performing initial splitting according to the timestamp of the teaching content text of the classroom teaching to obtain multiple segments of initial independent language text, distinguishing the main object of each initial independent language text, performing part-of-speech tagging on each initial independent language text, and judging whether each initial independent language text is semantically complete through the part-of-speech dependency tree and the main object;

[0095] For an initial independent language text with incomplete semantics, determine the missing parts of speech in the initial independent language text, so as to evaluate the degree of missing of the initial independent language text, and map a proximity according to the degree of missing, where the proximity is the length of time. Other initial independent language texts with incomplete semantics within the proximity of the context of the initial independent language text are classified into a set of texts to be divided, and the semantic association between each text in the set of texts to be divided and the initial independent language text is calculated respectively. Combined with the semantic association and the part-of-speech dependency tree, the texts in the set of texts to be divided are screened, and the screened texts and the initial independent language text are taken as independent language texts together, and the semantically complete initial independent language text is taken as an independent language text.

[0096] In this embodiment, the subject object of the initial independent language text refers to who is the speaker, and the initial split is performed according to the timestamp of the teaching content text of the classroom teaching, that is, the initial split is performed according to the time continuity. Part-of-Speech Tagging (POS Tagging for short) is a basic task in natural language processing (NLP). It involves marking each token in the text as its corresponding part of speech (such as noun, verb, adjective, etc.). The part-of-speech dependency tree and the subject object are used to judge whether each initial independent language text is semantically complete. By constructing the dependency tree (Dependency Parse Tree) of the sentence, the relationship between the various components in the sentence can be understood, thereby judging whether the sentence is semantically complete. The degree of missing is determined by the number and type of the missing parts of speech in the initial independent language text, and the greater the degree of missing, the longer the corresponding proximity. The semantic relevance is calculated by the correlation between the parts of speech and characters on the initial independent language text, and the initial independent language text with a higher semantic relevance and a relatively complete dependency tree judgment is retained as an independent language text.

[0097] It should be noted that when determining the independent language text, different subject objects are analyzed separately, and no correlation analysis is performed between the speech texts of different subject objects. In other words, the independent language text is the text at the level of a single subject object.

[0098] In some embodiments of the present application, two types of text, dual-object language text and single-object language text, are identified by using timestamps and semantic coherence indicators between independent language texts, including:

[0099] detecting whether there is a dialogue mark in the independent language texts according to the timestamps between the independent language texts, the subject objects of the independent language texts, and the interaction keywords between the independent language texts;

[0100] For an independent language text with a dialogue mark, the context of the independent language text is screened according to the timestamp, and the frequency of similar keyword occurrence, the spacing between similar keywords, the semantic relevance, and the degree of logical order between the independent language text in the screened context and the independent language text with the dialogue mark or between the independent language texts in the screened context are calculated, and the semantic coherence index is determined according to the frequency of similar keyword occurrence, the spacing between similar keywords, the semantic relevance, and the degree of logical order;

[0101]

[0102] Among them, G i，j is the semantic coherence index between the ith independent language text and the jth independent language text, τ1 is the weight corresponding to similar keywords, β1 and β2 are the conversion coefficients of similar keyword occurrence frequency and similar keyword spacing, respectively, W1 i，j 、W2 i，j are the frequency of similar keywords and the distance between similar keywords between the i-th independent language text and the j-th independent language text, τ2 is the semantic weight of the independent language text, β3 is the conversion coefficient of semantic relevance, E1 i，j 、E2 i，j are the semantic association degree and logical order degree between the ith independent language text and the jth independent language text, respectively, and k is a preset constant;

[0103] According to the semantic coherence index, the independent language texts in the context after the independent language texts with dialogue marks are screened are traversed, and the independent language texts that meet the requirements of the semantic coherence index are taken as dual-object language texts;

[0104] For independent language texts without dialogue marks, matching is performed between the independent language texts according to preset matching conditions, thereby determining the single object language text.

[0105] In this embodiment, the dialogue mark is a comprehensive judgment of time, subject object and interaction keywords.

[0106] For time and subject objects, the start of a conversation is determined by observing the change and continuity of timestamps. For example, when two or more different timestamps are detected to be closely connected and their corresponding speakers are different, this is likely to be a sign of the start of a conversation.

[0107] For interaction keywords, certain specific introductory words or phrases may indicate the beginning of a conversation, such as question words, such as "What do you think?", "Are there any different opinions?", etc. These questioning methods are often inviting the other party to participate in the conversation. Words such as "but", "however", "I think" that express opinions or refute may also be a sign of the beginning of a conversation, because these words are usually used to respond to or refute the previous content.

[0108] In this embodiment, for the independent language text with dialogue marks, the keywords in the text are first analyzed to determine which keywords are similar to the keyword. The frequency of similar keywords between the independent language text with the selected context and the independent language text with the dialogue mark or between the independent language texts with the selected context, the spacing between similar keywords, the semantic relevance and the degree of logical order are calculated. Here, the independent language text with the dialogue mark is recorded as the first text, and the text of its context is recorded as the second text. The frequency of similar keywords (the number of similar or similar similar keywords in the two texts), the spacing between similar keywords (character distance), the semantic relevance (using the word embedding model (such as Word2Vec, GloVe, etc.) in natural language processing technology to convert keywords into vector representations, and then using methods such as cosine similarity to calculate the semantic similarity between them) and the degree of logical order are generally the data between the first text and the second text that are closely adjacent in time or between the second texts. The degree of logical order describes the logical rationality between the two texts. For example, time sequence analysis: identify the time words and phrases involved in the dialogue, and analyze whether there is a reasonable time sequence between them to ensure the coherence of the dialogue or narrative. Identification of causal and conditional relationships: Use natural language processing technology to identify logical relationship words such as causal and conditional relationships in texts, and analyze whether these relationships are logically reasonable and naturally connected. This can be calculated using a pre-trained text logic analysis model.

[0109] In this embodiment, It indicates the correction or adjustment of the degree of logical order to the degree of semantic relevance.

[0110] Step S104, synchronize the dual-object language text, single-object language text and other voice content to each teaching link, analyze the classroom teaching quality according to the dual-object language text, single-object language text, audio proportion and other voice content, and visualize the multi-faceted teaching quality and comprehensive classroom teaching quality to help adjust or deploy teaching strategies.

[0111] In this embodiment, the dialogue between students and teachers, as well as their respective monologues and other voice contents are analyzed to comprehensively analyze the classroom teaching quality of the classroom teaching.

[0112] In some embodiments of the present application, the classroom teaching quality of classroom teaching is analyzed according to the dual-object language text, the single-object language text, the audio proportion and other voice content, including:

[0113] The teaching behavior is evaluated according to the dual-object language text, single-object language text, and other voice content in each teaching link, and the teaching quality index of a single teaching link is obtained. The classroom teaching quality of the classroom teaching is determined by combining the audio proportion and the quality index of a single teaching link.

[0114] In this embodiment, the classroom teaching quality of the classroom teaching is determined by combining the audio proportion and the quality index of a single teaching link, which is achieved through weighted summation.

[0115] Compared with the prior art, the present invention has the following beneficial effects:

[0116] 1. Configure the audio weight of each teaching link according to the importance of audio data and the nature of the link. Consider the nature of teaching and the importance of audio data to standardize the audio weight of different teaching links, providing a reliable basis for subsequent teaching behavior analysis.

[0117] 2. The audio separation in the multi-audio mixed signal segment is performed through ICA processing parameters and ICA methods, which ensures the purity of the audio signal and can analyze each audio signal source independently. The two types of texts, dual-object language text and single-object language text, are identified through the timestamps between independent language texts and semantic coherence indicators, and the dialogue between teachers and students and the monologues of teachers or students are identified, so as to accurately capture the interactive behavior between teachers and students in the classroom.

[0118] 3. Analyze the teaching quality of classroom teaching according to dual-object language text, single-object language text, audio proportion and other voice content, and analyze other voice content, dialogues and monologues to accurately analyze the situation of classroom teaching behavior, which improves the adaptability and accuracy of classroom teaching behavior analysis and ensures the adaptation of subsequent teaching strategy adjustments.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present invention.

[0120] Correspondingly, this application also provides a classroom teaching behavior analysis system based on semantic understanding, such as Figure 2 As shown, including,

[0121] A determination module is used to obtain a class schedule for classroom teaching, analyze the class schedule for classroom teaching to determine the type of classroom teaching that exists, and process the teaching content of classroom teaching in a standardized process according to the type of classroom teaching, obtain a standardized process for classroom teaching under each type, and define each standardized process into multiple teaching links, and configure the audio weight under each teaching link according to the importance of the audio data of the teaching link and the nature of the link;

[0122] A separation module is used to obtain audio data of classroom teaching, identify multi-audio mixed signal segments in the audio data of classroom teaching, analyze the mixed properties of the multi-audio mixed signal segments, determine ICA processing parameters based on the mixed properties, and perform audio separation in the multi-audio mixed signal segments through the ICA processing parameters and ICA method to obtain multiple independent audio signals;

[0123] The recognition module is used to perform speech recognition and text conversion on independent audio signals through NLP technology to obtain the teaching content text and other speech content of classroom teaching, and to split the teaching content text of classroom teaching into multiple independent language texts, and to identify two types of texts, dual-object language text and single-object language text, through the timestamps and semantic coherence indicators between the independent language texts;

[0124] The analysis module is used to synchronize dual-object language texts, single-object language texts and other voice contents to each teaching link, analyze the classroom teaching quality according to the dual-object language texts, single-object language texts, audio proportions and other voice contents, and visualize the multi-faceted teaching quality and the comprehensive quality of classroom teaching, so as to help adjust or deploy teaching strategies.

[0125] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present invention.

[0126] Those skilled in the art will appreciate that the modules in the system in the implementation scenario can be distributed in the system of the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more systems different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0127] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. The classroom teaching behavior analysis method based on semantic understanding is characterized by: include, Obtain the class schedule of classroom teaching, analyze the class schedule of classroom teaching to determine the type of classroom teaching that exists, and process the teaching content of classroom teaching in a standardized process according to the type of classroom teaching, obtain the standardized process of classroom teaching under each type, and define each standardized process into multiple teaching links, and configure the audio weight of each teaching link according to the importance of audio data of the teaching link and the nature of the link; Acquire audio data of classroom teaching, identify multi-audio mixed signal segments in the audio data of classroom teaching, analyze the mixed properties of the multi-audio mixed signal segments, determine ICA processing parameters based on the mixed properties, and perform audio separation in the multi-audio mixed signal segments by using the ICA processing parameters and ICA method to obtain multiple independent audio signals; Through NLP technology, the independent audio signals are recognized and converted into text to obtain the teaching content text and other voice content of the classroom teaching, and the teaching content text of the classroom teaching is split into multiple independent language texts. The two types of text, dual-object language text and single-object language text, are identified through the timestamps and semantic coherence indicators between the independent language texts. Synchronize dual-object language texts, single-object language texts and other voice content to each teaching link, analyze the classroom teaching quality according to dual-object language texts, single-object language texts, audio proportions and other voice content, and visualize the multi-faceted teaching quality and comprehensive classroom teaching quality to help adjust or deploy teaching strategies; in, Analyze the mixed properties of the multi-audio mixed signal segment and determine the ICA processing parameters based on the mixed properties, including: The mixed properties of the multi-audio mixed signal segment include the degree of sound source overlap and the relative signal-to-noise ratio. The processing parameters of ICA are a combination of the learning rate and the number of iterations. The sound source overlap degree is obtained according to the size of the overlapping area between the multiple clumps, and the single audio mixed signal segment with the highest signal-to-noise ratio is found among the multiple single audio mixed signal segments, and the single audio mixed signal segment is used as the reference signal segment; Dividing the multi-audio mixed signal segment into a plurality of partial signal segments, calculating the similarity between each partial signal segment and a reference signal segment, and determining the contribution power of each partial signal segment according to the similarity, thereby determining the signal power and noise power of the multi-audio mixed signal segment, and calculating the relative signal-to-noise ratio through the signal power and the noise power; Generate processing levels based on the degree of sound source overlap and relative signal-to-noise ratio; Wherein, L is the processing level, α1 is the conversion coefficient of the sound source overlap degree, Q1 is the sound source overlap degree, α2 is the conversion coefficient of the relative signal-to-noise ratio, Q2 is the relative signal-to-noise ratio, and b is a preset constant; The learning rate and number of iterations of ICA are set according to the processing level. Different processing levels correspond to different combinations of learning rate and number of iterations of ICA.

2. The classroom teaching behavior analysis method based on semantic understanding according to claim 1 is characterized in that: Configure the audio weight of each teaching link according to the importance of the audio data and the nature of the link, including: Determine the number of data types of multi-source data and the reflection ratio of each type of data according to the nature of the teaching link, and determine the audio weight range of audio data in the multi-source data according to the number of data types of multi-source data and the reflection ratio of each type of data; Acquire the collection records of multi-source data in the teaching link, analyze the audio quality of the audio data and the effective data volume in the audio data and the effective data volume of other multi-source data in the collection records of the multi-source data, determine the effective data volume ratio of the audio data by the effective data volume in the audio data and the effective data volume of other multi-source data, and determine the adjustment ratio in combination with the audio quality and the effective data volume ratio of the audio data; The audio weight in each teaching link is configured according to the median and adjustment ratio within the audio weight range of the audio data in each teaching link.

3. The classroom teaching behavior analysis method based on semantic understanding according to claim 1 is characterized in that: Identify multiple audio mixed signal segments in the audio data of classroom teaching, including: Obtaining a complete audio data segment according to the audio data of the classroom teaching, and dividing the complete audio data segment into a plurality of audio sub-segments according to the distribution of valid data in the complete audio data segment; The audio signal in the audio sub-segment is converted into a time domain representation to obtain a spectrogram. In the spectrogram, potential sound sources are identified by using an energy threshold, and are classified into a potential sound source set after statistics. A time-frequency mask is generated for each potential sound source in the potential sound source set. Clusters in the spectrogram are identified by using the time-frequency mask and a preset algorithm. The overlap of the time-frequency mask in each audio sub-segment is described according to the size of the overlapping area between multiple clusters, thereby identifying multi-audio mixed signal segments and single-audio mixed signal segments.

4. The classroom teaching behavior analysis method based on semantic understanding according to claim 1 is characterized in that: Through NLP technology, independent audio signals are recognized and converted into text to obtain the teaching content text and other voice content of classroom teaching, including: By converting and analyzing independent audio signals through NLP technology, we can obtain the teaching content text and other voice content of classroom teaching. The teaching content text of classroom teaching is a continuous long content text, and other voice content includes emotional features, speaking speed features, intonation features and non-voice features.

5. The classroom teaching behavior analysis method based on semantic understanding according to claim 4 is characterized in that: And split the teaching content text of classroom teaching into multiple independent language texts, including: Performing initial splitting according to the timestamp of the teaching content text of the classroom teaching to obtain multiple segments of initial independent language text, distinguishing the main object of each initial independent language text, performing part-of-speech tagging on each initial independent language text, and judging whether each initial independent language text is semantically complete through the part-of-speech dependency tree and the main object; For an initial independent language text with incomplete semantics, determine the missing parts of speech in the initial independent language text, so as to evaluate the degree of missing of the initial independent language text, and map a proximity according to the degree of missing, where the proximity is the length of time. Other initial independent language texts with incomplete semantics within the proximity of the context of the initial independent language text are classified into a set of texts to be divided, and the semantic association between each text in the set of texts to be divided and the initial independent language text is calculated respectively. Combined with the semantic association and the part-of-speech dependency tree, the texts in the set of texts to be divided are screened, and the screened texts and the initial independent language text are taken as independent language texts together, and the semantically complete initial independent language text is taken as an independent language text.

6. The classroom teaching behavior analysis method based on semantic understanding according to claim 5 is characterized in that: Two types of texts, dual-object language texts and single-object language texts, are identified by using timestamps and semantic coherence indicators between independent language texts, including: detecting whether there is a dialogue mark in the independent language texts according to the timestamps between the independent language texts, the subject objects of the independent language texts, and the interaction keywords between the independent language texts; For independent language texts with dialogue marks, the context of the independent language text is screened according to the timestamp, and the frequency of similar keywords, the distance between similar keywords, the semantic relevance and the degree of logical order between the independent language texts in the screened context and the independent language texts with dialogue marks or between the independent language texts in the screened context are calculated. The semantic coherence index is determined based on the frequency of similar keywords, the distance between similar keywords, the semantic relevance and the degree of logical order. Among them, G i，j is the semantic coherence index between the ith independent language text and the jth independent language text, τ1 is the weight corresponding to similar keywords, β1 and β2 are the conversion coefficients of the frequency of similar keywords and the spacing between similar keywords, respectively, and W1 i，j 、W2 i，j are the frequency of similar keywords and the distance between similar keywords between the i-th independent language text and the j-th independent language text, τ2 is the semantic weight of the independent language text, β3 is the conversion coefficient of semantic relevance, E1 i，j 、E2 i，j are the semantic association degree and logical order degree between the ith independent language text and the jth independent language text, respectively, and k is a preset constant; According to the semantic coherence index, the independent language texts in the context after the independent language texts with dialogue marks are screened are traversed, and the independent language texts that meet the requirements of the semantic coherence index are taken as dual-object language texts; For independent language texts without dialogue marks, matching is performed between the independent language texts according to preset matching conditions, thereby determining the single object language text.

7. The classroom teaching behavior analysis method based on semantic understanding according to claim 1 is characterized in that: Analyze the classroom teaching quality of classroom teaching based on dual-object language text, single-object language text, audio proportion and other voice content, including, The teaching behavior is evaluated according to the dual-object language text, single-object language text, and other voice content in each teaching link, and the teaching quality index of a single teaching link is obtained. The classroom teaching quality of the classroom teaching is determined by combining the audio proportion and the quality index of a single teaching link.

8. The classroom teaching behavior analysis system based on semantic understanding is characterized by: For implementing the classroom teaching behavior analysis method based on semantic understanding as described in any one of claims 1 to 7, the system comprises: A determination module is used to obtain a class schedule for classroom teaching, analyze the class schedule for classroom teaching to determine the type of classroom teaching that exists, and process the teaching content of classroom teaching in a standardized process according to the type of classroom teaching, obtain a standardized process for classroom teaching under each type, and define each standardized process into multiple teaching links, and configure the audio weight under each teaching link according to the importance of the audio data of the teaching link and the nature of the link; A separation module is used to obtain audio data of classroom teaching, identify multi-audio mixed signal segments in the audio data of classroom teaching, analyze the mixed properties of the multi-audio mixed signal segments, determine ICA processing parameters based on the mixed properties, and perform audio separation in the multi-audio mixed signal segments through the ICA processing parameters and ICA method to obtain multiple independent audio signals; The recognition module is used to perform speech recognition and text conversion on independent audio signals through NLP technology to obtain the teaching content text and other speech content of classroom teaching, and to split the teaching content text of classroom teaching into multiple independent language texts, and to identify two types of texts, dual-object language text and single-object language text, through the timestamps and semantic coherence indicators between the independent language texts; The analysis module is used to synchronize dual-object language texts, single-object language texts and other voice contents to each teaching link, analyze the classroom teaching quality according to the dual-object language texts, single-object language texts, audio proportions and other voice contents, and visualize the multi-faceted teaching quality and the comprehensive quality of classroom teaching, so as to help adjust or deploy teaching strategies.

Images