A method and system for joint identification of dual classroom states based on individual teacher baseline alignment

Abstract

This invention discloses a method and system for joint recognition of dual classroom states based on individual teacher baseline alignment. The method includes: acquiring classroom audio and separating the teacher's speech sequence; extracting multi-dimensional acoustic features from the teacher's speech in each time window to construct a current classroom acoustic vector; obtaining an individual baseline acoustic vector constructed based on the teacher's historical normal teaching samples; extracting fatigue-related features and interaction-related features from the current vector and baseline vector respectively to form fatigue state sub-vectors and interaction state sub-vectors; calculating fatigue deviation values ​​and interaction deviation values; determining state evolution features based on the deviation value sequence of multiple consecutive time windows; and jointly recognizing the teacher's acoustic fatigue state and classroom interaction state based on fatigue, interaction deviation values, and state evolution features. This invention achieves accurate recognition of the teacher's classroom teaching state by introducing individual baseline alignment, dual-state sub-vector separation calculation, and time-series evolution analysis.

Description

Technical Field

[0001] This invention relates to the field of classroom audio analysis and educational informatization technology, and more specifically to a method and system for joint recognition of dual classroom states based on individual teacher baseline alignment. Background Technology

[0002] Currently, in the field of classroom teaching quality assessment, how to objectively and accurately identify teachers' teaching status is an important issue. Existing technologies mainly approach this from two independent directions:

[0003] The first category is voice fatigue detection technology. For example, the scheme disclosed in patent document publication number CN118588114A: collects voice data of the person being tested, preprocesses the voice data, segments it, and extracts features, then uses a fatigue detection model to predict the degree of fatigue, and triggers an early warning when the degree of fatigue exceeds a threshold. The focus of this type of scheme is to identify and warn of the fatigue state of a general object.

[0004] The second category is classroom teaching audio monitoring and teaching segment management technology. For example, the solution disclosed in patent document publication number CN119850385A involves recording the teacher's lectures, dividing the lecture sentences, further constructing teaching monitoring segments, and classifying these segments into efficient or inefficient segments based on indicators such as teaching emotion values, before implementing subsequent management. The focus of this type of solution is on remote teaching monitoring and teaching segment quality management.

[0005] In summary, the existing technology has the following drawbacks: (1) Existing technologies are usually only designed for a single task such as fatigue state recognition or teaching emotion monitoring, and lack the ability to jointly recognize the "vocal fatigue state" and "interaction state" of teachers in the classroom, resulting in a relatively simple dimension of classroom state analysis.

[0006] (2) Existing technologies mostly use general speaker models or uniform thresholds, without establishing individual baseline models for individual differences in timbre, speech rate, pause habits, volume style, etc., which can easily lead to cross-teacher misjudgment.

[0007] (3) Most existing technologies are designed to judge single audio segments or static teaching segments, lacking the evolutionary analysis of the teacher's state changes within a continuous time window in the classroom, and are difficult to reflect the state fluctuations in the actual teaching process.

[0008] Therefore, a technical solution is needed that can combine individual differences among teachers and continuously, jointly, and accurately identify vocal fatigue and interactive states in classroom scenarios. Summary of the Invention

[0009] To address the problems of existing technologies that only identify voice fatigue or manage teaching emotions and teaching segments, and lack individual baseline and temporal evolution analysis mechanisms, this invention provides a classroom dual-state joint identification method and system based on teacher individual baseline alignment.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a method for joint identification of dual classroom states based on individual teacher baseline alignment, comprising: Collect classroom audio data and extract the teacher's speech sequence from the classroom audio data; For the teacher's speech segments divided by time windows in the teacher's speech sequence, acoustic features are extracted and a current classroom acoustic vector is constructed; Obtain the individual baseline vocal vector of the teacher, which is constructed based on audio samples of the teacher's historical normal teaching. From the current classroom acoustic vector and the individual baseline acoustic vector, fatigue-related features are extracted to form a fatigue state sub-vector, and interaction-related features are extracted to form an interaction state sub-vector. Calculate the fatigue deviation between the fatigue state subvector of the current window and the corresponding baseline subvector, and the interaction deviation between the interaction state subvector of the current window and the corresponding baseline subvector. Based on the fatigue deviation value and the interaction deviation value of multiple consecutive time windows, or based on the deviation between the current teacher's overall classroom acoustic vector and the individual baseline acoustic vector, the state evolution characteristics are determined. Based on the fatigue deviation value, the interaction deviation value, and the state evolution characteristics of the current window, the teacher's vocal fatigue state and classroom interaction state are jointly identified to obtain a dual-state identification result.

[0011] In one embodiment, extracting acoustic features and constructing the current classroom acoustic vector includes: The extracted acoustic features include at least two of the following: fundamental frequency features, prosodic features, energy features, spectral features, noise quality features, and interaction-related features; The fundamental frequency characteristics include: fundamental frequency mean, fundamental frequency standard deviation, and fundamental frequency fluctuation amplitude; The prosodic features include: speech rate, average pause duration, and pause percentage; The energy characteristics include: short-term energy average and short-term energy fluctuation; The spectral characteristics include: formant position, spectral centroid, and harmonic-to-noise ratio; The noise quality characteristics include: jitter and amplitude perturbation; The interactive features include: continuous speaking duration, proportion of question-type speech segments, frequency of intensity boosts, and number of pitch rises; The extracted acoustic feature values ​​are normalized and combined in a preset dimensional order to form the current classroom acoustic vector.

[0012] In one embodiment, the fatigue-related features include at least two of the following: jitter, amplitude perturbation, average pause duration, pause percentage, mean fundamental frequency, standard deviation of fundamental frequency, and fundamental frequency fluctuation amplitude. The interaction-related features include at least two of the following: speech rate, continuous speaking duration, proportion of question-type speech segments, frequency of intensity boosts, number of pitch rises, short-term energy mean, short-term energy fluctuations, and intonation variation features; the intonation variation features include: formant position, spectral centroid, or harmonic-to-noise ratio.

[0013] In one embodiment, the calculation of the fatigue deviation value and the interaction deviation value includes: The fatigue deviation value is obtained by calculating the weighted sum of the feature differences of each dimension between the fatigue state sub-vector and its baseline sub-vector; and the interaction deviation value is obtained by calculating the weighted sum of the feature differences of each dimension between the interaction state sub-vector and its baseline sub-vector. The weighting coefficients are either pre-set based on the importance of features or learned from historical data.

[0014] In one embodiment, determining state evolution characteristics based on the fatigue deviation values ​​and the interaction deviation values ​​over multiple consecutive time windows includes: Trend analysis is performed on the fatigue deviation value sequence of the consecutive multiple time windows to obtain the fatigue state change trend characteristics; Trend analysis is performed on the interaction deviation value sequence of the multiple consecutive time windows to obtain the trend characteristics of interaction status changes; The fatigue state change trend features and the interaction state change trend features are combined and input into a preset evolution function to obtain the state evolution features.

[0015] In one embodiment, the trend analysis includes calculating the slope of the sequence, a moving average, or extracting features using a time series prediction model.

[0016] In one embodiment, the joint identification of the teacher's vocal fatigue state and classroom interaction state includes: The fatigue deviation value is compared with multiple preset fatigue level threshold ranges to determine the acoustic fatigue state level; The interaction deviation value is compared with multiple preset interaction level threshold ranges to determine the classroom interaction status level; Based on the changing trends indicated by the state evolution characteristics, the acoustic fatigue state level and the classroom interaction state level are corrected or composite state labels are generated.

[0017] Secondly, embodiments of the present invention provide a classroom dual-state joint identification system based on individual teacher baseline alignment, using the classroom dual-state joint identification method based on individual teacher baseline alignment as described in any one of the first aspects. The system includes: The acquisition and separation module is used to acquire classroom audio data and separate the teacher's speech sequence from the classroom audio data; The extraction and construction module is used to extract acoustic features and construct the current classroom acoustic vector for teacher speech segments divided by time windows in the teacher speech sequence. The baseline acquisition module is used to acquire the individual baseline acoustic vector of the teacher, which is constructed based on audio samples of the teacher's historical normal teaching. The fatigue and interaction vector extraction module is used to extract fatigue-related features to form a fatigue state sub-vector and extract interaction-related features to form an interaction state sub-vector from the current classroom acoustic state vector and the individual baseline acoustic state vector, respectively. The deviation calculation module is used to calculate the fatigue deviation value between the fatigue state sub-vector of the current window and the corresponding baseline sub-vector, as well as the interaction deviation value between the interaction state sub-vector of the current window and the corresponding baseline sub-vector. The state evolution determination module is used to determine state evolution characteristics based on the fatigue deviation value and the interaction deviation value of multiple consecutive time windows, or based on the deviation between the current teacher's overall classroom acoustic state vector and the individual baseline acoustic state vector. The identification module is used to jointly identify the teacher's vocal fatigue state and classroom interaction state based on the fatigue deviation value, the interaction deviation value, and the state evolution characteristics of the current window, and obtain a dual-state identification result.

[0018] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the classroom dual-state joint recognition method based on teacher individual baseline alignment as described in any one of the first aspects.

[0019] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the classroom dual-state joint identification method based on teacher individual baseline alignment as described in any one of the first aspects.

[0020] The descriptions of the second and third aspects of this invention can be referred to the detailed description of the first aspect; and the beneficial effects described in the second and third aspects can be referred to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0021] As can be seen from the above technical solution, compared with the prior art, the present invention has the following technical advantages: This invention collects classroom audio and separates the teacher's voice to extract multidimensional acoustic features reflecting the teacher's vocal state, constructing a current classroom vocal state vector. Further, it constructs an individual baseline vocal state vector based on historical normal teaching samples of the same teacher, and decomposes the current classroom vocal state vector into fatigue state sub-vectors and interaction state sub-vectors, calculating the deviation from the corresponding baselines. Based on this, it constructs state evolution features by combining the deviation change trend within a continuous time window, and jointly identifies the teacher's vocal fatigue state and interaction state, thereby achieving dynamic analysis and result output of the teacher's teaching state in classroom scenarios.

[0022] This invention differs from existing general speech fatigue detection methods, as it does not focus on single fatigue recognition and over-threshold warnings; it also differs from existing classroom teaching audio monitoring schemes, as it does not focus on teaching sentence segmentation, teaching emotion value calculation, and high / low efficiency teaching segment management; instead, it achieves continuous, objective, and interpretable identification of teachers' teaching status during classroom teaching by introducing a teacher individual baseline alignment mechanism, a bivariate vector deviation calculation mechanism, and a continuous time window state evolution analysis mechanism.

[0023] In practice, relying solely on a single fatigue indicator for identification can easily misjudge short-term increases in speaking speed or energy caused by high interaction as abnormal fatigue. Conversely, relying solely on interaction indicators makes it difficult to distinguish the cumulative fatigue caused by high-intensity lecturing. Therefore, a dual-state joint identification mechanism is needed to improve state discrimination. Since teachers in the classroom exhibit multiple complex states such as "high interaction but mild fatigue," "low interaction but no fatigue," and "high fatigue and low interaction," identifying only the fatigue state or the interaction state separately can lead to misjudgments. Therefore, this application improves the discrimination and accuracy of classroom teaching state determination through a dual-state joint identification mechanism. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1This is a flowchart of the classroom dual-state joint identification method based on individual teacher baseline alignment provided in this embodiment of the invention; Figure 2 This is a block diagram of a classroom dual-state joint recognition system based on individual teacher baseline alignment provided in an embodiment of the present invention; Figure 3 This is a structural diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1: Reference Figure 1 As shown, this invention discloses a classroom dual-state joint recognition method based on teacher individual baseline alignment. This method separates, models, and calculates the state of the teacher's voice in classroom audio to form a joint recognition result of the teacher's teaching state for classroom scenarios; specifically, it includes the following steps S1~S7: S1. Collect classroom audio data and extract the teacher's speech sequence from the classroom audio data; This step acquires raw audio data from the classroom setting. The classroom audio can come from one or more of the following: classroom microphones, recording equipment, sound reinforcement equipment, or wearable microphones. The acquired audio must include at least the teacher's speech, and may further include student speech, ambient noise, and equipment noise.

[0028] To improve the accuracy of subsequent recognition, noise reduction, echo suppression, silence detection, and gain normalization can be performed on the classroom audio to obtain a standardized classroom audio stream.

[0029] Teacher speech is separated from standardized classroom audio streams to obtain teacher speech sequences. Teacher speech separation can be achieved through speaker detection, pickup channel filtering, speaker separation, or a combination of these methods.

[0030] S2. For the teacher speech segments divided by time window in the teacher speech sequence, extract acoustic features and construct the current classroom acoustic vector; wherein, in the process of acoustic feature extraction, the feature dimensions can be adjusted according to specific application needs, such as adding spectral entropy, zero-crossing rate, short-time energy change rate or stop-connection ratio features.

[0031] The teacher's speech is divided into multiple continuous speech segments according to a preset time window, with each segment corresponding to a time window t. For each teacher's speech segment, multidimensional acoustic features are extracted to characterize the teacher's vocal state. These acoustic features include at least two or more of the following: (1) Fundamental frequency characteristics, including fundamental frequency mean, fundamental frequency standard deviation, and fundamental frequency fluctuation amplitude; (2) Prosodic features, including speech rate, average pause duration, and pause percentage; (3) Energy characteristics, including short-term energy average and short-term energy fluctuation; (4) Spectral characteristics, including formant position, spectral centroid, and harmonic-to-noise ratio; (5) Voice quality characteristics, including vibrato and amplitude perturbation; (6) Interactive features, including continuous speaking duration, proportion of question-type speech segments, frequency of intensity increase, and number of pitch rises.

[0032] For example, mapping each acoustic feature to a current classroom acoustic vector according to a preset dimensional order: V t =[p t , r t , e t , s t , f t , j t , h t , q t ] in: p t It represents the characteristic value of fundamental frequency fluctuation, such as the fundamental frequency mean, fundamental frequency standard deviation, or fundamental frequency fluctuation amplitude; r t This represents the speech rate characteristic value; that is, the speech rate in the prosodic features. e t It represents energy characteristic values; for example, it could be a short-term energy average or a short-term energy fluctuation. s t This indicates a pause characteristic value; for example, it could be the average pause duration or the percentage of pauses. f t These represent stable spectral characteristics; for example, they could be the formant positions, spectral centroids, or harmonic-to-noise ratios that indicate pitch variations. j t Indicates jitter characteristic value; h t This represents the characteristic value of amplitude perturbation; q t These represent interactive-related features, such as continuous speaking duration, the proportion of question-type speech segments, the frequency of volume increases, and the number of times the tone rises.

[0033] To eliminate the influence of different dimensions, the features of each dimension can be processed by min-max normalization, Z-score standardization, or piecewise mapping, and then concatenated to form a current classroom acoustic vector with a unified dimension.

[0034] S3. Obtain the individual baseline acoustic vector of the teacher, which is constructed based on audio samples of the teacher's historical normal teaching. In the process of constructing the individual baseline, the mean method of historical normal classroom samples, the sliding baseline update method, or the scenario-based baseline construction method can be used.

[0035] Based on classroom audio samples from the same teacher during normal teaching sessions, individual baseline acoustic vectors are constructed: V base = [p0, r0, e0, s0, f0, j0, h0, q0] Among them, V base This represents the reference acoustic baseline of the teacher under normal teaching conditions, and each sub-vector represents the reference acoustic baseline of each feature value of the teacher under normal teaching conditions.

[0036] Historical normal teaching samples can be obtained from pre-labeled normal classroom segments or from multiple classroom audio segments without abnormal markers after statistical analysis. The purpose of this step is to reduce the interference of inherent differences in timbre, speaking habits, natural speaking speed, and volume style among different teachers on the recognition results.

[0037] S4. Extract fatigue-related features from the current classroom acoustic vector and the individual baseline acoustic vector to form a fatigue state sub-vector, and extract interaction-related features to form an interaction state sub-vector; For example, fatigue-related dimensions and interaction-related dimensions can be extracted from the current classroom acoustic vector and the individual baseline acoustic vector, respectively. The selection of dimensions for the fatigue and interaction sub-vectors can be adjusted according to different classroom scenarios.

[0038] The fatigue state sub-vector is: F t = [j t , h t , s t , p t ] F base = [j0, h0, s0, p0] F t F represents the acoustic vector related to current classroom fatigue; base This represents the vocal state vector related to individual baseline fatigue. The fatigue state sub-vector is used to characterize the teacher's current vocal fatigue-related state, involving at least the degree of tremor, amplitude perturbation, pause ratio, and fundamental frequency fluctuation instability.

[0039] The interaction state subvector is: I t = [r t , q t , e t , f t ] I base = [r0, q0, e0, f0] Among them, I t f represents the acoustic vector related to the current classroom interaction state; t These represent characteristic values ​​related to intonation changes, such as formant positions, spectral centroids, or harmonic-to-noise ratios. base This represents the vocal state vector related to the individual's baseline classroom interaction state. The interaction state sub-vector is used to characterize the teacher's current classroom interaction guidance state, involving at least changes in speech rate, interaction-related features, energy changes, and tone changes.

[0040] S5. Calculate the fatigue deviation value between the fatigue state sub-vector of the current window and the corresponding baseline sub-vector, and the interaction deviation value between the interaction state sub-vector of the current window and the corresponding baseline sub-vector; wherein, in the process of calculating the dual-state deviation, weighted Manhattan distance, Euclidean distance, regular function or similarity calculation method can be used.

[0041] In step S5, the deviations between the fatigue state sub-vector and the interaction state sub-vector and their respective baselines are calculated: ΔF t = F t - F base ΔI t = I t - I base For example, further calculate fatigue deviation values ​​and interaction deviation values: D F (t) = α1·|j t -j0| + α2·|h t -h0| + α3·|s t -s0| + α4·|p t -p0| D I (t) = β1·|r t -r0| + β2·|q t -q0| + β3·|e t -e0| + β4·|f t -f0| Wherein, α1~α4 and β1~β4 are the weighting coefficients corresponding to each dimension of the feature. The weighting coefficients for fatigue state and interaction state can be determined by preset empirical parameters or based on the statistical results of historical classroom audio samples.

[0042] S6. Determine the state evolution characteristics based on the fatigue deviation value and the interaction deviation value of multiple consecutive time windows, or based on the deviation between the current teacher's overall classroom acoustic vector and the individual baseline acoustic vector. This step involves two methods to determine the state evolution characteristics: (1) In order to reflect the continuous changes in the teacher's state during the classroom process, D can be further analyzed within multiple consecutive time windows. F (t) and D I (t) Perform time series analysis to obtain state evolution characteristics: E t = g(D F (tn), …, D F (t), D I (tn), …, D I (t)) Among them, E t The state evolution characteristics corresponding to time t (the index of the time window currently being analyzed and identified) are represented by g(·), where g(·) represents the moving average function, trend fitting function, difference function, or time series modeling function; and n represents the offset of the time window.

[0043] (2) Calculate the deviation between the current teacher's classroom vocal dynamics vector and the individual baseline vocal dynamics vector: ΔV t =V t -V base Further calculate the overall state deviation value: D t = w1·|p t -p0| + w2·|r t -r0| + w3·|e t -e0| + w4·|s t -s0| + w5·|f t -f0| +w6·|j t -j0| + w7·|h t -h0| + w8·|q t -q0| Where w1 to w8 are the weight coefficients corresponding to each feature dimension.

[0044] To reflect the continuous changes in the teacher's state during the classroom process, a time series analysis can be performed on the deviation values ​​of multiple consecutive time windows to obtain the state evolution characteristics: E t = g(D (t-n) , …, D t ) Among them, E t The state evolution characteristics corresponding to time t are represented by g(·), which represents the moving average function, trend fitting function, difference function, or time series modeling function; n represents the offset of the time window.

[0045] S7. Based on the fatigue deviation value, the interaction deviation value, and the state evolution characteristics of the current window, the teacher's vocal fatigue state and classroom interaction state are jointly identified to obtain a dual-state identification result. In the dual-state joint identification process, rule-based judgment, traditional machine learning classification, time-series model prediction, or multi-model fusion methods can be used.

[0046] Based on fatigue deviation value D F (t), Interaction Deviation Value D I (t) and state evolution characteristics E t The system performs joint identification of both classroom states for the teacher's current status. In one implementation, the following two types of state results can be output: (1) Acoustic fatigue state: normal, mild fatigue, moderate fatigue, severe fatigue; (2) Interaction status: low interaction, normal interaction, high interaction, strong guided interaction.

[0047] In another implementation, a joint state label may be further formed, for example: Normal lecture + general interaction; Mild fatigue + high level of interaction; Moderate fatigue + low interaction; High-pressure regulation + strong guidance and interaction; Among them, the determination of vocal fatigue state is mainly based on characteristics such as jitter, amplitude perturbation, pause ratio and fundamental frequency instability; the determination of interactive state is mainly based on characteristics such as speech rate change, interactive related characteristics, frequency of intensity increase and number of pitch rises.

[0048] Finally, the recognition results can be output to the upper-level application system for analysis of teachers' classroom status, auxiliary assessment of classroom quality, monitoring of teachers' teaching workload, or support for teaching and research review.

[0049] The output may include: (1) The teacher's vocal fatigue level corresponding to the current time window; (2) The teacher interaction status level corresponding to the current time window; (3) The dual-state change curve throughout the entire classroom process; (4) Joint analysis report of dual classroom states.

[0050] The thresholds for each status level can be dynamically set based on sample distribution, expert annotation results, or normal ranges from historical classroom lessons. Furthermore, during the output process, real-time streaming output, post-class batch processing output, or a combination of real-time and offline output can be used.

[0051] The implementation example is as follows: Numerical samples for joint identification of dual classroom states based on teacher baseline alignment; Taking a 40-minute math lesson audio recording as an example, the teacher's voice is segmented into 20-second fixed windows, resulting in 120 continuous voice segments. Based on 10 previous normal teaching lessons, the teacher's individual baseline vocal state vector is statistically obtained as follows: V base = [p0, r0, e0, s0, f0, j0, h0, q0]=[0.58, 0.64, 0.60, 0.22, 0.66,0.11, 0.10, 0.52] in: p0 is the baseline value of the fundamental frequency fluctuation characteristic, which can be the fundamental frequency mean, fundamental frequency standard deviation, or fundamental frequency fluctuation amplitude. r0 is the baseline value for speech rate characteristics; e0 is the baseline value of energy characteristics, such as short-term energy average or short-term energy fluctuation; s0 is the baseline value for the pause characteristics, which may be the average pause duration or the percentage of pauses. f0 is the baseline value of the spectral stability feature, such as the formant position, spectral centroid, or harmonic-to-noise ratio representing the pitch change; j0 is the baseline value of the jitter characteristic; h0 is the baseline value of the amplitude perturbation characteristic; q0 is the baseline value of interaction-related features, which can be continuous speaking duration, proportion of question-type speech segments, frequency of intensity boosts, number of pitch rises, etc.

[0052] In the 61st time window, that is, from 20 minutes and 20 seconds to 20 minutes and 40 seconds into the lesson, the current classroom acoustic vector is extracted as follows: V t =[p t , r t , e t , s t , f t , j t , ht , q t = [0.72, 0.81, 0.76, 0.34, 0.61, 0.28, 0.24, 0.79] Furthermore, fatigue state sub-vectors and interaction state sub-vectors are extracted from the current classroom acoustic state vector and the individual baseline acoustic state vector.

[0053] 1. Extraction of fatigue state sub-vectors; Let the fatigue state subvector be: F t = [j t , h t , s t , p t = [0.28, 0.24, 0.34, 0.72] The corresponding fatigue baseline subvector is: F base = [j0, h0, s0, p0]=[0.11, 0.10, 0.22, 0.58] The fatigue state deviation vector is then: ΔF t = F t - F base = [0.28-0.11, 0.24-0.10, 0.34-0.22, 0.72-0.58] = [0.17,0.14, 0.12, 0.14] Let the weight coefficients of each dimension of the fatigue state feature be: α = [α1, α2, α3, α4] = [0.30, 0.28, 0.22, 0.20] The formula for calculating fatigue deviation is: D F (t) = α1·|j t -j0| + α2·|h t -h0| + α3·|s t -s0| + α4·|p t -p0| Substituting into the above formula, we get: D F (t)= 0.30×0.17 + 0.28×0.14 + 0.22×0.12 + 0.20×0.14 D F (t)= 0.051 + 0.0392 + 0.0264 + 0.028 = 0.1446 That is: DF (t) = 0.1446.

[0054] 2. Extraction of interactive state sub-vectors; Let the interaction state subvectors be: I t = [r t , q t , e t , f t ] Among them, f t This represents feature values ​​related to intonation changes, which can be formant positions, spectral centroids, or harmonic-to-noise ratios. Let the current window extract: f t = 0.68; The historical baseline value is: f0 = 0.49; Then: I t = [0.81, 0.79, 0.76, 0.68] The corresponding interaction baseline subvector is: I base = [r0, q0, e0, f0]= [0.64, 0.52, 0.60, 0.49] The interaction state deviation vector is then: ΔI t =I t -I base =[0.81-0.64, 0.79-0.52, 0.76-0.60, 0.68-0.49]=[0.17, 0.27,0.16, 0.19]; Let the weight coefficients of each dimension of the interactive state feature be: β = [β1, β2, β3, β4] = [0.22, 0.32, 0.20, 0.26]; The formula for calculating the interaction deviation value is: D I (t) = β1·|r t -r0| + β2·|q t -q0| + β3·|e t -e0| + β4·|f t -f0| Substituting into the above formula, we get: D I (t) = 0.22×0.17 + 0.32×0.27 + 0.20×0.16 + 0.26×0.19 D I(t) = 0.0374 + 0.0864 + 0.032 + 0.0494 = 0.2052 That is: D I (t) = 0.2052.

[0055] 3. Calculation of evolution characteristics within a continuous time window; To reflect the teacher's continuous state changes during the classroom process, fatigue deviation values ​​and interaction deviation values ​​from the five consecutive time windows preceding the current window are selected and denoted as follows: Fatigue deviation sequence: [D F (t-4), D F (t-3), D F (t-2), D F (t-1), D F (t)]=[0.0810, 0.0975, 0.1138,0.1294, 0.1446]; Interactive bias sequence: [D I (t-4), D I (t-3), D I (t-2), D I (t-1), D I (t)] = [0.1620, 0.1785, 0.1914,0.1988, 0.2052]; The fatigue evolution slope is defined as: K f = (D F (t) - D F (t-4)) / 4 = (0.1446 - 0.0810) / 4 = 0.0159; The slope of the interaction evolution is defined as: K i = (D I (t) - D I (t-4)) / 4 = (0.2052 - 0.1620) / 4 = 0.0108; Further define the state evolution characteristics as: E t = [K f , K i ] = [0.0159, 0.0108]; Among them, K f K represents the rate at which fatigue deviation increases with the time window. iThis indicates the rate at which the interaction deviation increases over the time window.

[0056] 4. Dual-state joint identification and judgment; The rules for joint recognition of dual classroom states are as follows: (1) When D F When (t) < 0.08, it is judged as a normal acoustic fatigue state; When 0.08 ≤ D F When (t) < 0.16, it is judged to be a mild acoustic fatigue state; When 0.16 ≤ D F When (t) < 0.24, it is judged to be a moderate acoustic fatigue state; When D F When (t)≥ 0.24, it is determined to be a state of severe acoustic fatigue.

[0057] (2) When D I When (t) < 0.10, it is determined to be a low interaction state; When 0.10 ≤ D I When (t) < 0.16, it is determined to be a normal interactive state; When 0.16 ≤ D I When (t) < 0.22, it is determined to be a high-interaction state; When D I When (t) ≥ 0.22, it is determined to be a strongly guided interactive state.

[0058] (3) If K f If the value is greater than 0.01, it is considered that the fatigue state has a continuous upward trend. If K i If the value is greater than 0.008, it is considered that the interaction status shows a continuous increasing trend.

[0059] Based on the aforementioned calculation results: D F (t) = 0.1446, falling within the interval [0.08, 0.16), therefore it is determined to be: mild acoustic fatigue state.

[0060] D I (t) = 0.2052, falling within the interval [0.16, 0.22), therefore it is determined to be a high-interaction state.

[0061] at the same time: K f = 0.0159>0.01, indicating that the fatigue level continues to rise; K i = 0.0108>0.008, indicating that the interaction status continues to increase.

[0062] Therefore, the joint identification result of the dual states in the classroom during this time window is: mild vocal fatigue + high interaction.

[0063] 5. Example of subsequent window determination; At the 70th time window, the extracted data is as follows: D F (70) = 0.1824; D I (70) = 0.1480; According to the above rules: D F (70) = 0.1824, falling within the interval [0.16, 0.24), and is judged as: moderate acoustic fatigue state; D I (70) = 0.1480, falling within the interval [0.10, 0.16), and is determined to be: normal interactive state.

[0064] Therefore, the 70th time window can be identified as: moderate acoustic fatigue + general interaction.

[0065] 6. Output the results; Based on the above calculation process, the output can be: (1) Identification results of the 61st time window: mild acoustic fatigue + high interaction; (2) Identification results of the 70th time window: moderate acoustic fatigue + general interaction; (3) Throughout the class, the teacher's state gradually evolved from "high interaction accompanied by mild fatigue" to "decreased interaction and increased fatigue".

[0066] This enables continuous joint identification of the teacher's dual states during classroom teaching.

[0067] This invention provides a classroom dual-state joint recognition method based on individual teacher baseline alignment. First, by constructing a current classroom vocal state vector, it achieves a unified expression of multi-dimensional acoustic features in classroom speech, making the teacher's teaching state calculable and analyzable. Furthermore, by introducing individual baseline vocal state vectors, it reduces recognition errors caused by individual differences among teachers in timbre, speech rate, volume habits, etc., improving recognition accuracy and stability. Next, by decomposing the current classroom vocal state vector into fatigue state sub-vectors and interaction state sub-vectors, and calculating the deviations separately, it achieves separate modeling and collaborative analysis of the teacher's dual classroom states. Then, by performing evolutionary analysis on the dual deviation values ​​within a continuous time window, it reflects the dynamic changes in the teacher's classroom teaching state, rather than just providing a static single-point judgment. Finally, by jointly recognizing vocal fatigue state and interaction state, it overcomes the limitations of single fatigue recognition or single teaching emotion management in existing technologies, realizing a transformation from "general voice fatigue detection" or "teaching emotion monitoring" to "joint recognition of teacher dual states for classroom teaching scenarios."

[0068] Example 2: Based on the same inventive concept, this invention also provides a classroom dual-state joint identification system based on teacher individual baseline alignment. Since the principle of the problem solved by this system is similar to the aforementioned classroom dual-state joint identification method based on teacher individual baseline alignment, the implementation of this system can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0069] like Figure 2 As shown, this embodiment provides a classroom dual-state joint identification system based on individual teacher baseline alignment, using the classroom dual-state joint identification method based on individual teacher baseline alignment as described in any of Embodiment 1. The system includes: The acquisition and separation module is used to acquire classroom audio data and separate the teacher's speech sequence from the classroom audio data; The extraction and construction module is used to extract acoustic features and construct the current classroom acoustic vector for teacher speech segments divided by time windows in the teacher speech sequence. The baseline acquisition module is used to acquire the individual baseline acoustic vector of the teacher, which is constructed based on audio samples of the teacher's historical normal teaching. The fatigue and interaction vector extraction module is used to extract fatigue-related features to form a fatigue state sub-vector and extract interaction-related features to form an interaction state sub-vector from the current classroom acoustic state vector and the individual baseline acoustic state vector, respectively. The deviation calculation module is used to calculate the fatigue deviation value between the fatigue state sub-vector of the current window and the corresponding baseline sub-vector, as well as the interaction deviation value between the interaction state sub-vector of the current window and the corresponding baseline sub-vector. The state evolution determination module is used to determine state evolution characteristics based on the fatigue deviation value and the interaction deviation value of multiple consecutive time windows, or based on the deviation between the current teacher's overall classroom acoustic state vector and the individual baseline acoustic state vector. The identification module is used to jointly identify the teacher's vocal fatigue state and classroom interaction state based on the fatigue deviation value, the interaction deviation value, and the state evolution characteristics of the current window, and obtain a dual-state identification result.

[0070] The modules are connected sequentially and work together to complete the entire process from audio input to dual-state result output.

[0071] Example 3: Reference Figure 3 As shown, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the classroom dual-state joint recognition method based on teacher individual baseline alignment as described in any one of Embodiments 1.

[0072] The electronic device may include a processor 10, a communication interface 20, a memory 30, and a communication bus 40, wherein the processor 10, the communication interface 20, and the memory 30 communicate with each other via the communication bus 40. The processor 10 can call logical instructions from the memory 30 to execute a classroom dual-state joint recognition method based on teacher individual baseline alignment, the method including: S1. Collect classroom audio data and extract the teacher's speech sequence from the classroom audio data; S2. Extract acoustic features from the teacher speech segments divided by time windows in the teacher speech sequence and construct the current classroom acoustic vector; S3. Obtain the individual baseline vocal vector of the teacher, which is constructed based on audio samples of the teacher's historical normal teaching. S4. Extract fatigue-related features from the current classroom acoustic vector and the individual baseline acoustic vector to form a fatigue state sub-vector, and extract interaction-related features to form an interaction state sub-vector; S5. Calculate the fatigue deviation value between the fatigue state sub-vector of the current window and the corresponding baseline sub-vector, and the interaction deviation value between the interaction state sub-vector of the current window and the corresponding baseline sub-vector. S6. Determine the state evolution characteristics based on the fatigue deviation value and the interaction deviation value of multiple consecutive time windows, or based on the deviation between the current teacher's overall classroom acoustic vector and the individual baseline acoustic vector. S7. Based on the fatigue deviation value, the interaction deviation value, and the state evolution characteristics of the current window, the teacher's vocal fatigue state and classroom interaction state are jointly identified to obtain a dual-state identification result.

[0073] Furthermore, the logical instructions in the aforementioned memory 30 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0074] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0075] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for joint identification of dual classroom states based on individual teacher baseline alignment, characterized in that, include: Collect classroom audio data and extract the teacher's speech sequence from the classroom audio data; For the teacher's speech segments divided by time windows in the teacher's speech sequence, acoustic features are extracted and a current classroom acoustic vector is constructed; Obtain the individual baseline vocal vector of the teacher, which is constructed based on audio samples of the teacher's historical normal teaching. From the current classroom acoustic vector and the individual baseline acoustic vector, fatigue-related features are extracted to form a fatigue state sub-vector, and interaction-related features are extracted to form an interaction state sub-vector. Calculate the fatigue deviation between the fatigue state subvector of the current window and the corresponding baseline subvector, and the interaction deviation between the interaction state subvector of the current window and the corresponding baseline subvector. Based on the fatigue deviation value and the interaction deviation value of multiple consecutive time windows, or based on the deviation between the current teacher's overall classroom acoustic vector and the individual baseline acoustic vector, the state evolution characteristics are determined. Based on the fatigue deviation value, the interaction deviation value, and the state evolution characteristics of the current window, the teacher's vocal fatigue state and classroom interaction state are jointly identified to obtain a dual-state identification result.

2. The method according to claim 1, characterized in that, The extraction of acoustic features and construction of the current classroom acoustic vector include: The extracted acoustic features include at least two of the following: fundamental frequency features, prosodic features, energy features, spectral features, noise quality features, and interaction-related features; The fundamental frequency characteristics include: fundamental frequency mean, fundamental frequency standard deviation, and fundamental frequency fluctuation amplitude; The prosodic features include: speech rate, average pause duration, and pause percentage; The energy characteristics include: short-term energy average and short-term energy fluctuation; The spectral characteristics include: formant position, spectral centroid, and harmonic-to-noise ratio; The noise quality characteristics include: jitter and amplitude perturbation; The interactive features include: continuous speaking duration, proportion of question-type speech segments, frequency of intensity boosts, and number of pitch rises; The extracted acoustic feature values ​​are normalized and combined in a preset dimensional order to form the current classroom acoustic vector.

3. The method according to claim 2, characterized in that, The fatigue-related characteristics include at least two of the following: jitter, amplitude perturbation, average pause duration, pause percentage, mean fundamental frequency, standard deviation of fundamental frequency, and fundamental frequency fluctuation amplitude. The interaction-related features include at least two of the following: speech rate, continuous speaking duration, proportion of question-type speech segments, frequency of intensity boosts, number of pitch rises, short-term energy mean, short-term energy fluctuations, and intonation variation features; the intonation variation features include: formant position, spectral centroid, or harmonic-to-noise ratio.

4. The method according to claim 1, characterized in that, The calculation of the fatigue deviation value and the interaction deviation value includes: The fatigue deviation value is obtained by calculating the weighted sum of the feature differences of each dimension between the fatigue state sub-vector and its baseline sub-vector; and the interaction deviation value is obtained by calculating the weighted sum of the feature differences of each dimension between the interaction state sub-vector and its baseline sub-vector. The weighting coefficients are either pre-set based on the importance of features or learned from historical data.

5. The method according to claim 1, characterized in that, Based on the fatigue deviation values ​​and interaction deviation values ​​across multiple consecutive time windows, the state evolution characteristics are determined, including: Trend analysis is performed on the fatigue deviation value sequence of the consecutive multiple time windows to obtain the fatigue state change trend characteristics; Trend analysis is performed on the interaction deviation value sequence of the multiple consecutive time windows to obtain the trend characteristics of interaction status changes; The fatigue state change trend features and the interaction state change trend features are combined and input into a preset evolution function to obtain the state evolution features.

6. The method according to claim 5, characterized in that, The trend analysis includes calculating the slope of the sequence, moving average, or extracting features using a time series prediction model.

7. The method according to claim 1, characterized in that, The joint identification of teachers' vocal fatigue state and classroom interaction state includes: The fatigue deviation value is compared with multiple preset fatigue level threshold ranges to determine the acoustic fatigue state level; The interaction deviation value is compared with multiple preset interaction level threshold ranges to determine the classroom interaction status level; Based on the changing trends indicated by the state evolution characteristics, the acoustic fatigue state level and the classroom interaction state level are corrected or composite state labels are generated.

8. A classroom dual-state joint recognition system based on individual teacher baseline alignment, characterized in that, Using the classroom dual-state joint identification method based on teacher individual baseline alignment as described in any one of claims 1-7, the system comprises: The acquisition and separation module is used to acquire classroom audio data and separate the teacher's speech sequence from the classroom audio data; The extraction and construction module is used to extract acoustic features and construct the current classroom acoustic vector for teacher speech segments divided by time windows in the teacher speech sequence. The baseline acquisition module is used to acquire the individual baseline acoustic vector of the teacher, which is constructed based on audio samples of the teacher's historical normal teaching. The fatigue and interaction vector extraction module is used to extract fatigue-related features to form a fatigue state sub-vector and extract interaction-related features to form an interaction state sub-vector from the current classroom acoustic state vector and the individual baseline acoustic state vector, respectively. The deviation calculation module is used to calculate the fatigue deviation value between the fatigue state sub-vector of the current window and the corresponding baseline sub-vector, as well as the interaction deviation value between the interaction state sub-vector of the current window and the corresponding baseline sub-vector. The state evolution determination module is used to determine state evolution characteristics based on the fatigue deviation value and the interaction deviation value of multiple consecutive time windows, or based on the deviation between the current teacher's overall classroom acoustic state vector and the individual baseline acoustic state vector. The identification module is used to jointly identify the teacher's vocal fatigue state and classroom interaction state based on the fatigue deviation value, the interaction deviation value, and the state evolution characteristics of the current window, and obtain a dual-state identification result.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the classroom dual-state joint identification method based on teacher individual baseline alignment as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the classroom dual-state joint identification method based on teacher individual baseline alignment as described in any one of claims 1 to 7.

Images