A multi-modal interaction collaborative control method and device based on a recording and broadcasting central control smart classroom and a storage medium

Abstract

The application discloses a kind of multi-modal interactive collaborative control method, device and storage medium based on recording broadcast central control wisdom classroom. The network, hardware and scene data of each teaching point configuration are acquired, the network adaptation coefficient is calculated and the hardware level is divided, the electronic teaching aid state is determined based on the teacher operation signal captured by the central control equipment and the teaching aid correlation degree is calculated;Fusion cross-teaching point collaborative interaction value, teaching aid correlation degree, network and hardware adaptation comprehensive value, interactive intention recognition value and historical interaction portrait value Five-dimensional feature calculates collaborative interaction comprehensive value;Determine the stage trigger threshold value through three-level adjustment of basic threshold value, scene correction and individualized correction, and the network adaptation coefficient simultaneously acts on evaluation and threshold value to form dynamic coupling balance;Based on the priority ranking and distribution of stage mode based on the comparison of comprehensive value and threshold value;Through model iteration, closed-loop optimization is formed.The application solves the problem of insufficient adaptation across teaching points, evaluation and threshold value fragmentation.

Description

Technical Field

[0001] This invention relates to the field of educational informatization and intelligent interactive control technology for recording and broadcasting, specifically to a multimodal interactive collaborative control method, device and storage medium for smart classrooms based on recording and broadcasting central control equipment (hereinafter referred to as central control equipment) for synchronous teaching scenarios at multiple teaching points. Background Technology

[0002] With the advancement of educational informatization, smart classrooms utilize recording and broadcasting control equipment and remote synchronous teaching technology to extend high-quality educational resources to less developed teaching points. A typical form of this is a cross-space collaborative teaching model where "the main teaching point leads the instruction, and multiple branch teaching points participate simultaneously." In this model, the automatic student presentation (i.e., the student's audio, video, or text is switched to the main display area by the recording and broadcasting control equipment for viewing by all teaching points) is a core element ensuring cross-teaching-point interaction.

[0003] Current smart classroom stage-attending control solutions generally adopt a model combining a fixed threshold and a single activity indicator. This means that a single indicator, such as the number of times a student speaks or raises their hand, is compared with a pre-set fixed threshold to determine whether a student can go on stage. This approach has the following shortcomings: First, the role differences between main teaching sites and branch teaching sites were not distinguished. Main teaching sites are usually led by teachers, and the interaction opportunities for students in branch teaching sites are easily squeezed out by the high-frequency interaction in the main teaching sites, resulting in insufficient participation from students in weaker teaching sites. At the same time, the terminal hardware conditions of different teaching sites vary greatly. Some rural teaching sites have low terminal performance and limited network bandwidth. A unified online control strategy cannot adapt to this hardware heterogeneity, which can easily lead to uneven network load, where high-bandwidth users in the main teaching sites concentrate on uplink resources while users in branch teaching sites are marginalized.

[0004] Secondly, the criteria for triggering interaction on stage are too simplistic. Relying solely on the number of times students speak or the frequency of raising their hands cannot accurately identify their true intentions for interaction. This can easily lead to accidental triggering due to students accidentally pressing the hand-raising button, as well as missed triggering when students at different teaching points express their interaction needs through text-based questions or other non-hand-raising methods. Furthermore, it fails to adapt to the current teaching process and the usage of electronic teaching aids.

[0005] Third, the threshold for students to go on stage remains fixed. A fixed threshold cannot be adjusted according to dynamic factors such as the number of students attending classes in real time, fluctuations in network bandwidth, and the type of teaching content. In scenarios with low network quality, multiple students going on stage at the same time can easily cause video lag. In scenarios with high interaction requirements, an excessively high threshold can lead to no one going on stage, making it difficult to strike a balance between interactive experience and network resource consumption.

[0006] Fourth, the evaluation dimensions and threshold adjustments are disconnected. In the existing scheme, the calculation of student interaction evaluation indicators and the adjustment of the on-stage trigger threshold are two independent processes. The same network state parameter will not act on the evaluation side and the threshold side at the same time, and a dynamic coupling balance cannot be formed, resulting in a lag or over-adjustment of the on-stage control response when network conditions fluctuate.

[0007] Therefore, there is an urgent need for a smart classroom interactive collaborative control scheme that can adapt to heterogeneous environments of multiple teaching sites, integrate multi-dimensional evaluation indicators, and support dynamic coupling of evaluation and thresholds. Summary of the Invention

[0008] In view of the above-mentioned shortcomings of the prior art, the main technical problem to be solved by the present invention is: how to achieve a dynamic coupling balance between evaluation and threshold in a multi-teaching-point collaborative teaching scenario, so that the on-stage control can simultaneously adapt to the differences in roles across teaching points, heterogeneous hardware conditions and dynamic network states, and achieve a dynamic balance between interactive participation and network resource consumption.

[0009] Secondary technical challenges include: how to integrate multi-dimensional features to accurately identify students' real interaction needs; how to ensure interaction stability during the presentation process by adapting to network and device status; and how to continuously improve evaluation accuracy through closed-loop feedback of model iteration.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A multimodal interactive collaborative control method based on a recording and broadcasting central control smart classroom is executed by the recording and broadcasting central control equipment. It includes six steps: multi-dimensional scene preprocessing, multimodal collaborative interaction evaluation, dynamic threshold adaptive adjustment, differentiated on-stage control, on-stage state adaptation optimization, and model iterative optimization.

[0011] The core concept of this invention lies in: solving the adaptation problem of heterogeneous environments across teaching points by allocating role weights at teaching points and adapting to network and hardware in a dual-dimensional manner; constructing a collaborative interaction evaluation model by integrating five dimensions of features: cross-teaching point collaborative interaction value, teaching aid correlation, comprehensive network and hardware adaptation value, interactive intent recognition value, and student historical interaction profile value, thus solving the problem of insufficient accuracy of single indicator judgment; achieving dynamic threshold adaptation through a three-level adjustment mechanism of basic threshold calculation superimposed with scene correction and personalized correction; and achieving dynamic balance through an "evaluation-threshold dynamic coupling balance" mechanism, so that the network adaptation coefficient simultaneously acts on the calculation of the comprehensive collaborative interaction value and the basic threshold calculation of the stage trigger threshold, with the two acting in opposite directions to form a dynamic balance—the technical principle of which is: when the network deteriorates, by reducing the scoring item (Q dimension) and reducing the threshold item (S_th) in a synergistic effect, so that... The selection strategy smoothly shifts from "prioritizing high-quality interactions" to "ensuring basic connectivity," thereby maintaining system responsiveness and interactive continuity when network resources are limited, and resolving the disconnect between evaluation dimensions and threshold adjustments. Through the inverse linkage of dynamic quotas and thresholds, the number of users participating in online sessions is controlled collaboratively when network conditions change. Differentiated online session control is achieved through priority ranking and multi-mode allocation. The priority ranking rule balances the concurrent online sessions at different teaching points, addressing the technical issues of network congestion caused by high-bandwidth users at the main teaching point vying for uplink resources and uneven load distribution among teaching points. Real-time status monitoring and the linkage adjustment of bitrate, status, and quotas ensure the stability of the online session process. The iterative optimization of the S6 model forms a true closed loop based on feedback from online session performance, which, together with the feedforward parameter coupling and balancing of S3, constitutes a complete adaptive system.

[0012] It is important to note that this invention differs fundamentally from existing technologies that rely solely on threshold adjustment or priority ranking. Firstly, existing threshold adjustment uses network status as the sole input to adjust the threshold, while this invention uses a network adaptation coefficient to simultaneously act in opposite directions on the scoring item (Q dimension) and the threshold item (S_th0), forming a feedforward dual-path coupling that allows the screening strategy to switch smoothly according to network conditions. Secondly, existing priority ranking typically aims at interactive fairness or participation, while this invention's four-level priority rule aims at network load balancing. Its technical effect is to evenly distribute uplink resources among teaching points, avoiding system congestion caused by high-bandwidth users at the main teaching point vying for resources. The synergistic effect of these two distinguishing features constitutes the core inventive contribution of this invention.

[0013] like Figure 1 As shown, the specific technical solutions for each step are as follows:

[0014] The central control device acquires basic configuration data, real-time network status data, terminal hardware data, and teaching scenario data from each teaching point in the smart classroom. In this article, N represents the number of students attending class in real time, and N_total represents the total number of students at all teaching points; normally, the two are equal.

[0015] The basic configuration data includes the main teaching point identifier, the number and location distribution of sub-teaching points, the teaching point role type (leading, participating, and auxiliary), the teaching content type, the subject type, and the grade level. Role weights are assigned based on the teaching point role type: leading main teaching points receive higher weights, participating sub-teaching points receive medium weights, auxiliary sub-teaching points receive lower weights, and sub-teaching points of the same type receive equal weights. These role weights are used to subsequently calculate the cross-teaching point collaborative interaction value I, reflecting the contribution of each teaching point to the overall interaction.

[0016] Real-time network status data includes network bandwidth (B), network latency (D), network jitter (J), and packet loss rate (L) at each teaching location. The network adaptation coefficient (K) is calculated based on this data using the following formula: K=α×min(1,B / B0)+β×max(0,1-D / D0)+γ×max(0,1-J / J0)+δ×max(0,1-L / L0) Where B0, D0, J0, and L0 are the preset baseline bandwidth, baseline latency, baseline jitter, and baseline packet loss rate, respectively, and α, β, γ, and δ are preset weighting coefficients, with α+β+γ+δ=1. This formula normalizes the four indicators—bandwidth, latency, jitter, and packet loss rate—to the [0,1] interval and then sums them by weight. The calculated result K naturally falls within the [0,1] interval, requiring no additional truncation. A higher network adaptability coefficient indicates that the network conditions at the teaching location are more suitable for audio-visual interactive presentations.

[0017] Terminal hardware data includes terminal type (PC, tablet, smartphone, smart blackboard, etc.), camera resolution, microphone sampling rate, and terminal computing power level. Based on terminal hardware performance, each teaching point is divided into three levels: high adaptation level, medium adaptation level, and basic adaptation level. The division method is as follows: a weighted score is calculated based on camera resolution, microphone sampling rate, and terminal computing power level, and this weighted score is mapped to one of the three adaptation levels—a score greater than or equal to a preset first threshold indicates a high adaptation level; a score greater than or equal to a preset second threshold but less than the preset first threshold indicates a medium adaptation level; and a score less than the preset second threshold indicates a basic adaptation level. Each level corresponds to a quantization value H (1.0 for high adaptation level, 0.7 for medium adaptation level, and 0.4 for basic adaptation level). Basic adaptation level terminals preferentially use voice or text-based on-site communication by default.

[0018] The teaching scenario data includes the type of electronic teaching aids currently used by teachers at the main teaching point, the usage duration of the electronic teaching aids, the current teaching segment, and the teaching objectives. The electronic teaching aids refer to digital teaching tools whose usage status can be directly captured by the central control device, including but not limited to electronic whiteboard courseware, virtual experiment software, teaching multimedia resources (such as presentations, animations, and videos), and interactive quiz systems. The usage status of the electronic teaching aids is determined by teacher operation signals captured by the central control device. These teacher operation signals include software interaction events such as switching electronic whiteboard courseware pages, starting or exiting virtual experiment software, accessing teaching multimedia resources, and issuing interactive quiz tasks. Based on a preset electronic teaching aid and interactive scenario association library, the association degree C of the teaching aids is calculated by the ratio of the usage duration of the electronic teaching aids to the total duration of the current teaching segment, i.e., C = T. t / T n T t For the duration of use of electronic teaching aids, T n This represents the total duration of the current teaching segment. A scenario is considered highly correlated when C is greater than or equal to a preset correlation threshold.

[0019] In an optional implementation, for physical teaching aids, video streams captured by classroom cameras connected to the central control device can be used to identify the type and usage status of the physical teaching aids on the teaching desk using a target detection algorithm (such as the YOLO series), and the identification results can be incorporated into the teaching scenario data. This method is an optional supplement and does not affect the independent implementation of the electronic teaching aid solution.

[0020] The system simultaneously collects student behavioral data (hand raising, standing, facial expressions, skeletal key point sequences, etc.), voice data (speech content, volume, speech rate, Mel-frequency cepstral coefficients, etc.), and text data (chat box questions, answers, etc.) to provide multimodal data support for subsequent interactive intent recognition. This data collection is solely for classroom interaction control; informed consent from student guardians has been obtained prior to collection, and data storage and processing comply with relevant laws and regulations on personal information protection. Behavioral features are extracted and anonymized locally on the edge computing module of the central control device; the original video stream is not uploaded or stored, only irreversibly mapped feature vectors are output, and these features are not used for personal identification.

[0021] like Figure 2 As shown, a multimodal collaborative interaction evaluation model is constructed based on historical smart classroom data, and the comprehensive collaborative interaction value S is calculated by integrating five dimensions of features: S=Wᵢ×I+W_c×C+W_q×Q+W_r×R+W_p×P Where Wᵢ+W_c+W_q+W_r+W_p=1, the meanings and calculation methods of each dimension are as follows: Cross-teaching-point collaborative interaction value I: I is a global indicator representing the overall classroom collaborative interaction activity at the current moment. All students share the same I value at the same time. It is obtained by weighting the role weight and interaction frequency of each teaching point and normalizing it to the [0,1] interval. The calculation formula is I = (W m ×I m +Σ(W s ⱼ×I s ⱼ)) / I_max, where I m Number of interactions with the main teaching point, I s ⱼ represents the number of interactions at each teaching location, W m W s ⱼ represents the role weight of the corresponding teaching point, and I_max is the interaction saturation benchmark value. I_max is determined by multiplying the total number of students (N_total) across all teaching points by the preset saturation coefficient ρ, i.e., I_max = ρ × N_total, where ρ is the preset interaction saturation coefficient, indicating that interaction saturation is considered achieved when the weighted interaction count reaches ρ times the total number of students. The value of ρ can be determined based on historical classroom interaction statistics. For example, according to statistics, a highly active classroom is considered to have a weighted interaction count reaching 1 / 6 of the total number of students, so ρ = 1 / 6 can be chosen. This setting links the normalization benchmark of I to the classroom size, avoiding uncontrollable ranges of I values ​​due to arbitrary benchmark settings. This dimension reflects the overall interaction activity of the entire classroom. The main teaching point usually contributes more to the I value due to its large interaction base, but this is an objective reflection of the overall classroom atmosphere, not a bias towards individual students at the main teaching point.

[0022] Teaching aid relevance C: Taken from the calculation results of step S1, it reflects the degree of relevance between the current teaching scenario and the use of electronic teaching aids. In scenarios with high relevance, students' interactive needs are usually stronger.

[0023] The comprehensive network and hardware compatibility value Q is obtained by weighted fusion of the network compatibility coefficient K and the hardware compatibility level quantification value H, i.e., Q = w1 × K + w2 × H, where w1 + w2 = 1. This dimension reflects the degree of compatibility between network conditions and hardware conditions for on-stage interaction.

[0024] The interactive intent recognition value R is calculated using a hybrid model that serially connects a convolutional neural network (CNN) and a bidirectional long short-term memory network (Bi-LSTM). The input data preprocessing method for this model is as follows: the behavioral feature vector is encoded from the coordinate sequence of skeletal keypoints extracted by a skeletal keypoint extraction algorithm (such as OpenPose) from the student video stream, preferably 256 dimensions; the speech feature vector is obtained by preprocessing the student audio stream to extract Mel-frequency cepstral coefficients (MFCC) and filter bank features, and then mapping it through a fully connected layer, preferably 128 dimensions; the text feature vector is obtained from the student text input stream through mean pooling of the last hidden state layer of a pre-trained language model (such as BERT), preferably 128 dimensions. The three features are concatenated and fused into a multimodal feature vector (preferably 512-dimensional); local temporal features are extracted through at least one convolutional layer (preferably two one-dimensional convolutional layers, kernel size 3, output channels 256, ReLU activation) and max pooling; then, the vector is input into at least one bidirectional long short-term memory network (preferably two layers, hidden units 256, dropout = 0.2) to capture long-term temporal dependencies; finally, the vector is passed through a fully connected layer and a sigmoid activation function to output a numerical value R representing the intensity of the student's willingness to interact, with a value range of [0,1]. When R is greater than or equal to a preset intention threshold R0, the student is determined to have a high intention to interact.

[0025] In another implementation, a Transformer encoder can be used to replace the convolutional layers and the bidirectional long short-term memory network to capture the temporal dependencies of multimodal features through a self-attention mechanism; in yet another implementation, a GRU (gated recurrent unit) can be used to replace the bidirectional long short-term memory network to reduce computational cost.

[0026] The hybrid model employs supervised training, with the training set consisting of labeled smart classroom interaction segments (no fewer than 50,000 segments). The loss function is binary cross-entropy, the optimizer is Adam (learning rate 1e-4), and the training runs for 30 epochs. An early stopping strategy is used (training stops if the validation set loss does not decrease for 5 consecutive epochs). After training, the model is deployed in the edge computing module of the central control device, with an inference latency of no more than 200ms.

[0027] Student historical interaction profile value P: A profile is constructed based on the student's historical interaction frequency, interaction quality and interaction type and quantified as a value in the range [0,1]. The lower the P value, the less the student has participated in historical interactions, and appropriate preference should be given to it in the priority ranking.

[0028] The weight coefficients Wᵢ, W_c, W_q, W_r, and W_p of the above five dimensions can be determined by combining the analytic hierarchy process with the entropy weight method, or they can be adjusted according to the actual application scenario.

[0029] like Figure 3As shown, based on real-time status data, teaching scenario data, and student profile data, the threshold S_th for triggering students to go on stage is determined through a three-level adjustment logic: Level 1 is based on the threshold calculation: S_th0=a-b×(N / N0)+c×K_avg Where N represents the number of students attending class in real time, N0 represents the preset baseline number of students, K_avg represents the average network adaptation coefficient of all teaching locations, a represents the baseline threshold constant, b represents the student number influence coefficient (the more students there are, the lower the threshold, to ensure that a certain proportion of students have the opportunity to participate), and c represents the network influence coefficient (the better the network conditions, the higher the threshold can be to filter high-quality interactions). When N / N0 exceeds the preset upper limit (e.g., 2), the threshold is truncated to the upper limit to avoid the baseline threshold becoming negative when there are too many students.

[0030] The dynamic coupling balance mechanism of this invention lies in the fact that K_avg appears simultaneously in the calculation of the collaborative interaction comprehensive value S (through the Q dimension) and the calculation of the basic threshold S_th0, and their effects are opposite. The technical principle is as follows: when network conditions are poor, K_avg is low, the Q dimension in the student's S value decreases (the scoring item tightens, making it harder to reach the threshold), but at the same time, the threshold S_th0 decreases due to the reduction of the c×K_avg term (the threshold item relaxes, making it easier to reach the threshold). The synergistic effect of these two factors allows the selection strategy to smoothly switch from "preferring high-quality interactions" when the network is good to "ensuring basic connectivity" when the network deteriorates, ensuring that network conditions do not become an absolute obstacle for students to go on stage, nor does it lead to a large number of students going on stage simultaneously when the network is congested. It should be noted that this mechanism belongs to feedforward parameter coupling balance, that is, it adjusts the scoring and threshold simultaneously based on real-time network state parameters; while the model iterative optimization of S6 belongs to true closed-loop optimization based on feedback from the stage performance. Together, they constitute the complete adaptive system of this invention.

[0031] The second level is scenario correction: the correction amount ΔS1 is determined according to a preset scenario correction coefficient table. The scenario correction coefficient table includes: coefficients corresponding to teaching content types ∈ [-0.15, +0.10] (negative values ​​for experimental teaching and interactive Q&A scenarios, positive values ​​for theoretical lectures and summary / extension scenarios); coefficients corresponding to subject categories ∈ [-0.08, +0.05] (negative values ​​for science courses, positive values ​​for humanities courses); and coefficients corresponding to teaching stages ∈ [-0.12, +0.08] (negative values ​​for experimental operations and Q&A stages, positive values ​​for classroom introduction and summary stages). ΔS1 is the sum of the three types of coefficients, S_th1 = S_th0 + ΔS1.

[0032] The third level is personalized adjustment: Thresholds are further adjusted for specific student groups. The adjustment amount ΔS2 is determined based on the student's historical interaction profile value P and hardware compatibility level: When P is lower than the preset profile threshold P0, ΔS2 includes a reduction of -(P0-P)×η (η is the personalized adjustment coefficient, such as 0.2), meaning that the individual threshold for students who rarely participate in interaction is appropriately lowered to encourage participation; when the hardware compatibility level is basic and the teaching location is a sub-teaching location, ΔS2 includes a preset fixed reduction amount ε (such as 0.07) to compensate for hardware disadvantages. S_th=S_th1+ΔS2.

[0033] Ultimately, upper and lower limits were set for the threshold S_th for triggering participation on stage, limiting it to the range of [0.15, 0.75]. The technical rationale for setting the lower limit is that if the threshold is too low, a large number of students with low engagement or poor network quality will simultaneously qualify to participate on stage, exceeding the concurrent processing capacity and uplink bandwidth of the central control equipment, causing system overload and failure (i.e., video stuttering, latency spikes, or even service interruptions due to request overload). The technical rationale for setting the upper limit is that if the threshold is too high, no one will participate on stage, rendering the cross-teaching point interaction mechanism ineffective. This range of upper and lower limits provides effective space for personalized adjustments, ensuring that lowering the threshold for students with low participation and those with hardware disadvantages takes effect in most scenarios. The lower limit constraint is only triggered in extreme scenarios with extremely poor network conditions and significant personalized adjustments. In such cases, the lower limit constraint itself serves as a necessary protection for system stability.

[0034] like Figure 4 As shown, the collaborative interaction comprehensive value S of each student is compared with its corresponding stage trigger threshold S_th. When S≥S_th, the student obtains the qualification to go on stage.

[0035] When the number of students who qualify for the stage at the same time exceeds the dynamic quota, the top N_q students are selected to present according to a multi-level priority rule. The formula for calculating the dynamic quota N_q is as follows: N_q=min(N_max, floor(N_total×λ×K_avg)) Where N_max is the preset maximum number of students allowed on stage, N_total is the total number of students at all teaching locations, λ is the basic quota coefficient, and K_avg is the average network adaptation coefficient. When the calculated result is less than 1, N_q is set to 1. The technical effect of this formula is that when network conditions are extremely poor (K_avg approaches 0), floor(N_total×λ×K_avg) approaches 0, but through the constraint of "taking 1 when less than 1", the system still retains the minimum quota of 1 person allowed on stage, so as to avoid interaction freezing due to network fluctuations, ensure the minimum availability of the system in extreme network environments, and avoid the complete closure of the interaction channel.

[0036] Another linkage mechanism of this invention is that the dynamic quota N_q and the threshold S_th are both driven by K_avg, and their effects are synergistic—when the network is poor, K_avg is low, the threshold decreases (more students meet the requirements), but the quota decreases (fewer people ultimately get to perform on stage), and the two work together to avoid network congestion; when the network is good, K_avg is high, the threshold increases (selecting high-quality interactions), but the quota increases (allowing more people to perform on stage), and the two work together to improve the interactive experience. This inverse linkage between the "threshold selection breadth" and the "quota limit quantity" allows the number of people performing on stage to be smoothly adjusted according to network conditions.

[0037] The multi-level priority ranking rule, from highest to lowest priority, is as follows: First priority: Students at branch teaching sites take precedence over students at main teaching sites. The technical effect of this rule is that main teaching sites typically have higher network bandwidth and hardware conditions. Without restrictions, students at main teaching sites would concentrate on consuming uplink resources and the concurrent processing channels of the central control equipment, marginalizing students at branch teaching sites and causing severe uneven network load among teaching sites. By prioritizing students at branch teaching sites, uplink resources can be evenly distributed among teaching sites, avoiding system congestion caused by a single teaching site monopolizing bandwidth. Second priority: Students whose interaction intent recognition value R is higher than the preset intent threshold R0 are prioritized over students whose value is lower than this threshold. This is used to filter genuine interaction needs and reduce false triggers that consume system resources. Third priority: Students with a basic hardware adaptation level and whose teaching site is a branch teaching site are prioritized over other students. This is used to compensate for the uplink bandwidth bottleneck of teaching sites with low hardware adaptation levels, preventing these students from being unable to gain the opportunity to perform due to hardware disadvantages. Fourth priority: Students whose historical interaction profile value P is lower than the preset profile threshold P0 are prioritized over students whose value is higher than this threshold. Within the same priority level, rules are sorted from highest to lowest based on the overall collaborative interaction value S. The first two rules have higher priority than the last two.

[0038] The aforementioned priority rules employ a hierarchical lexicographical selection mechanism: First, candidate students are divided into sub-teaching point groups and main teaching point groups based on the highest priority, with the sub-teaching point groups ranked first overall; within each group, they are further divided into high-intent groups and low-intent groups based on the second priority, with the high-intent groups ranked first; this process continues up to the fourth priority; among students with identical priority conditions, they are ranked from highest to lowest based on their collaborative interaction comprehensive value S. This mechanism ensures the strict enforceability of the priority rules and avoids the ambiguity of "the more conditions met, the higher the priority."

[0039] The aforementioned priority rules and collaborative evaluation constitute a two-tiered mechanism: the evaluation layer (comparing the S-value with a threshold) determines candidate eligibility, while the priority layer (multi-level rules) determines the ranking when there are too many candidate students. The role weight of each teaching point is reflected in the I-value calculation of the evaluation layer, reflecting the overall interactive atmosphere of the teaching point; the priority rules in the ranking layer favor students from different teaching points, reflecting network load balancing. The two have clear divisions of labor and work synergistically, without any logical contradiction.

[0040] The on-stage interaction mode is determined and assigned in ascending order of network adaptability coefficient K: when K < K2, regardless of hardware adaptability level, a plain text on-stage mode is assigned; when K2 ≤ K < K1, or when the hardware adaptability level is basic and K ≥ K2, a voice plus text on-stage mode is assigned; when K ≥ K1 and the hardware adaptability level is high or medium, a video plus voice on-stage mode is assigned. K1 > K2. This determination order ensures that when network conditions are extremely poor (K < K2), the connectivity of plain text interaction is prioritized, avoiding rule conflicts caused by hardware level determination.

[0041] like Figure 5 As shown, after a student goes on stage, the central control equipment continuously monitors their network status, device status, and interaction status, and makes dynamic adjustments based on the monitoring results: Network condition adaptation: Real-time monitoring of students' bandwidth and packet loss rate. When bandwidth falls below the preset lower limit or packet loss rate exceeds the preset upper limit, the video bitrate and resolution are gradually reduced (e.g., from 1080p to 720p and then to 480p). When network conditions recover, the bitrate and resolution are gradually restored. The bitrate adjustment step size is 256kbps, and the adjustment interval is no less than 5 seconds to avoid frequent fluctuations.

[0042] Interaction Status Adaptation: If a student does not engage in any voice or text interaction for a preset duration (e.g., 60 seconds), the system automatically switches to a background standby state and releases the student's speaking quota. When a new interactive action is detected, the student's speaking status is automatically restored. Interaction requests from students who have not yet been granted speaking privileges are moved to a background standby queue and filled sequentially according to the multi-level priority rules after the student releases their quota.

[0043] Teaching aid association and adaptation: When the teacher at the main teaching point switches the type of electronic teaching aid, the teaching aid association degree C is updated in real time, the comprehensive value of collaborative interaction of relevant students S is recalculated, and the priority of going on stage is dynamically adjusted.

[0044] Abnormal status handling: When the network of a teaching point is interrupted for more than the preset time (e.g., 10 seconds), the students of that teaching point will be automatically removed from the stage and their interaction requests will be recorded; after the network is restored, students of that teaching point will be given priority in being assigned stage quotas.

[0045] Real-time data, interactive feedback data, and teacher manual adjustment records are stored during the teaching process. When preset trigger conditions are met, the feature weights of the collaborative interaction evaluation model and the network parameters of the interactive intent recognition model are updated using the gradient descent algorithm.

[0046] The preset triggering conditions include: the accumulated course data reaching a preset threshold (e.g., 100 class hours), or the interactive intent recognition accuracy falling below a preset accuracy threshold (e.g., 85%). During updates, a mini-batch gradient descent algorithm is used with a learning rate of 1e-4, a batch size of 32, and no more than 10 training epochs. An early stopping strategy is employed to prevent overfitting. The updated model can only be deployed when its validation set accuracy is not lower than the original model, thus avoiding model degradation. This step constitutes a true closed-loop optimization based on feedback from the performance on stage, complementing the feedforward parameter coupling and balancing of S3: S3 performs rapid feedforward adjustment based on real-time network status, while S6 performs slow closed-loop optimization based on long-term performance data, forming a complementary relationship on the time scale.

[0047] Corresponding to the above methods, such as Figure 6 As shown, this invention also provides a multimodal interactive collaborative control device based on a recorded and broadcasted central control smart classroom, including a scene multi-dimensional preprocessing module, a multimodal collaborative interaction evaluation module, a dynamic threshold adaptive adjustment module, a differentiated on-stage control module, an on-stage state adaptation optimization module, and a model iterative optimization module. Each module performs the corresponding steps in the above method, which will not be elaborated here.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0049] Compared with the prior art, the present invention has the following beneficial effects: First, it has strong adaptability across teaching sites. This invention distinguishes the interactive contribution of the main teaching site and the sub-teaching sites by assigning role weights to the teaching sites. Combined with a two-dimensional evaluation of network adaptability coefficient and hardware adaptability level, the platform control strategy can adapt to the differences in network conditions and hardware performance of different teaching sites, effectively ensuring the interactive participation opportunities for students in weaker teaching sites.

[0050] Second, the accuracy of triggering on stage is high. This invention integrates five dimensions of features: cross-teaching point collaborative interaction value, teaching aid correlation degree, network and hardware adaptation comprehensive value, interactive intent recognition value, and student historical interaction profile value. Combined with multimodal deep learning intent recognition based on deep temporal neural network, it can more accurately identify students' real interactive needs and reduce false triggers and missed triggers.

[0051] Third, the evaluation and threshold are dynamically coupled and balanced. The core innovation of this invention lies in the fact that the network adaptation coefficient simultaneously affects the calculation of the comprehensive value of collaborative interaction and the basic threshold calculation of the trigger threshold. The two are in opposite directions, forming a dynamic balance. The technical principle is that when the network deteriorates, the screening strategy smoothly switches from "preferring high-quality interactions" to "ensuring basic connectivity," which solves the problem of the evaluation dimension and threshold adjustment being mutually exclusive in the prior art, enabling the platform control to smoothly and adaptively adjust according to network conditions. Compared with the scheme of independent adjustment of evaluation and threshold, this scheme can significantly reduce the fluctuation range of the number of participants and the interaction interruption rate caused by network congestion in network fluctuation scenarios, and improve the stability of the system in dynamic network environments.

[0052] Fourth, quotas and thresholds are linked inversely. This invention utilizes a mechanism where dynamic onboarding quotas and thresholds are both driven by the network adaptation coefficient and act in a coordinated manner. This collaboratively controls the number of users onboard when network conditions change, avoiding response lag or over-adjustment caused by adjusting a single threshold or quota. The minimum quota constraint in the dynamic quota formula ensures that the system retains one user even in extremely poor network conditions, preventing interaction freezes due to network fluctuations and maintaining minimum availability under extreme network environments.

[0053] Fifth, network load balancing. This invention prioritizes students at branch teaching sites and those with lower hardware compatibility levels, ensuring a balanced distribution of uplink resources across all teaching sites. This avoids the uneven load distribution caused by high-bandwidth users at the main teaching site monopolizing resources, which marginalizes users at branch teaching sites. This improves the overall resource utilization efficiency and system stability of the central control equipment in multi-teaching-site scenarios. This mechanism prioritizes network load balancing; even if students at the main teaching site have higher individual collaborative interaction scores, priority is given to ensuring opportunities for students at branch teaching sites to participate, thus preventing the main teaching site from monopolizing high-quality resources.

[0054] Sixth, interactive stability is guaranteed. This invention continuously monitors the network, devices, and interactive status after going live, and effectively avoids interactive interruptions or stuttering caused by network fluctuations or device differences through adaptive bitrate adjustment, dynamic switching of live status, quota reallocation, and anomaly recovery mechanisms.

[0055] Seventh, a two-layer adaptive system. This invention achieves rapid adjustment based on real-time network status through feedforward parameter coupling balancing in S3, and achieves slow optimization based on long-term performance data through closed-loop feedback optimization in S6. The two complement each other on the time scale, forming a complete adaptive system that continuously improves evaluation accuracy and control effect.

[0056] Eighth, promoting educational equity. This invention prioritizes students at different teaching locations, those with lower hardware compatibility levels, and those with fewer historical interactions, ensuring that high-quality educational resources are distributed more equitably across various teaching locations and student groups. Compared to a fixed threshold scheme, this scheme significantly improves the participation rate of students at different teaching locations and reduces the false trigger rate in low network quality scenarios. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall process of the method of the present invention, showing the sequential execution order of the six steps: multi-dimensional scene preprocessing (S1), multi-modal collaborative interaction evaluation (S2), dynamic threshold adaptive adjustment (S3), differentiated on-stage control (S4), on-stage state adaptation optimization (S5), and model iterative optimization (S6), as well as the closed-loop feedback relationship from the model iterative optimization step to the multi-modal collaborative interaction evaluation step and the dynamic threshold adaptive adjustment step. Figure 2 This is a schematic diagram of the five-dimensional feature fusion logic of the collaborative interaction evaluation model of the present invention. It shows that the five-dimensional features of cross-teaching point collaborative interaction value I, teaching aid correlation degree C, network and hardware adaptation comprehensive value Q, interactive intention recognition value R, and student historical interaction profile value P are respectively fused according to the corresponding weight coefficients to calculate the collaborative interaction comprehensive value S, and the logic of triggering judgment on the stage (S≥S_th) is executed by S. Figure 3 This is a schematic diagram of the dynamic threshold three-level adjustment and evaluation-threshold dynamic coupling balance logic of the present invention. It shows the process of determining the basic threshold calculation (S_th0), scene correction (ΔS1), personalized correction (ΔS2), upper and lower limit constraints and the final trigger threshold S_th, as well as the relationship that the network adaptation coefficient K / K_avg simultaneously acts on the comprehensive value S of the collaborative interaction on the evaluation side and the basic threshold calculation on the threshold side, with opposite directions to form a dynamic coupling balance. Figure 4 This is a schematic diagram of the priority ranking and mode allocation logic of the present invention. The priority ranking adopts a process structure of step-by-step group screening. It shows the process of the candidate student pool being screened step by step from the first to the fourth priority, and the list of students to be screened (the top N_q) being determined by sorting the collaborative interaction comprehensive value S in descending order within the same level. It also shows the process of determining and allocating three screen interaction modes, plain text, voice plus text, and video plus voice, in order from low to high according to the network adaptation coefficient K. Figure 5 This is a schematic diagram of the workflow for the on-stage status adaptation and optimization of the present invention. It shows that the network status, device status and interaction status of the on-stage students are continuously monitored, and a closed-loop monitoring and adjustment process is executed, including step-by-step bitrate adjustment, on-stage status switching and quota release / filling, and abnormal status removal and recovery. Figure 6This is a block diagram of the modular structure of the device of the present invention, showing the layout and data flow relationship of the scene multi-dimensional preprocessing module, multi-modal collaborative interactive evaluation module, dynamic threshold adaptive adjustment module, differentiated on-stage control module, on-stage state adaptation optimization module and model iterative optimization module on the hardware platform of the recording and broadcasting central control equipment, as well as the on-stage control signal output. Detailed Implementation

[0058] The following is combined Figures 1 to 6 The technical solution of the present invention will be further described in detail below with specific embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0059] This embodiment is applied to a multi-teaching-point synchronous teaching scenario in junior high school physics experiment teaching (convex lens imaging), with the central control equipment performing the upper stage control.

[0060] Basic configuration: The main teaching site M is a high-quality urban middle school (leading type), and the branch teaching sites S1, S2, and S3 are rural weak middle schools (participating type); the teaching content type is experimental teaching, the subject is physics (science), and the teaching process is experimental operation and question answering.

[0061] Real-time status: The number of students in class is N = 60 (20 at school M, 15 at school S1, 12 at school S2, and 13 at school S3), and the total number of students at all teaching sites is N_total = 60. Network data for each teaching site is as follows: School M: bandwidth 4Mbps, latency 80ms, jitter 15ms, packet loss rate 1%; School S1: bandwidth 1.8Mbps, latency 120ms, jitter 30ms, packet loss rate 3%; School S2: bandwidth 2.2Mbps, latency 90ms, jitter 25ms, packet loss rate 2%; School S3: bandwidth 1.5Mbps, latency 150ms, jitter 35ms, packet loss rate 4%.

[0062] Set the baseline bandwidth B0 = 2Mbps, baseline latency D0 = 100ms, baseline jitter J0 = 20ms, baseline packet loss rate L0 = 5%, and weighting coefficients α = 0.4, β = 0.3, γ = 0.2, and δ = 0.1. Calculate the network adaptation coefficient for each teaching location: School M: K = 0.4 × min(1, 4 / 2) + 0.3 × max(0, 1 - 80 / 100) + 0.2 × max(0, 1 - 15 / 20) + 0.1 × max(0, 1 - 0.01 / 0.05) = 0.4 × 1 + 0.3 × 0.2 + 0.2 × 0.25 + 0.1 × 0.8 = 0.4 + 0.06 + 0.05 + 0.08 = 0.59 S1 school: K=0.4×min(1,1.8 / 2)+0.3×max(0,1-120 / 100)+0.2×max(0,1-30 / 20)+0.1 ×max(0,1-0.03 / 0.05)=0.4×0.9+0.3×0+0.2×0+0.1×0.4=0.36+0+0+0.04=0.40 S2 school: K = 0.4 × min(1, 2.2 / 2) + 0.3 × max(0, 1 - 90 / 100) + 0.2 × max(0, 1 - 25 / 20) + 0.1 × max(0, 1 - 0.02 / 0.05) = 0.4 × 1 + 0.3 × 0.1 + 0.2 × 0 + 0.1 × 0.6 = 0.4 + 0.03 + 0 + 0.06 = 0.49 S3 school: K = 0.4 × min(1, 1.5 / 2) + 0.3 × max(0, 1 - 150 / 100) + 0.2 × max(0, 1 - 35 / 20) + 0.1 × max(0, 1 - 0.04 / 0.05) = 0.4 × 0.75 + 0.3 × 0 + 0.2 × 0 + 0.1 × 0.2 = 0.3 + 0 + 0 + 0.02 = 0.32 The average network adaptation coefficient K_avg = (0.59 + 0.40 + 0.49 + 0.32) / 4 = 1.80 / 4 = 0.45.

[0063] Hardware configuration: M school terminal is PC plus smart blackboard (high adaptation level H1, H=1.0), S1 school terminal is tablet (medium adaptation level H2, H=0.7), S2 school terminal is old PC (basic adaptation level H3, H=0.4), S3 school terminal is smartphone (medium adaptation level H2, H=0.7).

[0064] Teaching Scenario: At the main teaching site, teachers use virtual experiment software (convex lens imaging simulation) and projection equipment to demonstrate experimental steps. The central control device determines the type of electronic teaching aid by capturing the operation signal of the teacher starting the virtual experiment software, and the usage time T of the electronic teaching aid. t =20 minutes, the total duration T of the current teaching segment n =25 minutes, the teaching aid correlation C=20 / 25=0.8, which is greater than the preset correlation threshold of 0.3, and is judged as a high correlation scenario.

[0065] Multimodal data collection: Three students from school S1 raised their hands, two students from school S2 submitted experimental questions via chat, one student from school S3 asked about the operating steps, and five students from school M raised their hands. Behavioral data was collected from video streams captured by classroom cameras, and the OpenPose algorithm was used to extract the student skeletal keypoint sequences; speech data was collected from audio streams captured by terminal microphones, and MFCC features were extracted; text data was collected from student input text via chat.

[0066] Teaching point role weight allocation: Dominant main teaching point W m =0.65, Participatory sub-teaching point W s =0.35 (equally distributed among the three teaching sites, approximately 0.117 each).

[0067] Cross-teaching site collaborative interaction value: Number of interactions at school M m =5, S1 school interaction count I s1 =3, S2 school I s2 =2, S3 school I s3 =1. Set the interaction saturation coefficient ρ = 1 / 6, and the total number of students at all teaching points N_total = 60. Then I_max = ρ × N_total = (1 / 6) × 60 = 10. The weighted sum = 0.65 × 5 + 0.117 × (3 + 2 + 1) = 3.25 + 0.702 = 3.952, I = 3.952 / I_max = 3.952 / 10 = 0.3952. I is a global indicator for the entire classroom, shared by all students.

[0068] Interactive Intent Recognition: The intent type and value of each student are identified using a CNN-BiLSTM hybrid model. An intent threshold of R0 = 0.6 is set. Students raising their hands in School S1 are identified as question-and-answer type, R≈0.7 > R0; students submitting text in School S2 are identified as actively seeking knowledge, R≈0.85 > R0; students speaking in School S3 are identified as question-and-answer type, R≈0.75 > R0; students raising their hands in School M are identified as collaborative participants, R≈0.5 < R0.

[0069] Student history interaction profile: 2 students from school S2 had relatively little history interaction, P=0.2; 3 students from school S1 had P=0.4; 1 student from school S3 had P=0.35; 5 students from school M had P=0.6.

[0070] The overall network and hardware compatibility value Q = 0.6 × K + 0.4 × H: School M: Q = 0.6 × 0.59 + 0.4 × 1.0 ≈ 0.354 + 0.4 = 0.754 S1 school: Q=0.6×0.40+0.4×0.7≈0.24+0.28=0.52 School S2: Q = 0.6 × 0.49 + 0.4 × 0.4 ≈ 0.294 + 0.16 = 0.454 S3 school: Q=0.6×0.32+0.4×0.7≈0.192+0.28=0.472 The overall collaborative interaction value of students at each teaching point (with weights set as Wᵢ=0.35, W_c=0.15, W_q=0.20, W_r=0.20, W_p=0.10): Students at School S2: I = 0.3952, C = 0.8, Q = 0.454, R = 0.85, P = 0.2, S = 0.35 × 0.3952 + 0.15 × 0.8 + 0.20 × 0.454 + 0.20 × 0.85 + 0.10 × 0.2 = 0.1383 + 0.12 + 0.0908 + 0.17 + 0.02 = 0.5391 Students at School S1: I = 0.3952, C = 0.8, Q = 0.52, R = 0.7, P = 0.4, S = 0.35 × 0.3952 + 0.15 × 0.8 + 0.20 × 0.52 + 0.20 × 0.7 + 0.10 × 0.4 = 0.1383 + 0.12 + 0.104 + 0.14 + 0.04 = 0.5423 S3 school student: I = 0.3952, C = 0.8, Q = 0.472, R = 0.75, P = 0.35, S = 0.35 × 0.3952 + 0.15 × 0.8 + 0.20 × 0.472 + 0.20 × 0.75 + 0.10 × 0.35 = 0.1383 + 0.12 + 0.0944 + 0.15 + 0.035 = 0.5377 Students at School M: I = 0.3952, C = 0.8, Q = 0.754, R = 0.5, P = 0.6, S = 0.35 × 0.3952 + 0.15 × 0.8 + 0.20 × 0.754 + 0.20 × 0.5 + 0.10 × 0.6 = 0.1383 + 0.12 + 0.1508 + 0.10 + 0.06 = 0.5691

[0071] Basic thresholds: set a = 0.6, b = 0.2, c = 0.3, and base number of people N0 = 50.

[0072] S_th0=0.6-0.2×(60 / 50)+0.3×0.45=0.6-0.24+0.135=0.495 Scenario correction: The correction coefficient for experimental teaching scenarios is -0.12, the correction coefficient for physics subjects is -0.05, and the correction coefficient for experimental operation procedures is -0.10.

[0073] ΔS1=-0.12-0.05-0.10=-0.27 S_th1 = 0.495 - 0.27 = 0.225 Personalized adjustment: For students in school S2, P = 0.2 < P0 = 0.3, the downward adjustment amount = -(0.3 - 0.2) × 0.2 = -0.02; for hardware level H3 and a sub-teaching point, the preset fixed downward adjustment amount ε = -0.07. ΔS2 = -0.02 - 0.07 = -0.09, S_th = 0.225 - 0.09 = 0.135, which is adjusted to 0.15 after lower limit constraint.

[0074] For students in schools S1, S3, and M, P is greater than or equal to P0 and their hardware is not at the basic level, with no personalized adjustments. S_th = 0.225.

[0075] Stage appearance trigger determination: The S value of each student is greater than or equal to its corresponding stage appearance trigger threshold (0.15 for students from S2 school, and 0.225 for students from S1, S3, and M schools). A total of 11 students (5 from M school, 3 from S1 school, 2 from S2 school, and 1 from S3 school) are qualified to be candidates.

[0076] Dynamic quota allocation: N_max = 8, basic quota coefficient λ = 0.15.

[0077] N_q=min(8, floor(60×0.15×0.45))=min(8, floor(4.05))=4.

[0078] Priority ranking (lexicographical selection): In the first priority comparison, students from schools S1, S2, and S3 are all students at branch teaching sites and are at the same level, while students from school M are ranked lower because they are at the main teaching site. Moving to the second priority comparison, the R values ​​of students from schools S1, S2, and S3 are all higher than R0 = 0.6, remaining at the same level; students from school M have R = 0.5 < R0, further lowering their ranking within the main teaching site group. Moving to the third priority comparison, students from school S2 have basic-level hardware (H3) and are at a branch teaching site, meeting the third priority condition and ranking first; both students from school S2 are selected. Students from schools S1 and S3 have medium-level hardware (H2) and do not meet the third priority condition, moving to the fourth priority comparison. In the fourth priority comparison, the P values ​​of students from schools S1 and S3 are both ≥ P0 = 0.3, not meeting the fourth priority condition; therefore, within the same priority, they are ranked from highest to lowest based on their individual collaborative interaction comprehensive value S. The score for students from school S1 is approximately 0.5423, while the score for students from school S3 is approximately 0.5377, indicating that the score for students from school S1 is higher. The remaining two slots for students to perform on stage were awarded to two students from school S1. The final list of students to perform on stage consists of two students from school S2 and two students from school S1.

[0079] It is worth noting that the collaborative interaction score of students from School M, S≈0.5691, was the highest in the entire event, yet they were not eligible to go on stage due to their first priority (the main teaching point was ranked last). This is not a loophole in the mechanism, but a direct manifestation of the principle of "network load balancing takes precedence over individual score optimization" in this invention—by forcibly constraining high-bandwidth users at the main teaching point to monopolize uplink resources, ensuring interaction opportunities for students at branch teaching points. This is the core innovation of this invention, which distinguishes it from existing technologies that use individual scores as the sole ranking criterion.

[0080] On-stage mode allocation: Set the first network threshold K1 = 0.5 and the second network threshold K2 = 0.35. Based on K from low to high: School S2 has K = 0.49, satisfying K2 ≤ K < K1, and is allocated voice plus text mode; School S1 has K = 0.40, satisfying K2 ≤ K < K1, and is also allocated voice plus text mode. If a student from School S3 is selected, because their K = 0.32 < K2, they will be directly allocated plain text mode without any downgrading or switching.

[0081] When a student from School S2 goes on stage, if the network bandwidth fluctuates to 1.5Mbps and K drops to approximately 0.32 < K2, the on-stage mode automatically switches from voice plus text to plain text, and the bitrate is automatically adjusted to ensure smooth text interaction. Once the network is restored, the original mode is automatically reverted. After a student submits a question, the text on-stage window remains displayed until the teacher finishes answering it; if there is no further interaction, the quota is automatically released. When the teacher switches to a projection device to demonstrate the experimental steps, the electronic teaching aid type is updated, the teaching aid correlation C is updated to 0.9, the collaborative interaction comprehensive value of the relevant students is recalculated, and their on-stage eligibility is retained.

[0082] This embodiment is applied to a collaborative teaching scenario involving multiple teaching points in high school mathematics group discussions.

[0083] The main teaching sites are M1 (Urban School A, leading type) and M2 (Urban School B, auxiliary type), with branch teaching sites S1 and S2 (rural schools, participatory type). The teaching content is high school mathematics group discussion, the subject is mathematics (science), and the teaching method is group collaborative inquiry. The current number of students in class is N = 45, and the total number of students across all teaching sites is N_total = 45.

[0084] Network adaptation coefficients for each teaching site: M1 K = 0.8, M2 K = 0.75, S1 K = 0.4, S2 K = 0.35, K_avg = (0.8 + 0.75 + 0.4 + 0.35) / 4 = 2.3 / 4 = 0.575. Hardware adaptation levels for M1 and M2 are H1 (H = 1.0), and for S1 and S2 are H2 (H = 0.7). The electronic teaching aids are virtual whiteboards plus projection equipment. The central control device determines the usage status by capturing the teacher's writing operation signals on the electronic whiteboard. The teaching aid correlation degree C = 0.7 (high correlation).

[0085] Teaching point role weight allocation: W m1 =0.5, W m2 =0.25, W s =0.25 (equal distribution between the two teaching sites, 0.125 each).

[0086] Interaction Intent Recognition: The intent threshold R0 = 0.6 was set. Three students from school S1 were identified as proactive learners, with R ≈ 0.82 > R0; two students from school S2 were identified as collaborative participants, with R ≈ 0.55 < R0. The historical interaction profiles of students all had a P value ≥ 0.3.

[0087] The average collaborative interaction value of students at each teaching site was calculated as follows: approximately 0.62 for students at school S1, approximately 0.58 for students at school S2, approximately 0.55 for students at school M2, and approximately 0.52 for students at school M1.

[0088] Basic threshold: S_th0 = 0.6 - 0.2 × (45 / 50) + 0.3 × 0.575 = 0.6 - 0.18 + 0.1725 = 0.5925.

[0089] Scenario Correction: Group discussion scenario correction coefficient -0.10, mathematics subject correction coefficient +0.03, collaborative inquiry scenario correction coefficient -0.08.

[0090] ΔS1=-0.10+0.03-0.08=-0.15 S_th1=0.5925-0.15=0.4425.

[0091] Without personalized adjustments (P values ​​are all ≥ 0.3, and the hardware is not at the basic level), the final S_th = 0.4425.

[0092] Dynamic quota: N_q=min(8, floor(45×0.15×0.575))=min(8, floor(3.88125))=3.

[0093] Priority sorting (lexicographical filtering): In the first priority comparison, students from S1 and S2 schools are both students from branch teaching sites and are at the same level, while students from M1 and M2 schools are ranked lower. In the second priority comparison, students from S1 school (R > R0) are ranked higher; students from S2 school (R < R0) are ranked lower. With a quota of 3, all 3 students from S1 school meet the first two criteria and are ranked higher, so they are selected to participate. Students from S2 school, due to R < R0 ranking lower in the second priority comparison, are not eligible to participate in this round. Their interaction requests are placed in a background queue to be filled by S1 school students after their quotas are released, according to the multi-level priority rules.

[0094] On-stage mode allocation: S1 school K = 0.4 satisfying K2≤K<K1, all 3 students are assigned voice plus text mode.

[0095] When the network fluctuation of school S1 reaches K = 0.32 < K2, the platform mode is switched to plain text, and the bitrate is reduced from a higher value to an appropriate value; when a teacher from school M2 initiates a collaborative task, the comprehensive value of collaborative interaction is recalculated, and an additional platform quota is allocated to one student from school M2. Implementation

[0096] The technical solution of this invention can be extended to the following scenarios: collaborative teaching at multiple teaching points, blended online and offline teaching, cross-border remote synchronous classrooms, etc. Adaptation can be achieved by adjusting the teaching point role weight allocation strategy and the network adaptation coefficient calculation model.

[0097] When there are multiple main teaching points, the formula for calculating the cross-teaching point collaborative interaction value is expanded to I=(Σ(W mk ×I mk )+Σ(W s ⱼ×I s ⱼ)) / I_max, where k is the main teaching point number, W mk I mk These represent the role weight and interaction count of the k-th main teaching point, respectively.

[0098] The weighting coefficients and correction coefficients in the dynamic threshold adjustment can be customized according to the level of regional education informatization. For example, in rural areas of western China, the weight of the hardware adaptation dimension can be appropriately increased.

[0099] The network structure of the interactive intent recognition model can adopt various deep temporal neural network architectures: in one implementation, a CNN and a Bi-LSTM are sequentially connected; in another implementation, a Transformer encoder is used to capture the temporal dependencies of multimodal features through a self-attention mechanism; and in yet another implementation, a GRU (Gated Recurrent Unit) is used to replace the Bi-LSTM to reduce computational cost. The behavior feature extraction algorithm can be replaced with other skeletal keypoint extraction algorithms (such as AlphaPose, HRNet, etc.), speech feature extraction can be expanded to support formant features, fundamental frequency features, etc., and text feature extraction can be replaced with other pre-trained language models.

[0100] The initial assignment of the on-stage interaction mode can cover all three modes: when student K < K2, the on-stage interaction is directly initiated in plain text mode without the need for downgrading; when K2 ≤ K < K1, the on-stage interaction is initiated in voice plus text mode; when K ≥ K1 and the hardware is high or medium, the on-stage interaction is initiated in video plus voice mode.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal interactive collaborative control method for a smart classroom based on recording and broadcasting, characterized in that, Performed by the recording and broadcasting control equipment, the process includes the following steps: S1. Multi-dimensional Preprocessing of Scenarios: Acquire basic configuration data, real-time network status data, terminal hardware data, and teaching scenario data of each teaching point in the smart classroom. Determine the role type of each teaching point and assign corresponding role weights based on the basic configuration data. Calculate the network adaptation coefficient K based on the real-time network status data. Divide the hardware adaptation level based on the terminal hardware data. Establish a dynamic association between electronic teaching aids and teaching links based on the teaching scenario data and calculate the teaching aid association degree C. The usage status of the electronic teaching aids is determined by the teacher operation signals captured by the recording and broadcasting control equipment. S2. Multimodal Collaborative Interaction Evaluation: Collect students' behavioral data, voice data, and text data, integrate features from five dimensions: cross-teaching point collaborative interaction value I, teaching aid correlation degree C, network and hardware adaptation comprehensive value Q, interactive intent recognition value R, and student historical interaction profile value P, and calculate the collaborative interaction comprehensive value S by weighting; wherein the network and hardware adaptation comprehensive value Q includes the network adaptation coefficient K. S3. Dynamic Threshold Adaptive Adjustment: Based on the real-time number of students N, the average network adaptation coefficient K_avg of each teaching point, and teaching scenario data, the on-stage trigger threshold S_th is determined through three levels of adjustment: basic threshold calculation, scenario correction, and personalized correction. Upper and lower limits are set for the on-stage trigger threshold S_th. The basic threshold calculation includes the average network adaptation coefficient K_avg, so that the network adaptation coefficient simultaneously affects the calculation of the collaborative interaction comprehensive value and the calculation of the on-stage trigger threshold, forming a dynamic coupling balance between evaluation and threshold. S4. Differentiated on-stage control: The student's collaborative interaction comprehensive value S is compared with the on-stage trigger threshold S_th. Students who reach the threshold are sorted according to multi-level priority rules. On-stage students are selected according to the dynamic on-stage quota, and on-stage interaction modes are assigned to them according to the network adaptation coefficient K and hardware adaptation level. S5. Stage Status Adaptation and Optimization: Continuously monitor the network status, device status and interaction status of students on stage, and dynamically adjust the video bitrate, switch the stage status or reallocate the stage quota based on the monitoring results. S6. Model Iterative Optimization: Store real-time data, interactive effect feedback data, and teacher manual adjustment records during the teaching process. When the preset trigger conditions are met, update the feature weights of the collaborative interaction evaluation model and the network parameters of the interactive intent recognition model to form a closed-loop optimization based on the feedback of the stage performance.

2. The method according to claim 1, characterized in that, The formula for calculating the network adaptation coefficient K in step S1 is as follows: K=α×min(1,B / B0)+β×max(0,1-D / D0)+γ×max(0,1-J / J0)+δ×max(0,1-L / L0) Where B is the real-time bandwidth, D is the network latency, J is the network jitter, L is the packet loss rate, B0, D0, J0, and L0 are the preset baseline bandwidth, baseline latency, baseline jitter, and baseline packet loss rate, respectively, α, β, γ, and δ are preset weight coefficients and α+β+γ+δ=1, and the value range of K is [0,1].

3. The method according to claim 1, characterized in that, The hardware adaptation level classification method in step S1 is as follows: a weighted score is calculated based on the terminal camera resolution, microphone sampling rate and terminal computing power level, and the weighted score is mapped to three adaptation levels. A score greater than or equal to a preset first threshold is a high adaptation level, a score greater than or equal to a preset second threshold and less than a preset first threshold is a medium adaptation level, and a score less than a preset second threshold is a basic adaptation level.

4. The method according to claim 1, characterized in that, The formula for calculating the collaborative interaction comprehensive value S in step S2 is as follows: S=Wᵢ×I+W_c×C+W_q×Q+W_r×R+W_p×P Where I is the cross-teaching point collaborative interaction value, C is the teaching aid correlation degree, Q is the network and hardware adaptation comprehensive value, R is the interaction intention recognition value, P is the student historical interaction profile value, and Wᵢ, W_c, W_q, W_r, and W_p are the preset weight coefficients of the corresponding dimensions, and Wᵢ+W_c+W_q+W_r+W_p=1; The cross-teaching-point collaborative interaction value I is obtained by weighting the role weight and interaction frequency of each teaching point and then normalizing it to the [0,1] interval. The calculation formula is I = (W m ×I m +Σ(W s ⱼ×I s ⱼ)) / I_max, where I m Number of interactions with the main teaching point, I s ⱼ represents the number of interactions at each teaching location, W m W s ⱼ represents the role weight of the corresponding teaching point, and I_max is the interaction saturation benchmark value. The interaction saturation benchmark value I_max is determined based on the total number of students N_total in all teaching points and the preset interaction saturation coefficient ρ, that is, I_max=ρ×N_total; The comprehensive value Q for network and hardware adaptation is obtained by weighted fusion of network adaptation coefficient K and hardware adaptation level quantization value H, i.e. Q = w1 × K + w2 × H, where H takes different quantization values ​​according to the hardware adaptation level, and w1 and w2 are preset weight coefficients and w1 + w2 = 1.

5. The method according to claim 4, characterized in that, The interactive intent recognition value R is calculated through a hybrid model that serially connects a convolutional neural network and a bidirectional long short-term memory network. The structure of the hybrid model is as follows: behavioral feature vectors, speech feature vectors and text feature vectors are concatenated and fused into a multimodal feature vector. After extracting local temporal features through at least one convolutional layer, it is input into at least one bidirectional long short-term memory network to capture temporal dependencies. Finally, it outputs a value R representing the intensity of the student's interactive intent through a fully connected layer and a sigmoid activation function. The value range is [0,1]. The behavioral feature vector, speech feature vector, and text feature vector are numerical sequences obtained after preprocessing based on the collected student video stream, audio stream, and text input stream, respectively. The behavioral feature vector is encoded by the coordinate sequence of key points of the student skeleton and has a first preset dimension. The speech feature vector is obtained by mapping the acoustic features of the speech signal through a fully connected layer and has a second preset dimension. The text feature vector is obtained by encoding the text content through a pre-trained language model and has a third preset dimension. The multimodal feature vector has a fourth preset dimension.

6. The method according to claim 1, characterized in that, The three-level adjustment mentioned in step S3 specifically includes: Basic threshold calculation: S_th0=a-b×(N / N0)+c×K_avg, where N is the number of students in real time, N0 is the preset baseline number of students, K_avg is the average value of the network adaptation coefficient of each teaching point, a is the baseline threshold constant, b is the influence coefficient of the number of students, and c is the influence coefficient of the network. Scene correction: The correction amount ΔS1 is determined according to the preset scene correction coefficient table. The scene correction coefficient table includes the coefficient corresponding to the teaching content type ∈ [-0.15, +0.10], the coefficient corresponding to the subject category ∈ [-0.08, +0.05], and the coefficient corresponding to the teaching link ∈ [-0.12, +0.08]. ΔS1 is the sum of the three types of coefficients, and S_th1 = S_th0 + ΔS1. Personalized correction: The correction amount ΔS2 is determined based on the student's historical interaction profile value P and the hardware adaptation level. When P is lower than the preset profile threshold P0, ΔS2 includes a downward adjustment amount of -(P0-P)×η, where η is the personalized correction coefficient. When the hardware adaptation level is basic and the teaching point is a sub-teaching point, ΔS2 includes a preset fixed downward adjustment amount ε. S_th=S_th1+ΔS2. Finally, upper and lower limits are set for the trigger threshold S_th, which is limited to the range of [0.15, 0.75]. These upper and lower limits are used to prevent the number of requests to go on stage from exceeding the processing capacity of the recording and broadcasting control equipment in extreme scenarios, which could lead to system overload and failure.

7. The method according to claim 1, characterized in that, The calculation formula for the dynamic quota N_q mentioned in step S4 is as follows: N_q=min(N_max, floor(N_total×λ×K_avg)) Where N_max is the preset maximum number of students allowed on stage, N_total is the total number of students at all teaching points, λ is the basic quota coefficient, and K_avg is the average network adaptation coefficient; when the calculation result is less than 1, N_q is 1. The multi-level priority rules are arranged from high to low as follows: First priority, students at the teaching points take priority over students at the main teaching point, in order to avoid high-bandwidth users at the main teaching point from vying for uplink resources and causing uneven load among the teaching points. The second priority is to prioritize students whose interaction intent recognition value R is higher than the preset intent threshold R0, over students whose value is lower than the threshold. The third priority is to prioritize students with basic hardware adaptation level and whose teaching point is a sub-teaching point, in order to compensate for the uplink bandwidth bottleneck of teaching points with low hardware adaptation level; the fourth priority is to prioritize students with historical interaction profile value P below the preset profile threshold P0, in order to prioritize students above the threshold. Within the same priority level, they are sorted from high to low according to the comprehensive collaborative interaction value S.

8. The method according to claim 1, characterized in that, In step S4, the on-stage interaction mode is determined and allocated in order from low to high according to the network adaptation coefficient K: when K < K2, regardless of the hardware adaptation level, the on-stage mode of plain text is allocated; when K2 ≤ K < K1, or when the hardware adaptation level is basic and K ≥ K2, the on-stage mode of voice plus text is allocated; when K ≥ K1 and the hardware adaptation level is high or medium, the on-stage mode of video plus voice is allocated; where K1 > K2.

9. A multimodal interactive collaborative control device based on a recorded and broadcast-controlled smart classroom, characterized in that, include: The multi-dimensional scenario preprocessing module is used to acquire basic configuration data, real-time network status data, terminal hardware data and teaching scenario data of each teaching point, and to complete the allocation of teaching point role weights, calculation of network adaptation coefficients, classification of hardware adaptation levels and dynamic association between electronic teaching aids and scenarios. The multimodal collaborative interaction evaluation module is used to collect students' multimodal data and integrate five dimensions of features—cross-teaching point collaborative interaction value, teaching aid correlation, network and hardware adaptation comprehensive value, interactive intent recognition value, and student historical interaction profile value—to calculate the comprehensive collaborative interaction value. The dynamic threshold adaptive adjustment module is used to determine the on-stage trigger threshold through three levels of adjustment: basic threshold calculation, scene correction, and personalized correction. This allows the network adaptation coefficient to simultaneously affect the calculation of the collaborative interaction comprehensive value and the calculation of the on-stage trigger threshold, forming a dynamic coupling balance between evaluation and threshold. The differentiated platform control module is used to perform platform trigger determination, dynamic quota calculation, multi-level priority sorting, and platform mode allocation based on the comparison result between the collaborative interaction comprehensive value and the platform trigger threshold. The on-stage status adaptation and optimization module is used to continuously monitor the network, device and interaction status of students on stage, and dynamically adjust the bitrate, on-stage status and quota. The model iteration and optimization module is used to store teaching process data and update the feature weights of the collaborative interaction evaluation model and the parameters of the interactive intent recognition model when preset trigger conditions are met, forming a closed-loop optimization based on feedback of the performance on stage.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images