Smart blackboard interactive control method and system based on multimodal interaction

Abstract

This paper proposes a smart blackboard interactive control method and system based on multimodal interaction. The method includes: collecting multimodal classroom data and constructing a joint feature matrix; collecting student classroom behavior, constructing a spatial behavior map, and converting the spatial behavior map into spatial behavior features through a graph convolution layer; fusing the joint feature matrix with the spatial behavior features to generate joint features, which are input into a pre-built LSTM-MLP to output a probability distribution of teacher intentions, selecting the maximum probability intention as the teacher intention, representing the teacher's real-time control intention; generating a structured content tree based on the teacher intentions and spatial behavior features; binding the teacher intentions to nodes in the structured content tree, and using the spatial behavior features as trigger conditions to generate a control strategy vector. This invention can adapt to changing teaching environments and comprehensively improve the intelligence level and application value of the smart blackboard.

Description

Technical Field

[0001] The present invention belongs to the field of smart blackboard interactive control, and in particular relates to a smart blackboard interactive control method and system based on multimodal interaction. Background Art

[0002] With the rapid development of emerging educational models such as smart education and smart classrooms, smart blackboards, as a crucial component of information-based classrooms, have gradually become an indispensable interactive device in modern teaching. Smart blackboards integrate multiple functions, including touch, display, and handwriting recognition, significantly improving teaching efficiency and classroom interactivity. However, current interactive control methods for smart blackboards still have significant limitations. Traditional systems often rely on a single interactive signal input, such as gesture recognition to control the blackboard or voice recognition to issue operational commands. This single interactive model often struggles to cope with the complex and ever-changing dynamics of classrooms in real-world teaching environments. For example, during a lecture, a teacher's frequent movements, changes in standing position, noise interference in the voice signal, and the complexity of the blackboard content can significantly reduce the system's ability to accurately identify the teacher's true intentions. Furthermore, most existing systems employ a passive response control strategy, triggering corresponding actions only upon receiving clear control signals. This makes it difficult to proactively perceive and proactively respond to the teacher's teaching behavior, resulting in limited classroom flow and low interactive efficiency. At the same time, the smart blackboard also faces another core problem, that is, the real-time organization and presentation of blackboard content relies on manual management by teachers. Especially when the amount of information is large and the content structure is complex (such as multi-step formula derivation and multi-level knowledge point explanation), teachers must manually switch screens, partition the display, or manually store and expand the content, which can easily increase the operational burden and affect the teaching rhythm and fluency.

[0003] Therefore, how to integrate multimodal signals, accurately predict teacher intentions, and achieve intelligent self-organization and optimized display of blackboard content in a dynamic teaching environment has become a technical bottleneck that needs to be urgently solved in the current smart blackboard interactive control system. Summary of the Invention

[0004] The purpose of the present invention is to design a smart blackboard interactive control method and system based on multimodal interaction, which can effectively overcome the shortcomings of the existing smart blackboard system in terms of single signal perception, delayed control response, and inefficient content organization.

[0005] In order to achieve the above-mentioned object, the present invention provides a first aspect of a smart blackboard interactive control method based on multimodal interaction, the method comprising:

[0006] S1. Collect multimodal classroom data and construct a joint feature matrix; the joint feature matrix includes the teacher's voice signal, gesture signal, location data, environmental information, and blackboard content; the blackboard content is the semantic context of the current classroom;

[0007] S2. Collect students' classroom behaviors, combine them with the joint feature matrix as graph nodes, use the interaction between teachers and nodes as graph edges, construct a spatial behavior graph, and convert the spatial behavior graph into spatial behavior features through a graph convolution layer;

[0008] S3. Fusing the joint feature matrix and the spatial behavior feature to generate a joint feature, which is input into a pre-built LSTM-MLP to output the probability distribution of the teacher's intention. The maximum probability intention is selected as the teacher's intention, which represents the teacher's real-time control intention.

[0009] S4. Generate a structured content tree based on the teacher's intention and spatial behavior characteristics, including: tree nodes representing semantic types, tree edges representing semantic relationships, wherein each tree node carries the teacher's interaction weight and logical hierarchical relationship;

[0010] S5. Bind the teacher's intention with the nodes in the structured content tree, and use the spatial behavior characteristics as trigger conditions to generate a control strategy vector; for each tree node in the structured content tree, generate a control strategy vector based on the teacher's intention, the teacher's current spatial position, and the degree of conflict of the previous interactive control instructions, and each element of the control strategy vector corresponds to an interactive operation suggestion for a content node.

[0011] Preferably, the S4 specifically includes:

[0012] S41, using the updated blackboard content as the current blackboard content pool;

[0013] S42. For all semantic blocks in the current blackboard content, first determine the intention-content relevance, which is used to measure the semantic matching degree between the current teacher's intention and the content theme represented by the semantic block;

[0014] S43. Calculate the teacher's attention tendency value for the content area in the spatial behavior based on the teacher's position and interaction trajectory in the spatial behavior characteristics;

[0015] S44, calculating weights according to the semantic matching degree and the attention tendency value, and establishing a tree structure from high to low weights, while retaining the logical connections between tree nodes;

[0016] S45. After the structured content tree is generated, the display suggestion attributes of each tree node are recorded; wherein each tree node includes a semantic type, a weight, and a logical hierarchical relationship.

[0017] Preferably, the blackboard content is extracted in real time by OCR and converted into structured text data;

[0018] The signal-to-noise ratio of the multimodal data is calculated, and the dynamic weight of each multimodal data is calculated according to the signal-to-noise ratio; the highest the signal-to-noise ratio, the greatest weight of the signal in the overall decision.

[0019] Preferably, the spatial behavior map is constructed as follows:

[0020] The teacher's spatial location information is used as the main node coordinates for graph initialization, while the blackboard area coordinates and student distribution are preset from the classroom layout map;

[0021] Defining a spatial interaction graph ,in Represents the set of entity graph nodes in the classroom, Represents the set of spatial relationship edges between entities; the weight of the graph edge Calculated as:

[0022] ;

[0023] in, They are graph nodes in the blackboard area and The spatial position coordinates of is the spatial diffusion coefficient, which controls the sensitivity of distance perception; is the interaction intensity between the teacher and other graph nodes; is the behavioral interaction weight factor;

[0024] The spatial behavior characteristics are characterized by Activation function generation.

[0025] Preferably, the joint feature is generated by performing feature fusion on the joint feature matrix and the spatial behavior feature, and adjusting the interaction intensity between the teacher and other graph nodes as a dynamic regularization factor.

[0026] Preferably, the display suggestion attributes include whether folding is required and whether split-screen highlighting is required.

[0027] Preferably, the control strategy vector is generated as follows:

[0028] ;

[0029] in, For tree nodes The control strategy output; is the matching degree between the intention and the semantic type of the tree node; is the spatial proximity between the teacher's current spatial position and the content node; is the degree of conflict with the previous control strategy; 、 、 is the adjustment factor, the trade-off logic of the control strategy; is the Sigmoid function, which normalizes the strategy strength to 0~1.

[0030] Preferably, a control action sequence list is generated according to the control strategy vector for calling by the display module of the smart blackboard;

[0031] When the control strategy output is greater than 0.7, the system assigns a display action to the tree node; when the control strategy output is less than 0.3, the system marks the tree node as hidden; and the middle area retains its original state.

[0032] Preferably, the interactive operation suggestion includes a target content node number, a suggested action type, and a trigger condition.

[0033] In a second aspect of the present invention, a smart blackboard interactive control system based on multimodal interaction is provided, the system comprising:

[0034] A multimodal data acquisition unit is used to collect multimodal classroom data and construct a joint feature matrix; the joint feature matrix includes the teacher's voice signal, gesture signal, location data, environmental information, and blackboard content; the blackboard content is the semantic context of the current class;

[0035] The spatial behavior feature generation unit is used to collect students' classroom behaviors, combine the joint feature matrix as graph nodes, and use the interaction between teachers and nodes as graph edges in the graph to construct a spatial behavior graph. The spatial behavior graph is then converted into spatial behavior features through a graph convolution layer.

[0036] The teacher intention analysis unit is used to fuse the joint feature matrix and the spatial behavior feature to generate a joint feature, input it into the pre-built LSTM-MLP, output the probability distribution of the teacher's intention, and select the maximum probability intention as the teacher's intention, which represents the teacher's real-time control intention;

[0037] A structured content tree generating unit, configured to generate a structured content tree according to the teacher's intention and spatial behavior characteristics, comprising: tree nodes representing semantic types, tree edges representing semantic relationships, wherein each tree node carries a teacher interaction weight and a logical hierarchical relationship;

[0038] The interactive suggestion generation unit is used to bind the teacher's intention with the nodes in the structured content tree, and use the spatial behavior characteristics as trigger conditions to generate a control strategy vector; for each tree node in the structured content tree, a control strategy vector is generated based on the teacher's intention, the teacher's current spatial position and the degree of conflict of the previous interactive control instructions, and each element of the control strategy vector corresponds to an interactive operation suggestion for a content node.

[0039] The beneficial technical effects of the present invention are at least as follows:

[0040] The present invention integrates a variety of interactive signals, including but not limited to the teacher's voice instructions, gestures, spatial position changes, classroom rhythm characteristics, etc., and combines the real-time perception of the blackboard's historical content and the classroom space environment to establish a multi-modal deep fusion teaching intention understanding mechanism, which can accurately predict the teacher's real control intention and actively trigger the blackboard's control logic. At the same time, in response to the problems of heavy burden of blackboard content management and chaotic hierarchical structure, the present invention innovatively designs a dynamic self-organization and display optimization method for blackboard content, which can automatically complete the layering, storage, folding and display layout adjustment of content according to the teacher's intention and teaching process, reduce the teacher's operating burden, and improve the fluency of classroom interaction and teaching efficiency. Through the linkage control strategy of multimodal perception, intention prediction and content intelligent organization, the present invention not only effectively solves the problems of slow response, easy misjudgment and complex operation of traditional smart blackboard systems, but also gives the blackboard system a stronger active service capability, can adapt to the changing teaching environment, and comprehensively improve the intelligence level and application value of the smart blackboard. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0042] Figure 1 This is a flow chart of the smart blackboard interaction control method based on multimodal interaction of the present invention.

[0043] Figure 2 This is a framework diagram of the smart blackboard interactive control system based on multimodal interaction of the present invention. DETAILED DESCRIPTION

[0044] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0045] In one or more embodiments, Figure 1 As shown, a smart blackboard interactive control method based on multimodal interaction is disclosed, and the method includes the following steps 1 to 5:

[0046] S1. Collect multimodal classroom data and construct a joint feature matrix; the joint feature matrix includes the teacher's voice signal, gesture signal, location data, environmental information and blackboard content; the blackboard content is the semantic context of the current classroom.

[0047] Specifically, in a smart classroom, teacher behavior and blackboard content are the core signal sources for multimodal interaction. To ensure the accuracy of subsequent intent prediction and the rationality of content presentation, this step aims to jointly collect and preprocess various classroom signals (voice, gestures, location, ambient noise) with the current blackboard content to form standardized features that can support multimodal interaction decision-making. This process is mainly divided into the following steps:

[0048] Signal acquisition:

[0049] The classroom is equipped with multimodal sensors for data collection, including:

[0050] Voice signal The teacher's voice is collected through a microphone array and streamed in real time at a sampling rate of 16 kHz. If the ambient noise is high, sound source localization technology is used to enhance the signal quality.

[0051] Hand signals : The teacher’s gestures are collected through the camera, and the image resolution is , sampling 30 frames per second. Gesture features are extracted through convolutional neural networks (CNN);

[0052] Location data : Use the indoor positioning system to obtain the teacher's location coordinates in real time , and mark its position in the classroom. The sampling frequency is 10 Hz;

[0053] Environmental signals : Use the ambient noise sensor to monitor the background noise level in the classroom (in decibels). This data is sampled once per second to help determine the reliability of the current signal.

[0054] Blackboard content collection Blackboard content is extracted in real time using optical character recognition (OCR) and layout analysis modules, identifying text, diagrams, formulas, and other items written on the blackboard and converting them into structured text data. Through OCR, the present invention can extract written content (such as formulas, graphics, or text blocks) and assign them semantic labels (such as "definition," "formula," or "step").

[0055] For example, when the teacher writes on the blackboard " ”, OCR will recognize the formula and mark it as a physical formula type, generating structured information.

[0056] Furthermore, in order to ensure that different signal sources (voice, gesture, position, etc.) have appropriate contributions in different environments, this step introduces a dynamic signal calibration mechanism (D-SCAM):

[0057] The signal-to-noise ratio (SNR) is calculated for each collected signal (voice, gesture, location, environment), and its dynamic weight is calculated based on the SNR of each signal. The higher the SNR, the greater the weight of the signal in the overall decision; the formula is as follows:

[0058] ;

[0059] in, Indicates the The signal-to-noise ratio of the signal mode, is the total number of signals. This weight is used to dynamically adjust the weights of different signals.

[0060] For example: If the speech signal The SNR is 12dB, the gesture signal The SNR is 25dB, while the ambient noise The SNR is 30dB, and the calculated weight is: , , Therefore, the system will rely on gesture and position data first in a noisy environment and reduce its reliance on voice signals.

[0061] Furthermore, based on the preprocessing results of each signal, a joint feature matrix is generated , as input in subsequent steps (such as intent prediction). Specifically: each modality (voice, gesture, location, environment, blackboard content) is normalized by the feature extraction module and combined with the signal-to-noise ratio weight; the blackboard content It provides the semantic context of the current class, while other signals (voice, gesture, position) provide the teacher's immediate behavior information. They are weighted and fused to form a unified input feature matrix , which is used for subsequent teacher intention prediction.

[0062] Furthermore, the output joint feature matrix As input for subsequent steps (such as spatial behavior graph construction, intention prediction, etc.), this feature matrix contains the joint information of teacher behavior, classroom space, and blackboard content, providing a complete context for intelligent decision-making.

[0063] S2. Collect students' classroom behaviors, combine them with the joint feature matrix as graph nodes, use the interaction between teachers and nodes as graph edges, construct a spatial behavior graph, and convert the spatial behavior graph into spatial behavior features through a graph convolution layer.

[0064] Specifically, this step is a bridge between "signal perception" and "intention prediction" in the overall patent solution. Its main task is to generate the joint feature matrix based on the output of step 1. , integrating dynamic behavior information in the classroom space and constructing a spatial behavior map , providing a global spatial and interactive context for subsequent intention reasoning and blackboard content organization. Especially in the dynamic spatial environment of smart classrooms, teacher behavior is constrained by spatial locations (such as the podium area, blackboard area, and student area) and classroom behavior relationships (such as student interaction and group discussions). Therefore, spatial behavior must be modeled as a dynamic feature input to effectively support the goal of "active control."

[0065] Furthermore, this step receives the joint feature matrix output from the previous step ,in Contains the teacher's voice signal , hand signals , location data , environmental information And the blackboard content These multimodal data contain both the teacher's action features (gestures, voice, position information) and the context of the current teaching content (blackboard structure information). ), but lacks the overall modeling of “spatial dynamics” and “multi-agent interaction behavior”.

[0066] Furthermore, spatial behavior maps Build:

[0067] This step introduces the Multi-Agent Spatial Behavior Graph Modeling mechanism (MSGM mechanism), which uses key entities in the classroom, such as "teacher, students, blackboard, and podium" as nodes, and the interaction relationships between teachers and other entities (such as relative distance, attention direction, and interactive actions) as edges in the graph;

[0068] Teacher's spatial location information As the main node coordinates for graph initialization, the blackboard area coordinates and student distribution Then preset from the classroom layout map;

[0069] Defining a spatial interaction graph ,in Represents the set of entity graph nodes in the classroom, The weight of the edge represents the spatial relationship between entities. Calculated by the following spatial perception function:

[0070] ;

[0071] in, Graph nodes and The spatial position coordinates of

[0072] is the spatial diffusion coefficient, which controls the sensitivity of distance perception;

[0073] The intensity of interaction between the teacher and other graph nodes, such as whether the teacher is talking to the student or writing on the blackboard. ;

[0074] It is a customized behavioral interaction weight factor, specially designed in the smart classroom scenario, to strengthen the joint modeling of spatial distance and behavioral actions.

[0075] It is understandable that compared with the traditional GNN graph structure, the edge weight of this graph is Interaction strength is introduced in As a regularization factor, it solves the problem that a single "physical distance" map is difficult to reflect the "intensity of interaction intention".

[0076] Furthermore, the feature embedding of spatial behavior graph:

[0077] Based on the above construction , design the spatial context embedding model (SCEM model), and transform the graph structure into spatial behavior features through the graph convolution layer (GCN) , the output is as follows:

[0078] ;

[0079] in, Represents a graph node in the graph The input feature vector (i.e. Middle and Node associated signal characteristics);

[0080] is the learned graph convolution weight matrix;

[0081] For graph nodes The edge weight normalization factor of ;

[0082] For graph nodes The set of neighbors.

[0083] It can be understood that through this feature embedding, the system deeply integrates the "teacher behavior signal" with the "spatial relationship" to generate spatial behavior features that can be directly used for subsequent intention prediction. .

[0084] The final output It includes information such as the interaction pattern, spatial dynamics, and distance relationship between the teacher and entities in the space (such as the blackboard, students, and podium), which serves as a direct input for the subsequent step (intention prediction); It has the following dimensional features: behavioral signal features (teacher's voice, gesture, position) + spatial relationship features (dynamic edge weights with blackboard, students, and podium) + behavioral interaction features ( reflects the current interactive activity).

[0085] S3. Fuse the joint feature matrix and spatial behavior features to generate joint features, input them into the pre-built LSTM-MLP, output the probability distribution of teacher intention, select the maximum probability intention as the teacher intention, and represent the teacher's real-time control intention.

[0086] Specifically, the goal of this step is to generate multimodal signal features based on the output of step 1. The spatial behavior characteristics output from step 2 , jointly predicting teachers' classroom control intentions , serving as the key input for subsequent active interactive control of the smart blackboard. This step, as the "behavior decision layer" in the patent chain, focuses on "understanding the teacher's intention" and does not involve the processing or display of blackboard content.

[0087] Furthermore, the input is (teacher's voice, gestures, position signals and blackboard content characteristics) and (Spatial relationship characteristics between the teacher and the blackboard, students, podium, etc.) Output the teacher's control intention , for example: "Prepare blackboard writing", "Expand new knowledge points", "Clear some blackboard writing" and other labels.

[0088] Furthermore, a JIPM+ intention prediction model specifically for smart classrooms was designed. The core innovation of this model lies in:

[0089] Space-behavior dynamic embedding module and Deep feature fusion;

[0090] The classroom interaction intensity regularization term introduces the "interaction intensity" between teachers and spatial entities as a dynamic adjustment factor in the intention prediction process, which is different from the general intention prediction model.

[0091] Further, and Generate joint features through fusion layer , and introduce the spatial interaction intensity matrix (Output from the spatial behavior diagram in step 2, reflecting the interaction intensity between the teacher and the blackboard, students, and podium) as a dynamic regularization factor:

[0092] ;

[0093] in, is the fusion weight of each modality feature, is a feature fusion operation (such as concatenation or MLP mapping); It is a "classroom interaction regularization coefficient" designed specifically for smart classroom scenarios, used to dynamically amplify or suppress the impact of spatial relationships on intent prediction; Reflects the teacher's current interaction intensity with other entities, such as "whether he is at the podium" or "whether he is close to students".

[0094] Furthermore, the fusion features Input into the LSTM-MLP structure to capture the temporal characteristics and spatial dynamic changes of teacher behavior and output the probability distribution of teacher intention , select the maximum probability intention as the final result.

[0095] A special "spatial dependency gating mechanism" is designed to dynamically adjust spatial features in LSTM. The impact on the current intention state ensures that the output of the intention prediction model can be automatically optimized in different scenarios such as "the teacher is close to the podium" and "the teacher is far away from the blackboard".

[0096] Final output , indicating the teacher's real-time control intention (such as "split screen", "new blackboard writing", "interactive mode", etc.), for direct call by subsequent steps (blackboard content organization and interactive control).

[0097] S4. Generate a structured content tree based on the teacher's intention and spatial behavior characteristics, including: tree nodes representing semantic types, tree edges representing semantic relationships, where each tree node carries the teacher's interaction weight and logical hierarchical relationship.

[0098] Specifically, the goal of this step is to determine the teacher's intention based on the output of step 3. And the spatial behavior characteristics output in step 2 , generate a blackboard content structure representation that matches the current teaching scenario This step serves as the "knowledge organization layer," resulting in a systematic understanding of the "logical structure of content presentation" as intended by the teacher, such as: What are the key points? What is the blackboard writing area? Is a structural stratification (definition-deduction-conclusion) necessary?

[0099] Further, input: teacher's current intention (e.g., “explain the definition,” “show a chart,” “expand old content”);

[0100] Teachers' spatial behavior characteristics (e.g. the teacher is near the blackboard area and is moving to the student area);

[0101] Output: structured content tree , which represents the organizational unit, logical hierarchy, and spatial correlation of the content, and is used for subsequent control strategy judgment.

[0102] Furthermore, content structure generation framework (CTG mechanism): The present invention designs an intent-space driven content structure generation module (CTG: Contextualized Tree Generator) for dynamically generating a semantic structure tree.

[0103] Specifically, the system maintains an updateable "current blackboard content pool" The content includes the structured blackboard information extracted in step 1 (picture blocks after OCR recognition, etc.). On this basis, the present invention constructs a content semantic tree , the structure is as follows:

[0104] Tree nodes represent content units (e.g., definitions, examples, diagrams, derivations, etc.);

[0105] Tree edges represent semantic relations (e.g., “belongs to,” “derived from,” “exemplifies,” etc.);

[0106] Each node carries the teacher interaction weight ,Depend on and Calculate its current importance and display priority.

[0107] Furthermore, the structural weight calculation formula (core modeling formula) is:

[0108] In order to dynamically adjust the importance of each content unit in the structure, the present invention designs the following weight distribution mechanism:

[0109] ;

[0110] : Content node Display weight in the structure tree;

[0111] : Intent-content relevance function, which measures the semantic matching degree between the current teacher intention and the content theme represented by the semantic block;

[0112] : The teacher's attention tendency value towards the content area in spatial behavior, such as approaching the block, facing the area, etc.;

[0113] and : An adjustable coefficient used to balance the structural modeling effects of intention guidance and space guidance.

[0114] Furthermore, the structure tree generation process:

[0115] right All semantic blocks in the ,first calculated by means of keyword matching, semantic embedding matching, etc. ;

[0116] Based on The teacher position and interaction trajectory in , calculation ;

[0117] Comprehensive calculation weight , build a tree structure from high to low weight, and retain the logical connection between tree nodes (such as: for Extension description, etc.);

[0118] Structure tree After generation, record the display suggestion attributes of each tree node (such as whether it needs to be folded or whether it needs split-screen highlighting).

[0119] Output: Output is a content structure tree , where each tree node is labeled:

[0120] Semantic type (definition / formula / legend / interactive block, etc.);

[0121] Weight (Affects the subsequent display priority);

[0122] Logical hierarchical relationships (for subsequent expansion, folding, and grouping display);

[0123] This will serve as the basis for executing control actions in step 5 to ensure that the content display is highly consistent with the teacher's intention and space status.

[0124] S5. Bind the teacher's intention with the nodes in the structured content tree, and use the spatial behavior characteristics as trigger conditions to generate a control strategy vector; for each tree node in the structured content tree, generate a control strategy vector based on the teacher's intention, the teacher's current spatial position, and the degree of conflict of the previous interactive control instructions, and each element of the control strategy vector corresponds to an interactive operation suggestion for a content node.

[0125] Specifically, this step aims to establish a control strategy model to guide the smart blackboard system on how to , spatial behavior state and the content structure tree , actively generate interactive control strategies and pass it to the execution module for display and feedback.

[0126] Furthermore, input variables: : teacher's current intention; : Teacher spatial behavior context; : Content structure tree, including all teaching content units and their hierarchical structure;

[0127] The goal of this step: Output control strategy , used to instruct the execution system to complete the display behavior.

[0128] Furthermore, the present invention designs an "Intention-Space Collaborative Control Generation Model" (AICG: ActionIntent-Control Generator) to convert the intention and structure Bind the nodes in , taking into account the spatial behavior context As a trigger condition, interactive control instructions are generated.

[0129] Furthermore, for the structure tree Each tree node , the system is based on Determine whether it is the current control focus. The system defines the following control strategy functions:

[0130] ;

[0131] :For tree nodes The control strategy output (the value is 0~1, indicating the confidence of the action execution);

[0132] : The matching degree between the intention and the semantic type of the tree node (such as = Display formula, it is highly matched with the formula type node);

[0133] : The spatial proximity of the teacher's current spatial position to the content node (e.g., distance, facing);

[0134] : The degree of conflict with the previous control strategy (such as repeated highlighting and display overlap);

[0135] 、 、 : Adjustment factor, trade-off logic of control strategy;

[0136] : Sigmoid function, normalizing the strategy strength to 0~1.

[0137] Further, policy decoding and action definition:

[0138] when When , the system is a tree node Assign the "show" action;

[0139] when , the system marks the tree node as "hidden";

[0140] The middle area is "Keep original state".

[0141] The system is based on the entire Sequence, outputs a list of control action sequences for the executor (display module) to call.

[0142] The final output is the control strategy vector , each element corresponds to an interactive operation suggestion for a content node; each interactive operation suggestion includes: target content node number, recommended action type (display / fold / highlight, etc.), triggering condition (spatial behavior state, intention label);

[0143] Sample output:

[0144] U_c = [ {node: "t_3", action: "expand", trigger: "I_c=Expand explanation&& F_2=In front of the blackboard"}, {node: "t_7", action: "hide", trigger: "I_c=Switch topic&& F_2=Podium area"}, ... ].

[0145] In one or more embodiments, Figure 2 As shown, a smart blackboard interactive control system based on multimodal interaction is disclosed, and the system includes:

[0146] The multimodal data acquisition unit 101 is used to collect multimodal classroom data and construct a joint feature matrix; the joint feature matrix includes the teacher's voice signal, gesture signal, location data, environmental information, and blackboard content; the blackboard content is the semantic context of the current classroom;

[0147] The spatial behavior feature generation unit 102 is used to collect students' classroom behaviors, combine the joint feature matrix as graph nodes, and use the interaction between teachers and nodes as graph edges in the graph to construct a spatial behavior graph, and convert the spatial behavior graph into spatial behavior features through a graph convolution layer;

[0148] The teacher intention analysis unit 103 is used to perform feature fusion on the joint feature matrix and the spatial behavior feature to generate a joint feature, input it into the pre-built LSTM-MLP, output the probability distribution of the teacher intention, and select the maximum probability intention as the teacher intention, which represents the teacher's real-time control intention;

[0149] A structured content tree generating unit 104 is configured to generate a structured content tree according to the teacher's intention and spatial behavior characteristics, including: tree nodes representing semantic types, tree edges representing semantic relationships, wherein each tree node carries a teacher interaction weight and a logical hierarchical relationship;

[0150] The interactive suggestion generation unit 105 is used to bind the teacher's intention with the nodes in the structured content tree, and use the spatial behavior characteristics as trigger conditions to generate a control strategy vector; for each tree node in the structured content tree, a control strategy vector is generated based on the teacher's intention, the teacher's current spatial position and the degree of conflict of the previous interactive control instructions, and each element of the control strategy vector corresponds to an interactive operation suggestion for a content node.

[0151] It is worth noting that the specific workflow of the smart blackboard interactive control system based on multimodal interaction provided by the embodiment of the present invention is the same as the workflow of the smart blackboard interactive control method based on multimodal interaction described in the above embodiment, and will not be repeated here.

[0152] An embodiment of the present invention also provides a smart blackboard interactive control device based on multimodal interaction, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the steps in the embodiment of the smart blackboard interactive control method based on multimodal interaction are implemented, for example Figure 1 or, the processor implements the functions of the modules in the above-mentioned system embodiments when executing the computer program.

[0153] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the multimodal interaction-based smart blackboard interactive control device.

[0154] The multimodal interactive smart blackboard interactive control device can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The multimodal interactive smart blackboard interactive control device can include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the multimodal interactive smart blackboard interactive control device can also include input and output devices, network access devices, buses, and the like.

[0155] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASAC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the multimodal interaction-based intelligent blackboard interactive control device, and utilizes various interfaces and lines to connect the various parts of the multimodal interaction-based intelligent blackboard interactive control device.

[0156] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the multimodal interaction-based smart blackboard interactive control device by running or executing the computer programs and / or modules stored in the memory and accessing the data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the data storage area may store data generated based on the operation of the air conditioning controller. Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0157] If the integrated module of the multimodal interaction-based smart blackboard interactive control device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0158] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0159] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A smart blackboard interactive control method based on multimodal interaction, characterized in that: The method comprises: S1. Collect multimodal classroom data and construct a joint feature matrix; the joint feature matrix includes the teacher's voice signal, gesture signal, location data, environmental information, and blackboard content; the blackboard content is the semantic context of the current classroom; S2. Collect students' classroom behaviors, combine them with the joint feature matrix as graph nodes, use the interaction between teachers and graph nodes as graph edges, construct a spatial behavior graph, and convert the spatial behavior graph into spatial behavior features through a graph convolution layer; S3. Fusing the joint feature matrix and the spatial behavior feature to generate a joint feature, which is input into a pre-built LSTM-MLP to output the probability distribution of the teacher's intention. The maximum probability intention is selected as the teacher's intention, which represents the teacher's real-time control intention. S4. Generate a structured content tree based on the teacher's intention and spatial behavior characteristics, including: tree nodes representing semantic types, tree edges representing semantic relationships, wherein each tree node carries the teacher's interaction weight and logical hierarchical relationship; S5. Bind the teacher's intention with the tree nodes in the structured content tree, and use the spatial behavior characteristics as trigger conditions to generate a control strategy vector; for each tree node in the structured content tree, generate a control strategy vector based on the teacher's intention, the teacher's current spatial position, and the degree of conflict of the previous interactive control instructions, and each element of the control strategy vector corresponds to an interactive operation suggestion for a content node.

2. The intelligent blackboard interactive control method based on multimodal interaction according to claim 1 is characterized in that: Said S4 specifically includes: S41, using the updated blackboard content as the current blackboard content pool; S42. For all semantic blocks in the current blackboard content, first determine the intention-content relevance, which is used to measure the semantic matching degree between the current teacher's intention and the content theme represented by the semantic block; S43, calculating the teacher's attention tendency value for the semantic block area in the spatial behavior according to the teacher's position and interaction trajectory in the spatial behavior characteristics; S44, calculating weights according to the semantic matching degree and the attention tendency value, and establishing a tree structure from high to low weights, while retaining the logical connections between tree nodes; S45. After the structured content tree is generated, the display suggestion attributes of each tree node are recorded; wherein each tree node includes a semantic type, a teacher interaction weight, and a logical hierarchical relationship.

3. The intelligent blackboard interactive control method based on multimodal interaction according to claim 1 is characterized in that: The blackboard content is extracted in real time through OCR and converted into structured text data; The signal-to-noise ratio of the multimodal data is calculated, and the dynamic weight of each multimodal data is calculated according to the signal-to-noise ratio; the highest the signal-to-noise ratio, the greatest weight of the signal in the overall decision.

4. The intelligent blackboard interactive control method based on multimodal interaction according to claim 1 is characterized in that: The spatial behavior map is constructed as follows: The teacher's spatial location information is used as the main node coordinates for graph initialization, while the blackboard area coordinates and student distribution are preset from the classroom layout map; Defining a spatial interaction graph ,in Represents the set of entity graph nodes in the classroom, Represents the set of spatial relationship edges between entities; the weight of the graph edge Calculated as: ； in, They are graph nodes in the blackboard area and The spatial position coordinates of is the spatial diffusion coefficient, which controls the sensitivity of distance perception; is the interaction strength between the current teacher’s graph node i and other graph nodes j; is the behavioral interaction weight factor; The spatial behavior characteristics are characterized by Activation function generation.

5. The intelligent blackboard interactive control method based on multimodal interaction according to claim 4 is characterized in that: The joint feature is generated by fusing the joint feature matrix and the spatial behavior feature, and adjusting it based on the interaction strength between the teacher and other graph nodes as a dynamic regularization factor.

6. The intelligent blackboard interactive control method based on multimodal interaction according to claim 2 is characterized in that: The display suggestion attributes include whether folding is required and whether split-screen highlighting is required.

7. The intelligent blackboard interactive control method based on multimodal interaction according to claim 1 is characterized in that: Generate a control action sequence list according to the control strategy vector for the display module of the smart blackboard to call; Among them, when the control strategy vector output is greater than 0.7, the system assigns a display action to the tree node; when the control strategy vector output is less than 0.3, the system marks the tree node as hidden; the middle area retains the original state.

8. The intelligent blackboard interactive control method based on multimodal interaction according to claim 7 is characterized in that: The interactive operation suggestion includes a target content node number, a suggested action type, and a trigger condition.

9. The intelligent blackboard interactive control system based on multimodal interaction is characterized by: The system comprises: A multimodal data acquisition unit is used to collect multimodal classroom data and construct a joint feature matrix; the joint feature matrix includes the teacher's voice signal, gesture signal, location data, environmental information, and blackboard content; the blackboard content is the semantic context of the current class; The spatial behavior feature generation unit is used to collect students' classroom behaviors, combine the joint feature matrix as graph nodes, and use the interaction between teachers and graph nodes as graph edges in the graph to construct a spatial behavior graph. The spatial behavior graph is then converted into spatial behavior features through a graph convolution layer. The teacher intention analysis unit is used to fuse the joint feature matrix and the spatial behavior feature to generate a joint feature, input it into the pre-built LSTM-MLP, output the probability distribution of the teacher's intention, and select the maximum probability intention as the teacher's intention, which represents the teacher's real-time control intention; A structured content tree generating unit, configured to generate a structured content tree according to the teacher's intention and spatial behavior characteristics, comprising: tree nodes representing semantic types, tree edges representing semantic relationships, wherein each tree node carries a teacher interaction weight and a logical hierarchical relationship; The interactive suggestion generation unit is used to bind the teacher's intention with the tree nodes in the structured content tree, and use the spatial behavior characteristics as trigger conditions to generate a control strategy vector; for each tree node in the structured content tree, a control strategy vector is generated based on the teacher's intention, the teacher's current spatial position and the degree of conflict of the previous interactive control instructions, and each element of the control strategy vector corresponds to an interactive operation suggestion for a content node.

Images