Intelligent mobile blackboard writing real-time semantic recognition method based on multi-modal fusion

Abstract

The invention discloses an intelligent mobile blackboard writing real-time semantic recognition method based on multi-modal fusion, and the method comprises the steps: collecting multi-modal data, carrying out the preprocessing, and completing the time and space alignment; constructing an improved PSENet network, and detecting a blackboard-writing text instance area; generating a space-time alignment grid, and constructing a blackboard writing candidate semantic unit; writing a double-track semantic structure, and updating dominant and implicit semantic tracks; semantic consistency verification is executed, and semantic backtracking and version correction are triggered; and constructing a semantic conflict field, and scheduling and outputting a structured semantic recognition result. According to the method, real-time, continuous and consistent semantic recognition of the writing content of the intelligent mobile blackboard is realized through multi-modal fusion processing and combination with improved PSENet detection, double-track reversible semantic memory and semantic conflict field scheduling.

Description

Technical Field

[0001] This invention relates to the field of smart blackboard technology, and in particular to a real-time semantic recognition method for smart mobile blackboard writing based on multimodal fusion. Background Technology

[0002] With the continuous development of information processing technology and intelligent interactive devices, smart mobile blackboards are gradually being applied in classroom teaching scenarios. By digitally collecting and processing the content written on the blackboard, they enable the storage, retrieval, and reuse of this information. Existing technologies typically rely on image acquisition devices to capture the blackboard image and combine computer vision and data processing methods to identify and analyze the content, thus improving the automation level of blackboard information processing to some extent. These technologies essentially fall under the category of computer information processing, which involves the collection, encoding, analysis, and processing of multi-source digital information, and have high application value in teaching support systems.

[0003] However, in actual teaching, the content written on a smart mobile blackboard has significant dynamic characteristics. The writing often evolves with operations such as writing, erasing, and covering, and is also affected by factors such as writing occlusion, surface reflection, and changes in lighting. Existing technologies mostly focus on static text recognition of blackboard images within a single frame or short time window, lacking the ability to continuously model the evolution of blackboard content over time, making it difficult to accurately reflect the formation order and semantic relationships of the content. Existing blackboard recognition systems typically process only at the pixel or character level, failing to effectively combine multimodal data such as writing trajectories, resulting in insufficient stability and consistency of recognition results in complex teaching scenarios.

[0004] Existing technologies for processing whiteboard recognition results generally lack a mechanism for systematically managing and verifying the recognized semantics, making it difficult to address semantic conflicts arising from repeated modifications to the whiteboard content. Once a recognition error or semantic inconsistency occurs, the system can usually only correct it through re-recognition or manual intervention, lacking reversible semantic backtracking capabilities and differentiated processing strategies based on the degree of conflict.

[0005] Therefore, how to provide a real-time semantic recognition method for smart mobile blackboard writing based on multimodal fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a real-time semantic recognition method for smart mobile blackboard writing based on multimodal fusion. This invention collects and processes multimodal data generated during the teaching process on a smart mobile blackboard, constructing blackboard text instance regions, blackboard semantic units, and their spatiotemporal relationships, thereby achieving continuous semantic recognition and management of the blackboard content as it evolves over time. This invention comprehensively utilizes computer information processing technologies such as multimodal data fusion processing, blackboard text instance detection, semantic memory modeling, and conflict-aware scheduling. It systematically describes a method for real-time modeling, verification, and backtracking of blackboard semantics in dynamic blackboard scenarios, effectively addressing semantic changes caused by blackboard writing, erasing, and covering operations. Compared with existing technologies, this invention improves the continuity, consistency, and stability of blackboard semantic recognition by introducing a dual-track reversible blackboard semantic memory structure and a semantic conflict field-driven scheduling mechanism, exhibiting advantages such as strong real-time performance, high robustness, and strong semantic management capabilities.

[0007] The real-time semantic recognition method for smart mobile blackboard writing based on multimodal fusion according to embodiments of the present invention includes:

[0008] Collect multimodal data generated during the teaching process of the smart mobile blackboard, preprocess the multimodal data and unify the spatial coordinates to obtain preprocessed multimodal data;

[0009] Based on the preprocessed multimodal data, an improved PSENet network is constructed to perform whiteboard text instance region detection, generate text kernel regions and expand them step by step to obtain a set of whiteboard text instance regions;

[0010] Using a set of whiteboard text instance regions and preprocessed multimodal data, a spatiotemporal structured evidence alignment grid is constructed to generate candidate semantic units for whiteboard writing.

[0011] Write the candidate semantic units of the blackboard into the dual-track reversible blackboard semantic memory structure, perform semantic recording, semantic association and version evolution processing, generate blackboard semantic nodes and update the dual-track reversible blackboard semantic memory structure.

[0012] Based on the preset teaching causal relationship rules, the semantic consistency of newly generated or updated blackboard semantic nodes is verified. When the verification fails, semantic backtracking is triggered to correct or delay the confirmation of the semantic type, spatial boundary or version status of the blackboard semantic nodes.

[0013] Based on the semantic inconsistencies between explicit and implicit semantic tracks, the semantic consistency verification results, and the version status of the semantic nodes in the blackboard writing, a semantic conflict field is constructed. The blackboard writing semantic processing process is scheduled and controlled according to the conflict intensity distribution of the semantic conflict field, and a structured blackboard writing semantic recognition result is output.

[0014] Optionally, the multimodal data includes blackboard image data, writing trajectory data, and speech data corresponding to the blackboard image data and writing trajectory data on the time axis.

[0015] Optionally, the preprocessing of multimodal data and the unification of spatial coordinates includes performing time alignment processing on the multimodal data, performing board area calibration processing on the blackboard image data, and mapping the writing trajectory data to a board spatial coordinate system consistent with the blackboard image data to form a unified spatiotemporal coordinate representation.

[0016] Optionally, the step of generating a text kernel region and progressively expanding it to obtain a set of whiteboard text instance regions includes:

[0017] Blackboard image data is extracted from the preprocessed multimodal data and used as the input image;

[0018] The input image data is fed into the backbone feature extraction layer of the improved PSENet network to obtain the backbone feature map. The backbone feature extraction layer is connected to the first convolutional block, the second convolutional block, the third convolutional block and the fourth convolutional block in sequence. A plate surface geometric feature injection module is connected in series between the third convolutional block and the fourth convolutional block. The plate surface geometric calibration parameters in the preprocessing stage are received and converted into a geometric feature map of the same scale as the backbone feature map. The geometric feature map is injected into the backbone feature map through channel splicing and the geometrically enhanced backbone feature map is output.

[0019] The geometrically enhanced backbone feature map is input into the multi-scale feature fusion layer. The multi-scale feature fusion layer fuses features of different levels in a top-down and lateral connection manner. At each lateral connection of the multi-scale feature fusion layer, a reflection and occlusion suppression gating module is connected in series. The low-level texture features and high-level semantic features before fusion are used as inputs to generate corresponding gating weight maps. The low-level texture features are then gated and suppressed before participating in the fusion, and the multi-scale fused features after reflection and occlusion suppression are output.

[0020] Multi-scale fusion features are input into the text kernel prediction head and the extended guidance prediction head, which output the text kernel region map and the extended guidance map respectively. A trajectory consistency boundary calibration module is connected in parallel at the output end of the extended guidance map. The trajectory consistency boundary calibration module receives the writing trajectory data, generates a trajectory coverage heat map, performs boundary calibration on the extended guidance map, and outputs the calibrated extended guidance map.

[0021] Based on the text kernel region map at multiple scale levels and the calibrated extended guide map, a step-by-step expansion generation process is performed to output a set of whiteboard text instance regions, recording the spatial location and time range corresponding to a unified time axis for each whiteboard text instance region.

[0022] Optionally, the generation of candidate semantic units for whiteboard writing includes:

[0023] Obtain the set of whiteboard text instance regions and the writing trajectory data in the preprocessed multimodal data, and divide the writing trajectory data into multiple time-continuous trajectory segments according to a unified time axis;

[0024] A spatiotemporal structured evidence alignment grid is constructed, which consists of a spatial grid layer, a time slice layer, and an evidence index layer. The spatial grid layer divides the board surface into multiple spatial grid units according to the board surface coordinate system, the time slice layer divides time into multiple time slice units according to a unified time axis, and the evidence index layer records the image instance evidence and trajectory fragment evidence corresponding to each spatial grid unit in each time slice unit.

[0025] An adaptive raster thinning mechanism is introduced, which projects the set of blackboard text instance regions onto the board coordinate system, performs thinning and splitting processing on the spatial raster units that contain the boundaries of the blackboard text instance regions, and maintains the original raster scale for the spatial raster units that do not contain the boundaries of the blackboard text instance regions, thus forming a hierarchical spatial raster structure.

[0026] Write the set of whiteboard text instance areas into the evidence index layer. According to the occurrence time of each whiteboard text instance area on the unified time axis, register it to the spatial grid unit under the corresponding time slice unit. Write the trajectory fragments into the evidence index layer. According to the occurrence time of each trajectory fragment on the unified time axis, register it to the spatial grid unit under the corresponding time slice unit.

[0027] Alignment determination is performed on image instance evidence and trajectory fragment evidence registered in the same spatial grid cell within the same time slice cell. Image instance evidence and trajectory fragment evidence that satisfy temporal overlap and spatial coverage are grouped into an alignment pair, and the alignment pairs are aggregated into whiteboard candidate semantic units.

[0028] Optionally, the step of generating blackboard semantic nodes and updating the dual-track reversible blackboard semantic memory structure includes:

[0029] A dual-track reversible blackboard semantic memory structure is constructed, which includes explicit semantic tracks, implicit semantic tracks, a version chain index table, and an anchor point mapping table between tracks.

[0030] Based on the spatial location and temporal range of the candidate semantic units on the whiteboard, a matching whiteboard semantic node is retrieved in the explicit semantic track. When no matching whiteboard semantic node is found, a new whiteboard semantic node is created in the explicit semantic track. When a matching whiteboard semantic node is found, an update is performed on the matching whiteboard semantic node. Evidence association information corresponding to the candidate semantic units on the whiteboard is recorded in the explicit semantic track.

[0031] Establish structural associations for blackboard semantic nodes in the explicit semantic track. The structural associations include the adjacency associations between adjacent blackboard semantic nodes in the same time period and the inclusion associations between blackboard semantic nodes within the same blackboard text instance area. Write the structural associations into the dual-track reversible blackboard semantic memory structure.

[0032] Version evolution processing is performed on the whiteboard semantic nodes. When a writing overwrite event is detected, a new version record is generated for the whiteboard semantic node and written into the version chain index table. When an erase event is detected, an erase version record is generated for the whiteboard semantic node and written into the version chain index table. The current valid version mark is written into the explicit semantic track.

[0033] Based on the changes in the temporal order and structural association of the currently effective version of the blackboard semantic nodes in the explicit semantic track, latent semantic nodes corresponding to the blackboard semantic nodes are generated in the implicit semantic track. The correspondence between the latent semantic nodes and the blackboard semantic nodes is established through the inter-track anchor mapping table, and the dual-track reversible blackboard semantic memory structure is updated.

[0034] Optionally, the step of performing semantic consistency verification on newly generated or updated whiteboard semantic nodes, and triggering semantic backtracking processing when verification fails, includes:

[0035] Read newly generated or updated blackboard semantic nodes from the explicit semantic track, and read the context blackboard semantic nodes that intersect with the blackboard semantic nodes in the time range and are adjacent to the blackboard space to form a set of nodes to be verified and a set of context nodes.

[0036] Based on the preset teaching causal relationship rules, causal verification relationship records are generated for the set of nodes to be verified and the set of context nodes. The causal verification relationship records include pre-order dependency relationship records, successor association relationship records and structural hierarchy relationship records.

[0037] Based on the causal verification relationship record, semantic consistency verification is performed on each whiteboard semantic node in the node set to be verified. The semantic consistency verification includes dependency consistency verification, temporal consistency verification and layout consistency verification, and the corresponding verification result marker is output.

[0038] When the verification result flag indicates that the semantic consistency verification has failed, a backtracking trigger flag is written to the corresponding whiteboard semantic node, and the whiteboard semantic node is set to the backtracking state;

[0039] Semantic backtracking processing is performed on the blackboard semantic nodes that are in the pending backtracking state. The semantic backtracking process includes reading the historical version of the blackboard semantic node from the explicit semantic track and switching the current valid version, updating the semantic type, spatial boundary and version status of the blackboard semantic node, and updating the dual-track reversible blackboard semantic memory structure after setting the blackboard semantic node to the delayed confirmation state.

[0040] Optionally, the output structured whiteboard semantic recognition result includes:

[0041] Read the explicit semantic track's blackboard semantic nodes, implicit semantic track's latent semantic nodes, semantic consistency verification result markers, and version status information of the blackboard semantic nodes from the dual-track reversible blackboard semantic memory structure.

[0042] A spatial index of the semantic conflict field is constructed in the spatial coordinate system of the board surface. The board surface is divided into multiple conflict evaluation units, and a conflict record item is established for each conflict evaluation unit. The conflict record item includes semantic deviation record, verification failure record, and version change record.

[0043] A version-driven conflict propagation mechanism is introduced within each conflict assessment unit. The conflict record items corresponding to the blackboard semantic nodes that have overwritten or erased version records are propagated to adjacent conflict assessment units. The conflict record items of adjacent conflict assessment units are cumulatively updated according to the number of propagations and the propagation sequence to form a semantic conflict field.

[0044] A conflict intensity distribution is generated based on the conflict record items of each conflict assessment unit. The conflict intensity distribution is jointly determined by the cumulative value of semantic deviation records, the cumulative value of verification failure records, and the cumulative value of version change records. The conflict intensity distribution is mapped to the corresponding processing priority, processing frequency, and freeze flag.

[0045] The whiteboard semantic processing is scheduled and controlled based on processing priority, processing frequency, and freezing markers. The current valid version state of the whiteboard semantic node is maintained for the conflict evaluation unit corresponding to the freezing marker. For the conflict evaluation unit corresponding to the non-freezing marker, the whiteboard semantic node is updated according to the processing priority order, and the structured whiteboard semantic recognition result is output.

[0046] The beneficial effects of this invention are:

[0047] This invention establishes a stable data processing foundation across the spatial and temporal dimensions of the blackboard by uniformly collecting and processing multimodal data generated during the teaching process using a smart mobile blackboard. This enables the correlation analysis of blackboard image data and writing trajectory data within the same spatiotemporal coordinate system. Through effective detection and alignment of blackboard text instance regions, this invention can accurately obtain the spatial distribution and temporal evolution characteristics of the blackboard content, thereby providing reliable data input for subsequent blackboard semantic modeling and processing, and improving the stability and continuity of blackboard recognition in dynamic teaching scenarios.

[0048] This invention constructs a dual-track reversible blackboard semantic memory structure to continuously record and manage the evolution of blackboard semantics, ensuring that the content retains a clear semantic structure even after multiple writing, covering, and erasing operations. When inconsistencies or conflicts arise in the recognition results, the system can backtrack and adjust the blackboard semantics based on a semantic consistency verification mechanism, avoiding the accumulation of semantic errors and effectively ensuring the logical consistency and traceability of the blackboard semantics, thus improving the reliability of the blackboard semantic recognition results.

[0049] This invention constructs a semantic conflict field and introduces a conflict intensity-driven scheduling control mechanism, enabling the system to differentiate the scheduling of blackboard semantic processing based on the degree of semantic conflict in different areas of the board. While ensuring real-time processing capabilities, it prioritizes high-conflict areas and reduces the processing overhead of low-conflict areas, thus achieving a balance between real-time performance and recognition accuracy. This invention can achieve efficient, stable, and semantically consistent real-time semantic recognition of blackboard writing in complex teaching environments, possessing significant practical value and potential for wider application. Attached Figure Description

[0050] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0051] Figure 1 This is a flowchart of the real-time semantic recognition method for smart mobile blackboard writing based on multimodal fusion proposed in this invention;

[0052] Figure 2 This is a schematic diagram of the text instance region detection structure based on the improved PSENet network in the real-time semantic recognition method for smart mobile blackboard writing based on multimodal fusion proposed in this invention. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0054] refer to Figure 1and Figure 2 A real-time semantic recognition method for smart mobile blackboard handwriting based on multimodal fusion includes:

[0055] Collect multimodal data generated during the teaching process of the smart mobile blackboard, preprocess the multimodal data and unify the spatial coordinates to obtain preprocessed multimodal data;

[0056] Based on the preprocessed multimodal data, an improved PSENet network is constructed to perform whiteboard text instance region detection, generate text kernel regions and expand them step by step to obtain a set of whiteboard text instance regions;

[0057] Using a set of whiteboard text instance regions and preprocessed multimodal data, a spatiotemporal structured evidence alignment grid is constructed to generate candidate semantic units for whiteboard writing.

[0058] Write the candidate semantic units of the blackboard into the dual-track reversible blackboard semantic memory structure, perform semantic recording, semantic association and version evolution processing, generate blackboard semantic nodes and update the dual-track reversible blackboard semantic memory structure.

[0059] Based on the preset teaching causal relationship rules, the semantic consistency of newly generated or updated blackboard semantic nodes is verified. When the verification fails, semantic backtracking is triggered to correct or delay the confirmation of the semantic type, spatial boundary or version status of the blackboard semantic nodes.

[0060] Based on the semantic inconsistencies between explicit and implicit semantic tracks, the semantic consistency verification results, and the version status of the semantic nodes in the blackboard writing, a semantic conflict field is constructed. The blackboard writing semantic processing process is scheduled and controlled according to the conflict intensity distribution of the semantic conflict field, and a structured blackboard writing semantic recognition result is output.

[0061] In this embodiment, the multimodal data includes blackboard image data, writing trajectory data, and speech data corresponding to the blackboard image data and writing trajectory data on the time axis.

[0062] In this embodiment, the preprocessing of multimodal data and the unification of spatial coordinates include performing time alignment processing on the multimodal data, performing board area calibration processing on the blackboard image data, and mapping the writing trajectory data to a board surface spatial coordinate system consistent with the blackboard image data to form a unified spatiotemporal coordinate representation.

[0063] In this embodiment, the step of generating a text kernel region and progressively expanding it to obtain a set of whiteboard text instance regions includes:

[0064] Blackboard image data is extracted from the preprocessed multimodal data and used as the input image;

[0065] The input image data is fed into the backbone feature extraction layer of the improved PSENet network to obtain the backbone feature map. The backbone feature extraction layer is connected to the first convolutional block, the second convolutional block, the third convolutional block and the fourth convolutional block in sequence. A plate surface geometric feature injection module is connected in series between the third convolutional block and the fourth convolutional block. The plate surface geometric calibration parameters in the preprocessing stage are received and converted into a geometric feature map of the same scale as the backbone feature map. The geometric feature map is injected into the backbone feature map through channel splicing and the geometrically enhanced backbone feature map is output. The first to fourth convolutional blocks are the feature extraction units connected in sequence in the backbone feature extraction layer of the improved PSENet network.

[0066] The geometrically enhanced backbone feature map is input into a multi-scale feature fusion layer. This layer fuses features from different levels using a top-down and lateral connection approach. At each lateral connection point of the multi-scale feature fusion layer, a reflection and occlusion suppression gating module is connected in series. Using the low-level texture features and high-level semantic features before fusion as input, a corresponding gating weight map is generated. The low-level texture features are then gated and suppressed before participating in the fusion, outputting the multi-scale fused features after reflection and occlusion suppression. Wherein:

[0067] High-level semantic features: Feature maps output by the subsequent convolutional blocks of the backbone feature extraction layer;

[0068] Low-level texture features: Feature maps output by the preceding convolutional blocks of the backbone feature extraction layer are introduced into the lateral connection branches leading to the multi-scale feature fusion layer;

[0069] Multi-scale fusion features are input into the text kernel prediction head and the extended guidance prediction head, which output a text kernel region map and an extended guidance map, respectively. A trajectory consistency boundary calibration module is connected in parallel at the output of the extended guidance map. The trajectory consistency boundary calibration module receives the written trajectory data, generates a trajectory coverage heatmap, and performs boundary calibration on the extended guidance map, outputting the calibrated extended guidance map. Specifically, generating the trajectory coverage heatmap and performing boundary calibration on the extended guidance map are as follows:

[0070] The writing trajectory data is mapped to a pixel grid in the board coordinate system according to a unified time axis. The number of times the trajectory point falls into each pixel is counted and accumulated over a time window to obtain a trajectory coverage heat map. The heat map is then normalized and smoothed to form a trajectory coverage weight distribution.

[0071] Align the trajectory coverage heatmap and the extended guide map at the pixel level, increase the extension weight of pixels that fall into high heat areas in the extended guide map, and reduce the extension weight or set them to non-extension state for pixels that fall into low heat areas and lack trajectory support, to obtain the trajectory-constrained extended guide map.

[0072] Perform boundary calibration on the extended guide map after trajectory constraint, converge the extended boundary to the trajectory coverage edge, remove isolated extended segments and repair the boundary connectivity, and output the calibrated extended guide map;

[0073] Based on text kernel region maps at multiple scale levels and calibrated extended guide maps, a hierarchical expansion generation process is performed to output a set of whiteboard text instance regions. The spatial location and corresponding time range of each whiteboard text instance region are recorded. The hierarchical expansion generation process specifically involves:

[0074] Using the smallest-scale text kernel region as the instance seed, each text kernel region is labeled with connected components and a unique instance identifier is assigned to each connected component to form an initial instance set;

[0075] Expanding sequentially from small to large scale, pixels in the next scale that belong to the allowed expansion range of the calibrated expansion guide map and are spatially adjacent and connected to the marked instance are merged into the corresponding instance. Within the same scale, conflicting pixels are assigned according to the instance adjacency priority rule, and the instance boundaries are updated.

[0076] After completing the expansion of all scales, the corresponding whiteboard text instance area is output for each instance, and the spatial position is determined according to the outer boundary of the instance in the image. The time range corresponding to the unified time axis is determined according to the timestamps of the instance's first appearance and final stabilization.

[0077] This invention structurally improves the original PSENet framework of backbone feature extraction—multi-scale fusion—text kernel prediction and stepwise expansion, forming an improved PSENet network: the input image is sequentially passed through the first to fourth convolutional blocks to obtain backbone feature maps, and a plate-surface geometric feature injection module is connected in series between the third and fourth convolutional blocks. The preprocessed plate-surface geometric calibration parameters are converted into geometric feature maps of the same scale and injected through channel splicing to achieve geometric enhancement. The geometrically enhanced features enter the top-down and laterally connected multi-scale feature fusion layer. At each lateral connection, a reflection and occlusion suppression gating module is connected in series to control the glare. The low-level features are gated and suppressed before being fused to obtain robust multi-scale fused features. These fused features are input to the text kernel prediction head and the extended guidance prediction head to output the text kernel region map and the extended guidance map, respectively. A trajectory consistency boundary calibration module is connected in parallel at the output of the extended guidance map. The trajectory coverage heat map generated by the writing trajectory is used to constrain and calibrate the boundary of the extended guidance map. Finally, the multi-scale text kernel region map and the calibrated extended guidance map are used to perform a step-by-step extension generation process to output a set of whiteboard text instance regions and record the spatial location and temporal range, thus completing the construction and application of the improved PSENet network.

[0078] In this embodiment, the generation of candidate semantic units for whiteboard writing includes:

[0079] Obtain the set of whiteboard text instance regions and the writing trajectory data in the preprocessed multimodal data, and divide the writing trajectory data into multiple time-continuous trajectory segments according to a unified time axis;

[0080] A spatiotemporal structured evidence alignment grid is constructed, which consists of a spatial grid layer, a time slice layer, and an evidence index layer. The spatial grid layer divides the board surface into multiple spatial grid units according to the board surface coordinate system, the time slice layer divides time into multiple time slice units according to a unified time axis, and the evidence index layer records the image instance evidence and trajectory fragment evidence corresponding to each spatial grid unit in each time slice unit.

[0081] An adaptive raster thinning mechanism is introduced, projecting the set of whiteboard text instance regions onto the whiteboard coordinate system. Thinning and splitting processing is performed on spatial raster cells containing the boundaries of the whiteboard text instance regions, while maintaining the original raster scale for spatial raster cells not containing the boundaries of the whiteboard text instance regions, forming a hierarchical spatial raster structure. Specifically, the thinning and splitting processing for spatial raster cells containing the boundaries of the whiteboard text instance regions is as follows:

[0082] Obtain the intersection of spatial raster cells with the boundaries of the text instance area on the whiteboard, mark spatial raster cells that have boundary crossings or boundary falling into as cells to be refined, and record the boundary fragment positions corresponding to the cells to be refined.

[0083] Each unit to be refined is divided equally according to a preset refinement granularity to generate multiple sub-grid units, and the positions of the boundary segments are synchronously mapped to the plate coordinate range of each sub-grid unit, and the spatial index relationship of the sub-grid units is updated.

[0084] The split sub-raster cells are re-determined to see if they still contain the boundary of the whiteboard text instance area. If they still do, the splitting continues until the refinement termination condition is met. The final sub-raster cells are written into the hierarchical spatial raster structure as boundary high-resolution area cells.

[0085] Write the set of whiteboard text instance areas into the evidence index layer. According to the occurrence time of each whiteboard text instance area on the unified time axis, register it to the spatial grid unit under the corresponding time slice unit. Write the trajectory fragments into the evidence index layer. According to the occurrence time of each trajectory fragment on the unified time axis, register it to the spatial grid unit under the corresponding time slice unit.

[0086] Alignment determination is performed on image instance evidence and trajectory fragment evidence registered in the same spatial grid cell within the same time slice cell. Image instance evidence and trajectory fragment evidence that satisfy temporal overlap and spatial coverage are grouped into an alignment pair, and the alignment pairs are aggregated into whiteboard candidate semantic units.

[0087] In this embodiment, generating blackboard semantic nodes and updating the dual-track reversible blackboard semantic memory structure includes:

[0088] A dual-track reversible blackboard semantic memory structure is constructed, which includes explicit semantic tracks, implicit semantic tracks, version chain index table, and track anchor point mapping table. The version chain index table is used to record the historical version order of the same blackboard semantic node, and the track anchor point mapping table is used to record the correspondence between the blackboard semantic nodes in the explicit semantic track and the potential semantic nodes in the implicit semantic track.

[0089] Based on the spatial location and temporal range of the candidate semantic units on the whiteboard, a matching whiteboard semantic node is retrieved in the explicit semantic track. When no matching whiteboard semantic node is found, a new whiteboard semantic node is created in the explicit semantic track. When a matching whiteboard semantic node is found, an update is performed on the matching whiteboard semantic node. Evidence association information corresponding to the candidate semantic units on the whiteboard is recorded in the explicit semantic track.

[0090] Establish structural associations for blackboard semantic nodes in the explicit semantic track. The structural associations include the adjacency associations between adjacent blackboard semantic nodes in the same time period and the inclusion associations between blackboard semantic nodes within the same blackboard text instance area. Write the structural associations into the dual-track reversible blackboard semantic memory structure.

[0091] Version evolution processing is performed on the whiteboard semantic nodes. When a writing overwrite event is detected, a new version record is generated for the whiteboard semantic node and written into the version chain index table. When an erase event is detected, an erase version record is generated for the whiteboard semantic node and written into the version chain index table. The current valid version mark is written into the explicit semantic track.

[0092] Based on the changes in the temporal order and structural association of the currently effective version of the blackboard semantic nodes in the explicit semantic track, latent semantic nodes corresponding to the blackboard semantic nodes are generated in the implicit semantic track. The correspondence between the latent semantic nodes and the blackboard semantic nodes is established through the inter-track anchor mapping table, and the dual-track reversible blackboard semantic memory structure is updated.

[0093] In this embodiment, the step of performing semantic consistency verification on newly generated or updated whiteboard semantic nodes, and triggering semantic backtracking processing when verification fails, includes:

[0094] Read newly generated or updated blackboard semantic nodes from the explicit semantic track, and read the context blackboard semantic nodes that intersect with the blackboard semantic nodes in the time range and are adjacent to the blackboard space to form a set of nodes to be verified and a set of context nodes.

[0095] Based on preset teaching causal relationship rules, causal verification relationship records are generated for the set of nodes to be verified and the set of context nodes. These causal verification relationship records include preorder dependency records, successor association records, and structural hierarchy relationship records. The preset teaching causal relationship rules are specifically as follows:

[0096] Based on the relationship direction rules between node type and course structure, it is used to determine the generation direction of preorder dependencies, successor associations and structural hierarchy relationships;

[0097] Based on the relationship determination rules of textbook catalog, knowledge graph edge type, text trigger word pattern and learning order constraint, it is used to generate corresponding causal verification relationship records between the node to be verified and the context node;

[0098] The adjudication rules based on conflict resolution and priority are used to determine the final record type or assign confidence when multiple relations are satisfied simultaneously.

[0099] Relationship record field rules are used to standardize the field composition of pre-dependent relationship records, successor association relationship records, and structural hierarchy relationship records;

[0100] Based on the causal verification relationship record, semantic consistency verification is performed on each whiteboard semantic node in the set of nodes to be verified. The semantic consistency verification includes dependency consistency verification, temporal consistency verification, and layout consistency verification, and the corresponding verification result marker is output, wherein:

[0101] Dependency consistency verification refers to checking whether the prerequisites required by the semantic nodes of the whiteboard have appeared in the context and have been correctly introduced, and whether there are any missing prerequisites or incorrect dependency directions.

[0102] The temporal consistency verification checks whether the presentation order of the semantic nodes on the whiteboard conforms to the sequential relationship defined in the causal verification record, so as to avoid temporal conflicts such as presenting subsequent knowledge first and presenting preceding knowledge later.

[0103] The layout consistency verification checks whether the relative layout of the semantic nodes on the whiteboard matches the dependency and hierarchical relationship, so that the preceding and superior nodes are presented visually more forward, higher, or more concentrated, and the subsequent and inferior nodes are presented after or below them, so as to avoid the contradiction between the layout and the logical relationship.

[0104] When the verification result flag indicates that the semantic consistency verification has failed, a backtracking trigger flag is written to the corresponding whiteboard semantic node, and the whiteboard semantic node is set to the backtracking state;

[0105] Semantic backtracking processing is performed on the blackboard semantic nodes that are in the pending backtracking state. The semantic backtracking process includes reading the historical version of the blackboard semantic node from the explicit semantic track and switching the current valid version, updating the semantic type, spatial boundary and version status of the blackboard semantic node, and updating the dual-track reversible blackboard semantic memory structure after setting the blackboard semantic node to the delayed confirmation state.

[0106] In this embodiment, the output of structured whiteboard semantic recognition results includes:

[0107] Read the explicit semantic track's blackboard semantic nodes, implicit semantic track's latent semantic nodes, semantic consistency verification result markers, and version status information of the blackboard semantic nodes from the dual-track reversible blackboard semantic memory structure.

[0108] A spatial index of the semantic conflict field is constructed in the spatial coordinate system of the board surface. The board surface is divided into multiple conflict evaluation units, and a conflict record item is established for each conflict evaluation unit. The conflict record item includes semantic deviation record, verification failure record, and version change record.

[0109] A version-driven conflict propagation mechanism is introduced within each conflict assessment unit. Conflict records corresponding to whiteboard semantic nodes that have overwritten or erased version records are propagated to adjacent conflict assessment units. The conflict records in adjacent conflict assessment units are cumulatively updated based on the propagation frequency and sequence, forming a semantic conflict field. The version-driven conflict propagation mechanism refers to:

[0110] Using the version change event of the semantic node on the blackboard as the trigger source, the conflict record item corresponding to the node is gradually propagated to the adjacent conflict evaluation unit according to the spatial or temporal adjacency relationship. The conflict record items of the propagated unit are weighted and cumulatively updated according to the number of propagations and the order of occurrence, forming a semantic conflict field that can characterize the distribution of conflict intensity from local to global scope.

[0111] A conflict intensity distribution is generated based on the conflict record items of each conflict assessment unit. This distribution is jointly determined by the cumulative values ​​of semantic deviation records, verification failure records, and version change records. The conflict intensity distribution is then mapped to corresponding processing priorities, processing frequencies, and freeze flags, where:

[0112] The cumulative value of the semantic deviation record is obtained by summing the deviations generated during the semantic verification process for the blackboard semantic nodes that fall into the corresponding conflict assessment unit.

[0113] The cumulative value of the verification failure records is obtained by summing the failure flags of the dependency consistency verification, timing consistency verification and layout consistency verification according to the number or weight;

[0114] The cumulative value of the version change record is obtained by summing the number or weight of version change events such as overwriting and erasing that occur on the blackboard semantic nodes that fall into the corresponding conflict assessment unit, and can be further superimposed with the increment generated by the version-driven conflict diffusion mechanism.

[0115] The whiteboard semantic processing is scheduled and controlled based on processing priority, processing frequency, and freezing markers. The current valid version state of the whiteboard semantic node is maintained for the conflict evaluation unit corresponding to the freezing marker. For the conflict evaluation unit corresponding to the non-freezing marker, the whiteboard semantic node is updated according to the processing priority order, and the structured whiteboard semantic recognition result is output.

[0116] Example 1:

[0117] To verify the feasibility of this invention in practice, it was applied to a smart classroom in a regular middle school. This classroom was equipped with a smart mobile blackboard system. The blackboard surface was an electronic writing pad, and a fixed high-definition camera was installed above the surface. An integrated writing trajectory acquisition module was used to simultaneously collect blackboard image data and writing trajectory data when the teacher wrote, erased, and covered the blackboard content. The teaching content was selected as the chapter on systems of two linear equations in two variables in junior high school mathematics. The blackboard writing in this course typically includes headings, definition of concepts, formula derivations, and multi-step calculation processes. The blackboard content was frequently modified and covered during class, exhibiting obvious temporal evolution characteristics. This is a typical application scenario where existing blackboard recognition technologies are prone to semantic discontinuity and recognition instability.

[0118] During classroom teaching, the smart mobile blackboard system continuously collects multimodal data, preprocesses it, and unifies its spatial coordinates to correlate blackboard image data and writing trajectory data under a unified time axis and blackboard coordinate system. Subsequently, based on the preprocessed multimodal data, the system constructs an improved PSENet network to perform blackboard text instance region detection on the blackboard images, outputting a set of blackboard text instance regions that change over time. This process can stably identify blackboard text regions even under continuous writing and partial occlusion, avoiding recognition interruptions caused by fluctuations in the quality of single-frame images.

[0119] After generating the text instance region on the whiteboard, the system constructs a spatiotemporal structured evidence alignment grid, aligning the text instance region with the writing trajectory fragments in both spatial and temporal dimensions to generate candidate semantic units. In this way, each candidate semantic unit corresponds to a clear spatial location, temporal range, and evidence of writing behavior, providing a reliable data foundation for whiteboard semantic modeling. Subsequently, the system constructs a dual-track reversible whiteboard semantic memory structure, writing the candidate semantic units into the explicit semantic track and maintaining the potential evolutionary relationship of whiteboard semantics in the implicit semantic track, ensuring that semantic evolution information is retained even when the whiteboard is erased or covered.

[0120] During the generation and updating of blackboard semantic nodes, the system combines preset teaching causal relationship rules to perform semantic consistency verification on newly generated or updated blackboard semantic nodes. When inconsistencies are detected in the writing order, spatial layout, or causal relationship of the blackboard semantics, the system triggers semantic backtracking processing, performs version switching or delayed confirmation of the relevant semantic nodes, and avoids unreasonable semantics from directly entering the final result. The system constructs a semantic conflict field within the blackboard space, mapping the semantic deviation between explicit and implicit semantic tracks, semantic consistency verification results, and version change information to a conflict intensity distribution, and schedules and controls the blackboard semantic processing process according to the conflict intensity, so that areas with higher conflict receive higher processing frequency, thereby improving the overall recognition stability while ensuring real-time performance.

[0121] Table 1. Performance Comparison Results of Different Blackboard Writing Recognition Methods in Real-World Classroom Scenarios

[0122]

[0123] As can be seen from the data in Table 1, in terms of character recognition accuracy, the traditional single-frame method has an average accuracy of 88.1%, while the present invention improves it to 93.2%, an increase of 5.1 percentage points. This indicates that under the conditions of writing occlusion, reflection, and the gradual formation of blackboard writing, the present invention, through multimodal preprocessing and improved PSENet text instance region detection, can output the blackboard writing text region more stably and reduce character errors caused by missed detections and false detections.

[0124] In terms of semantic continuity and consistency, the semantic node consistency pass rate of this invention increased from 82.7% to 90.4%, and the number of semantic breaks decreased from 6.0 times per lesson to 2.9 times. This reflects that this invention generates candidate semantic units based on spatiotemporal structured evidence alignment grids and combines a dual-track reversible blackboard semantic memory structure for semantic node evolution management, which can maintain a more coherent and consistent semantic chain when the blackboard is constantly updated.

[0125] In terms of write / erase adaptation and real-time performance, this invention achieves a semantic recovery success rate of 87.6% after overwrite / erase, which is 20.2 percentage points higher than the traditional method. The average processing latency is reduced from 118 milliseconds to 97 milliseconds, and the stability score is improved from 0.76 to 0.88. This indicates that semantic consistency verification and semantic conflict field scheduling not only improve error recovery capabilities but also reduce invalid calculations through differentiated processing, achieving more stable real-time output. Therefore, under the same number of courses, blackboard size, and class duration, the overall usability is improved from average to good.

[0126] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A real-time semantic recognition method for smart mobile blackboard handwriting based on multimodal fusion, characterized in that, include: Collect multimodal data generated during the teaching process of the smart mobile blackboard, preprocess the multimodal data and unify the spatial coordinates to obtain preprocessed multimodal data; Based on the preprocessed multimodal data, an improved PSENet network is constructed to perform whiteboard text instance region detection, generate text kernel regions and expand them step by step to obtain a set of whiteboard text instance regions; Using a set of whiteboard text instance regions and preprocessed multimodal data, a spatiotemporal structured evidence alignment grid is constructed to generate candidate semantic units for whiteboard writing. Write the candidate semantic units of the blackboard into the dual-track reversible blackboard semantic memory structure, perform semantic recording, semantic association and version evolution processing, generate blackboard semantic nodes and update the dual-track reversible blackboard semantic memory structure. Based on the preset teaching causal relationship rules, the semantic consistency of newly generated or updated blackboard semantic nodes is verified. When the verification fails, semantic backtracking is triggered to correct or delay the confirmation of the semantic type, spatial boundary or version status of the blackboard semantic nodes. Based on the semantic inconsistencies between explicit and implicit semantic tracks, the semantic consistency verification results, and the version status of the semantic nodes in the blackboard writing, a semantic conflict field is constructed. The blackboard writing semantic processing process is scheduled and controlled according to the conflict intensity distribution of the semantic conflict field, and a structured blackboard writing semantic recognition result is output.

2. The real-time semantic recognition method for intelligent mobile blackboard writing based on multimodal fusion according to claim 1, characterized in that, The multimodal data includes blackboard image data, writing trajectory data, and speech data corresponding to the blackboard image data and writing trajectory data on the time axis.

3. The real-time semantic recognition method for intelligent mobile blackboard writing based on multimodal fusion according to claim 1, characterized in that, The preprocessing of multimodal data and the unification of spatial coordinates include time alignment of the multimodal data, board area calibration of the blackboard image data, and mapping of the writing trajectory data to a board spatial coordinate system consistent with the blackboard image data to form a unified spatiotemporal coordinate representation.

4. The real-time semantic recognition method for intelligent mobile blackboard writing based on multimodal fusion according to claim 1, characterized in that, The generated text kernel region is expanded level by level to obtain a set of whiteboard text instance regions, including: Blackboard image data is extracted from the preprocessed multimodal data and used as the input image; The input image data is fed into the backbone feature extraction layer of the improved PSENet network to obtain the backbone feature map. The backbone feature extraction layer is connected to the first convolutional block, the second convolutional block, the third convolutional block and the fourth convolutional block in sequence. A plate surface geometric feature injection module is connected in series between the third convolutional block and the fourth convolutional block. The plate surface geometric calibration parameters in the preprocessing stage are received and converted into a geometric feature map of the same scale as the backbone feature map. The geometric feature map is injected into the backbone feature map through channel splicing and the geometrically enhanced backbone feature map is output. The geometrically enhanced backbone feature map is input into the multi-scale feature fusion layer. The multi-scale feature fusion layer fuses features of different levels in a top-down and lateral connection manner. At each lateral connection of the multi-scale feature fusion layer, a reflection and occlusion suppression gating module is connected in series. The low-level texture features and high-level semantic features before fusion are used as inputs to generate corresponding gating weight maps. The low-level texture features are then gated and suppressed before participating in the fusion, and the multi-scale fused features after reflection and occlusion suppression are output. Multi-scale fusion features are input into the text kernel prediction head and the extended guidance prediction head, which output the text kernel region map and the extended guidance map respectively. A trajectory consistency boundary calibration module is connected in parallel at the output end of the extended guidance map. The trajectory consistency boundary calibration module receives the writing trajectory data, generates a trajectory coverage heat map, performs boundary calibration on the extended guidance map, and outputs the calibrated extended guidance map. Based on the text kernel region map at multiple scale levels and the calibrated extended guide map, a step-by-step expansion generation process is performed to output a set of whiteboard text instance regions, recording the spatial location and time range corresponding to a unified time axis for each whiteboard text instance region.

5. The real-time semantic recognition method for intelligent mobile blackboard writing based on multimodal fusion according to claim 1, characterized in that, The generated candidate semantic units for whiteboard writing include: Obtain the set of whiteboard text instance regions and the writing trajectory data in the preprocessed multimodal data, and divide the writing trajectory data into multiple time-continuous trajectory segments according to a unified time axis; A spatiotemporal structured evidence alignment grid is constructed, which consists of a spatial grid layer, a time slice layer, and an evidence index layer. The spatial grid layer divides the board surface into multiple spatial grid units according to the board surface coordinate system, the time slice layer divides time into multiple time slice units according to a unified time axis, and the evidence index layer records the image instance evidence and trajectory fragment evidence corresponding to each spatial grid unit in each time slice unit. An adaptive raster thinning mechanism is introduced, which projects the set of blackboard text instance regions onto the board coordinate system, performs thinning and splitting processing on the spatial raster units that contain the boundaries of the blackboard text instance regions, and maintains the original raster scale for the spatial raster units that do not contain the boundaries of the blackboard text instance regions, thus forming a hierarchical spatial raster structure. Write the set of whiteboard text instance areas into the evidence index layer. According to the occurrence time of each whiteboard text instance area on the unified time axis, register it to the spatial grid unit under the corresponding time slice unit. Write the trajectory fragments into the evidence index layer. According to the occurrence time of each trajectory fragment on the unified time axis, register it to the spatial grid unit under the corresponding time slice unit. Alignment determination is performed on image instance evidence and trajectory fragment evidence registered in the same spatial grid cell within the same time slice cell. Image instance evidence and trajectory fragment evidence that satisfy temporal overlap and spatial coverage are grouped into an alignment pair, and the alignment pairs are aggregated into whiteboard candidate semantic units.

6. The real-time semantic recognition method for intelligent mobile blackboard writing based on multimodal fusion according to claim 1, characterized in that, The process of generating blackboard semantic nodes and updating the dual-track reversible blackboard semantic memory structure includes: A dual-track reversible blackboard semantic memory structure is constructed, which includes explicit semantic tracks, implicit semantic tracks, a version chain index table, and an anchor point mapping table between tracks. Based on the spatial location and temporal range of the candidate semantic units on the whiteboard, a matching whiteboard semantic node is retrieved in the explicit semantic track. When no matching whiteboard semantic node is found, a new whiteboard semantic node is created in the explicit semantic track. When a matching whiteboard semantic node is found, an update is performed on the matching whiteboard semantic node. Evidence association information corresponding to the candidate semantic units on the whiteboard is recorded in the explicit semantic track. Establish structural associations for blackboard semantic nodes in the explicit semantic track. The structural associations include the adjacency associations between adjacent blackboard semantic nodes in the same time period and the inclusion associations between blackboard semantic nodes within the same blackboard text instance area. Write the structural associations into the dual-track reversible blackboard semantic memory structure. Version evolution processing is performed on the whiteboard semantic nodes. When a writing overwrite event is detected, a new version record is generated for the whiteboard semantic node and written into the version chain index table. When an erase event is detected, an erase version record is generated for the whiteboard semantic node and written into the version chain index table. The current valid version mark is written into the explicit semantic track. Based on the changes in the temporal order and structural association of the currently effective version of the blackboard semantic nodes in the explicit semantic track, latent semantic nodes corresponding to the blackboard semantic nodes are generated in the implicit semantic track. The correspondence between the latent semantic nodes and the blackboard semantic nodes is established through the inter-track anchor mapping table, and the dual-track reversible blackboard semantic memory structure is updated.

7. The real-time semantic recognition method for intelligent mobile blackboard writing based on multimodal fusion according to claim 1, characterized in that, The process of performing semantic consistency verification on newly generated or updated whiteboard semantic nodes, and triggering semantic backtracking when verification fails, includes: Read newly generated or updated blackboard semantic nodes from the explicit semantic track, and read the context blackboard semantic nodes that intersect with the blackboard semantic nodes in the time range and are adjacent to the blackboard space to form a set of nodes to be verified and a set of context nodes. Based on the preset teaching causal relationship rules, causal verification relationship records are generated for the set of nodes to be verified and the set of context nodes. The causal verification relationship records include pre-order dependency relationship records, successor association relationship records and structural hierarchy relationship records. Based on the causal verification relationship record, semantic consistency verification is performed on each whiteboard semantic node in the node set to be verified. The semantic consistency verification includes dependency consistency verification, temporal consistency verification and layout consistency verification, and the corresponding verification result marker is output. When the verification result flag indicates that the semantic consistency verification has failed, a backtracking trigger flag is written to the corresponding whiteboard semantic node, and the whiteboard semantic node is set to the backtracking state; Semantic backtracking processing is performed on the blackboard semantic nodes that are in the pending backtracking state. The semantic backtracking process includes reading the historical version of the blackboard semantic node from the explicit semantic track and switching the current valid version, updating the semantic type, spatial boundary and version status of the blackboard semantic node, and updating the dual-track reversible blackboard semantic memory structure after setting the blackboard semantic node to the delayed confirmation state.

8. The real-time semantic recognition method for intelligent mobile blackboard writing based on multimodal fusion according to claim 1, characterized in that, The output structured whiteboard semantic recognition results include: Read the explicit semantic track's blackboard semantic nodes, implicit semantic track's latent semantic nodes, semantic consistency verification result markers, and version status information of the blackboard semantic nodes from the dual-track reversible blackboard semantic memory structure. A spatial index for the semantic conflict field is constructed in the spatial coordinate system of the board surface. The board surface is divided into multiple conflict evaluation units, and a conflict record item is established for each conflict evaluation unit. The conflict record item includes semantic deviation record, verification failure record, and version change record. A version-driven conflict propagation mechanism is introduced within each conflict assessment unit. The conflict record items corresponding to the blackboard semantic nodes that have overwritten or erased version records are propagated to adjacent conflict assessment units. The conflict record items of adjacent conflict assessment units are cumulatively updated according to the number of propagations and the propagation sequence to form a semantic conflict field. A conflict intensity distribution is generated based on the conflict record items of each conflict assessment unit. The conflict intensity distribution is jointly determined by the cumulative value of semantic deviation records, the cumulative value of verification failure records, and the cumulative value of version change records. The conflict intensity distribution is mapped to the corresponding processing priority, processing frequency, and freeze flag. The whiteboard semantic processing is scheduled and controlled based on processing priority, processing frequency, and freezing markers. The current valid version state of the whiteboard semantic node is maintained for the conflict evaluation unit corresponding to the freezing marker, and the whiteboard semantic node is updated according to the processing priority order for the conflict evaluation unit corresponding to the non-freezing marker, and the structured whiteboard semantic recognition result is output.

Images