Blackboard content recognition method and device based on conditional generative adversarial network, equipment and storage medium

Abstract

The application discloses a blackboard content recognition method and device based on a conditional generative adversarial network, equipment and a storage medium. First, the conditional generative adversarial network is combined with a handwriting style condition vector to generate diversified blackboard images, and an enhanced data set is constructed to improve the generalization ability of the model to different writing styles. Secondly, a noise suppression module with an encoder-decoder structure is used, which embeds an improved self-attention mechanism based on channel dimension calculation, effectively suppresses complex environmental noise and enhances character features. Then, a lightweight Transformer combining local and global attention is used for accurate character recognition of the denoised image. Finally, the recognition sequence is time-series modeled by a long short-term memory network, and the semantic correction is performed by using a natural language processing technology, and the accurate and coherent text is output. The application significantly improves the accuracy, robustness and practicality of blackboard recognition in diversified writing, noise interference and other actual scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a method, apparatus, device, and storage medium for blackboard writing content recognition based on conditional generative adversarial networks. Background Technology

[0002] Blackboard handwriting recognition is an important research direction in the fields of computer vision and pattern recognition, and it has significant application value in the process of educational informatization. By converting the teacher's handwritten content on the blackboard into editable and storable digital text in real time, this technology can assist students in after-class review, support teachers in summarizing teaching and archiving content, and facilitate the digital management and sharing of educational resources.

[0003] Early blackboard handwriting recognition primarily relied on traditional image processing and pattern recognition methods, such as feature extraction (e.g., histogram of oriented gradients, contour features) combined with rule matching or classic classifiers (e.g., support vector machines, K-nearest neighbors). These methods are effective in relatively structured and simple background scenarios, but their limitations are also significant: they lack adaptability to the vast differences in handwriting styles among individual writers, such as stroke thickness, cursive writing habits, character spacing, and tilt angles; they are not robust to common complex background interference on blackboards, such as chalk dust residue, reflective spots, shadows, and stains; and feature design often depends on human experience, resulting in limited generalization ability and ultimately failing to meet the recognition accuracy requirements of practical applications.

[0004] With the rapid development of deep learning technology, especially the introduction of architectures such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Transformers, the performance of blackboard handwriting recognition has been significantly improved. CNN-based methods can automatically learn hierarchical features in images and perform excellently in character detection and classification tasks; RNNs and Long Short-Term Memory (LSTM) networks are used to model temporal dependencies between character sequences, improving the recognition coherence of continuous text; while Transformer models, with their powerful global context modeling capabilities, have shown potential in capturing long-range dependencies and layout information between characters.

[0005] However, existing deep learning methods still face many technical challenges when applied to blackboard recognition in real classroom environments:

[0006] 1. Insufficient adaptability to diverse handwriting styles: Different teachers have significant differences in their writing habits, font shapes, and stroke styles. Even the handwriting of the same teacher may vary in different contexts. Existing models are usually trained on datasets with limited styles, resulting in weak generalization ability to unseen handwriting styles and a significant drop in recognition accuracy when faced with diverse handwriting styles.

[0007] 2. Robustness to complex environmental interference needs improvement: Real-world blackboard images often contain various noises and interferences, such as chalk dust, highlights or shadows caused by uneven lighting, uneven background color due to aging of the blackboard surface, and blurring and distortion introduced by the camera. These interferences can contaminate the character stroke areas and reduce image quality. Existing models often lack targeted and efficient noise reduction methods in the preprocessing stage, directly affecting the reliability of subsequent feature extraction and recognition.

[0008] 3. Poor semantic coherence and accuracy in text recognition: Existing recognition systems mostly focus on recognizing single characters or local sequences, lacking sufficient ability to understand the contextual semantics of entire sentences or paragraphs. This easily leads to typos, disordered word order, or grammatical errors in the output text, thus affecting the readability and usability of the final recognition results.

[0009] In summary, while current blackboard handwriting recognition technology has made progress driven by deep learning, it still has shortcomings in handling diverse writing styles, resisting complex environmental noise, and ensuring the semantic accuracy of the recognized text. Therefore, there is an urgent need for a comprehensive solution that can effectively integrate multi-style data generation, robust noise suppression, and deep semantic understanding to improve the practicality and reliability of blackboard handwriting recognition systems in real-world educational scenarios. Summary of the Invention

[0010] To address the problems of poor adaptability to diverse writing styles, weak resistance to environmental interference, and insufficient semantic coherence of recognized text in existing blackboard handwriting recognition methods, this invention proposes a blackboard writing content recognition method, device, equipment, and storage medium based on conditional generative adversarial networks. This method can meet the actual needs of blackboard handwriting recognition applications and achieve efficient and accurate recognition of blackboard writing content with diverse writing styles of teachers.

[0011] In the blackboard writing content recognition method proposed in this invention, a generative adversarial model is trained and training data is expanded to improve the model's generalization ability, enabling the recognition system to adapt to different teachers' writing styles and habits. This solves the problem of low accuracy in recognizing multiple writing styles in current handwriting recognition algorithms, significantly improving recognition accuracy. Furthermore, by designing a lightweight noise suppression module in the model, complex background interference can be effectively filtered out, enhancing the recognizability of target characters and addressing the impact of noise and interference factors (such as chalk dust, lighting changes, etc.) on the recognition results, thereby improving the robustness of the recognition method in complex backgrounds. In addition, a lightweight Transformer module with a hybrid attention mechanism is designed in the model to accurately recognize characters. Combined with LSTM and NLP technologies, deep semantic understanding of the context of the recognized text is performed to solve the problems of incoherent context and typos in the recognition results, effectively reducing typos and grammatical errors and improving the accuracy and readability of the final text.

[0012] The technical solution of this invention is as follows:

[0013] This invention first proposes a method for recognizing blackboard writing content based on conditional generative adversarial networks, including the following steps:

[0014] Step S1: Constructing diverse blackboard data.

[0015] Obtain the original whiteboard image dataset. Construct a Conditional Generative Adversarial Network (CGAN), whose generator takes the original whiteboard images and handwriting style conditional vectors as input. These handwriting style conditional vectors are categorically encoded based on the style dataset. Through adversarial training between the generator and discriminator, synthetic whiteboard images with diverse handwriting styles are generated. The generated synthetic images and the original images are then combined to construct an augmented dataset. This step, by actively generating multi-style samples, significantly expands the style diversity of the model's training data, effectively alleviating the model's overfitting to a limited number of handwriting styles and fundamentally improving the recognition system's generalization ability to unknown handwriting styles.

[0016] Step S2: Noise suppression and image enhancement.

[0017] Images from the augmented dataset are input into a noise suppression module for preprocessing. This module employs a three-level encoder-decoder structure, its innovation lying in the integration of an improved self-attention mechanism. This mechanism computes attention in the channel dimension rather than the spatial dimension: first, the input features are layer-normalized; then, cross-channel information is aggregated sequentially through 1×1 convolutions, and local spatial context is captured through 3×3 convolutions, generating query, key, and value projections rich in multi-scale context. Multi-head attention computation is then performed in the channel dimension. Finally, the attention output is added to the original input features through residual connections. This design efficiently suppresses noise such as chalk dust and uneven lighting with linear complexity, while enhancing character stroke features, significantly improving the model's input quality and robustness in complex real-world environments.

[0018] Step S3: Character recognition based on lightweight Transformer.

[0019] The denoised and enhanced image is input into the handwriting recognition model for training or recognition. The core of this model is a lightweight Transformer encoder, which first segments the image into blocks and linearly embeds them into a sequence. The key to the encoder lies in its multi-head attention mechanism, which employs a hybrid design combining local and global attention: one attention head is set to local attention, limiting the receptive field to adjacent regions through masking, focusing on capturing detailed features such as strokes and edges; the other part is set to global attention, used to understand the overall layout relationships between characters and words. This hybrid design combining local and global attention allows the model to simultaneously extract discriminative features from both local details and global structure, achieving accurate recognition of diverse handwritten characters and outputting an initial character sequence.

[0020] Step S4: Temporal modeling and semantic post-processing.

[0021] The initial character sequence output by the recognition model is fed into a Long Short-Term Memory (LSTM) network layer. The LSTM, through its gating mechanism, models long-range temporal dependencies in the character sequence, correcting potential order errors caused by the unordered output of the recognition model and ensuring the coherence of the text sequence. Subsequently, the LSTM output sequence is fed into the Natural Language Processing (NLP) module. This module performs deep semantic analysis using techniques such as lexicalization and context encoding, including utilizing context vectors from pre-trained language models. Based on the context, it detects and corrects potential typos and grammatical errors, ultimately generating high-quality recognized text that conforms to natural language norms.

[0022] Based on the above method, the present invention also proposes a blackboard writing content recognition device based on conditional generative adversarial networks, comprising:

[0023] Data augmentation module: Used to implement step S1, including conditional vector generation unit, CGAN generator and discriminator unit, responsible for building augmented datasets with diverse styles.

[0024] Noise suppression module: used to implement step S2, its structure is as follows Figure 1 As shown, it includes a multi-level encoder-decoder unit and a processing unit that integrates an improved self-attention mechanism, which is responsible for image denoising and feature enhancement.

[0025] The whiteboard recognition module is used to implement step S3. Its core is a lightweight Transformer encoder, which is responsible for recognizing character content from images.

[0026] Post-processing module: Used to implement step S4, including LSTM timing modeling unit and NLP semantic correction unit, responsible for optimizing and correcting character sequences.

[0027] The present invention also proposes an electronic device and a computer-readable storage medium:

[0028] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0029] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described method.

[0030] Beneficial effects

[0031] The present invention has the following beneficial effects:

[0032] 1. Improved generalization ability: This invention generates handwritten blackboard images of various styles through conditional generative adversarial networks, constructs a diverse dataset of blackboard images, and enables the model to better adapt to the diverse handwriting of different teachers, significantly improving the robustness of the recognition system to unknown writing styles.

[0033] 2. Enhanced environmental robustness: This invention designs a lightweight noise suppression module that combines channel attention and local convolution to remove interference from the complex background of the blackboard, ensuring the clarity of the character area and improving recognition stability in a real classroom environment.

[0034] 3. Ensure text quality: This invention uses a hybrid attention Transformer to accurately identify characters, and then performs temporal and semantic corrections through LSTM and NLP modules, ensuring that the final output text is of high quality in terms of character accuracy, sequential coherence, and language standardization.

[0035] 4. Balancing performance and efficiency: The overall solution adopts a lightweight design, such as channel attention and lightweight Transformer, which controls the computational overhead while maintaining high recognition accuracy, meeting the requirements of real-time performance and easy deployment in educational applications.

[0036] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0038] Figure 1 This is a schematic diagram of the noise suppression module;

[0039] Figure 2 A schematic diagram of the improved self-attention module;

[0040] Figure 3 A schematic diagram of a conditional generative adversarial network module;

[0041] Figure 4 This is a schematic diagram of the residual module;

[0042] Figure 5 A schematic diagram of a multi-head lightweight Transformer module;

[0043] Figure 6 This is a schematic diagram of the training process;

[0044] Figure 7 This is a flowchart illustrating the identification process. Detailed Implementation

[0045] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0046] This embodiment proposes a blackboard writing content recognition method based on conditional generative adversarial networks. Its core implementation process mainly includes two stages: offline training and online recognition. The overall framework can be referenced from [reference needed]. Figure 5 (Training process) and Figure 6 (Recognition Process). The training phase aims to build a robust model that can adapt to diverse handwriting styles, resist environmental noise, and understand semantics; the recognition phase utilizes the trained model to efficiently and accurately convert real-time captured whiteboard images into text. The following section elaborates on the specific network structure and parameter settings.

[0047] Step S1: Obtain the original whiteboard image dataset. Using a Conditional Generative Adversarial Network (CGAN), take the original whiteboard images and handwriting style conditional vectors as input to generate synthetic whiteboard images with diverse handwriting styles. Combine the synthetic images with the original images to construct an augmented dataset. Specifically, this includes:

[0048] 1. Data preparation and preprocessing

[0049] First, video processing tools (such as FFmpeg) are used to extract frames or keyframes from recorded classroom blackboard teaching videos at equal intervals to obtain the original set of blackboard images. The images are then standardized, such as resizing and converting to grayscale, and extremely blurry or irrelevant frames are preliminarily cleaned up to form the original training dataset. Simultaneously, to construct conditional vectors, category labels from publicly available handwriting style datasets (such as MNIST) are introduced and converted to one-hot encoded form as discrete conditional labels for handwriting styles.

[0050] 2. Diverse blackboard data generation

[0051] Data augmentation is achieved through Conditional Generative Adversarial Networks (CGANs), and its structural diagram is shown below. Figure 3 As shown.

[0052] Network Structure: CGAN consists of a generator (G) and a discriminator (D). The generator receives two inputs: the original whiteboard image and a one-hot conditional vector controlling the handwriting style of the text. This conditional vector is mapped to style features through an embedding layer and fused with image features at multiple levels. For example, it is linearly projected to the gain and bias of each convolutional layer to reduce computational and memory costs. Fully connected layers map these inputs to a feature space suitable for subsequent convolutional layer processing. The generator employs an encoder-decoder structure with residual blocks. Two residual blocks reduce the number of input channels, refine and process the features, and merge the input features with high-level image features to prepare for final generation. Another residual block reduces the output channels to two, used to generate the final image feature representation, thus progressively synthesizing a whiteboard image that matches the style specified by the conditional vector. The discriminator also receives a combination of the image and the conditional vector. Through convolutional and pooling layers, it determines whether the input image is a real sample or a generated sample, and also determines whether its style matches the condition.

[0053] For residual block structures, such as Figure 4As shown, its internal data processing is as follows: First, the input feature map undergoes channel dimension adjustment and preliminary feature fusion through a fully connected layer; then, batch normalization layers are used to standardize the data for stable training, and activation functions (such as ReLU or LeakyReLU) are used to introduce non-linearity; next, a 1×1 convolution operation is used to change the number of channels while maintaining the spatial size, followed by bilinear interpolation for upsampling, and then a 3×3 convolution kernel is used to extract more complex local features; finally, a skip connection is used to add the original input feature map element-wise to the output feature map after the above multi-layer transformations, forming the final output of the residual block. This structural design ensures the effective flow of gradients in deep networks, alleviates the gradient vanishing problem, and significantly improves the network's feature learning ability and the overall training stability and expressive power of the model by retaining and reusing input information.

[0054] Adversarial training: The generator and discriminator are trained alternately. The generator aims to produce realistic images that can "fool" the discriminator, while the discriminator strives to accurately distinguish between real and fake images. After training, the generator is used to generate batches of synthetic whiteboard images covering various handwriting styles, which are then merged with the original images to construct a larger and more stylistically diverse augmented dataset.

[0055] Step S2: Input the images in the enhanced dataset into the noise suppression module for processing to obtain a denoised and enhanced image; the noise suppression module adopts an encoder-decoder structure and includes an improved self-attention mechanism based on channel dimension calculation.

[0056] The structure of the noise suppression module is as follows: Figure 1As shown, a three-level U-Net-type encoder-decoder structure is adopted. The encoder progressively compresses the spatial size (H×W) through convolution and downsampling, while increasing the number of channels (C) to extract deep features. Specifically, the input image is first embedded into low-dimensional features through a 3×3 convolutional layer, maintaining a size of H×W×C, where H×W represents the spatial dimension of the image and C represents the dimension of the feature layer. Each level of the encoder-decoder contains multiple improved self-attention modules. Compared to the traditional transformer, whose computational and storage complexity increases quadratically with the spatial resolution of the input image, to ensure the model's computational efficiency, the improved self-attention modules are applied to the channel dimension instead of the traditional spatial dimension, thus achieving linear complexity. Starting with a high-resolution input, the encoder progressively reduces the spatial size while increasing the capacity of the feature channels. The decoder takes low-resolution latent features as input, with a size of H / 4×W / 4×4C, and progressively restores the high-resolution display. Finally, for the refined feature map H×W×C, a convolutional layer is used to generate a residual image of size H×W×C, which is then added to the original image to preserve low-level features (preventing information loss) and obtain a noise-suppressed image. The decoder ultimately outputs a residual map of the same size as the input image, which is added to the original input image to obtain a clear image after noise suppression. This process effectively reduces interference from noise such as chalk dust, light spots, and surface stains.

[0057] The improved self-attention module is the core of the noise suppression module, such as... Figure 1 As shown, at the jump connection and bottleneck between the encoder and decoder, a method such as... is introduced. Figure 2 The improved self-attention module is shown below. The specific operation is as follows:

[0058] Input features are normalized layer by layer, which standardizes all channels of the input, reduces internal covariate bias, and makes deep networks easier to optimize. The input features are then passed through a 1×1 convolutional layer and a 3×3 depthwise separable convolutional layer. The 1×1 convolution handles cross-channel information interaction and recombination, enhancing feature representation. The 3×3 convolution captures local spatial context (such as relationships between adjacent pixels), encoding spatial structure information. The local receptive field of the convolution explicitly models pixel-level spatial dependencies, avoiding the blindness of random matrices and enhancing the physical meaning of feature encoding. The outputs of the two convolutions are fused to generate Query (Q), Key (K), and Value (V) projections. Unlike traditional random initialization weight matrix multiplication, this method integrates explicit channel and spatial semantics into Q, K, and V, containing not only the original features but also multi-scale context (channel interaction of the 1×1 convolution + spatial correlation of the 3×3 convolution), making it easier to extract local features such as edges and textures.

[0059] Furthermore, to enrich the model with global semantic information, a multi-head attention mechanism is introduced. Multi-head attention is calculated in the channel dimension rather than the spatial dimension (e.g., 8 heads), reducing computational complexity from the square of the spatial size to linear, significantly improving efficiency when processing high-resolution images. Each group of attention is calculated independently, with different heads focusing on different feature subspaces; for example, some heads focus on local details while others capture global structure. This multi-head mechanism is similar to a "multi-expert system," extracting complementary features from different subspaces and avoiding the limitations of a single attention pattern. Each query and key is multiplied and scaled to prevent gradient vanishing, and SoftMax normalization is applied to the attention weights, reflecting the importance of other positions to the current position. The multi-head results are concatenated and integrated through a linear layer, allowing the features at each position to incorporate multi-faceted contextual semantics, such as complementary local and global information, enhancing model robustness. The output of the multi-head attention is added to the original input of the module through residual connections, effectively preserving low-level features, preventing information loss, and mitigating the gradient vanishing problem. The residuals are then normalized again to stabilize the data distribution. The concatenated result serves as the input to the next layer, forming a hierarchical feature transfer mechanism. Through the above improvements, the self-attention module significantly enhances its perception of local details and cross-channel interactions while maintaining its global modeling capabilities.

[0060] Step S3: Input the denoised and enhanced image into a whiteboard recognition model using a lightweight Transformer architecture for training and recognition, generating an initial character sequence; the lightweight Transformer architecture includes a multi-head attention mechanism that combines local attention heads and global attention heads.

[0061] This step uses the denoised image as input to train a lightweight whiteboard recognition model. This model employs an existing pre-trained lightweight Transformer encoder-decoder architecture to reduce the number of parameters. The specific process is as follows:

[0062] Image serialization: The input image is resized to 384×384, and then the 384×384 image is divided into 16×16 non-overlapping image patches, for a total of 24×24=576 patches. Each 16×16×3 image patch is mapped to a D-dimensional vector through a learnable fully connected layer, thereby converting the two-dimensional image into a one-dimensional sequence.

[0063] Hybrid Attention Encoder: In the lightweight Transformer encoder for sequence input, local and global attention heads are introduced. Q, K, and V are split into 6 groups, and attention is calculated independently, allowing different heads to focus on different feature subspaces. For example, among the 6 heads, 2 are set as local attention heads, whose receptive fields are limited to a nearby 8×8 area through spatial masking, focusing on capturing the subtle structure of strokes; the other 4 are set as global attention heads, responsible for establishing long-range dependencies between characters and even words, understanding the overall layout. The outputs of different heads are concatenated and fused through a linear layer, achieving complementarity between local details and global semantics, avoiding the limitations of a single attention mode, and obtaining richer feature information.

[0064] Prediction and Training: The feature sequence output by the hybrid attention encoder is mapped to the character category space through a classification head (fully connected layer) to generate the initial character recognition sequence. The model is trained end-to-end using cross-entropy loss.

[0065] Step S4: The initial character sequence is sequentially passed through a Long Short-Term Memory (LSTM) network and a Natural Language Processing (NLP) module to perform temporal relationship modeling and semantic correction, and the final recognized text is output.

[0066] Because the Transformer model's ability to model the positional order of sequences is relatively unordered, one or more bidirectional LSTM layers are added in this step. LSTM, through its input gate, forget gate, and output gate mechanisms, explicitly models the temporal dependencies between characters in the sequence, corrects possible character order disorder, and ensures that the output text sequence is temporally coherent.

[0067] The sequence output by the LSTM is fed into a lightweight NLP module. This module first performs lexicalization, converting the character sequence into words or sub-word units. Then, a context-dependent vector representation of each word is obtained through a pre-trained or co-trained context encoder, such as a small bidirectional Transformer or RNN. Finally, based on this contextual semantic information, a language model layer, such as a Conditional Random Field (CRF) or directly using Softmax, decodes and corrects the text, identifying and replacing typos that do not conform to the context, and outputting the final fluent and accurate text.

[0068] The training process in this embodiment is as follows: Figure 5 As shown, the above steps S1 to S3 are executed sequentially. First, CGAN is trained with the original data and an augmented dataset is generated. Then, the noise suppression module and the lightweight Transformer recognition model are trained with the augmented dataset. Finally, the recognition model parameters are fixed, and the LSTM and NLP modules are jointly trained at the recognition output.

[0069] The above training used the Adam optimizer with the parameters set. and To ensure high stability of the optimizer in the early stages of training, especially during the competition between the generator and discriminator, the learning rate of the discriminator was set to 0.0004, while that of the generator was set to 0.0001. These settings help prevent the generator from updating too quickly, allowing it to gradually approximate the real data distribution. The training process lasted for 300 training epochs, with each epoch consisting of 32 batches. Mini-batch stochastic gradient descent was used to optimize the model weights during this training process. The learning rate was reduced by a factor of 0.5 on the 200th and 250th training epochs to accommodate minor adjustments to the model later in the training process.

[0070] The recognition process is as follows Figure 6 As shown, for the real-time whiteboard image to be recognized, it is first denoised by a trained noise suppression module; then the denoised image is input into a trained lightweight Transformer recognition model to obtain an initial character sequence; finally, the sequence is passed through LSTM and NLP modules in sequence to output the final corrected text.

[0071] Experimental verification:

[0072] To verify the effectiveness of the proposed method, 2000 real blackboard images extracted from classroom recordings of five different schools were used as the original training set, and the MNIST dataset was introduced as a style conditional source. An additional 500 independent images were collected as a test set, covering handwriting samples from three new teachers (whose styles were not present in the training set), and including varying degrees of uneven lighting and chalk dust interference. The experimental hardware platform was an NVIDIA RTX 3090 GPU. Two benchmark models were selected for comparison: the CRNN model based on a classic convolutional neural network and the standard Transformer recognition model.

[0073] Character-level recognition accuracy, complete sentence accuracy, and average inference time were used as the main evaluation metrics. The performance comparison of each method on the same test set is as follows:

[0074]

[0075] Experimental results show that the character-level accuracy of the complete process of this invention reaches 94.6%, and the whole-sentence accuracy reaches 72.8%, both significantly higher than the two comparative models. This verifies the comprehensive effectiveness of this invention in complex real-world scenarios through the combination of data augmentation, noise suppression, and hybrid attention mechanisms. Furthermore, ablation experiments are presented. When the conditional generative adversarial network (CGAN) data augmentation step is removed (i.e., training is done using only the original 2000 images), the whole-sentence accuracy drops to 55.4% when faced with new teacher handwriting. This demonstrates that CGAN-generated diverse style data plays a crucial role in improving the model's generalization ability and adapting to unknown handwriting styles. Removing the noise suppression module results in a significant decrease in recognition accuracy on test sets containing interference. This indicates that the proposed noise suppression module is essential for improving the system's robustness in real-world noisy environments. In addition, the average inference time of this invention is 45 milliseconds per image, falling between the fast CRNN and the computationally intensive standard Transformer. This shows that while introducing more powerful modeling capabilities, this method achieves a good balance between recognition accuracy and computational efficiency through lightweight design, meeting the real-time or near-real-time application requirements of classrooms.

[0076] The numerical examples demonstrate that the method proposed in this invention can effectively solve key technical problems such as adapting to diverse writing styles and environmental noise interference. While ensuring high processing efficiency, it achieves better accuracy in blackboard content recognition and text coherence than traditional methods.

[0077] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for recognizing blackboard writing content based on conditional generative adversarial networks, characterized in that, Includes the following steps: Step S1: Obtain the original whiteboard image dataset, use the conditional generative adversarial network CGAN, take the original whiteboard image and handwriting style conditional vector as input, generate synthetic whiteboard images with diverse handwriting styles, and construct an augmented dataset together with the original image; Step S2: Input the images in the enhanced dataset into the noise suppression module for processing to obtain a denoised and enhanced image; The noise suppression module adopts an encoder-decoder structure and includes an improved self-attention mechanism based on channel dimension calculation; Step S3: Input the denoised and enhanced image into a whiteboard recognition model using a lightweight Transformer architecture for training and recognition, generating an initial character sequence; the lightweight Transformer architecture includes a multi-head attention mechanism that combines local attention heads and global attention heads; Step S4: The initial character sequence is sequentially passed through a Long Short-Term Memory (LSTM) network and a Natural Language Processing (NLP) module to perform temporal relationship modeling and semantic correction, and the final recognized text is output.

2. The method according to claim 1, characterized in that, In step S1, the Conditional Generative Adversarial Network (CGAN) includes a generator and a discriminator. The generator receives the original whiteboard image and the handwriting style condition vector, and generates a handwriting style whiteboard image corresponding to the condition vector. The discriminator receives the combination of the real whiteboard image and the condition vector, as well as the combination of the generated image and the condition vector, and performs authenticity judgment. Through adversarial training between the generator and the discriminator, the quality and diversity of the generated images are optimized.

3. The method according to claim 1, characterized in that, In step S2, the noise suppression module is a three-level encoder-decoder structure; the improved self-attention mechanism is implemented in the following way: after layer normalization of the input features, 1×1 convolution and 3×3 convolution operations are performed in sequence to fuse cross-channel information and local spatial context to generate Query, Key and Value projections; multi-head attention is calculated in the channel dimension, and the attention output is added to the module input through residual connection.

4. The method according to claim 1, characterized in that, In step S3, the local attention head restricts the receptive field to a preset adjacent spatial region through a masking mechanism to capture stroke details; the global attention head is used to model the overall layout relationship between characters; the outputs of the local attention head and the global attention head are merged after concatenation and linear transformation.

5. The method according to claim 1, characterized in that, In step S4, the Long Short-Term Memory (LSTM) network is used to model the temporal dependencies of character sequences; the Natural Language Processing (NLP) module is used to perform lexicalization, context encoding, and semantic understanding-based typo correction on the sequences output by the LSTM.

6. A blackboard writing content recognition device based on conditional generative adversarial networks, characterized in that, include: The data enhancement module is used to implement step S1 as described in claim 1, thereby enabling the construction of diverse whiteboard data; A noise suppression module is used to implement step S2 as described in claim 1, which involves denoising and enhancing the image. The whiteboard recognition module is used to implement step S3 of claim 1, which involves recognizing characters in the image. The post-processing module is used to implement step S4 of claim 1, which performs temporal optimization and semantic correction on the character sequence.

7. The apparatus according to claim 6, characterized in that, The noise suppression module includes a three-level encoder-decoder and an improved self-attention unit embedded therein; the improved self-attention unit includes a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence to generate Query, Key and Value, and includes a multi-head attention calculation unit for calculating attention in the channel dimension.

8. The apparatus according to claim 6, characterized in that, The whiteboard recognition module includes a lightweight Transformer encoder whose multi-head attention mechanism is configured to include at least one local attention head that limits the receptive field through a spatial mask, and at least one global attention head for capturing the global context.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images