Handwritten sample generation method based on decoupling control and mask guidance

Abstract

The invention relates to the technical field of automatic generation of image samples, and discloses a handwriting sample generation method based on decoupling control and mask guidance, which decouples character types and personal morphological styles of handwritings through a structural mask generation model based on an SDF (Software Development Function) network, obtains controllable types, and improves the generation efficiency of the handwritings. According to the method, the handwritten character structure mask with the adjustable shape is obtained, the obtained structure mask is input into the diffusion model based on structure mask guidance, the handwritten image samples are output, a worker can directionally generate a large number of handwritten image samples with reasonable structures and consistent writing styles by using existing rare handwritten images, the data collection cost is reduced, and the data collection efficiency is improved. And meanwhile, rare and difficult example styles can be accurately enhanced, so that the accuracy and generalization of a handwritten form recognition model in smart government affairs are improved.

Description

Technical Field

[0001] This invention relates to the field of automatic image sample generation technology, specifically to a handwritten sample generation method based on decoupling control and mask guidance. Background Technology

[0002] Driven by the increasing maturity of artificial intelligence and deep learning technologies, society is rapidly entering the "intelligent society" stage. Traditional government services often face problems such as long queues, cumbersome procedures, and excessive documentation, thus giving rise to smart government services. Smart government services involve the pre-screening of images of forms, documents, and materials, which requires the rapid recognition of handwriting from specific groups using deep learning models. However, obtaining handwriting samples from specific groups is difficult, significantly impacting the accuracy of handwriting recognition models.

[0003] Meanwhile, existing handwritten sample generation models cannot accurately model the character type features and individual shape features of handwritten characters. The generated handwritten samples may have problems such as distorted character structure and mismatched individual shapes, which in turn affects the training effect of handwritten recognition models.

[0004] (1) There are essential structural differences between different types of characters in handwriting (such as characters, numbers and symbols), which may result in the generation of unrecognizable and structurally malformed handwriting samples;

[0005] (2) The handwriting of different individuals is complex. Even for the same type of character (such as the number "2"), the shape of the character will show significant individual differences due to the individual's writing habits (such as slant, thickness, cursive, decoration, etc.).

[0006] In summary, existing technologies, due to the scarcity of training data and the insufficient modeling capabilities of generative models, are unable to meet the demand for high-precision and highly generalizable models in handwriting recognition in smart government. Summary of the Invention

[0007] The purpose of this invention is to provide an intelligent weld point flaw detection structure and an automated welding device, which solves the problems of existing weld point flaw detection devices being unable to perform continuous flaw detection, having low flaw detection efficiency, and being unable to automatically flip the steel pipe, thus making it impossible to comprehensively detect flaws in the weld points of the steel pipe.

[0008] To address the aforementioned problems, the present invention aims to provide a handwritten sample generation method based on decoupling control and mask guidance. By using a structure mask generation model based on an SDF network, the character type and individual style of the handwritten text are decoupled, obtaining a controllable and adjustable handwritten character structure mask. This structure mask is then input into a diffusion model guided by the structure mask, outputting handwritten image samples. This method can directionally generate a large number of handwritten image samples with reasonable structure and consistent writing style, thereby improving the accuracy and generalization of handwritten recognition models in smart government applications. The technical solution is as follows:

[0009] A method for generating handwritten samples based on decoupling control and mask guidance includes the following steps:

[0010] Step S1: Collect handwritten images containing different character types and handwriting styles, and perform data annotation to construct a handwritten image set;

[0011] Step S2: Construct and train a structure mask generation model based on an SDF network to generate structure masks with controllable character types and adjustable morphology;

[0012] Step S3: Construct and train a handwritten sample generation model based on structure masking to generate handwritten sample images with reasonable structure and consistent shape;

[0013] Step S4: Select the target character type and convert it to one-hot encoding; input the one-hot encoding of the target character type and the handwritten image to be generated into the structure mask generation model, and output the structure mask to be generated; input the structure mask to be generated into the handwritten sample generation model, and output the handwritten image sample.

[0014] A further technical solution is that the data annotation of handwritten images in step S1 includes manually annotating the individuals and character types to which all handwritten images belong.

[0015] A further technical solution is that the structure mask generation model in step S2 is based on the DeepSDF model architecture, including:

[0016] A type representation module based on an attention mechanism, a reversible deformation module, and an SDF decoder;

[0017] The type representation module is composed of Transformer Blocks, which include a multi-head self-attention network, a feedforward neural network, and layer normalization.

[0018] The reversible deformation module includes an input layer, an MLP layer, and an output layer. Both the input and output layers consist of a linear layer and a ReLU activation function layer.

[0019] A further technical solution involves vectorizing the acquired handwritten image in the type representation module to obtain the vector contours of the handwritten characters in the image; constructing a signed distance field (SDF) for the handwritten image; and calculating the true signed distance value s for the vector contour of each character, which is then used as the true SDF value for supervised learning. Obtain the zero-level set of SDF values ​​for handwritten images. The zero level set of the SDF value s is the outline of the character, as shown in the following formula:

[0020] .

[0021] A further technical solution involves converting the category of handwritten characters into a one-hot vector and inputting it into a type representation module based on an attention mechanism. After layer normalization, the one-hot vector is input into a multi-head self-attention network. The output of the multi-head self-attention network is added to the input and then normalized before being input into a feedforward neural network. The output of the feedforward neural network is then added to the output of the multi-head self-attention network to obtain the implicit variable of the handwritten character type. The formula is as follows;

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] Zero-level set of SDF values ​​in handwritten images implicit variables for learnable shapes Perform linear interpolation to generate implicit shape features. .

[0027] A further technical solution is to use the zero-level set of the SDF values ​​of the handwritten image. and shape implicit features Input reversible deformation module, for zero level set After performing Fourier position encoding, it is combined with implicit shape features The data is concatenated, and the concatenated data is then passed sequentially through the input layer, MLP layer, and output layer to obtain... and Adding them together gives the deformation update position. ;

[0028] Update the position by deformation. Implicit variables of character type Input to the SDF decoder to obtain the handwritten SDF prediction value. And the structure mask for generating handwritten characters is determined by a threshold;

[0029] The structure mask generation model uses data loss and regularization loss as the overall loss function, defining the collected handwritten image set as follows: Where N is the total number of samples in the handwritten image set. The sample index for the handwritten image set is given by the following formula: ; ; ,in, For the first Data loss per handwritten image sample For the first SDF prediction values ​​for a sample of handwritten images. For the first SDF ground truth value of a sample of handwritten image. For variance scalar, For the first The shape of each handwritten image sample is an implicit variable. Indicates the first The shape of each handwritten image sample is an implicit variable subject to L2 regularization constraints. This represents the total loss.

[0030] A further technical solution is that, in step S3, the handwritten sample generation model is based on a diffusion model, and U-Net is used as the denoising network. The standard ResNet50 network is used as the noise-adding network; during the denoising process, the acquired handwritten image is compared with the corresponding structural mask. Concatenate along the channel dimension and from the conditional distribution. Sampling is performed in the structure mask. The semantic information of C handwritten character types is gradually restored through iterative denoising operations to generate handwritten image samples that conform to semantic constraints.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] This invention uses a structure mask generation model based on an SDF network to decouple the character type and individual style of handwritten characters, obtaining a controllable and adjustable structure mask for handwritten characters. The obtained structure mask is then input into a diffusion model guided by the structure mask, outputting handwritten image samples. Staff can use existing rare handwritten images to generate a large number of structurally sound and style-consistent handwritten image samples, reducing data collection costs and improving work efficiency. At the same time, it can accurately enhance rare and difficult styles, thereby improving the accuracy and generalization of handwritten recognition models in smart government. Attached Figure Description

[0033] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration:

[0034] Figure 1 This is a flowchart illustrating the method.

[0035] Figure 2 This is a diagram illustrating the overall framework of this method.

[0036] Figure 3 This is a structural framework diagram of the Transformer Block.

[0037] Figure 4 This is a structural framework diagram of a reversible deformation module. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] like Figures 1 to 4 As shown, to alleviate the performance limitations of handwritten digit training samples caused by scarcity, this invention employs a sample generation and enhancement framework based on decoupled control and mask guidance. This architecture aims to generate handwritten digit samples with reasonable character structure and controllable form through a generative model, thereby expanding the scale of the training dataset and improving the performance of the handwritten digit recognition model. The generative model of this invention adopts a dual-model collaborative architecture, including two independent but closely cooperating models (a handwritten digit sample generation model and a structure mask generation model). The structure mask generation model is responsible for generating handwritten digit character structure masks that conform to specific individual writing habits (controllable form). Guided by these handwritten digit character structure masks, the other handwritten digit sample generation model generates high-quality handwritten digit image samples, achieving a balance between the diversity of individual handwritten digit styles and structural consistency. Specifically, this includes the following steps:

[0040] Step S1: Collect handwritten images containing different character types and handwriting styles, and perform data annotation to construct a handwritten image set;

[0041] Specifically, 110 handwritten images from 10 individuals were collected. The handwritten images included Chinese characters, letters, symbols, and numbers. The individuals and character types of all handwritten images were manually labeled to construct a handwritten image set.

[0042] Step S2: Construct and train a structure mask generation model based on SDF network to generate handwritten structure masks with controllable character type and adjustable shape;

[0043] The present invention proposes a structure mask generation model based on decoupling control. The structure mask generation model uses the symbolic distance field (SDF) to construct a continuous implicit representation of the handwritten structure, and generates realistic and controllable handwritten image samples by effectively decoupling the character type and personal morphological features of the handwritten characters.

[0044] The structural mask generation model is improved and optimized based on the DeepSDF model architecture, and proposes a type representation module, a reversible deformation module, and an SDF decoder based on the attention mechanism.

[0045] The acquired handwritten images are vectorized to obtain the vector contours of the handwritten characters in the images. A signed distance field (SDF) is constructed for the handwritten images, and the true signed distance value s for each character's vector contour is calculated and used as the SDF ground truth value for supervised learning. That is, the signed distance field SDF will sample the image points. The distance from the image sampling point p to the character vector contour is mapped to a value s, where positive and negative signs indicate that the image sampling point p is outside and inside the character, respectively. This yields the zero-level set of the SDF value s of the handwritten image. (The set of image sampling points p when s=0), the zero level set of the SDF value s is the outline of the character, as shown in the following formula:

[0046] .

[0047] The type representation module based on the attention mechanism is composed of Transformer Blocks (the basic building blocks of the Transformer model), such as... Figure 3 The diagram shows the structural framework of the Transformer Block, which includes a Self Multi-Attention (FMI) network, a Feedforward Neural Network (FFN), and a Layer Normalization (LayerNorm).

[0048] The handwritten character categories (Chinese characters, letters, symbols, and numbers) are converted into one-hot vectors (vectors containing true and false values). These one-hot encoded vectors are then input into an attention-based type representation module. After layer normalization, the one-hot encoded vectors are input into a multi-head self-attention network (MNB). The output and input of the MNB are summed and normalized before being input into a feedforward neural network. Finally, the output of the feedforward neural network is summed with the output of the MNB to obtain the implicit variable representing the handwritten character type. The formula is as follows:

[0049] ;

[0050] ;

[0051] ;

[0052] ,in, Let be the one-hot encoded vector for each handwritten character category. Linear1, Linear2, and Linear3 are linear layers. Q(x) is the query, K(x) is the key, V(x) is the value, and Softmax is the softmax activation function. Let x be the dimension of the key vector, MHSA(x) be the output of the multi-head self-attention mechanism, and LayerNorm represent layer normalization. For Transformer Block, FFN represents a feedforward neural network.

[0053] The attention-based type representation module encodes the type information of handwritten characters into an implicit variable of character type. , It captures the standard, ideal structural information of the character. For example, for the letter "A", it learns the basic topological structure that must have two diagonal lines and one horizontal line, ensuring that the generated character is structurally correct.

[0054] like Figure 4 As shown, the reversible deformation module includes an input layer, an MLP layer, and an output layer. Both the input and output layers consist of a linear layer and a ReLU activation function layer.

[0055] The reversible deformation module is based on learnable implicit shape variables. , It is a three-dimensional implicit variable (a latent variable directly associated with the training dataset samples). In this embodiment, (Number of channels is 32). At the zero-level set of SDF values ​​s in the handwritten image. The learnable shape implicit variable at the set of image sampling points p when s=0 Perform linear interpolation to generate implicit shape features. (i.e., the stylistic features of the handwritten sample).

[0056] Zero-level set of SDF values ​​of handwritten images and shape implicit features Input reversible deformation module, for zero level set After performing Fourier position encoding, it is combined with implicit shape features The data is concatenated, and the concatenated data is then passed sequentially through the input layer, MLP layer, and output layer to obtain... and Adding them together gives the deformation update position. , indicating the position update after processing by the reversible deformation module (the deformed outline of the handwritten character).

[0057] Update the position by deformation. Implicit variables of character type Input to the SDF decoder to obtain the handwritten SDF prediction value. And judged by threshold ( Generate the final binary mask, that is, generate a handwritten structure mask with controllable character type and adjustable shape.

[0058] The structure mask generation model uses data loss and regularization loss as the overall loss function, defining the collected handwritten image set as follows: Where N is the total number of samples in the handwritten image set. The sample index for the handwritten image set is N=220 in this embodiment, and the formula is as follows:

[0059] ; ; ,in, For the first The data loss for each handwritten image sample (i.e., the supervision loss for reconstructing the handwritten character structure). For the first SDF prediction values ​​for a sample of handwritten images. For the first SDF ground truth value of a sample of handwritten image. For variance scalar, For the first The shape of each handwritten image sample is an implicit variable. Indicates the first The shape of each handwritten image sample is an implicit variable subject to L2 regularization constraints. This represents the total loss.

[0060] Step S3: Construct and train a handwritten sample generation model based on structure mask to generate handwritten image samples with reasonable structure and consistent shape.

[0061] Existing diffusion models generate images by simulating the diffusion process. Their core idea is to achieve image generation through two stages: forward noise addition and reverse denoising. Specifically, given an original image... The diffusion process involves continuously adding Gaussian noise. Gradually destroy the image structure to form a data distribution Then, a noise denoising network was trained to predict noise. Gradually distribute the data Restore to the original image .

[0062] This invention constructs a handwritten digit sample generation model guided by a structure mask, aiming to introduce semantic structure masks to guide the denoising process. The handwritten digit sample generation model is based on a diffusion model and uses U-Net as the denoising network. The standard ResNet50 network is used as the noise-adding network (encoder part). During the denoising process, the acquired handwritten image is concatenated with the corresponding structural mask m in the channel dimension, and the conditional distribution is used as the basis for further processing. Sampling is performed in the structure mask. The semantic information contains C types of handwritten characters. In this embodiment, C=4 (Chinese characters, letters, numbers, and symbols). The information is gradually restored through iterative denoising operations to generate handwritten image samples that conform to semantic constraints.

[0063] Step S4: Select the target character type and convert it to one-hot encoding; input the one-hot encoding of the target character type and the handwritten image to be generated into the structure mask generation model, and output the structure mask to be generated; input the structure mask to be generated into the handwritten sample generation model, and output the handwritten image sample, thus realizing the directional generation of handwritten samples.

[0064] This invention uses a collection of handwritten images to train the model. The AdamW optimizer is used for training, with an initial learning rate of 0.001. The learning rate is dynamically adjusted using a cosine annealing strategy, and the batch size is set to 8. The entire training process lasts for 400 epochs, with the first 20 epochs using a linear warm-up mechanism to improve training stability.

[0065] GAN, unconditional diffusion model, and cGAN were selected as comparison models. Accuracy and long-tail class accuracy (specifically calculated for characters with the fewest samples in the original dataset, such as the numbers "1" and "7", the letters "O" and "Q") were used as evaluation metrics. The experimental results are shown in Table 1.

[0066] Table 1. Accuracy of different generative models:

[0067] Handwritten Image Sample Generation Method accuracy Long-tail category accuracy GAN 68% 57% Unconditional diffusion model 71% 60% cGAN 76% 62% Ours 83% 71%

[0068] As shown in Table 1, the method of the present invention achieves the best accuracy (83%) compared with the existing generative model, especially in the long-tail category (71%). This indicates that the present invention can generate a large number of handwritten image samples with reasonable structure and consistent writing style, and can accurately enhance rare and difficult styles, thereby improving the accuracy and generalization of the handwritten recognition model in smart government affairs.

[0069] Although the invention has been described herein with reference to several illustrative embodiments, it should be understood that many other modifications and implementations can be devised by those skilled in the art, which will fall within the scope and spirit of the principles disclosed herein. More specifically, various variations and modifications can be made to the components and / or layout of the subject matter combination within the scope of the disclosure, drawings, and claims. Besides variations and modifications to the components and / or layout, other uses will be apparent to those skilled in the art.

Claims

1. A method for generating handwritten samples based on decoupling control and mask guidance, characterized in that, Includes the following steps: Step S1: Collect handwritten images containing different character types and handwriting styles, and perform data annotation to construct a handwritten image set; Step S2: Construct and train a structure mask generation model based on an SDF network to generate structure masks with controllable character types and adjustable morphology; Step S3: Construct and train a handwritten sample generation model based on structure masking to generate handwritten sample images with reasonable structure and consistent shape; Step S4: Select the target character type and convert it to one-hot encoding; Input the one-hot encoding of the target character category and the handwritten image to be generated into the structure mask generation model, and output the structure mask to be generated; input the structure mask to be generated into the handwritten sample generation model, and output the handwritten image sample.

2. The handwritten sample generation method based on decoupling control and mask guidance as described in claim 1, characterized in that, The data annotation of handwritten images in step S1 includes manually annotating the individuals and character types of all handwritten images.

3. The handwritten sample generation method based on decoupling control and mask guidance as described in claim 1, characterized in that, The structure mask generation model in step S2 is based on the DeepSDF model architecture and includes: A type representation module based on an attention mechanism, a reversible deformation module, and an SDF decoder; The type representation module is composed of Transformer Blocks, which include a multi-head self-attention network, a feedforward neural network, and layer normalization. The reversible deformation module includes an input layer, an MLP layer, and an output layer. Both the input and output layers consist of a linear layer and a ReLU activation function layer.

4. The handwritten sample generation method based on decoupling control and mask guidance as described in claim 3, characterized in that, In the type representation module, the acquired handwritten images are vectorized to obtain the vector contours of the handwritten characters in the images; a signed distance field (SDF) of the handwritten images is constructed, and the true signed distance value s of each character's vector contour is calculated and used as the SDF ground truth value for supervised learning. Obtain the zero-level set of SDF values ​​for handwritten images. The zero level set of the SDF value s is the outline of the character, as shown in the following formula: 。 5. The handwritten sample generation method based on decoupling control and mask guidance as described in claim 4, characterized in that, The handwritten character category is converted into a one-hot vector and input into the attention-based type representation module. After layer normalization, the one-hot vector is input into a multi-head self-attention network. The output of the multi-head self-attention network is added to the input and normalized before being input into a feedforward neural network. The output of the feedforward neural network is then added to the output of the multi-head self-attention network to obtain the implicit variable of the handwritten character type. The formula is as follows; ； ； ； ； Zero-level set of SDF values ​​in handwritten images implicit variables of learnable shape Perform linear interpolation to generate implicit shape features. .

6. The handwritten sample generation method based on decoupling control and mask guidance as described in claim 5, characterized in that, Zero-level set of SDF values ​​of handwritten images and shape implicit features Input reversible deformation module, for zero level set After performing Fourier position encoding, it is combined with implicit shape features The data is concatenated, and the concatenated data is then passed sequentially through the input layer, MLP layer, and output layer to obtain... and Adding them together gives the deformation update position. ; Update the position by deformation. Implicit variables of character type Input to the SDF decoder to obtain the handwritten SDF prediction value. And the structure mask for generating handwritten characters is determined by a threshold; The structure mask generation model uses data loss and regularization loss as the overall loss function, defining the collected handwritten image set as follows: Where N is the total number of samples in the handwritten image set. The sample index for the handwritten image set is given by the following formula: ; ; ,in, For the first Data loss per handwritten image sample For the first SDF prediction values ​​for a sample of handwritten images. For the first SDF ground truth value of a sample of handwritten image. For variance scalar, For the first The shape of each handwritten image sample is an implicit variable. Indicates the first The shape of each handwritten image sample is an implicit variable subject to L2 regularization constraints. This represents the total loss.

7. The handwritten sample generation method based on decoupling control and mask guidance as described in claim 1, characterized in that, In step S3, the handwritten sample generation model is based on a diffusion model and uses U-Net as the denoising network. The standard ResNet50 network is used as the noise-adding network; during the denoising process, the acquired handwritten image is compared with the corresponding structural mask. Concatenate along the channel dimension and from the conditional distribution. Sampling is performed in the structure mask. The semantic information of C handwritten character types is gradually restored through iterative denoising operations to generate handwritten image samples that conform to semantic constraints.

Images