Document repairing method based on graph neural network

Abstract

The invention discloses a document restoration method based on a graph neural network, which comprises the following steps: acquiring document image data, and preprocessing the document image data; performing feature extraction on the preprocessed document image data to obtain image feature data, and constructing a document feature map according to the image feature data; a document repair model is constructed, the document repair model is a graph neural network model, a document feature graph is input into the document repair model, and the document repair model outputs a repair feature graph; and constructing a generative adversarial network model, inputting the repair feature map to the generative adversarial network model, and outputting a repaired document image by the generative adversarial network model. According to the method, the features of the document image are captured through the graph neural network, and the features of the document image are repaired according to the semantic information of the document image, so that the generative adversarial network is utilized to perform document image reconstruction according to the repaired features of the document image, and the problems that the repairing quality is low and various document types cannot be considered in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a document repair method based on graph neural network. Background Art

[0002] With the advent of the digital information age, the storage and dissemination of documents in image form has become increasingly common. However, due to factors such as the quality limitations of scanning equipment, changes in ambient lighting, paper aging, and human damage, captured document images often exhibit quality issues such as blur, distortion, noise, and shadows. These defects not only affect the readability of document content but also pose significant challenges to subsequent tasks such as optical character recognition (OCR) and information extraction.

[0003] However, existing document image repair methods mainly include:

[0004] 1. Methods based on traditional image processing: Traditional methods typically use filtering, edge detection, morphological processing, and other techniques to eliminate noise, enhance contrast, or correct distortion. These methods primarily operate on the pixel level, directly processing images and lack an understanding of high-level semantic information.

[0005] 2. Deep learning methods based on convolutional neural networks (CNNs): By stacking multiple convolutional and pooling layers, CNNs can extract hierarchical features from images and are widely used in tasks such as image classification, object detection, and image segmentation. Through large-scale data training, CNNs can automatically learn image feature representations and perform well for common image restoration tasks. However, CNNs primarily focus on local features and have difficulty capturing the overall structural information of a document.

[0006] In summary, existing document restoration methods cannot take into account image quality, semantic information and overall image structure, cannot generalize processing for different types of documents, and are difficult to meet existing document image restoration needs. Summary of the Invention

[0007] To solve the above problems, the present invention provides a document repair method based on graph neural network, which captures the features of document images through graph neural network and repairs the document image features according to the semantic information of the document image, and then uses generative adversarial network to reconstruct the document image according to the repaired document image features, solving the problem of low repair quality and inability to take into account various document types in the existing technology.

[0008] To achieve the above objectives, the present invention provides the following technical solutions:

[0009] A document repair method based on graph neural network includes the following steps:

[0010] S1. Obtain document image data and preprocess the document image data;

[0011] S2. Feature extraction is performed on the preprocessed document image data to obtain image feature data, and a document feature map is constructed based on the image feature data;

[0012] S3. Build a document repair model, which is a graph neural network model. Input a document feature map to the document repair model, and output a repair feature map.

[0013] S4. Build a generative adversarial network model, input the repair feature map into the generative adversarial network model, and the generative adversarial network model outputs the repaired document image.

[0014] Furthermore, the pre-processing of the document image data specifically includes:

[0015] The document image data is converted into a grayscale image, binarized, and filtered.

[0016] Furthermore, the document image data is binarized, and its specific implementation is as follows:

[0017] The Otsu threshold segmentation method is used to calculate the binarization threshold, and each pixel in the document image data is threshold segmented according to the binarization threshold.

[0018] Furthermore, in step S2, the feature extraction is performed on the pre-processed document image data, and the specific implementation method is as follows:

[0019] Construct a feature extraction model, which is a pre-trained convolutional neural network. Input the preprocessed document image data into the feature extraction model, and the feature extraction model extracts the feature map of the preprocessed document image data. According to the feature map, several feature vectors are generated, and each feature vector corresponds to an element in the document image data.

[0020] Furthermore, in step S2, the document feature map is constructed based on the image feature data, and the specific implementation method is as follows:

[0021] The document feature graph includes a plurality of nodes, each node represents a feature vector, and the nodes are connected by edges, and the edges represent the semantic associations between the nodes. An adjacency matrix is constructed according to the document feature graph.

[0022] Furthermore, in step S3, the node feature update formula in the document repair model is:

[0023]

[0024] in, is the feature vector of node j in the lth layer of the document repair model, n is the number of nodes, W is the weight matrix, σ is the activation function, is the feature vector of node i in the document repair model l+1, α ij is the attention weight between node i and node j, and the calculation formula of the attention weight is:

[0025]

[0026] Among them, e ij is the correlation score between node i and node j, exp() represents the natural exponential function, e ik represents the correlation score between node i and node k.

[0027] Furthermore, the training process of the document repair model is as follows:

[0028] S31. Obtain a document image dataset, divide the document image dataset into a training set and a validation set, and perform data enhancement on the training set;

[0029] S32. Input the data-enhanced training set to the document repair model and train the document repair model using the Adam optimizer to obtain a trained document repair model.

[0030] S33. Input the validation set into the trained document repair model, evaluate the model performance of the trained document repair model through cross-validation, and when the model performance is less than the preset threshold, retrain the document repair model until the model performance is greater than the preset threshold.

[0031] Furthermore, in step S4, the specific structure of the generative adversarial network model includes a generator and a discriminator, and the generator includes a node feature decoding module, a multi-layer decoding module and a residual connection module.

[0032] Furthermore, the training process of the generative adversarial network is specifically as follows:

[0033] S41. Obtaining a generated data set, wherein the generated data set includes a plurality of image data and the same number of feature maps, each image data corresponding to a feature map;

[0034] S42. Input a feature map to the generator. The generator generates pseudo image data based on the input feature map. The image data corresponding to the input feature map is real image data. The pseudo image data and the real image data are input to the discriminator. The discriminator distinguishes the pseudo image data from the real image data, calculates the discrimination loss, and optimizes the parameters of the generator and the discriminator based on the discrimination loss.

[0035] S43. Repeat step S42 until the discrimination loss is less than a preset threshold or the number of repetitions reaches a preset number, then end the training and obtain a trained generative adversarial network.

[0036] Furthermore, in step S4, the repair feature map is input into the generative adversarial network model, and the generative adversarial network model outputs the repaired document image. The specific implementation method is: the repair feature map is input into the generator in the generative adversarial network, and the generator outputs the repaired document image.

[0037] Through the above technical solution, the present invention has the following beneficial effects: based on graph structure modeling, information transmission and node feature updating of graph neural networks, the restoration of document images can take into account deeper semantic and structural relationships, and is combined with generative adversarial networks to make the restored document images more natural and realistic, and ensure the semantic consistency of the image, improve the quality of image restoration, avoid the loss of information after restoration, and have generalization for different types of document images, and can repair various documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the overall process of a document repair method based on graph neural network of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1 As shown, a document repair method based on graph neural network includes the following steps:

[0042] S1. Obtain document image data and preprocess the document image data;

[0043] S2. Feature extraction is performed on the preprocessed document image data to obtain image feature data, and a document feature map is constructed based on the image feature data;

[0044] S3. Build a document repair model, which is a graph neural network model. Input a document feature map to the document repair model, and output a repair feature map.

[0045] S4. Build a generative adversarial network model, input the repair feature map into the generative adversarial network model, and the generative adversarial network model outputs the repaired document image.

[0046] In an optional embodiment, the preprocessing of the document image data specifically includes:

[0047] The document image data is converted into a grayscale image, binarized, and filtered.

[0048] Specifically, the filtering algorithm adopts Gaussian filtering, which can smooth and remove noise, and has good edge preservation, high computational efficiency, and conforms to the statistical characteristics of natural images while removing image noise.

[0049] In an optional embodiment, the binarization of the document image data is specifically implemented as follows:

[0050] The Otsu threshold segmentation method is used to calculate the binarization threshold, and each pixel in the document image data is threshold segmented according to the binarization threshold.

[0051] The Otsu threshold segmentation method is used to automatically determine the optimal threshold value, convert the grayscale image into a binary image, highlight the text and graphic areas, thereby reducing noise interference and enhancing the recognizability of important information. The specific formula of the Otsu threshold segmentation method is:

[0052]

[0053] Among them, T is the binarization threshold, w1, w2 are the weights of background and foreground, σ1 2 ,σ2 2 is the between-class variance, and max means taking the maximum value.

[0054] In an optional embodiment, in step S2, the feature extraction is performed on the pre-processed document image data, and the specific implementation method is as follows:

[0055] Construct a feature extraction model, which is a pre-trained convolutional neural network. Input the preprocessed document image data into the feature extraction model, and the feature extraction model extracts the feature map of the preprocessed document image data. According to the feature map, several feature vectors are generated. Each feature vector corresponds to an element in the document image data, and the element includes a character, word, paragraph or chart.

[0056] The pre-trained convolutional neural networks include EfficientNetV2, ConvNeXt, Swin Transformer, etc. The pre-trained convolutional neural networks use weights obtained by training on large-scale data sets to extract universal and efficient feature representations, reduce dependence on small data sets, and improve the generalization ability of the model.

[0057] In addition, feature pyramids (FPN) at different levels are fused to preserve detail information while enhancing global features.

[0058] In an optional embodiment, in step S2, the document feature map is constructed based on the image feature data, and the specific implementation method is as follows:

[0059] The document feature graph includes several nodes, each representing a feature vector. Nodes are connected by edges, which represent semantic associations between nodes. An adjacency matrix is constructed based on the document feature graph. Specifically, semantic associations are established based on feature vector similarity.

[0060] In an optional embodiment, in step S3, the node feature update formula in the document repair model is:

[0061]

[0062] in, is the feature vector of node j in the lth layer of the document repair model, n is the number of nodes, W is the weight matrix, σ is the activation function, is the feature vector of node i in the document repair model l+1, α ij is the attention weight between node i and node j, and the calculation formula of the attention weight is:

[0063]

[0064] Among them, e ij is the correlation score between node i and node j, exp() represents the natural exponential function, e ik represents the correlation score between node i and node k. The correlation score is calculated by an attention function, which is a dot product or a multi-layer perceptron to calculate the similarity between nodes.

[0065] The document repair model in this embodiment performs repairs based on a hierarchical repair strategy. At the lowest level, the model learns character-level relationships, ensuring that even small areas of damaged characters can be restored from neighboring characters. A local graph structure update strategy is employed, giving characters in damaged areas greater weights, thereby prioritizing repair. Furthermore, the model learns relationships between sentences or paragraphs to ensure coherence, dynamically adjusting the adjacent relationships between paragraphs based on the text structure to ensure optimal repair strategies for all document types. Before final output, the model calculates topological structure loss and uses this loss to repair the entire page layout, ensuring that the repaired document's overall structure is consistent with the original format.

[0066] Graph neural networks aggregate information layer by layer, incorporating information from both direct and indirect neighbors at each layer. This enables the network to capture deep patterns and semantic relationships within document images. At each layer, nodes interact with their neighbors' features, generating new node features. This interaction is not limited to spatial adjacency but also includes semantic similarity. By capturing higher-order relationships between nodes, graph neural networks can complete the structure of damaged areas, making the repaired document image more complete and reasonable in both structure and content.

[0067] In an optional embodiment, the training process of the document repair model is:

[0068] S31. Obtain a document image dataset, divide the document image dataset into a training set and a validation set, and perform data enhancement on the training set;

[0069] The data augmentation includes augmenting the training data by random rotation, scaling, cropping, etc., to increase the generalization ability of the model and adapt to different types of document damage;

[0070] S32. Input the data-augmented training set to the document repair model and train the document repair model using the Adam optimizer to obtain a trained document repair model. The Adam optimizer has a learning rate of 0.01, which decays every 10 epochs. Forward propagation and backward propagation are performed once per epoch using the complete training set.

[0071] S33. Input the validation set into the trained document repair model, evaluate the model performance of the trained document repair model through cross-validation, and when the model performance is less than the preset threshold, retrain the document repair model until the model performance is greater than the preset threshold.

[0072] In an optional embodiment, in step S4, the specific structure of the generative adversarial network model includes a generator and a discriminator, and the generator includes a node feature decoding module, a multi-layer decoding module and a residual connection module.

[0073] Generative adversarial networks are used to improve the realism and detail of inpainted images. Through the generation and discrimination process, the naturalness and structural consistency of inpainted images are enhanced. The generator is responsible for image inpainting based on the output of the graph neural network. The generator's role is to map the high-order relationships and inpainted features transmitted between nodes into the actual image space.

[0074] The generator decodes the node features passed through the graph neural network to generate restored image pixel values. Furthermore, a multi-layer decoder gradually restores the document image's structure and details. The decoder, which includes a Transformer decoder or a CNN decoder, must not only restore pixel information but also ensure the document's semantic coherence and spatial layout. Finally, residual connections incorporate information about the tractor, preventing information loss and gradient explosion, and enhancing model stability.

[0075] The role of the discriminator is to determine whether the generated repaired image is similar enough to the real document image. By distinguishing between the repaired image and the real image, the discriminator helps the generator learn images that are more consistent with the characteristics of the real document.

[0076] The discriminator is a convolutional neural network. It feeds the inpainted image and the real image into a convolutional neural network. The convolutional neural network extracts local features from the images and outputs a binary classification result: either the input image is real or generated. The discriminator's goal is to maximize its ability to distinguish the generated image from the real image as much as possible. The generator's goal is to minimize the discriminator's ability to distinguish, meaning to trick the discriminator into mistaking the generated image for the real one.

[0077] In an optional embodiment, the training process of the generative adversarial network is specifically as follows:

[0078] S41. Obtaining a generated data set, wherein the generated data set includes a plurality of image data and the same number of feature maps, each image data corresponding to a feature map;

[0079] S42. Input a feature map to the generator. The generator generates pseudo image data based on the input feature map. The image data corresponding to the input feature map is real image data. The pseudo image data and the real image data are input to the discriminator. The discriminator distinguishes the pseudo image data from the real image data, calculates the discrimination loss, and optimizes the parameters of the generator and the discriminator based on the discrimination loss.

[0080] S43. Repeat step S42 until the discrimination loss is less than a preset threshold or the number of repetitions reaches a preset number, then end the training and obtain a trained generative adversarial network.

[0081] In an optional embodiment, in step S4, the repair feature map is input into the generative adversarial network model, and the generative adversarial network model outputs the repaired document image. The specific implementation method is: the repair feature map is input into the generator in the generative adversarial network, and the generator outputs the repaired document image.

[0082] The embodiment disclosed in this specification is merely an illustration of one aspect of the present invention. The scope of protection of the present invention is not limited to this embodiment. Any other functionally equivalent embodiments fall within the scope of protection of the present invention. Those skilled in the art can make various other corresponding changes and modifications based on the technical solutions and concepts described above, and all such changes and modifications should fall within the scope of protection of the claims of the present invention.

Claims

1. A document repair method based on graph neural network, characterized in that: The following steps are involved: S1. Obtain document image data and preprocess the document image data; S2. Feature extraction is performed on the preprocessed document image data to obtain image feature data, and a document feature map is constructed based on the image feature data; S3. Build a document repair model, which is a graph neural network model. Input a document feature map to the document repair model, and output a repair feature map. S4. Build a generative adversarial network model, input the repair feature map into the generative adversarial network model, and the generative adversarial network model outputs the repaired document image.

2. A document repair method based on graph neural network according to claim 1, characterized in that: The preprocessing of the document image data specifically includes: The document image data is converted into a grayscale image, binarized, and filtered.

3. The document repair method based on graph neural network according to claim 2 is characterized in that: The document image data is binarized, and its specific implementation is as follows: The Otsu threshold segmentation method is used to calculate the binarization threshold, and each pixel in the document image data is threshold segmented according to the binarization threshold.

4. The document repair method based on graph neural network according to claim 1, characterized in that: In step S2, feature extraction is performed on the pre-processed document image data, and the specific implementation method is as follows: Construct a feature extraction model, which is a pre-trained convolutional neural network. Input the preprocessed document image data into the feature extraction model, and the feature extraction model extracts the feature map of the preprocessed document image data. According to the feature map, several feature vectors are generated, and each feature vector corresponds to an element in the document image data.

5. The document repair method based on graph neural network according to claim 4 is characterized in that: In step S2, the document feature map is constructed based on the image feature data, and the specific implementation method is as follows: The document feature graph includes a plurality of nodes, each node represents a feature vector, and the nodes are connected by edges, and the edges represent the semantic associations between the nodes. An adjacency matrix is constructed according to the document feature graph.

6. The document repair method based on graph neural network according to claim 5 is characterized in that: In step S3, the node feature update formula in the document repair model is: in, is the feature vector of node j in the lth layer of the document repair model, n is the number of nodes, W is the weight matrix, σ is the activation function, is the feature vector of node i in the document repair model l+1, α ij is the attention weight between node i and node j, and the calculation formula of the attention weight is: Among them, e ij is the correlation score between node i and node j, exp() represents the natural exponential function, e ik represents the correlation score between node i and node k.

7. The document repair method based on graph neural network according to claim 1, characterized in that: The training process of the document repair model is as follows: S31. Obtain a document image dataset, divide the document image dataset into a training set and a validation set, and perform data enhancement on the training set; S32. Input the data-enhanced training set to the document repair model and train the document repair model using the Adam optimizer to obtain a trained document repair model. S33. Input the validation set into the trained document repair model, and evaluate the model performance of the trained document repair model through cross-validation. When the model performance is less than the preset threshold, retrain the document repair model until the model performance is greater than the preset threshold.

8. The document repair method based on graph neural network according to claim 1, characterized in that: In step S4, the specific structure of the generative adversarial network model includes a generator and a discriminator, and the generator includes a node feature decoding module, a multi-layer decoding module and a residual connection module.

9. The document repair method based on graph neural network according to claim 8, characterized in that: The training process of the generative adversarial network is specifically as follows: S41. Obtaining a generated data set, wherein the generated data set includes a plurality of image data and the same number of feature maps, each image data corresponding to a feature map; S42. Input a feature map to the generator. The generator generates pseudo image data based on the input feature map. The image data corresponding to the input feature map is real image data. The pseudo image data and the real image data are input to the discriminator. The discriminator distinguishes the pseudo image data from the real image data, calculates the discrimination loss, and optimizes the parameters of the generator and the discriminator based on the discrimination loss. S43. Repeat step S42 until the discrimination loss is less than a preset threshold or the number of repetitions reaches a preset number, then end the training and obtain a trained generative adversarial network.

10. The document repair method based on graph neural network according to claim 9, characterized in that: In step S4, the repair feature map is input into the generative adversarial network model, and the generative adversarial network model outputs the repaired document image. The specific implementation method is: the repair feature map is input into the generator in the generative adversarial network, and the generator outputs the repaired document image.

Images