Ancient character defect font reconstruction method and device based on graph structure and deep learning

Abstract

The invention provides an ancient character defect font reconstruction method and device based on a graph structure and deep learning, and relates to the technical field of image processing, and the method comprises the steps: carrying out the graph structure representation extraction of a defective ancient character image, and obtaining a target contour graph and a target skeleton graph corresponding to the defective ancient character image; according to the target contour map and the target skeleton map, determining a defect type corresponding to the defective ancient character image, determining a font reconstruction strategy based on the defect type, and determining a map neural network corresponding to the defective ancient character image based on the font reconstruction strategy; the graph neural network comprises a contour graph reconstruction network, a skeleton graph reconstruction network or a cross-modal reconstruction network; and inputting the defective ancient character image into a pre-trained graph neural network based on a font reconstruction strategy to obtain a reconstruction result. According to the method, high-quality ancient character defect font reconstruction is realized by using the graph neural network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a method and device for reconstructing damaged characters of ancient scripts based on graph structure and deep learning. BACKGROUND

[0002] As an important carrier of human civilization, ancient scripts carry rich historical and cultural information. However, due to factors such as long time, poor preservation conditions, and the like, existing ancient script materials are generally damaged to varying degrees, which seriously affects the recognition, interpretation, and research of ancient scripts. The technology for reconstructing damaged characters of ancient scripts is of great significance for the digital protection of cultural heritage, automatic recognition of ancient scripts, and historical literature research.

[0003] At present, the method for reconstructing damaged characters of ancient scripts is a traditional rule-based method, which is as follows: mainly relying on the character structure rules and stroke combination rules summarized by experts in ancient scripts, and combining traditional machine learning methods to reconstruct characters. For example, by constructing a complete character library, the most similar template to the damaged character is retrieved, that is, the template matching-based retrieval reconstruction. These methods can achieve certain results in dealing with simple damage with strong rules, but they face problems such as incomplete rule coverage, poor adaptability to character variants, and difficulty in dealing with complex damage patterns. SUMMARY

[0004] In view of the above problems in the prior art, the present application provides a method and device for reconstructing damaged characters of ancient scripts based on graph structure and deep learning, which converts characters into structured representations of contour graphs and skeleton graphs, replaces traditional pixel representations, combines self-supervised pre-training and contrast learning techniques, automatically learns deep representations of characters from a large amount of unannotated structured data of ancient scripts, and thus realizes high-quality reconstruction of damaged characters of ancient scripts using graph neural networks.

[0005] In a first aspect, the present application provides a method for reconstructing damaged characters of ancient scripts based on graph structure and deep learning, which comprises the following steps: extracting graph structure representations from damaged ancient script images to obtain target contour graphs and target skeleton graphs of the damaged ancient script images; determining the damage type corresponding to the damaged ancient script image according to the target contour graph and the target skeleton graph, determining a character reconstruction strategy based on the damage type, and determining a graph neural network corresponding to the damaged ancient script image based on the character reconstruction strategy; the graph neural network comprises a contour graph reconstruction network, a skeleton graph reconstruction network, or a cross-modal reconstruction network; inputting the damaged ancient character image into the pre-trained graph neural network based on the glyph reconstruction strategy to obtain a reconstruction result corresponding to the damaged ancient character image; the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0006] According to the ancient character damaged glyph reconstruction method based on a graph structure and deep learning provided by the application, the damaged ancient character image is graph structure characterized and extracted to obtain a target contour graph of the damaged ancient character image, including: The Canny edge detection algorithm is used to extract the main contour line of the damaged ancient character image. The Douglas-Peucker algorithm is used to simplify the main contour line and retain key turning points to obtain a simplified contour line. A first node on the simplified contour line is identified, and edge connections between the first nodes are constructed based on the first nodes and contour adjacency relationship potentials to obtain an initial contour graph; the first nodes include corner points, turning points, connection points and feature points, and different first nodes correspond to different attribute encodings. The initial contour graph is subjected to scale and position normalization processing to obtain the target contour graph.

[0007] According to the ancient character damaged glyph reconstruction method based on a graph structure and deep learning provided by the application, the damaged ancient character image is graph structure characterized and extracted to obtain the target skeleton graph, including: The Zhang-Suen thinning algorithm is used to extract the central axis skeleton of the damaged ancient character image and eliminate the sawtooth effect. A second node in the central axis skeleton is identified; the second node includes skeleton end points, intersection points and turning points. Based on skeleton connectivity, edge connections between the second nodes are constructed to obtain the target skeleton graph; different second nodes correspond to different attribute encodings.

[0008] According to the ancient character damaged glyph reconstruction method based on a graph structure and deep learning provided by the application, the glyph reconstruction strategy is determined based on the damage type, and the graph neural network corresponding to the damaged ancient character image is determined based on the glyph reconstruction strategy, including: In the case that the damage type corresponding to the damaged ancient character image is skeleton serious loss but relatively complete boundary, the glyph reconstruction strategy is determined as a contour priority strategy, and the graph neural network corresponding to the damaged ancient character image is determined as a pre-trained contour graph reconstruction network. In a case where the missing type corresponding to the missing ancient character image is that the internal structure is identifiable but the boundary is blurred, the glyph reconstruction strategy is determined as a skeleton priority strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained skeleton graph reconstruction network; In a case where the missing type corresponding to the missing ancient character image is that the contour and skeleton information are partially available, the glyph reconstruction strategy is determined as a collaborative reconstruction strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained cross-modal reconstruction network.

[0009] According to the ancient character missing glyph reconstruction method based on a graph structure and deep learning provided by the application, the glyph reconstruction strategy is used to input the missing ancient character image into the pre-trained graph neural network to obtain a reconstruction result corresponding to the missing ancient character image, which includes: In a case where the glyph reconstruction strategy is a contour priority strategy, the missing ancient character image is input into a pre-trained contour graph reconstruction network to obtain a reconstruction result of the missing ancient character image; In a case where the glyph reconstruction strategy is a skeleton priority strategy, the missing ancient character image is input into a pre-trained skeleton graph reconstruction network to obtain a reconstruction result of the missing ancient character image; In a case where the glyph reconstruction strategy is a collaborative reconstruction strategy, the target contour graph and the target skeleton graph are feature-aligned, the fusion weights of the target contour graph and the target skeleton graph are dynamically adjusted based on the modal confidence, and aligned feature information is obtained; The aligned feature information is input into a pre-trained cross-modal reconstruction network to obtain a reconstruction result of the missing ancient character image.

[0010] According to the ancient character missing glyph reconstruction method based on a graph structure and deep learning provided by the application, the pre-trained contour graph reconstruction network, the pre-trained skeleton graph reconstruction network and the pre-trained cross-modal reconstruction network are trained by using a progressive learning strategy, and the training steps of the pre-trained contour graph reconstruction network include: An unannotated ancient character dataset is obtained; the unannotated ancient character dataset includes a plurality of sample ancient characters; A positive and negative sample pair of a contour graph reconstruction network is generated based on each sample ancient character; the positive and negative sample pair includes a positive sample pair and a negative sample pair, the positive sample pair is different enhanced contour graphs of the same character, and the negative sample pair is contour graphs of different characters; The contour graph reconstruction network is iteratively trained based on the positive and negative sample pair of the contour graph reconstruction network and an InfoNCE loss function; The iteration terminates when the InfoNCE loss function reaches its minimum value, and the pre-trained contour reconstruction network is determined based on the model parameters of the contour reconstruction network at the time of iteration termination.

[0011] According to the present invention, a method for reconstructing missing ancient Chinese characters based on graph structure and deep learning is provided, wherein the training steps of the pre-trained cross-modal reconstruction network include: Positive sample pairs for the cross-modal reconstruction network are generated based on the contour maps and skeleton maps of the same character in each of the sample ancient characters, and negative sample pairs for the cross-modal reconstruction network are generated based on the contour maps and skeleton maps of different characters. The cross-modal reconstruction network is iteratively trained based on the positive sample pairs and negative sample pairs of the cross-modal reconstruction network and the InfoNCE loss function; The iteration terminates when the InfoNCE loss function reaches its minimum value. Based on the model parameters of the cross-modal reconstruction network at the time of iteration termination, the pre-trained cross-modal reconstruction network is determined.

[0012] According to the present invention, a method for reconstructing missing ancient Chinese characters based on graph structure and deep learning, after extracting graph structure representations from the missing ancient Chinese character image to obtain the target contour map and target skeleton map corresponding to the missing ancient Chinese character image, the method further includes: Generate a multi-dimensional feature set corresponding to the nodes of the target contour map and the target skeleton map; The multi-dimensional feature set includes: geometric features, topological features, philological features, and document context features; The step of inputting the missing ancient character image into the pre-trained graph neural network based on the character reconstruction strategy to obtain the reconstruction result corresponding to the missing ancient character image includes: The missing ancient character image and the multi-dimensional feature set are input into the pre-trained graph neural network to obtain the reconstruction result corresponding to the missing ancient character image.

[0013] According to the present invention, a method for reconstructing the missing glyphs of ancient Chinese characters based on graph structure and deep learning is provided, the method further comprising: The reconstruction results corresponding to the missing ancient characters are verified; The verification includes at least one of topological rationality testing, geometric consistency assessment, and multimodal cross-validation.

[0014] Secondly, the present invention also provides a device for reconstructing the shape of missing ancient characters based on graph structure and deep learning, the device comprising the following modules: The figure structure representation extraction module is configured to perform figure structure representation extraction on the damaged ancient character image to obtain a structured representation of the damaged ancient character image; the structured representation includes a target contour graph and a target skeleton graph. The strategy determination module is configured to determine a damage type corresponding to the damaged ancient character image according to the structured representation, and determine a character shape reconstruction strategy based on the damage type; the character shape reconstruction strategy represents a graph neural network corresponding to the damaged ancient character image; the graph neural network includes a contour graph reconstruction network, a skeleton graph reconstruction network, and a cross-modal reconstruction network. The reconstruction module is configured to input the damaged ancient character image into the pre-trained graph neural network based on the character shape reconstruction strategy to obtain a reconstruction result corresponding to the damaged ancient character image; the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0015] In a third aspect, the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned ancient character damaged character reconstruction methods based on graph structure and deep learning.

[0016] In a fourth aspect, the present application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement any of the above-mentioned ancient character damaged character reconstruction methods based on graph structure and deep learning.

[0017] In a fifth aspect, the present application further provides a computer program product, including a computer program, wherein the computer program is executed by a processor to implement any of the above-mentioned ancient character damaged character reconstruction methods based on graph structure and deep learning.

[0018] The ancient character damaged character reconstruction method and device based on graph structure and deep learning provided by the present application first perform figure structure representation extraction on the damaged ancient character image to obtain a target contour graph and a target skeleton graph; then, according to the target contour graph and the target skeleton graph, a damage type corresponding to the damaged ancient character image is determined, a character shape reconstruction strategy is determined based on the damage type, a graph neural network corresponding to the damaged ancient character image is determined based on the character shape reconstruction strategy, the graph neural network includes a contour graph reconstruction network, a skeleton graph reconstruction network, and a cross-modal reconstruction network; further, based on the character shape reconstruction strategy, the damaged ancient character image is input into the pre-trained graph neural network to obtain a reconstruction result corresponding to the damaged ancient character image, and the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0019] The application replaces the traditional pixel representation by converting the character into a structured representation of contour graph and skeleton graph, automatically learns the deep representation of the character from a large amount of unannotated ancient character structured data by combining self-supervised pre-training and contrast learning technology, and realizes high-quality ancient character missing character reconstruction by using a graph neural network. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is one of the flowcharts of the ancient character missing character reconstruction method based on graph structure and deep learning provided by the present application.

[0022] Figure 2 is the second flowchart of the ancient character missing character reconstruction method based on graph structure and deep learning provided by the present application.

[0023] Figure 3 is a structural schematic diagram of the ancient character missing character reconstruction device based on graph structure and deep learning provided by the present application.

[0024] Figure 4 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application will be described clearly and completely in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the present application.

[0026] In order to more clearly understand the embodiments provided by the present application, first, the technical content related to the present application is introduced as follows: In the prior art, there are some other methods to realize character reconstruction: 1. Simple deep learning-based methods. These include convolutional neural network (CNN)-based end-to-end reconstruction (modeling the reconstruction task as an image-to-image mapping problem, using an encoder-decoder architecture to learn pixel-level reconstruction), U-Net-based image inpainting (utilizing skip connections to preserve detail information, migrating from medical image inpainting to ancient text scenarios), and generative adversarial network (GAN)-based adversarial training (improving the authenticity of reconstruction results through adversarial learning between generator and discriminator). These methods have improved reconstruction quality compared to traditional methods, but still have limitations such as excessive focus on pixel details, neglect of the essence of character structure, and high computational resource requirements.

[0027] 2. Extended deep learning-based methods. These mainly include transformer-based sequence modeling methods (block processing of images, utilizing self-attention mechanisms to capture long-range dependencies, improving structural consistency to some extent) and diffusion model-based generation methods (generating high-quality images through step-by-step denoising, demonstrating good performance in image generation tasks).

[0028] These methods have achieved good results in certain scenarios, but for the special field of ancient texts, they still face challenges such as scarcity of training data, insufficient model interpretability, and difficulty in fully utilizing the structured characteristics of ancient texts.

[0029] The following will be described Figures 1-4 The ancient text missing character reconstruction method and device based on graph structure and deep learning provided by the present application.

[0030] Figure 1 is one of the flowcharts of the ancient text missing character reconstruction method based on graph structure and deep learning provided by the present application, as Figure 1 shown, the method comprises the following steps: Step 101, extracting graph structure representation of the missing ancient text image to obtain the target contour graph and the target skeleton graph corresponding to the missing ancient text image.

[0031] It should be noted that the execution subject of the present application is an electronic device, and the present application can realize high-quality ancient text missing character reconstruction using graph neural networks.

[0032] First, the missing ancient text image is subjected to graph structure representation extraction, wherein the structured representation includes a target contour graph and a target skeleton graph.

[0033] Target contour mapping refers to a graph representation based on the topological structure of the external boundaries of character shapes. It transforms the continuous contour lines of ancient character images into a network composed of nodes (key points) and edges (contour segments), preserving the abstract structure of the character's geometric shape. Extracting contour maps can accurately capture the external morphology of characters (such as the thick strokes in bronze inscriptions) and is robust to differences in writing styles (edge ​​deformation of the same character can still be matched across different rubbings). Contour maps can effectively represent the external shape features and boundary variation patterns of character shapes.

[0034] A target skeleton graph refers to the central axis framework of ancient Chinese character strokes. Nodes represent key points of the skeleton (endpoints / intersections / turning points), and edges represent connecting paths between strokes, constructing a graph network that describes the internal topology. Both contour graphs and skeleton graphs contain nodes, edges, and corresponding feature information. Skeleton graphs directly reflect the topological structure of the character and the connection relationships between strokes.

[0035] The graph structure representation of the missing ancient characters is extracted, that is, the contour map and skeleton map corresponding to the ancient character image are extracted. Then, the character shape is reconstructed based on the contour map and skeleton map combined with a graph neural network.

[0036] Step 102: Based on the target contour map and the target skeleton map, determine the type of defect corresponding to the missing ancient character image, determine the character reconstruction strategy based on the type of defect, and determine the graph neural network corresponding to the missing ancient character image based on the character reconstruction strategy; the graph neural network includes a contour map reconstruction network, a skeleton map reconstruction network, or a cross-modal reconstruction network.

[0037] After determining the structured representation, the missing type of the ancient character image is further determined based on the structured representation. The missing type is also the distribution and type of the missing area.

[0038] The types of defects include severe missing skeletons but relatively complete boundaries, identifiable internal structures but blurred boundaries, and partially available contour and skeleton information.

[0039] Subsequently, based on different loss types, corresponding character reconstruction strategies are determined. A character reconstruction strategy can be understood as focusing on reconstructing a specific region of the ancient character image during the character reconstruction process; correspondingly, graph neural networks with different parameters are employed. The graph neural networks in this invention include contour map reconstruction networks, skeleton map reconstruction networks, and cross-modal reconstruction networks.

[0040] The improvement of the graph neural network, such as a graph convolution network (GCN), a graph attention network (GAT) and a graph transformer, lies in the addition of a class embedding layer.

[0041] In step 103, the damaged ancient character image is input into the pre-trained graph neural network based on the glyph reconstruction strategy, and a reconstruction result corresponding to the damaged ancient character image is obtained.

[0042] Specifically, after determining the corresponding glyph reconstruction strategy, the graph neural network for glyph reconstruction of the damaged ancient character image is determined.

[0043] Further, the damaged ancient character image is input into the pre-trained graph neural network, and a reconstruction result corresponding to the damaged ancient character image is obtained.

[0044] The pre-trained graph neural network is trained based on self-supervised pre-training and contrastive learning.

[0045] The self-supervised pre-training uses unlabeled ancient character graph structure data to automatically generate a supervision signal for pre-training, learns a general glyph structure representation, and alleviates the problem of insufficient labeled data. The implementation steps in the ancient character graph convolution network include: 1. Construct a proxy task (Pretext Task) Mask node prediction: randomly mask part of the node features (such as coordinates and curvature) in the outline graph / skeleton graph, and let the graph convolution network reconstruct the masked features based on the neighborhood structure Graph structure contrast: generate two graph structures (such as outline graphs with different simplification thresholds) for the same glyph, and train the graph convolution network to determine whether they come from the same original glyph Patent protection point: the proxy task design needs to bind the graph structure characteristics (such as the topological relationship of the 3 key points of the skeleton) 2. Graph neural network encoding Input: graph structure with mask / perturbation Encoder: GCN / GAT network learns node representation Output: reconstruction features or similarity scores 3. Pre-training objective function Make the model master the glyph structure rules (such as the correlation between turning points and neighborhood curvature) Contrastive learning narrows the graph representation of similar glyphs and widens the graph representation of dissimilar glyphs, enhancing the model's ability to distinguish writing variants.

[0046] The implementation steps of contrastive learning in ancient script GNNs include: 1. Construct positive and negative sample pairs: Positive samples are, for example, different variants of the same character (such as the 10 variant character skeleton diagrams of the oracle bone script for "horse"), while negative samples are, for example, similar glyphs of different characters (such as the outline diagrams of "mouth" vs. "say").

[0047] 2. Visual comparison The graph structure representation is extracted using a GNN encoder; the similarity between positive and negative samples is calculated; and the loss function is compared.

[0048] The training of a pre-trained graph neural network includes the following steps: Unlabeled ancient script dataset, self-supervised pre-training: agent task learning, graph representation initialization, labeled data comparison learning: positive and negative sample fine-tuning, end-to-end network training.

[0049] The method provided in this embodiment first extracts graph structure representations from the damaged ancient character image to obtain a structured representation of the damaged ancient character image, wherein the structured representation includes a target contour map and a target skeleton map; then, based on the structured representation, the type of damage corresponding to the damaged ancient character image is determined, and a character reconstruction strategy is determined based on the type of damage, wherein the character reconstruction strategy represents the graph neural network corresponding to the damaged ancient character image, and the graph neural network includes a contour map reconstruction network, a skeleton map reconstruction network, and a cross-modal reconstruction network; furthermore, based on the character reconstruction strategy, the damaged ancient character image is input into the pre-trained graph neural network to obtain the reconstruction result corresponding to the damaged ancient character image.

[0050] This invention converts character shapes into structured representations of contour and skeleton diagrams, replacing traditional pixel representations. By combining self-supervised pre-training and contrastive learning techniques, it automatically learns deep representations of character shapes from a large amount of unlabeled ancient character structured data, thereby using graph neural networks to achieve high-quality reconstruction of ancient character shapes with defects.

[0051] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0052] According to the present invention, a method for reconstructing missing ancient Chinese characters based on graph structure and deep learning is provided. This method extracts graph structure representations from the missing ancient Chinese character images to obtain target contour maps corresponding to the missing ancient Chinese character images, including: The Canny edge detection algorithm was used to extract the main contour lines of the damaged ancient text image; The Douglas-Peucker algorithm is used to simplify the main contour lines while retaining key inflection points, resulting in simplified contour lines. A first node on the simplified contour line is identified, and an edge connection between the first nodes is constructed based on the first nodes and contour adjacency relationship potential, to obtain an initial contour graph; the first nodes include corner points, turning points, connection points, and feature points, and different first nodes correspond to different attribute encodings; The initial contour graph is subjected to scale and position normalization processing, to obtain a target contour graph.

[0053] Specifically, in some embodiments, the process of skeleton graph structure representation extraction in step 101 includes three steps of contour extraction and simplification, graph structure construction, and graph standardization. Specifically as follows: (1) Contour extraction and simplification: First, the Canny edge detection algorithm is used to extract the main contour line of the damaged ancient character image.

[0054] The Canny edge detection algorithm is a classic edge detection method in computer vision, which processes images through multiple steps to extract edges with high precision while suppressing noise. The core steps include: Gaussian filtering (noise reduction), calculating gradient amplitude and direction, non-maximum suppression (NMS), double threshold detection and lag threshold processing, and outputting the edge map (i.e., the main contour line).

[0055] Then, the Douglas-Peucker algorithm is used to simplify the main contour line and retain key turning points to obtain the simplified contour line. The Douglas-Peucker algorithm is a classic algorithm for simplifying polyline or polygon contour, which retains the main shape features by reducing redundant points. The following are the core steps and details: 1. Algorithm input and initialization Input: ordered point set P { p 1, p 2,..., p n} (such as pixel coordinates of the contour line).

[0056] Initialization: select the starting point p 1 and the end point p n as the initial key points, and add them to the simplified result set Q .

[0057] 2. Recursive simplification process Step 1: Find the maximum deviation point For the current line segment , calculate the perpendicular distance of all intermediate points p i ( i ∈[2, n −1])to the line segmentd i ; record maximum distance d max and its corresponding point p max .

[0058] Step 2: Determine whether to split If d max > e ( e is a preset threshold value): Add p max to the key point set Q . Recursively process two sub-chord lines: p 1,..., p max}, { p max ,..., p n}; If d max ≤ e : Discard all intermediate points and only keep p 1and p n .

[0059] 3. Termination condition When all sub-chord lines cannot find a deviation point exceeding e , the recursion is terminated.

[0060] 4. Output the simplified result The final key point set Q is connected in the original order to form the simplified contour line.

[0061] Further, after simplification, redundant points are removed and key turning points are retained, contour breaks and noise are processed to ensure the continuity and integrity of the contour.

[0062] (2) Graph structure construction: Identify the first node on the simplified contour line, which includes corner points, turning points, connection points and feature points. Different first nodes correspond to different attribute encodings.

[0063] Further, based on the first node and the contour adjacency relationship potential, the edge connection between the first nodes is constructed to obtain the initial contour graph.

[0064] (3) Graph standardization: The initial contour map is subjected to scale and position normalization to obtain a target contour map. For example, the contour map is finally subjected to scale and position normalization, the node attribute range is unified, the comparability of the graph is ensured through a graph isomorphism algorithm, the normalized representation of the graph is established, and the influence of position and scale difference is eliminated.

[0065] The method provided in the embodiment extracts a graph structure representation from a damaged ancient character image, obtains a contour map and a skeleton map, converts a traditional pixel representation into a contour map, and the contour map and stroke skeleton structure map of a character form contain rich geometric and topological information, which can provide more reliable basis for a reconstruction task and improve the reliability of the character form reconstruction task.

[0066] According to the ancient character damaged character form reconstruction method based on a graph structure and deep learning provided in the application, a graph structure representation is extracted from a damaged ancient character image, and a target skeleton map is obtained, including: The Zhang-Suen thinning algorithm is used to extract the central axis skeleton of the damaged ancient character image, and the sawtooth effect is eliminated. Second nodes in the central axis skeleton are identified; the second nodes include skeleton end points, intersection points and turning points. Based on skeleton connectivity, edge connections between the second nodes are constructed to obtain the target skeleton map; different second nodes correspond to different attribute encodings.

[0067] Specifically, in some embodiments, the step 101 of structured representation (skeleton map) extraction includes three steps of skeleton extraction, graph structure construction and attribute encoding, and the details are as follows: (1) Skeleton extraction: For example, the Zhang-Suen thinning algorithm is used to extract the central axis skeleton of the damaged ancient character image, and the sawtooth effect is eliminated.

[0068] The Zhang-Suen thinning algorithm is a classical binary image skeleton extraction method, which is suitable for central axis processing of complex shapes such as ancient characters. The core idea is to delete the boundary pixels that meet the conditions through iteration, and finally retain the single-pixel-wide central axis skeleton.

[0069] Then, the skeleton is subjected to smoothing processing to eliminate the sawtooth effect introduced by the algorithm, so as to maintain the topological structure integrity of the character form and avoid skeleton breakage and pseudo-branching.

[0070] (2) Graph structure construction: First, the second nodes in the central axis skeleton are identified, that is, the key points on the skeleton are identified, and the key points (second nodes) on the skeleton include skeleton end points (stroke starting and ending), intersection points (stroke intersection) and turning points (direction change).

[0071] Then, based on the skeleton connectivity, the edge connection between the second nodes is constructed to obtain a target skeleton graph, form a complete graph structure representation, handle complex intersection conditions, and ensure the accuracy and consistency of the graph structure.

[0072] (3) Attribute coding: Different second nodes correspond to different attribute codes. Detailed attribute coding of the nodes lays a foundation for subsequent node feature design.

[0073] The method provided in the embodiment extracts the graph structure representation of the damaged ancient character image, obtains the contour graph and the skeleton graph, converts the traditional pixel representation into the contour graph, and the contour graph and the stroke skeleton structure graph of the character form contain rich geometric and topological information, which can provide more reliable basis for the reconstruction task and improve the reliability of the character reconstruction task.

[0074] According to the ancient character damaged character reconstruction method based on the graph structure and deep learning provided by the application, the character reconstruction strategy is determined based on the damage type, and the graph neural network corresponding to the damaged ancient character image is determined based on the character reconstruction strategy, including: In the case that the damage type corresponding to the damaged ancient character image is that the skeleton is severely missing but the boundary is relatively complete, the character reconstruction strategy is determined as a contour priority strategy, and the graph neural network corresponding to the damaged ancient character image is determined as a pre-trained contour graph reconstruction network. In the case that the damage type corresponding to the damaged ancient character image is that the internal structure is identifiable but the boundary is blurred, the character reconstruction strategy is determined as a skeleton priority strategy, and the graph neural network corresponding to the damaged ancient character image is determined as a pre-trained skeleton graph reconstruction network. In the case that the damage type corresponding to the damaged ancient character image is that the contour and skeleton information are partially available, the character reconstruction strategy is determined as a collaborative reconstruction strategy, and the graph neural network corresponding to the damaged ancient character image is determined as a pre-trained cross-modal reconstruction network.

[0075] Specifically, in some embodiments, the strategy determination stage of step 102 can be implemented by the following steps: First, based on the target contour graph and the target skeleton graph, the damage type is determined.

[0076] For example, the damage structure analysis stage pre-processes the input image, identifies the distribution and type of the damaged area, and the damage type includes that the skeleton is severely missing but the boundary is relatively complete, the internal structure is identifiable but the boundary is blurred, and the contour and skeleton information are partially available. The reconstruction strategy corresponding to each damage type is different.

[0077] The process of determining the character reconstruction strategy based on the damage type is as follows: For example, in the case that the missing type corresponding to the missing ancient character image is that the skeleton is severely missing but the boundary is relatively complete, the glyph reconstruction strategy is determined as a contour priority strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained contour graph reconstruction network.

[0078] The contour priority strategy is suitable for the case that the skeleton is severely missing but the boundary is relatively complete. This strategy focuses on utilizing the available boundary information to learn the spatial relationship and geometric constraints between contour nodes through the graph neural network to infer the missing boundary segments. In the processing process, the algorithm prioritizes maintaining the continuity and closure of the contour to ensure that the reconstructed boundary conforms to the morphological characteristics of ancient characters.

[0079] For example, in the case that the missing type corresponding to the missing ancient character image is that the internal structure is identifiable but the boundary is blurred, the glyph reconstruction strategy is determined as a skeleton priority strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained skeleton graph reconstruction network.

[0080] The skeleton priority strategy is suitable for the case that the internal structure is identifiable but the boundary is blurred. Through the analysis of the connection relationship and topological structure of strokes, this strategy can infer the missing stroke segments and connection points. The algorithm utilizes the writing rules of ancient characters, such as the relationship of strokes, to guide the reconstruction process of the skeleton structure.

[0081] For example, in the case that the missing type corresponding to the missing ancient character image is that the contour and skeleton information are both partially available, the glyph reconstruction strategy is determined as a collaborative reconstruction strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained cross-modal reconstruction network.

[0082] The collaborative reconstruction strategy takes advantage of the case that the contour and skeleton information are both partially available. This strategy ensures the consistency of the two representations in the semantic space through a feature alignment mechanism and dynamically weighs the reliability of different information sources using an attention fusion network. When the confidence of one representation is low, the algorithm relies more on the information of the other representation for compensation.

[0083] The method provided in this embodiment pre-processes the input image in the missing structure analysis stage, identifies the distribution and type of the missing area, then determines the reconstruction strategy based on the missing type, and determines the graph neural network used for glyph reconstruction based on the reconstruction strategy, which facilitates targeted glyph reconstruction and improves the glyph reconstruction effect.

[0084] According to the ancient character missing glyph reconstruction method based on graph structure and deep learning provided by the application, the missing ancient character image is input into the pre-trained graph neural network based on the glyph reconstruction strategy, and the reconstruction result corresponding to the missing ancient character image is obtained, including: In the case of the glyph reconstruction strategy being the contour priority strategy, inputting the damaged ancient character image into the pre-trained contour graph reconstruction network to obtain a reconstruction result of the damaged ancient character image. In the case of the glyph reconstruction strategy being the skeleton priority strategy, inputting the damaged ancient character image into the pre-trained skeleton graph reconstruction network to obtain a reconstruction result of the damaged ancient character image. In the case of the glyph reconstruction strategy being the collaborative reconstruction strategy, performing feature alignment on the target contour graph and the target skeleton graph, dynamically adjusting a fusion weight of the target contour graph and the target skeleton graph based on respective modal confidence degrees, and obtaining aligned feature information. Inputting the aligned feature information into the pre-trained cross-modal reconstruction network to obtain a reconstruction result of the damaged ancient character image.

[0085] Specifically, in some embodiments, step 103 can be implemented by the following steps, including: In the case of the glyph reconstruction strategy being the contour priority strategy, inputting the damaged ancient character image into the pre-trained contour graph reconstruction network to obtain a reconstruction result of the damaged ancient character image.

[0086] The pre-trained contour graph reconstruction network focuses on using available boundary information, learns spatial relationships and geometric constraints between contour nodes through a graph neural network, infers missing boundary segments, and realizes glyph reconstruction of ancient characters.

[0087] In the case of the glyph reconstruction strategy being the skeleton priority strategy, inputting the damaged ancient character image into the pre-trained skeleton graph reconstruction network to obtain a reconstruction result of the damaged ancient character image.

[0088] The pre-trained skeleton graph reconstruction network can infer missing stroke segments and connection points by analyzing the connection relationships and topological structures of strokes, and effectively reconstructs damaged ancient characters.

[0089] In the case of the glyph reconstruction strategy being the collaborative reconstruction strategy, performing feature alignment on the target contour graph and the target skeleton graph, dynamically adjusting a fusion weight of the target contour graph and the target skeleton graph based on respective modal confidence degrees, and obtaining aligned feature information; inputting the aligned feature information into the pre-trained cross-modal reconstruction network to obtain a reconstruction result of the damaged ancient character image.

[0090] The strategy ensures the consistency of the two kinds of representations in the semantic space through a feature alignment mechanism, and dynamically weighs the reliability of different information sources by using an attention fusion network. When the confidence degree of a certain representation is low, the algorithm relies more on the information of another representation for compensation.

[0091] Optionally, the method further comprises an integrated decision mechanism: combining the output results of multiple strategies, selecting the optimal reconstruction scheme through uncertainty quantification and consistency check.

[0092] The method provided by the embodiment selects a corresponding character shape reconstruction strategy according to the type of the defect, different character shape reconstruction strategies correspond to different graph neural networks, in the case of the character shape reconstruction strategy being a skeleton priority strategy, inputs the defective ancient character image into the pre-trained skeleton graph reconstruction network to obtain a reconstruction result of the defective ancient character image, in the case of the character shape reconstruction strategy being a collaborative reconstruction strategy, performs feature alignment on the target contour graph and the target skeleton graph, dynamically adjusts the fusion weight of the target contour graph and the target skeleton graph based on the confidence of each modality, and obtains aligned feature information, and inputs the aligned feature information into the pre-trained cross-modal reconstruction network to obtain the reconstruction result of the defective ancient character image, and the strategy-based processing can realize high-quality ancient character defective character shape reconstruction.

[0093] According to the ancient character defective character shape reconstruction method based on a graph structure and deep learning provided by the application, the pre-trained contour graph reconstruction network, the pre-trained skeleton graph reconstruction network and the pre-trained cross-modal reconstruction network are obtained by using a progressive learning strategy for training, and the training steps of the pre-trained contour graph reconstruction network include: An unannotated ancient character data set is obtained; the unannotated ancient character data set includes a plurality of sample ancient characters; Based on each sample ancient character, a positive and negative sample pair of the contour graph reconstruction network is generated; the positive and negative sample pair includes a positive sample pair and a negative sample pair, the positive sample pair is different enhanced contour graphs of the same character, and the negative sample pair is contour graphs of different characters; Based on the positive and negative sample pair of the contour graph reconstruction network and an InfoNCE loss function, the contour graph reconstruction network is iteratively trained; The iteration is terminated when the value of the InfoNCE loss function reaches a minimum value, and the pre-trained contour graph reconstruction network is determined based on the model parameters of the contour graph reconstruction network at the time of iteration termination.

[0094] Specifically, in some embodiments, the pre-trained contour graph reconstruction network, the pre-trained skeleton graph reconstruction network and the pre-trained cross-modal reconstruction network are trained by adopting a progressive learning strategy. Through a three-stage learning process of "self-supervised pre-training - contrastive learning optimization - mask reconstruction training", strong glyph structure understanding and reconstruction capabilities are gradually built. The necessity of this design lies in that: the self-supervised pre-training phase establishes the basic structure representation capability, enabling the model to understand the basic patterns of contour graphs and skeleton graphs; the contrastive learning phase enhances the discriminative ability and semantic consistency of the representation, ensuring that different variants of the same glyph have similar representations and different glyphs have distinctive representations; the mask reconstruction phase is specifically trained for reconstruction capability, based on the high-quality representation established in the previous two phases, learning how to infer the complete structure from the missing structure. This progressive learning strategy avoids the training difficulties and performance instability problems when directly training complex reconstruction tasks. The self-supervised pre-training design targets multiple pre-training tasks for contour graph and skeleton graph structures, learning the basic structure representation of glyphs from unannotated data; the contrastive learning optimization enhances the discriminative ability of the representation by constructing positive and negative sample pairs; the mask reconstruction training specifically trains the structure reconstruction capability of the model based on the high-quality representation established in the previous two phases.

[0095] Next, the training process of the contour graph reconstruction network (graph neural network) is illustrated by way of example.

[0096] The contour graph reconstruction network supports multiple graph neural network architectures such as graph convolutional network (GCN), graph attention network (GAT), graph sampling and aggregation network (GraphSAGE), graph isomorphism network (GIN), simplified graph convolutional network (SGC) to process contour graphs, introduces geometric graph convolution to consider the geometric properties of nodes and edges, and designs multi-scale graph encoding to capture structure information in different ranges. By reconstructing the complete contour graph, the topological patterns and geometric laws of the graph are learned.

[0097] Contour graph contrastive learning uses different augmented versions of the same glyph as positive sample pairs and contour graphs of different glyphs as negative sample pairs to train the model to learn the similarity and difference of contour features using the InfoNCE loss function.

[0098] The training steps of the pre-trained contour graph reconstruction network include: Firstly, an unlabeled ancient character data set is obtained, and the unlabeled ancient character data set includes a plurality of sample ancient characters; then, a positive and negative sample pair of a contour graph reconstruction network is generated based on the sample ancient characters. The positive and negative sample pair includes a positive sample pair and a negative sample pair, the positive sample pair is different enhanced contour graphs of the same character, and the negative sample pair is contour graphs of different characters; further, the contour graph reconstruction network is iteratively trained based on the positive and negative sample pair of the contour graph reconstruction network and an InfoNCE loss function; finally, the iteration is terminated when a value of the InfoNCE loss function reaches a minimum value, and a pre-trained contour graph reconstruction network is determined based on model parameters of the contour graph reconstruction network at the time of iteration termination.

[0099] Next, the training process of the skeleton graph reconstruction network (graph neural network) is exemplified.

[0100] The skeleton graph reconstruction network also supports a plurality of graph neural network architectures to process the skeleton graph, introduces geometric graph convolution to consider the geometric properties of nodes and edges, and designs multi-scale graph encoding to capture structural information in different ranges. By reconstructing the complete skeleton graph, the topological patterns and structural rules of the graph are learned.

[0101] The skeleton graph contrast learning adopts the same positive and negative sample construction strategy, and optimizes the discriminativeness of the skeleton structure representation through a contrast loss.

[0102] The training steps of the skeleton graph reconstruction network are similar to those of the contour graph reconstruction network, which will not be described here.

[0103] The method provided in the embodiment can learn general glyph structure rules from limited labeled data by self-supervised pre-training and contrast learning technology to train the contour graph reconstruction network and the skeleton graph reconstruction network. The method has certain adaptability when processing missing patterns that have not been seen during training, and can provide relatively stable reconstruction effect for ancient character samples of different writing styles and historical periods.

[0104] According to the ancient character missing glyph reconstruction method based on graph structure and deep learning provided by the application, the training steps of the pre-trained cross-modal reconstruction network include: generating a positive sample pair of the cross-modal reconstruction network based on the contour graph and the skeleton graph of the same character in each sample ancient character, and generating a negative sample pair of the cross-modal reconstruction network based on the contour graph and the skeleton graph of different characters; iteratively training the cross-modal reconstruction network based on the positive sample pair of the cross-modal reconstruction network and the negative sample pair of the cross-modal reconstruction network and an InfoNCE loss function; terminating the iteration when a value of the InfoNCE loss function reaches a minimum value, and determining a pre-trained cross-modal reconstruction network based on model parameters of the cross-modal reconstruction network at the time of iteration termination.

[0105] Specifically, in some embodiments, the training step of the pre-trained cross-modal reconstruction network comprises the following: It should be noted that the cross-modal reconstruction network learns the correspondence between the contour map and the skeleton map, including graph-to-graph prediction and structure consistency verification, to enhance the model's understanding of the internal structure of the character.

[0106] The cross-modal contrast learning takes the contour map and the skeleton map of the same character as a positive sample pair, maps the two modalities to a 128-dimensional shared feature space through a linear projection layer, calculates the feature distance using cosine similarity, and ensures that different modal representations of the same character remain consistent in the semantic space.

[0107] The training step specifically comprises: First, based on the contour map and the skeleton map of the same character in each sample ancient text, a positive sample pair of the cross-modal reconstruction network is generated, and based on the contour map and the skeleton map of different characters, a negative sample pair of the cross-modal reconstruction network is generated; Then, based on the positive sample pair of the cross-modal reconstruction network, the negative sample pair of the cross-modal reconstruction network and the InfoNCE loss function, the cross-modal reconstruction network is iteratively trained; Finally, when the value of the InfoNCE loss function reaches a minimum value, the iteration is terminated, and based on the model parameters of the cross-modal reconstruction network at the time of iteration termination, a pre-trained cross-modal reconstruction network is determined.

[0108] The method provided in this embodiment can learn general character structure rules from limited labeled data by training a cross-modal reconstruction network through self-supervised pre-training and contrast learning technology. This method has certain adaptability when dealing with missing patterns that have not been seen during training, and can provide relatively stable reconstruction results for ancient text samples of different writing styles and historical periods.

[0109] Optionally, during model training, after extracting graph structure representations, structured data augmentation strategies can be employed to increase sample diversity. These strategies include contour graph augmentation, skeleton graph augmentation, and cross-modal augmentation. Contour graph augmentation primarily uses geometric transformations, such as small-angle rotations (±15°), moderate scaling (0.8-1.2 times), and minor translations (±5% of image size). It also perturbs the graph structure by adding Gaussian noise with a standard deviation of 0.01 to node coordinates and randomly deleting no more than 10% of non-critical nodes. Furthermore, it increases the structural diversity of samples by extracting connected subgraphs and randomly sampling subsets of nodes. Skeleton graph augmentation employs similar strategies for node position perturbation and edge connection adjustment. It enhances the model's adaptability to attribute changes by adding noise to node attributes and randomly masking some attribute values. Finally, it enriches the topological variation patterns of the skeleton through operations such as local subgraph reconnection and decomposition followed by reconstruction. Cross-modal enhancement focuses on maintaining the spatial correspondence between the contour map and the skeleton map, applying the same geometric transformation parameters to both representations simultaneously, and using contour information to constrain the reasonable range of skeleton enhancement.

[0110] Optionally, the mask reconstruction training, based on the high-quality representations established in the first two stages, specifically trains the model's structural reconstruction capabilities. Masking strategies simulating natural defects are employed, including semantic masking (masking complete semantic units based on glyph components), progressive masking (learning from simple to complex lessons), and multi-scale masking (multi-level local and global masking), simulating real glyph defect patterns such as missing radicals and broken strokes. The structured reconstruction network design includes contour graph reconstruction networks, skeleton graph reconstruction networks, and cross-modal reconstruction. The contour graph reconstruction network reconstructs the structure of the masked subgraph based on neighborhood information while maintaining topological rationality; the skeleton graph reconstruction network reconstructs the structure of the masked subgraph based on neighborhood information while maintaining topological rationality; and cross-modal reconstruction utilizes complementary information from the two modalities to ensure consistency in the reconstruction results.

[0111] According to the method for reconstructing missing ancient Chinese characters based on graph structure and deep learning provided by the present invention, after extracting graph structure representations from the missing ancient Chinese character image to obtain the target contour map and target skeleton map corresponding to the missing ancient Chinese character image, the method further includes: Generate a multi-dimensional feature set corresponding to the nodes of the target contour map and the target skeleton map; The multi-dimensional feature set includes: geometric features, topological features, philological features, and document context features; Based on the character shape reconstruction strategy, the missing ancient character image is input into a pre-trained graph neural network to obtain the reconstruction result corresponding to the missing ancient character image, including: The damaged ancient character image and the multi-dimensional feature set are input into a pre-trained graph neural network to obtain a reconstruction result corresponding to the damaged ancient character image.

[0112] Specifically, in some embodiments, to sufficiently describe the structural information of ancient character glyphs and support high-quality reconstruction of missing parts, the step of extracting graph structure representation further includes multi-dimensional feature design of nodes. This is a key technical basis for achieving accurate structured reconstruction. The node feature design follows the principles of hierarchy, complementarity, and reconstructability, ensuring that the features can accurately describe the essential properties of the glyphs and provide sufficient information support for the reconstruction task.

[0113] The multi-dimensional feature set includes geometric features, topological features, philological features, and literature context features.

[0114] The geometric feature set contains the basic geometric properties of the nodes, providing spatial positioning and shape constraint information for reconstruction. The coordinate feature records the two-dimensional position information of the node in the normalized coordinate system, including absolute coordinates and relative coordinates with respect to the glyph center, providing a spatial positioning reference for reconstruction. The curvature feature calculates the local curvature value at the node, including discrete curvature, Gaussian curvature, and average curvature, reflecting the bending degree of the glyph at that point, which is of great significance for reconstructing the natural bending shape of strokes. The direction feature includes the tangent direction, normal direction, and principal direction at the node, using both angle and unit vector representations, providing constraints on stroke orientation and connection direction for reconstruction. The distance feature calculates the Euclidean distance and geodesic distance from the node to key positions such as the glyph boundary, skeleton endpoint, and centroid, providing global spatial relationship information, which helps maintain the overall coordination of the reconstruction result.

[0115] The topological feature set describes the connection relationship and topological status of the node in the graph structure, providing structural integrity constraints for reconstruction. The connectivity feature includes the degree of the node (the number of connected edges), the number of neighbor nodes, and the connected component identifier, reflecting the importance and connection pattern of the node in the graph structure. The path feature calculates the shortest path length, path number, and path diversity from the node to other key nodes, describing the position of the node in the global topological structure. The loop feature identifies the number of loops passing through the node, the loop length distribution, and the loop type, reflecting the closed structure characteristics of the glyph, which has guiding significance for reconstructing closed strokes and component boundaries. The hierarchy feature calculates the hierarchy level, subgraph affiliation, and structure importance score of the node through graph decomposition algorithms, providing a basis for hierarchical reconstruction.

[0116] The set of graphology features combines ancient graphology knowledge to design features reflecting graphology attributes such as stroke type, component combination, and structure rule. The stroke type feature is based on ancient graphology theory, marks nodes as different basic stroke types, and records the start, middle, and end position attributes of strokes to provide stroke semantic constraints for reconstruction. The component attribution feature identifies the component type, component boundary, and component relationship by a character shape analysis algorithm to support hierarchical reconstruction strategies based on components. The structure rule feature encodes the character construction rules of ancient characters, such as left-right structure, up-down structure, surrounding structure, and corresponding proportion relationship and symmetry features to provide overall layout constraints for reconstruction.

[0117] The set of document context features considers the context information of characters in documents to provide semantic and historical constraints for reconstruction. The co-occurrence character feature records other characters that frequently co-occur with the current character in the same document, including adjacent characters, same sentence characters, and same paragraph characters, and provides semantic consistency constraints for reconstruction by statistically analyzing the co-occurrence frequency and semantic association degree. The document type feature identifies the type of the document to which the character belongs, such as oracle bone inscriptions, bronze inscriptions, and stone inscriptions, and different types of documents have different writing standards and style characteristics to provide type-based constraints for reconstruction. The era background feature encodes the historical period information of characters, including time, region, and other spatiotemporal attributes, reflects the evolution characteristics and regional differences of ancient characters in different historical periods, and provides historical evolution constraints for reconstruction. The semantic field feature analyzes the semantic environment of characters in documents to identify their semantic fields and theme domains, such as sacrifice, military, and agriculture, and provides theme-related semantic guidance for reconstruction.

[0118] The method provided in this embodiment designs a multi-dimensional set of node features, fully describes the structure information of ancient character shapes, and supports high-quality missing reconstruction.

[0119] According to the ancient character missing character reconstruction method based on a graph structure and deep learning provided by the application, the method further comprises: verifying the reconstruction result corresponding to the missing ancient character image; The verification includes at least one of topological rationality testing, geometric consistency evaluation, and multi-modal cross-validation.

[0120] Specifically, in some embodiments, the method further comprises result verification and optimization to realize quality evaluation and post-processing of the reconstruction output.

[0121] Specifically, the topological rationality of the reconstruction result corresponding to the missing ancient character image is tested. For example, the topological rationality of the reconstruction result is first checked to ensure that the stroke connection relationship conforms to the construction rule of ancient characters, Then, the geometric consistency of the reconstructed result corresponding to the damaged ancient character image is evaluated. For example, the morphological coordination of the reconstructed part and the original part is verified.

[0122] Finally, the multi-modal cross-validation of the reconstructed result corresponding to the damaged ancient character image is performed. For example, the complementarity of the contour and the skeleton information is used to identify and correct potential reconstruction errors.

[0123] Correspondingly, the system outputs include the reconstructed complete character structure and the corresponding confidence score.

[0124] The method provided in the embodiment improves the accuracy of character reconstruction by performing quality evaluation and post-processing on the reconstruction output in the result verification and optimization stage.

[0125] Figure 2 is a flowchart of the ancient character damaged character reconstruction method based on graph structure and deep learning provided by the present application, as shown in Figure 2 The method comprises the following steps. Training stage: The ancient character image is subjected to graph structure representation extraction, and the structured representation comprises a contour graph and a skeleton graph. Based on the structured representation, joint representation learning is performed, and the model is trained.

[0126] Inference stage: The damaged ancient character is subjected to structured analysis to obtain a damage type, and then the trained model is used for reconstruction inference to obtain a complete character output.

[0127] The present application has the following advantages: 1. An ancient character representation method based on contour graph and skeleton graph structure: the method realizes the structured representation of ancient character by extracting contour graph and skeleton graph structure. This representation method replaces the traditional pixel representation, which can more effectively capture the essential features and topological relationships of the character.

[0128] 2. A multi-dimensional node feature design method: the method designs a multi-dimensional node feature set that integrates geometric features, topological features, philological features and literature context features, providing rich feature description and semantic guidance for the graph structure.

[0129] 3. A self-supervised learning method for structured representation: the method designs multiple self-supervised learning tasks, including contour graph mask autoencoder, skeleton graph mask autoencoder and cross-modal prediction task. This method can learn the general structure representation of the character from unlabeled data, greatly reducing the dependence on manual annotation and improving the generalization ability of the model.

[0130] 4. A structured contrastive learning and cross-modal fusion technique: This technique includes contour graph contrastive learning, skeleton graph contrastive learning, and cross-modal contrastive learning. By enhancing the discriminative ability and semantic consistency of the representation, the quality and robustness of the reconstruction result are improved. At the same time, through feature alignment and multiple fusion strategies, the effective combination of contour graph and skeleton graph representations is realized, fully utilizing the complementary advantages of the two representations, providing more comprehensive and robust glyph representations.

[0131] 5. A structured mask strategy based on natural defect simulation: This strategy adopts a mask method that simulates the actual glyph defect situation, including semantic mask, progressive mask, and multi-scale mask. This method can better train the model to handle actual ancient character defect problems, improving the application effect of the model in real scenarios.

[0132] 6. A multi-strategy structured reconstruction method: This method designs three reconstruction strategies: pure contour graph modeling, pure skeleton graph modeling, and fusion modeling. These strategies can handle contour graph and skeleton graph representations separately or jointly, achieving flexible and efficient glyph structure reconstruction, which can adapt to different types and degrees of glyph defect situations.

[0133] The following describes the ancient character missing glyph reconstruction device based on graph structure and deep learning provided by the present application. The ancient character missing glyph reconstruction device based on graph structure and deep learning described below can be mutually corresponding and referenced with the ancient character missing glyph reconstruction method based on graph structure and deep learning described above.

[0134] Figure 3 is a structural schematic diagram of the ancient character missing glyph reconstruction device based on graph structure and deep learning provided by the present application, as Figure 3 shown, the ancient character missing glyph reconstruction device based on graph structure and deep learning 300 includes the following modules: The graph structure representation extraction module 310 is configured to perform graph structure representation extraction on the missing ancient character image to obtain a target contour graph and a target skeleton graph corresponding to the missing ancient character image. The strategy determination module 320 is configured to determine a missing type corresponding to the missing ancient character image according to the target contour graph and the target skeleton graph, determine a glyph reconstruction strategy based on the missing type, and determine a graph neural network corresponding to the missing ancient character image based on the glyph reconstruction strategy; the graph neural network includes a contour graph reconstruction network, a skeleton graph reconstruction network, or a cross-modal reconstruction network. The reconstruction module 330 is configured to input the missing ancient character image into the pre-trained graph neural network based on the glyph reconstruction strategy to obtain a reconstruction result corresponding to the missing ancient character image; the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0135] The device provided by the embodiment comprises a graph structure representation extraction module 310, a strategy determination module 320 and a reconstruction module 330. Firstly, the graph structure representation extraction module 310 is used for performing graph structure representation extraction on a damaged ancient character image to obtain a target contour graph and a target skeleton graph. Then, the strategy determination module 320 is used for determining a damage type corresponding to the damaged ancient character image according to the target contour graph and the target skeleton graph, determining a character shape reconstruction strategy based on the damage type, and determining a graph neural network corresponding to the damaged ancient character image based on the character shape reconstruction strategy. The graph neural network comprises a contour graph reconstruction network, a skeleton graph reconstruction network and a cross-modal reconstruction network. Further, the reconstruction module 330 is used for inputting the damaged ancient character image into a pre-trained graph neural network based on the character shape reconstruction strategy to obtain a reconstruction result corresponding to the damaged ancient character image. The pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0136] The application replaces the traditional pixel representation by converting the character shape into the structured representation of the contour graph and the skeleton graph, and automatically learns the deep representation of the character shape from a large amount of unannotated ancient character structured data by combining the self-supervised pre-training and the contrastive learning technology, so as to realize the high-quality ancient character damaged character shape reconstruction by using the graph neural network.

[0137] According to the ancient character damaged character shape reconstruction device 300 based on the graph structure and the deep learning provided by the application, the graph structure representation extraction 310 is specifically used for: using a Canny edge detection algorithm to extract the main contour line of the damaged ancient character image; simplifying the main contour line by using a Douglas-Peucker algorithm and retaining key turning points to obtain a simplified contour line; identifying a first node on the simplified contour line, constructing edge connection between the first nodes based on the first nodes and contour adjacency relationship potential to obtain an initial contour graph; the first nodes comprise corner points, turning points, connection points and feature points, and different first nodes correspond to different attribute encodings; performing scale and position normalization processing on the initial contour graph to obtain the target contour graph.

[0138] According to the ancient character damaged character shape reconstruction device 300 based on the graph structure and the deep learning provided by the application, the graph structure representation extraction 310 is further used for: extracting the central axis skeleton of the damaged ancient character image by using a Zhang-Suen thinning algorithm and eliminating the sawtooth effect; identifying a second node in the central axis skeleton; the second node comprises a skeleton end point, an intersection point and a turning point; Based on the skeleton connectivity, an edge connection between the second nodes is constructed to obtain the target skeleton graph; different second nodes correspond to different attribute encodings.

[0139] According to the ancient character image, the strategy determination module 320 is specifically used for: In the case that the missing type corresponding to the missing ancient character image is that the skeleton is seriously missing but the boundary is relatively complete, the character shape reconstruction strategy is determined as a contour priority strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained contour graph reconstruction network; In the case that the missing type corresponding to the missing ancient character image is that the internal structure is identifiable but the boundary is blurred, the character shape reconstruction strategy is determined as a skeleton priority strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained skeleton graph reconstruction network; In the case that the missing type corresponding to the missing ancient character image is that the contour and skeleton information are partially available, the character shape reconstruction strategy is determined as a collaborative reconstruction strategy, and the graph neural network corresponding to the missing ancient character image is determined as a pre-trained cross-modal reconstruction network.

[0140] According to the ancient character image, the strategy determination module 320 is specifically used for: In the case that the character shape reconstruction strategy is a contour priority strategy, the missing ancient character image is input into the pre-trained contour graph reconstruction network to obtain a reconstruction result of the missing ancient character image; In the case that the character shape reconstruction strategy is a skeleton priority strategy, the missing ancient character image is input into the pre-trained skeleton graph reconstruction network to obtain a reconstruction result of the missing ancient character image; In the case that the character shape reconstruction strategy is a collaborative reconstruction strategy, the target contour graph and the target skeleton graph are feature-aligned, the fusion weights of the target contour graph and the target skeleton graph are dynamically adjusted based on the modal confidence, and the aligned feature information is obtained; The aligned feature information is input into the pre-trained cross-modal reconstruction network to obtain a reconstruction result of the missing ancient character image.

[0141] According to the ancient character image, the strategy determination module 320 is specifically used for: The device further comprises a training module; The training module is used for: obtain an unlabeled ancient character data set, wherein the unlabeled ancient character data set comprises a plurality of sample ancient characters; generate a positive and negative sample pair of a contour reconstruction network based on each of the sample ancient characters, wherein the positive and negative sample pair comprises a positive sample pair and a negative sample pair, the positive sample pair is different enhanced contour graphs of the same character, and the negative sample pair is contour graphs of different characters; iteratively train the contour reconstruction network based on the positive and negative sample pair of the contour reconstruction network and an InfoNCE loss function; terminate the iteration when a value of the InfoNCE loss function reaches a minimum value, and determine the pre-trained contour reconstruction network based on model parameters of the contour reconstruction network at the time of termination of the iteration.

[0142] According to the ancient character missing character reconstruction device 300 based on the graph structure and the deep learning provided by the application, the training module is further used for: generate a positive sample pair of a cross-modal reconstruction network based on contour graphs and skeleton graphs of the same character in each of the sample ancient characters, and generate a negative sample pair of the cross-modal reconstruction network based on contour graphs and skeleton graphs of different characters; iteratively train the cross-modal reconstruction network based on the positive sample pair of the cross-modal reconstruction network and the negative sample pair of the cross-modal reconstruction network and an InfoNCE loss function; terminate the iteration when a value of the InfoNCE loss function reaches a minimum value, and determine the pre-trained cross-modal reconstruction network based on model parameters of the cross-modal reconstruction network at the time of termination of the iteration.

[0143] According to the ancient character missing character reconstruction device 300 based on the graph structure and the deep learning provided by the application, the graph structure representation extraction 310 is further used for: generate a multi-dimensional feature set corresponding to nodes of the target contour graph and the target skeleton graph; The multi-dimensional feature set comprises geometric features, topological features, philological features and literature context features. The graph neural network is pre-trained based on the character reconstruction strategy, the missing ancient character image is input into the pre-trained graph neural network, and a reconstruction result corresponding to the missing ancient character image is obtained. The missing ancient character image and the multi-dimensional feature set are input into the pre-trained graph neural network, and a reconstruction result corresponding to the missing ancient character image is obtained.

[0144] According to the ancient character missing character reconstruction device 300 based on the graph structure and the deep learning provided by the application, the device further comprises a result verification module. The result verification module is configured to: verify the reconstruction result corresponding to the damaged ancient character image; The verification includes at least one of topological rationality verification, geometric consistency evaluation and multi-modal cross verification.

[0145] Figure 4 is a structural schematic diagram of an electronic device provided by the present application, as Figure 4 indicated, the electronic device can include a processor (processor) 410, a communications interface (communications interface) 420, a memory (memory) 430 and a communications bus 440, wherein the processor 410, the communications interface 420, the memory 430 complete mutual communication through the communications bus 440. The processor 410 can call the logical instructions in the memory 430 to execute the ancient character damaged character reconstruction method based on the graph structure and deep learning, which includes: extracting a graph structure representation from a damaged ancient character image to obtain a target contour graph and a target skeleton graph of the damaged ancient character image; According to the target contour graph and the target skeleton graph, determine the damage type corresponding to the damaged ancient character image, determine the character reconstruction strategy based on the damage type, and determine the graph neural network corresponding to the damaged ancient character image based on the character reconstruction strategy; the graph neural network includes a contour graph reconstruction network, a skeleton graph reconstruction network or a cross-modal reconstruction network; Based on the character reconstruction strategy, input the damaged ancient character image into the pre-trained graph neural network to obtain the reconstruction result corresponding to the damaged ancient character image; the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0146] In addition, the logical instructions in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk and various program code storage media.

[0147] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and the computer program being capable of executing the ancient character missing glyph reconstruction method based on graph structure and deep learning provided by the above-mentioned methods when executed by a processor, the method comprising: extracting a graph structure representation of the missing ancient character image to obtain a target contour graph and a target skeleton graph of the missing ancient character image; determining a missing type corresponding to the missing ancient character image according to the target contour graph and the target skeleton graph, determining a glyph reconstruction strategy based on the missing type, and determining a graph neural network corresponding to the missing ancient character image based on the glyph reconstruction strategy; the graph neural network comprises a contour graph reconstruction network, a skeleton graph reconstruction network, or a cross-modal reconstruction network; inputting the missing ancient character image into the pre-trained graph neural network based on the glyph reconstruction strategy to obtain a reconstruction result corresponding to the missing ancient character image; the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0148] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being capable of implementing the ancient character missing glyph reconstruction method based on graph structure and deep learning when executed by a processor, the method comprising: extracting a graph structure representation of the missing ancient character image to obtain a target contour graph and a target skeleton graph of the missing ancient character image; determining a missing type corresponding to the missing ancient character image according to the target contour graph and the target skeleton graph, determining a glyph reconstruction strategy based on the missing type, and determining a graph neural network corresponding to the missing ancient character image based on the glyph reconstruction strategy; the graph neural network comprises a contour graph reconstruction network, a skeleton graph reconstruction network, or a cross-modal reconstruction network; inputting the missing ancient character image into the pre-trained graph neural network based on the glyph reconstruction strategy to obtain a reconstruction result corresponding to the missing ancient character image; the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training.

[0149] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0150] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0151] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for reconstructing missing characters of ancient scripts based on graph structure and deep learning, characterized in that, The method comprises the following steps: a graph structure representation of a damaged ancient character image is extracted to obtain a target contour graph and a target skeleton graph corresponding to the damaged ancient character image; a damage type corresponding to the damaged ancient character image is determined according to the target contour graph and the target skeleton graph, a character shape reconstruction strategy is determined based on the damage type, and a graph neural network corresponding to the damaged ancient character image is determined based on the character shape reconstruction strategy; the graph neural network comprises a contour graph reconstruction network, a skeleton graph reconstruction network or a cross-modal reconstruction network; the damaged ancient character image is input into the pre-trained graph neural network based on the character shape reconstruction strategy, and a reconstruction result corresponding to the damaged ancient character image is obtained; the pre-trained graph neural network is obtained based on self-supervised pre-training and contrastive learning training. 2.The ancient character missing character reconstruction method based on graph structure and deep learning according to claim 1, characterized in that, The graph structure representation of the damaged ancient character image is extracted to obtain the target contour graph, comprising: a Canny edge detection algorithm is used to extract the main contour line of the damaged ancient character image; a Douglas-Peucker algorithm is used to simplify the main contour line and retain key turning points to obtain a simplified contour line; a first node on the simplified contour line is identified, and edge connections between the first nodes are constructed based on the first nodes and contour adjacency relationship potentials to obtain an initial contour graph; the first nodes include corner points, turning points, connection points and feature points, and different first nodes correspond to different attribute encodings; scale and position normalization processing is performed on the initial contour graph to obtain the target contour graph. 3.The ancient character missing character reconstruction method based on graph structure and deep learning according to claim 1, characterized in that, The graph structure representation of the damaged ancient character image is extracted to obtain the target skeleton graph, comprising: a Zhang-Suen thinning algorithm is used to extract the central axis skeleton of the damaged ancient character image and eliminate the sawtooth effect; a second node in the central axis skeleton is identified; the second node includes skeleton end points, intersection points and turning points; based on skeleton connectivity, edge connections between the second nodes are constructed to obtain the target skeleton graph; different second nodes correspond to different attribute encodings. 4.The ancient character missing character reconstruction method based on graph structure and deep learning according to claim 1, characterized in that, The character shape reconstruction strategy is determined based on the damage type, and the graph neural network corresponding to the damaged ancient character image is determined based on the character shape reconstruction strategy, comprising: in a case where the damage type corresponding to the damaged ancient character image is that the skeleton is severely missing but the boundary is relatively complete, the character shape reconstruction strategy is determined as a contour priority strategy, and the graph neural network corresponding to the damaged ancient character image is determined as a pre-trained contour graph reconstruction network; in a case where the damage type corresponding to the damaged ancient character image is that the internal structure is identifiable but the boundary is blurred, the character shape reconstruction strategy is determined as a skeleton priority strategy, and the graph neural network corresponding to the damaged ancient character image is determined as a pre-trained skeleton graph reconstruction network; in a case where the damage type corresponding to the damaged ancient character image is that the contour and skeleton information are partially available, the character shape reconstruction strategy is determined as a collaborative reconstruction strategy, and the graph neural network corresponding to the damaged ancient character image is determined as a pre-trained cross-modal reconstruction network. 5.The ancient character missing character reconstruction method based on graph structure and deep learning according to claim 1, characterized in that, The inputting the damaged ancient character image into the pre-trained graph neural network based on the glyph reconstruction strategy to obtain a reconstruction result corresponding to the damaged ancient character image comprises: In the case that the glyph reconstruction strategy is a contour-first strategy, inputting the damaged ancient character image into a pre-trained contour graph reconstruction network to obtain a reconstruction result of the damaged ancient character image; In the case that the glyph reconstruction strategy is a skeleton-first strategy, inputting the damaged ancient character image into a pre-trained skeleton graph reconstruction network to obtain a reconstruction result of the damaged ancient character image; In the case that the glyph reconstruction strategy is a collaborative reconstruction strategy, performing feature alignment on the target contour graph and the target skeleton graph, dynamically adjusting the fusion weight of the target contour graph and the target skeleton graph based on the modal confidence, and obtaining aligned feature information; Inputting the aligned feature information into a pre-trained cross-modal reconstruction network to obtain a reconstruction result of the damaged ancient character image. 6.The ancient character missing character reconstruction method based on graph structure and deep learning according to claim 5, characterized in that, The pre-trained contour graph reconstruction network, the pre-trained skeleton graph reconstruction network and the pre-trained cross-modal reconstruction network are trained by using a progressive learning strategy, and the training steps of the pre-trained contour graph reconstruction network comprise: Obtaining an unannotated ancient character dataset; the unannotated ancient character dataset comprises a plurality of sample ancient characters; Based on each sample ancient character, generating a positive and negative sample pair of the contour graph reconstruction network; the positive and negative sample pair comprises a positive sample pair and a negative sample pair, the positive sample pair is different enhanced contour graphs of the same character, and the negative sample pair is contour graphs of different characters; Based on the positive and negative sample pair of the contour graph reconstruction network and an InfoNCE loss function, iteratively training the contour graph reconstruction network; Terminating iteration when the value of the InfoNCE loss function reaches a minimum value, and determining the pre-trained contour graph reconstruction network based on the model parameters of the contour graph reconstruction network at the time of iteration termination.

7. The ancient character missing character reconstruction method based on graph structure and deep learning according to claim 6, characterized in that, The training steps of the pre-trained cross-modal reconstruction network comprise: Based on the contour graph and the skeleton graph of the same character in each sample ancient character, generating a positive sample pair of the cross-modal reconstruction network, and based on the contour graph and the skeleton graph of different characters, generating a negative sample pair of the cross-modal reconstruction network; Based on the positive sample pair of the cross-modal reconstruction network and the negative sample pair of the cross-modal reconstruction network and an InfoNCE loss function, iteratively training the cross-modal reconstruction network; Terminating iteration when the value of the InfoNCE loss function reaches a minimum value, and determining the pre-trained cross-modal reconstruction network based on the model parameters of the cross-modal reconstruction network at the time of iteration termination.

8. The graph structure and deep learning based ancient character missing character reconstruction method according to any one of claims 1-7, characterized in that, After the damaged ancient character image is subjected to graph structure representation extraction to obtain a target contour graph and a target skeleton graph corresponding to the damaged ancient character image, the method further comprises: Generating a multi-dimensional feature set corresponding to the nodes of the target contour graph and the target skeleton graph; wherein the multi-dimensional feature set comprises geometric features, topological features, philological features and literature context features; The method comprises the following steps: The method comprises the following steps:

9. The graph structure and deep learning based ancient character missing character reconstruction method according to any one of claims 1-7, characterized in that, The method further comprises: The method further comprises: The method further comprises: 10.A device for reconstructing missing characters of ancient scripts based on graph structure and deep learning, characterized in that, The method further comprises: The method further comprises: The method further comprises: The method further comprises:

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The method further comprises:

12. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises

Images